A cup of...

Lucene and Hibernate Search - a small overview

2011-02-28T11:17:00.001+01:00

Background
For a user visiting your web site it would certainly be nice if the search experience felt like when using Google. To achieve this one can use Hibernate Search and Lucene. A user might then search for almost anything in relation to e.g. an ad, even combining search terms like ad title, location and using wild cards like '*'. If a user misspells a search term suggestions can be shown like: 'Did you mean ABC'.
Well since we are not Google we can not develop our own search framework, instead we us open source frameworks to helps us achieve what we want. The Apache Software Foundation hosts an excellent framework for doing full text searches - Lucene. On top of Lucene we use Hibernate Search, which provides our domain model with the full text capabilities of Lucene and acts like a bridge between the index model and the object model.
I thought I would write something about how Lucene and Hibernate Search works while I still remember how. It is just an overview and I really recommend the books: Lucene in Action, Second Edition (ISBN: 1933988177) and Hibernate Search in Action (ISBN: 1933988649) for deeper understanding and as reference material.

Overview
Before we start delve into details have a look at the below mind map. You will notice that it contains two "legs": Indexing and Searching. These two legs will form the base of the rest of our discussion. To view the whole mind map click here (created with Freemind). Indexing is obviously done prior to searching.

Full pic

Indexing
We will start with left leg - Indexing.

Full pic

Indexing Overview
To be able to do really fast searches indexes need to be created in advance, perhaps triggered from a scheduled job or if you use Hibernate Search when a transaction completes. Indexing is the process that grabs data in the form of plain text (Lucene only understands plain text), builds Documents, analyzes the data and finally creates Indexes. Before the Indexing process starts raw data (e.g. from files or a database ) must be extracted into a format which can be used to create the most fundamental element in Lucene - a Document. When the extraction part is done Documents are built - although still not part of any Index files. When Documents have been built they are analyzed by an Analyzer and tokens are created. Analyzers can remove irrelevant words like "and", "a" etc, which are common but not that relevant when later on searching in the indexes. Finally the Documents are attached and stored into physical Indexes using an IndexWriter.

Indexing Main classes
Some of the main classes used when Indexing are shown in this class diagram.

A Document represents a collection of fields. You can think of a Document as a database row, where fields in the Document are represented by columns in the table of the row.
The analogy with a database has some differences though:

In contrast to the database world a Document does not have different types, in fact it only has String types. Lucene 2.9 added a new type, NumericField (marked as experimental), which can be used for numerical range filtering and sorting.
Another difference is that one Document can have x number of fields while yet another one can have y number of fields. The fields do not even have to have same names in different Documents.
A Document can also have a Field multiple times, e.g. one Book Document could have id, title and isbn10 and another one could have id, title, isbn10, isbn10, isbn10 etc.

A sample code snippet to add a Document to an Index using the Lucene API could look like:

In Hibernate Search you only need to save an instance of an entity an that instance will also be stored or updated in the corresponding Lucene index. The way to inform Hibernate Search of this is by adding a couple of annotations to your entity. E.g. when you persist an instance of Book the Book data will also be persisted to the index of the Book, if the transaction is committed succesfully (Hibernate Search specific annotations are marked in yellow below)

A Field is part of a Document and has a name, text or binary value, and a series of detailed options that describes what Lucene should do with the Field’s value. A Document can have more than one Field with the same name, meaning that when a search is issued both fields will be searched for a given search token.
For a specific Field we can optionally store the original string value in its entirety in the index, which means that the value can be retrieved by an IndexReader directly from the index. This is useful if one needs a value to be shown in a search result list e.g. Although Index files of course get bigger the more data you store in them - but for a Book title in above example this is quite feasable.
If for a specific Field we choose to tokenize the Field's string value (renamed to analyze in Lucene 3) the Analyzer will break the Fields value into a stream of separate tokens (tokenstream) and make each token search able (tokenization). This is useful for normal text fields (body, title, abstract, etc.) for which we want to have full text search.
Some combinations of store and index options are shown in below table:

Example	Store	Index
Book ISBN, phone numbers, Social Security numbers, URLs, personal names, Dates	Field.Store.YES Hibernate Search: @Field(store = Store.YES)	Field.Index.UN_TOKENIZED Hibernate Search: @Field(index = Index.UN_TOKENIZED)
Book title, book abstract	Field.Store.YES	Field.Index.TOKENIZED
Book body	Field.Store.NO	Field.Index.TOKENIZED
Book type, database primary key (if not used for searching)	Field.Store.YES	Field.Index.NO
Book tags, hidden Keywords	Field.Store.NO	Field.Index.UN_TOKENIZED

In Hibernate Search annotations are used for mapping domain entity attributes to Lucene Fields. E.g. a Field mapping a ManyToOne relationship in Hibernate Search could look like below, which when inspected in Luke would reveal the Book attributes (yellow) as fields in the Ad index:

Tokenizing is the process of cleaning up and breaking down a text into Tokens which the Indexing process can use. Tokenizing is done by an Analyzer which cleans up text from often meaningless words like e.g. "a", "or", "and" "to" which are generally to common and not relevant to a user when searching the indexes. During the tokenization process the Analyzer can also discard punctuation, remove accents and lowercase the tokens (normalizing).
The analyzer can also reduce words to a common root form called stemming (e.g. apples becomes apple) which gives the user more relevant results when searching. Lemmatisation is closely related to stemming but looks also at the context of a word, so e.g. the word "better" has the word "good" as its lemma. Lemmatisation can also be done when analyzing but is not as fast as stemming since it needs to look-up lemmas for words.
The content the Analyzer operates on needs to be plain text, so if a PDF file needs tokenization then the raw data first needs to be converted into plain text and only then the Analyzer can do its magic. Finally Analyzers are also used when we are searching. The search string needs to be processed the same way that the indexed text was processed (tokenized) and therefore it is crucial to use the same Analyzer for both indexing and searching.

The IndexWriter's job is to take the input (a Document), pass the Document through the Analyzer, and finally create the Index. The IndexWriter creates a new index or reopens an existing one, adds, removes or updates documents in the Index. Think of an IndexWriter as an something that gives you write access to the index but doesn’t let you read or search it.
With Hibernate this happens transparently when a flush triggers a change to an indexed entity or collection. Hibernate then adds this change to a queue scoped by a transaction. When a commit operation is issued Hibernate Search will process the queue and apply the changes to the index or indexes. When a transaction is rolled back the queue will just be discarded without processing pending modifications to the index.
The IndexWriter keeps pending changes in a memory buffer until a threshold is reached - default is 16MB ram - after which a write to the Index is done (buffering ). The IndexWriter then flushes pending changes to the Index as new segments if: the RAM threshold is reached, a certain number of deletes is reached or if a certain number on new Documents have been added. After a flush is done by an IndexWriter a Lucene commit needs to take place before any searches done by IndexSearcher can detect changes made. While an IndexWriter is making changes to the index, an IndexReader will not see any of these changes until commit() or close() is called.
When adding Documents to Lucene the Documents are not immediately written to disk due to performance reasons. How many to keep in memory before flushing out changes is governed by the mergefactor (default is 10 Documents). The mergeFactor value also means that once the number of segments on the disk has been reached, Lucene will merge these segments into a single segment.
In Hibernate Search the property hibernate.search.<indexname>.indexwriter.transaction.merge_factor passed to the SessionFactory is governing this behavior. There is also the property hibernate.search.<indexname>.indexwriter.batch.merge_factor which can be set for batch processes, i.e. when a large number of Documents are indexed (if not set defaults to the transaction version of this property).
When calling optimize() on an IndexWriter instance, all in-memory Documents are flushed and all index segments are merged into a single segment, reducing number of files that for the index. Optimizing an index improves only the speed of searching due to the less number of index files that need to be opened, processed, and searched. Optimize should be done at the end of the indexing process, when you know the index will remain unmodified for a while.
Lucene uses a black-list approach when committing deleted documents to the index. This means that the document is simply marked as deleted, which is a very quick operation, but the data corresponding to that document is left in the segment files consuming disk space in the index. It’s not until that segment is merged away, either by normal merging over time or by an explicit call to optimize, that these bytes are reclaimed.
Finally the Directory class represents the location of a Lucene index. It is an abstract class that allows its subclasses (three of which are included in Lucene) to store the index as they see fit - in memory (RAMDirectory) or on disk (FSDirectory).

Searching
Nex up is the right leg - Searching.

Full pic

Searching Overview
When a user performs a search a Query will be issued against one or more indexes. The actual Query object is built using a QueryParser which passes the Query over to an IndexSearcher. The IndexSearcher can optionally also use a Filter together with the Query, whereby a subset of the results are returned. A Filter can be anything that makes sense in the context of the search. A Filter could e.g. be created for all results valid only for a given City. Another example could be a Filter where some results are filtered out due to security restrictions.
The IndexSearcher returns a TopDocs instance which contains an ordered array with ScoreDoc instances. A ScoreDoc instance in turn is holding the id of the Document and a score (a numeric value of relevance) for that Document. A ScoreDoc is merely a pointer to a Document - not the Document itself. See below picture for an example.

Searching Main Classes
Most of the classes where mentioned above in the overview. Their relationship is roughly as depicted in below class diagram.

The Searcher class is abstract with various overloaded search methods extended by the IndexSearcher. The search method returns an ordered collection of TopDocs which holds Documents ranked by computed scores. Lucene calculates a score for each of the documents that match a given query. You can specify the number of top results that need to be retrieved by specifying it in the IndexSearcher’s search method.
When firing of a search a Query is created and issued against an Index using an IndexSearcher. The IndexSearcher only accepts Query objects and to create theses we use a QueryParser. An IndexSearcher is basically opening an index in a read-only mode. The IndexSearcher offers some methods to query indexes of which some are implemented in the inherited abstract class Searcher.
A QueryParser translates a user entered search string into a Query object. The syntax which the QueryParser understands is governed by JavaCC (e.g. mathemati* ). An analogy of the search string is to think of it as a SQL expression which needs to be translated into something that the database (or in our case the search engine) can understand. The QueryParser does the translation of the string to something that Lucene understands.
A QueryParser also needs an Analyzer to break pieces of the search string into Tokens (same one as when indexing). The Query instance created from the QueryParser is then passed to the IndexSearcher. There are many different flavors of Query objects, some of which are shown below:

In the database world when you want to restrict on something you use a "where" condition in the issued query. In Lucene you can do the same using a a Filter. E.g. a QueryWrappingFilter is wrapping a Query and is usable when you want to restrict on a certain field in your index. Consider a domain entity like Ad. If the Ad has not been verified we might not want to display this Ad in the results to the user. A QueryWrappingFilter is a good choice for doing this and could look like:

Instead of using a filter you could also have used a BooleanQuery like below which could yield a query like : +isVerified:true (which can be copied and run on the index in Luke)

In Hibernate Search the syntax for querying is slightly different. Have a look at this example which will return a collection of Hibernate manged instances back to caller. The caller can update or even delete instances whereby the index underneath will also be updated or marked for deletion transparently to the caller upon transaction completion.

Well that concludes our quick overview about Lucene and Hibernate Search. Of course a lot more could be added to this but maybe that will come in the future.

Pet projects are nice...

2010-11-15T10:36:00.006+01:00

...even nicer if they are finished ad go live. After two years of development during weekends and evenings we are getting close to launch time for our (Andreas an me) little pet project. We have been through 3 stacks of technologies and finally settled on one stack which will be the one we go live with.
Anyway to facilitate our small but fast growing company ;-) we decided to launch this little company web page:

http://andreasgeorges.com/

It is a kind of Ben and Jerry site - the ice cream site - although I promise our little pet project is not about ice creams.

More to come

Process heap, Eden, Tenured and Permgen

2010-06-03T09:39:00.011+02:00

A picture says more than...

Testing Syntaxhiglighter

2009-11-04T22:15:00.006+01:00

To add a nice syntax highlightning I to make code easier to read do:

1 Add this to you Blog template. Go into Layout > Edit template and add this to you head tag:

2 Surround your code snippet with

The result might look like below


   @Test

    public void testCreateUser_userIdAlreadyUsed() {

        // Set expectations. For void methods we can not use the normal  expect(...).andReturn(...) idiom

        // instead we use something EasyMock calls matchers - e.g. EasyMock.anyObject()

        mockedActivityLoggerDao.persist((String) anyObject(), (Date) anyObject());

        expectLastCall().anyTimes();

        expect(mockedUserDao.findUserById("existingUserId")).andReturn(Boolean.TRUE).anyTimes();



        // Stop recording expectations and add all mocks

        replay(mockedActivityLoggerDao, mockedUserDao);



        // Start the test and do the assertions. Instantiate our class beeing testet and pass it the mocked collaborator

        User user = instanceUnderTest.createNewUser("existingUserId", "password", "James", "Gosssling", UserStatus.pending);

        assertNull(user);

    }

Concordion - specification based testing part 2

2009-06-09T21:11:00.003+02:00

See original document at : Concordion - specification based testing part 2
First post : Concordion - specification based testing part 1

Background

The first part was an introduction to why you might want to use a tool like Concordion. It was more or less based on an internal presentation and used to promote Concordion to management and analysts where I currently work. In this part I will try to explain in more technical terms how it works, how we have setup Concordion in practice and also give a few examples of Concordion test we are running. Also please refer to the excellent Concordion site at : http://www.concordion.org (by David Peterson).

Concordion definitions

Let us see how we can achieve our goals as defined in part 1 using Concordion as our testing framework. Before we look at examples there are some basic Concordion definitions we need to be aware of.

A Concordion specification is a well-formed XHTML
document describing a set of functionality in plain English (or whatever language you would like to use) and also data which is used to instrument what is called a fixture. The specification is the document where your business analyst or product owner should tell you what the test should do and check. The specification should be verbose enough to enable other stakeholders reading and understanding it. The first draft of the specification could be written in an OpenOffice document a wiki page or a text file - the important thing is that at some point it is converted (most probably by a developer) into a valid XHTML document containing the specification.

A fixture is a Java class that uses the specification as input, runs the specification against application code - the Service layer - and produces a report with green and red colored assertions. For a developer the fixture is basically a JUnit test, only instead of having test data within a Java class we have it externalized into the specification. For a business developer the fixture not important, the report is. The report will tell the business developer if the requirements were met according to the specification.

Figure: From specification and fixture we receive a nice looking report (which we can publish for stakeholders).

Concordion specification

A resulting Concordion report layout can look anyway you want it to look - using pictures, links etc. The Concordion specification determines the resulting layout and also how you call your fixture class. How to call your fixture class can be done in different ways. One way I found useful is a table based specification in which you can run a test over and over using different sets of instrumentation data (resembles a Fit test). Concordion provides support for this kind of tests using concordion:execute on a table.

Below is an example of a specification using a a table based approach. The first part - concordion:execute - will call an underlying method in the fixture class called runConcordionIntegrationTest(....). The passed in values are #clientPartnerNo etc and the data passed in are given in under <tr><th> tags using concordion:set for each parameter. The assertions are done under the <tr><td> tags where concordion:assertEquals are comparing values from the #result object created by the runConcordionIntegrationTest(....) method. The Result object is implemented as an inner class of the fixture and has a bunch of helper methods returning values that we can use Concordion assertions on.

Figure: A concordion:execute on a table example

Concordion fixture

In our setup we have a small hierarchy of classes in which our main test class is inheriting the BusinessTestCase, which has functionality reused in all our Concordion tests. MyBusinessTest class extends a Spring test class (rollbacks transactions, holds a jdbctemplat etc) and is a convenience class which has helper methods reused in other Concordion tests.

Figure: Class hierarchy used.

In BusinessTestCase there is a method called runConcordion (see figure below):

public abstract class BusinessTestCase
{
....
public void runConcordionAndProcessSpecification() throws Throwable
{
     ConcordionBuilder concordionBuilder = new ConcordionBuilder();
     Concordion concordion = concordionBuilder.build();
     ResultSummary resultSummary;
     if (resource != null)
     {
         resultSummary = concordion.process(resource, this);
    }
     else
     {
         resultSummary = concordion.process(this);
     }
     resultSummary.print(System.out);
     // This will write a summary file with one line: 
     // Rate: 100.0%. Tests: 6, successes: 6, failures: 0, exceptions: 0. 3/5/09..
 // This file is then the used by the wiki portal through small Javascript to show red green statuses
     ConcordionResultWriter.write(this, resultSummary);
     resultSummary.assertIsSatisfied();
}
....
}

Figure: Small code running the Concordion test and also writing out a file used by our Concordion portal page (Twiki)

Concordion portal

At the end of the day we want to show the output from the Concordion test runs without forcing users to go into each Concordion report to see the individual outcome of a Concordion test run. What we would like is a nice portal page where all Concordion test reports are listed as links to each one of the Concordion reports. The individual detailed Concordion report would most probably only be viewed at when there is an error displayed on the Concordion portal page, or perhaps if somebody wants to read the specification and and get an understanding of a specific business process or use case. The portal page helps give an overivew over all Concordion tests.

To help us achieve this we used Twiki (was already used) in which were developers can enter a link to their Concordion report. The Twiki page also uses some Javascript to parse a also the summary file mentioned above - which helps produce red / green bullets in the twiki page. The Javascript also aggregates the total result and checks that the date of the generated file is todays date. When a Concordion test is not ran during the night it will be flagged with red due to the date check done by the Javascript.

Since we already were using CruiseControl to run our JUnit test suites we decided to let CruiseControl also run the Concordion test suites (any other build server supporting ANt / Junit will work of course). The reports generated by Concordion are stored on an Apache web server and this web server directory is fronted / decorated using a above mentioned wiki page.

We opted to use a Concordion test suite per business module within the system. Each buisness module has its own header in the Twiki - se below picture.

Figure: Nightly build portal page for Concordion tests. Different modules have their own section listing the Concordion tests giving a nice quick overview over all tests.

Concordion example reports

The ways to create and layout a report are endless. The layout and the contents of the report would be the responsibility of your business developer to define together with the developer/s. Any data that injected into the actual fixture and displayed in the report should be an business id (e.g. one which can be used from a GUI to look-up the actual data). Do not use an id that does not have meaning to someone outside your development department. An id which means that a business guy needs to use sql to understand what it means is probably not a good id to display in the Concordion report.

Figure: Concordion test where we used a concordion:table approach, although in this case we only have one row we could easily add more rows to test different paths. This is much like a Fit test - but much easier to understand and use.

Another example report is shown below, not so fancy and no pictures. The Concordion specification tries to use a domain language to express what is the purpose and expected result. Embedded in the Concordion specification are assertions made showing up in green (or red). One thing that Concordion does not have - currently - is reuse of other specifications. In the example below we actually would benefit from a reusable Concordion specification in the setup part of the specification.

Finally

I listed some of the things you might want to watch out for when writing Concordion below

Do not write to technical Concordion tests. Concordion tests that are not meant to replace unit tests - some tests are just better of as unit tests. Low level unit tests that does not have meaning to anyone except for the guy who actually wrote the test and other developers are better of in a unit test. Once you start writing Concordion tests and have all the "surrounding" environment setup it is really easy to fall into writing every single test as a Concordion test. To many Concordion test on lower level hides the "more important" business tests. An example of a not so intuitive Concordion test which in my opinion would be more suitable as a unit test is given below:

Do not write big Concordion specifications. Concordion specifications that try to cover to much - different use cases all rolled into one Concordion specification - are hard to read and hard to maintain. Some of my early Concordion tests where far too long and were later on split into to smaller parts.

Write Concordion tests early. Less motivation and the ability to forget things as time passes by makes it harder to write them after a Sprint has completed - and your business guy might not be so keen on writing it afterwards since they moved on to new assignments.

Keep them green. If a Concordion test is starting to show red more often than it shows green it will make stakeholders uninterested at them. Beware of "they are usually broken" comments. Sometimes it might be better to remove a Concordion test from a suite if it has been showing red too long. Keeping them green is easier for a small sized system - but when a system is somewhat big with a lot of interfaces to other systems it is much harder to keep
them green.

Concordion is a great tool to use for your integration test layer. For a developer it is easy to use - if you know JUnit you practically already know how to use Concordion. For your business guys it shine with its tranparency of the test results. Above I mentioned the portal page we use - in the latest Concordion release there is something like a portal page support implemented (di not have the time to check yet).

GWT, Tomcat lite and Intellij

2009-03-02T16:33:00.004+01:00

A normal layout directory of Tomcat lite looks like:

To point to this directory from Intellij use the -Dcatalina.base (Tomcat) property to point to the Tomcat "lite" directory where you have all your configuration and property files set up.

Concordion - specification based testing part 1

2008-12-30T13:07:00.002+01:00

See original document at Concordion - specification based testing part 1

Background

I am currently working in a banking and finance project where the system handles straight through processing of buy and sell orders coming from the stock exchange. As I started little over a year ago we did not have any automatic integration tests in place, the set of unit tests were in a miserable state with a success rate round 50% - and nobody seemed to care (or have time to care). In previous projects I was used to a strong focus on unit testing and in the long run they always seemed to pay of. Now, one year later our JUnit tests are working, and on top of that we have a large set of automated integration tests using Concordion which are increasing for each milestone. Concordion has really made the business side engaged and dare I say excited to be part of the test writing process. In the past before joining my this company I had been using Fit and also JUnit to help doing integration testing. Fit is a nice tool for writing specifications in html and exercising your system with glue code written in Java (other languages are supported as well). Fit was invented in 2002 by Ward Cunningham and has been further developed into Fitnesse (a wiki frontend to Fit basically) - more info at Wikipedia - Fit. Although Fit is nice it lacks the easy setup that Concordion has - no special Fit runners are needed just invoke your Concordion tests as you would invoke any other JUnit test in your development environment. And on top of being easy for developers, the output looks really great, the Concordion website is really good and finally, the guy who wrote it and is maintaining it (David Peterson) is very quick in answering any of your questions on the mailing list of Concordion. The first part of this blog entry is more about why you would want to use Concordion and the second is more technical about how it works and how we have set it up in my current project. The material was used to kick things of where I work - so perhaps it might help you do the same thing at your work.

What are our goals ?

At the end of the day we want to have a solid system and a happy customer. To achieve this we would like to have a few things like:

Automate testing. Focus on service layer where use case starts, typically an session ejb or transaction demarcated Spring bean.

Visualize input and output for developers as well as non developers, thereby minimizing conceptual barriers between developers and non developers.

Couple business specifications closer to running code, thereby avoiding divergence.

Instrument tests with data outside of actual test code.

So lets go through the goals one by one and later on have a look on more technical details on how Concordion works and how we set things up with a TWiki and CruiseControl.

Automating service layer integration tests

Fortunately as a Java developer there are a multitude of different testing tools to choose from. Some testing tools are more cross cutting like e.g. CruiseControl and help exercise the actual tests while others are more targeted on different layers of your application stack like JUnit or Selenium. One could picture it as a pyramid where you have most of the running application code at the bottom exercised by developer only tools like JUnit or TestNG. At the top of the pyramid one could use Selenium to test the system through recorded test cases that can run in your continuous integration server (like CruiseControl). Selenium is a bit fragile when user interfaces changes are frequent - so I would prefer to focus more on the service or business layer. I also use EasyMock for testing framework dependent code - e.g. code extending some nasty hierarchy of abstract classes. EasyMock works great and is quite easy to use with the Java5 support they have.

Visualize input and output, minimize barriers

JUnit is great as a developer tool for writing unit tests quickly. It makes refactoring easier and helps other developers understand what you as a developer coded. One problem with JUnit is that non developers, e.g. business analysts, do not know or understand what JUnit is actually doing or testing. And if you do not understand something you do not get involved. Concordion on the other side, has the advantage of JUnit (it actually is extending JUnit) and additionally also adds the benefit of visualizing the input and output of a test run. Concordion is a specification tool which encourages non developers to write the specification before actual implementation starts. This is really the biggest strength of Concordion - encouraging discussion between different members of a team and getting the "other side" involved. For a developer to have a specification that can actually be exercised instead of a text document is a great advantage and forces the business developer to think through the business case thoroughly. The requirements are clearer since it is obviously hard to write the actual business implementation if the Concordion specification is not understood. Requirements are interpreted by developers in one way and by testers in another way – and tested against the artifact of developers i.e. the code.

Tighter coupling, avoiding divergence.

Consider what happens with a specification written in a text document and what happens to the actual application code being developed over a period of time. Which of these two artifacts is closest to the truth on what the application is doing? Or consider an UML class diagram artifact describing the relationship between entities in your application compared to the actual entities as coded and annotated in your running application. Obviously, the application code is closer to the truth on what actually is implemented, rather than an UML artifact or a requirements document. Why? Well, what usually happens over time is that some newly written code is changing rapidly in the beginning and finally stabilizes somewhat. New requirements not foreseen are quickly implemented no meet the deadline - but the original requirement document or UML diagram is not changed accordingly and the description of the application is not up to date. The application code and the high level documents start to diverge from the code. Concordion tries to solve this divergence by tighter coupling of specifications into the code tree and actually having specifications version ed along with the application code. makes divergence less probable.

Instrumentation of test data, or test data injection

How many times have you not put some hard coded test data into your unit tests? Concordion lets you abstract these test data away and put the test data into the Concordion specification (basically a html file) so that the fixture (the glue between the specification and the application code being tested, basically a Java class) is clean. Furthermore - using a table format in the Concordion specification helps running the same test fixture over and over again with different input data. Below is an example where the same test fixture is run twice - once for each row. The 2 columns to the left are data injected into the test fixture for every run and the asserted data is the green columns. The "Generated Delivery" column is the business object being produced after a successful run and can be used within the actual system for further investigation - especially if the test fails.

On part 2 of this blog entry we will look into how Concordion works and how we used it at my current work location.

Spying on SQL in Weblogic using P6Spy

2008-04-07T08:36:00.007+02:00

Sometimes it might be good to actually see what is going on between your application server and the database. To set the stage for this we ar going to follow these little instructions:

1 Download P6Spy and unpack to get the jar and spy.properties file. P6Spy is old but still works nice - P6Spy

2 Copy the jar file into an arbitrary folder and let Weblogic know this by adding it
to the classpath. In startWebLogic.cmd do:
set CLASSPATH=%CLASSPATH%;%WEBLOGIC_CLASSPATH%;....C:/...../p6spy-1.3.jar

3 Copy the spy.properties file to the root location of your server (domain instance) so that it will be picked up on start.

4 Alter the spy.properties file and change according to what datbase you are using. E.g for Oracle use:
realdriver=oracle.jdbc.driver.OracleDriver
useprefix=true
deregisterdrivers=true
logfile = C:/bea/..../logs/spy.log

5 Finally change the config.xml file (root of your domain instance) so that the pool configuration will be:

Now you are able to tail the spy.log file for all SQL statements going on. Very useful for debugging Hibernate stuff.

Hibernate cache : UnsupportedOperationException

2008-04-04T20:29:00.005+02:00

Consider the exception below :

2008-04-01 15:34:22,828 ERROR org.hibernate.cache.ReadOnlyCache - Application attempted to edit read only item: org.grouter.dao.TheDAO.aCollection#org.grouter.dao.dao.TheClass@148ad
java.lang.UnsupportedOperationException: Can't write to a readonly object
at org.hibernate.cache.ReadOnlyCache.lock(ReadOnlyCache.java:43)
at org.hibernate.action.CollectionAction.beforeExecutions(CollectionAction.java:110)
at org.hibernate.engine.ActionQueue.prepareActions(ActionQueue.java:257)

If you have this exception it is probably due to you your caching strategy settings for Hibernate beeing set to read-only. This can be set on entity level and also on collection level. If an entity marked as read-only has a collection of another read-only collection then you need to ensure that both the originating entity and the collection of entities are both marked ar mutable="false". This should be set on both entity A and the entities of the collection belonging to entity A. Setting mutable="false" is the same as to say they are immutable, i.e. not changed after creation.
However, this might not solve everything for you. Consider an entity A is set to read-only and has a collection with read-only entities B. If you load A and use a setter on A after beeing loaded the entity will be marked as dirty - above exception will be thrown. Now if A is loaded and a new query is issued to get the collection of read-only B entites and then set on the entity A - setCollection(newColl) - then again the above exception will be thrown.

Remote debug Weblogic or JBoss from Intellij

2008-03-19T09:30:00.010+01:00

Although I tend to write unit tests for almost everything I write - one java class <-> one junit class - I sometimes have a need to be able to step into the executing code of an application server and debug the running code. Java has a standard way of debugging remotely for a running Java process which is very nice.
To remote deubg Weblogic from Intellij
1 - Start the application server up with a set of debug switches. In Weblogic 8.1 you would alter the start up script (setup.cmd or something like that) and put: set DEBUG_PORT=19001
set JAVA_OPTIONS_DEBUG=-Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,address=%DEBUG_PORT%,suspend=n,server=y
When weblogic starts it will listen for incoming request on port 19001
2 - In Intellij you create a debug configuration. Choose Run / Debug and click +. Choose, in this case, Weblogic and alter settings according below picture. In Weblogic the name of the server is the name of your application under your domain, something like config/mydomain/applicationsnameOfServer.

And on the Startup/Connection tab you alter accoring to:

To remote debug JBoss from Intellij:
1 Add this to run.sh
JAVA_OPTS="$JAVA_OPTS -Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=4142,suspend=n"
2 Then alter Intelliij debug confiuratino

Hibernate Caches

2008-01-31T08:28:00.000+01:00

Background
Hibernate comes with three different caching mechanisms - first level, second level and query cache. Truly understanding how the Hibernate caches work and interact with each other is important when you need to increase performance - just enabling caching in your entity with an annotation (or in classic .hbm.xml mapping file) is easy. But understanding what and how things happens behind the scenes is not. You might even end up with a less performing system if you do not know what you are doing.

SessionFactory and Session
The purpose of the Hibernate SessionFactory (called EntityManager in JEE) is to create Sessions, initialize JDBC connections and pool them (using a pluggable provider like C3P0). A SessionFactory is immutable and built from a Configuration holding mapping information, cache information and a lot of other information usually provided by means of a hibernate.cfg.cml file or through a Spring bean configuration.
A Session is a unit of work at its lowest level - representing a transaction in database lingua. When a Session is created and operations are done on Hibernate entities, e.g. setting an attribute of an entity, Hibernate does not go of and update the underlying table immediately. Instead Hibernate keeps track of the state of an entity, whether it is dirty or not, and flushes (commits) updates at the end at the end of a unit of work. This is what Hibernate calls the first level cache.

The 1st level cache
Definition: The first level cache is where Hibernate keeps track of the possible dirty states of the ongoing Session's loaded and touched entities. The ongoing Session represents a unit of work and is always used and can not be turned of. The purpose of the first level cache is to hinder to many SQL queries or updates beeing made to the database, and instead batch them together at the end of the Session. When you think about the 1st level cache think Session.

The 2nd level cache
The 2nd level cache is a process scoped cache that is associated with one SessionFactory. It will survive Sessions and can be reused in new Session by same SessionFactory (which usually is one per application). By default the 2nd level cache is not enabled.
The hibernate cache does not store instances of an entity - instead Hibernate uses something called dehydrated state. A dehydrated state can be thought of as a deserialized entity where the dehydrated state is like an array of strings, integers etc and the id of the entity is the pointer to the dehydrated entity. Conceptually you can think of it as a Map which contains the id as key and an array as value. Or something like below for a cache region:

{ id -> { atribute1, attribute2, attribute3 } }
{ 1 -> { "a name", 20, null } }
{ 2 -> { "another name", 30, 4 } }

If the entity holds a collection of other entities then the other entity also needs to be cached. In this case it could look something like:

{ id -> { atribute1, attribute2, attribute3, Set{item1..n} } }
{ 1 -> { "a name", 20, null , {1,2,5} } }
{ 2 -> { "another name", 30, 4 {4,8}} }

The actual implementation of the 2nd level cache is not done by Hibernate (there is a simple Hashtable cache available, not aimed for production though). Hibernate instead has a plugin concept for caching providers which is used by e.g. EHCache.

Enabling the 2nd level cache and EHCache
To get the 2nd level cache working you need to do 2 things:
1 Cache Strategy. Enable a cache strategy for your Hibernate entity - either in the class with an annotation or in the hibernate mapping xml file if you are stuck with pre java5. This can be done for an entity by providing this little snippet into your hbm.xml file (a better place is to store the cache setting strategy in hibernate.cg.xml file )

<class name="org.grouter.domain.entities.Router" table="ROUTER">
<cache usage="transactional|read-write|nonstrict-read-write|read-only" />
<id ...
</class>

or using an annotation for your entity (if you are on java5 or greater)

@Entity
@Cache(usage = CacheConcurrencyStrategy.NONSTRICT_READ_WRITE)
public class Router { ... }

And as mentioned above if you want to cache collections of an entity you need to specify caching on collection level:
<class name="org.grouter.domain.entities.Router" table="ROUTER">
    <cache usage="transactional|read-write|nonstrict-read-write|read-only"/>
    <id ...
    <set name="nodes">
        <cache usage="transactional|read-write|nonstrict-read-write|read-only"/>
        ...
    </set>
</class>

Hibernate has something called a cache region which by default will be the full qualified name of your Java class. And if you like me are a fan of convention over configuration you will use the default region for an entity. A cache region will also be needed for the collection using the full qualified name of the Java class plus the name of the collection name (i.e. org.grouter.domain.entities.Router.nodes)

2 Cache provider. Setting up the physical caching for a cache provider. If you are using EHCache - which is the most common choice i dear to say - then you will need to specify some settings for the cache regions of your entities in a file called ehcache.xml. The EHCache will look for this file in the classpath and if not found it will fallback to ehcache-failsafe.xml which resides in the ehcache.jar library A typical sample for an EHCache configuration could look like (see mind map below for explanations):

<cache name="org.grouter.domain.entities.Router" maxElementsInMemory="1000" eternal="false" timeToLiveSeconds="600" overflowToDisk="false"/>
and
<cache name="org.grouter.domain.entities.Router.nodes" maxElementsInMemory="1000" eternal="false" timeToLiveSeconds="600" overflowToDisk="false"/>

The name maps to the name of the cache region of your entity. The attribute maxelementsInMemory needs to be set so that Hibernate does not have to swap in and out elements from the cache. A good choice for a read only cache would be as many entities there are in the database table the entity represents. The attribute eternal, if set to true means that any time outs specified will be ignored and entities put into the cache from Hibernate will live for ever.
Below is a mindmap for the second level cache and how it relates to the SessionFactory and the 1st level cache.

The Query cache
The Query cache of Hibernate is not on by default. It uses two cache regions called org.hibernate.cache.StandardQueryCache and org.hibernate.cache.UpdateTimestampsCache. The first one stores the query along with the parameters to the query as a key into the cache and the last one keeps track of stale query results. If an entity part of a cached query is updated the the query cache evicts the query and its cached result from the query cache. Of course to utilize the Query cache the returned and used entities must be set using a cache strategy as discussed previously. A simple load( id ) will not use the query cache but instead if you have a query like:

Query query = session.createQuery("from Router as r where r.created = :creationDate");
query.setParameter("creationDate", new Date());
query.setCacheable(true);
List l = query.list(); // will return one instance with id 4321

Hibernate will cache using as key the query and the parameters the value of the if of the entity.
{ query,{parameters}} ---> {id of cached entity}
{"from Router as r where r.id= :id and r.created = :creationDate", [ new Date() ] } ----> [ 4321 ] ]

Pragmatic approach to the 2nd level cache
How do you now if you are hitting the cache or not? One way is using Hibernates SessionFactory to get statistics for cache hits. In your SessionFactory configuration you can enable the cache statistics by:

<prop key="hibernate.show_sql">true</prop>
<prop key="hibernate.format_sql">true</prop>
<prop key="hibernate.use_sql_comments">true</prop>
<prop key="hibernate.cache.use_query_cache">true</prop>
<prop key="hibernate.cache.use_second_level_cache">true</prop>
<prop key="hibernate.generate_statistics">true</prop>
<prop key="hibernate.cache.use_structured_entries">true</prop>

The you might want to write a unit test to verify that you indeed are hitting the cache. Below is some sample code where the unit test is extending Springs excellent AbstractTransactionalDataSourceSpringContextTests

public class MessageDAOTest extends AbstractDAOTests  // which extends AbstractTransactionalDataSourceSpringContextTests
{
   public void testCache() 
    {
        long numberOfMessages = jdbcTemplate.queryForInt("SELECT count(*) FROM message ");
        System.out.println("Number of rows :" + numberOfMessages);
        final String cacheRegion = Message.class.getCanonicalName();
        SecondLevelCacheStatistics settingsStatistics = sessionFactory.getStatistics().
            getSecondLevelCacheStatistics(cacheRegion);
        StopWatch stopWatch = new StopWatch();
        for (int i = 0; i < 10; i++)
        {
            stopWatch.start();
            messageDAO.findAllMessages();
            stopWatch.stop();
            System.out.println("Query time : " + stopWatch.getTime());
            assertEquals(0, settingsStatistics.getMissCount());
            assertEquals(numberOfMessages * i, settingsStatistics.getHitCount());
            stopWatch.reset();
            System.out.println(settingsStatistics);
            endTransaction();

            // spring creates a transaction when test starts - so we first end it then start a new in the loop
            startNewTransaction();
        }
    }
}

The output could looke something like:

30 Jan 08 23:37:14  INFO org.springframework.test.AbstractTransactionalSpringContextTests:323 - Began transaction (1): 
transaction manager [org.springframework.orm.hibernate3.HibernateTransactionManager@ced32d]; default rollback = true
Number of rows :6
Query time : 562
SecondLevelCacheStatistics[hitCount=0,missCount=0,putCount=6,elementCountInMemory=6,elementCountOnDisk=0,sizeInMemory=8814]
30 Jan 08 23:37:15  INFO org.springframework.test.AbstractTransactionalSpringContextTests:290 - Rolled back transaction 
after test execution
30 Jan 08 23:37:15  INFO org.springframework.test.AbstractTransactionalSpringContextTests:323 - Began transaction (2): 
transaction manager [org.springframework.orm.hibernate3.HibernateTransactionManager@ced32d]; default rollback = true
Query time : 8
SecondLevelCacheStatistics[hitCount=6,missCount=0,putCount=6,elementCountInMemory=6,elementCountOnDisk=0,sizeInMemory=8814]
30 Jan 08 23:37:15  INFO org.springframework.test.AbstractTransactionalSpringContextTests:290 - Rolled back transaction 
after test execution
30 Jan 08 23:37:15  INFO org.springframework.test.AbstractTransactionalSpringContextTests:323 - Began transaction (3): 
transaction manager [org.springframework.orm.hibernate3.HibernateTransactionManager@ced32d]; default rollback = true
Query time : 11

Another way to spy on what Hibernate is doing is to proxy the jdbc driver used by a proxy driver. One excellent one I use is p6spy which will show you exactly what is issued over a JDBC connection to the actual backend database. For other tips have a look below in the mindmap.

Maven book

2007-11-23T09:23:00.000+01:00

Found this very nice information about Maven at: Maven book

Maven tips - loading resource for unit test

2007-10-23T10:49:00.000+02:00

Got a question the other day about how to load a resource for a unit test in Maven. I knew I had the same problem over a year ago but could not remember the exact way to solve it. As always Maven's email list is the number one source of knowledge - and I knew I read this really good post about this problem (here is the post at link to post).
To recap the problem is how classloaders work in Java. In the case of running a Unit test in Maven using the Surefire plugin you end up with a class loader hierarchy like this:
1 System classloader - where Maven runs
1.2 Plugin classloader - created by maven
1.3 Surefire classloader - created by Surefire
1.4 Unit tests

So if you do load your resources like this using Classloader it will not work - instead ask the class for the class loader and get the reource from that classloader

ClassLoader.getSystemResourceAsStream("test.properties");  // does not work
this.getClass().getClassLoader().getSystemResourceAsStream("test.properties"); // works

Software lessons - do not forget df -h

2007-10-17T15:11:00.000+02:00

During a four months develoment period the project was reaching one major milestone: acceptance tests. Working long hours and fixing those final show stoppers is always an unpleasant thing to do - but I has to be done.
Friday afternoon and we have a final build for the customer arriving on Monday to do the acceptance testing. Puh we made it - or at least that was the fealing when leaving the office heading home.
Monday morning. I come into the office early and find out that we can not loginto the system. What? After 30 minutes I realize that the release and installation did not install that little jboss config file - the file that is cruicial for login to work. So on to my desk get some java and start reading emails.
"Hey Georges - Jboss is not starting. Could you and x have a look". Over to x desk and start the quest for "why jboss does not start". The installation of the software is a bit complex using jboss template xml files and replacing placeholders in those files using the sed command (check out Maven Assemblies to simplify building release). After a while see a strange behaviour of files beeing cropped. Our initial assumption was that the sed did not work if some fields where empty, our next assumption was that there was something wrong with buffers on this Solaris machine.

Finally after doing a df -h we see that we are out of disk space.

Desing pattern - generic dao

2007-08-09T22:38:00.000+02:00

Background
A year back I worked in a project where we wanted to extend an system handling an auction site. The system used DAOs and had CRUD operations spread in different DAOs. To simplify and generalize the CRUD operations the Generic DAO (Java5) pattern was implemented. The Generic DAO pattern was proposed on a Hibernate forum and picked up by Christian Bauer who implemented it in the Caveat example application. It is also discussed in the second edition of the Hibernate book - Java persistence with Hibernate and on a post at Hibernate.

Enter the pattern
The grouter (a small open source message router I am working on) system contains a set of components - as depicted below - where the core-domain component is responsible for providing persistence and transaction demarcation. The core-domain component uses Hibernate entities and Spring to provide this services to clients like the presentation-web component. There is also an option to use Ejb3 stateless beans instead of Spring - only that road needs more work since everything else in the grouter is using Spring.

To avoid duplicating CRUD operations I adopted the proposed Generic DAO pattern as described by Christian Bauer at the Hibernate website. The pattern relies on Java5 generics and consists of:
GenericDAO - The GenericDAO interface specifies the CRUD operations using Java5 generics. GenericDAO is a generic type interface that has two type parameters - T and ID. These type parameters are later replaced by type arguments when the generic type is instantiated.
GenericHibernateDAO - The abstract GenericHibernateDAO class provides an implementation of the GenericDAO interface using two type parameter. It subclasses the Spring class HibernateSuport to get Springs support for handling the Hibernate Sessionfactory and Sessions.
GenericEjb3DAO - Just like GenericHibernateDAO but in this case we are using Ejb3 and the EnitityManager instead of Hibernates SessionFactory which is injected from the JEE container. There is also a MessageDAO implemenation called MessageDAOBean (only it is not used in the grouter).
MessageDAO - This interface specifies non CRUD operations specific for the Message enitity.
MessageDAOImpl - MessageDAOImpl implements the MessageDAO interface and also extends the abstract class GenericHibernateDAO and instantiates a parametrized type of that generic type in its constructor.

Below is an UML diagram of the classes discussed above.

You are free to check out the source code for the grouter - see this page on how to do that.

Design pattern - command

2007-08-06T23:20:00.000+02:00

Background
When designing a message router (see open source project grouter ) I wanted to decouple the senders from the receivers using an internal queue and potentially a JMS queue. The Command Pattern let me do just that.

Enter the pattern
Definition: In a command pattern objects represent actions - command objects. A command pattern decouples the sender from the receiver. The sender creates commands which are read by the receiver. The receiver then calls execute (or some other method defined by the interface for the commands) not knowing what and how to do the unit of work - which the command itself knows.
The grouter contains inbound EndPoint readers and outbound EndPoint writers using an internal queue. An EndPoint in the grouter domain represents a source or destination of different types (EndPointType ), e.g. JMS or ftp. A reader should ideally only read from an inbound EndPoint, create a command (AbstractCommand ) and pass it to the internal queue. On the other side of the queue an EndPoint writer (command consumer) would pop the internal queue and call a special method on the command object instance ( execute() ). The command consumer should be designed in a general way and thereby able to handle different kind of commands - still maintaining a template for how to handle commands (backing up, validating and sending).
EndPoint writers (command consumers) and EndPoint readers (command producers) could then be grouped together to form nodes in the message router.

Full size

Maven tips - sources and intellij project files

2007-08-03T11:40:00.000+02:00

To generate Idea Intellij project files and download sources for 3rd party dependencies:

mvn idea:idea -DdownloadSources=true

This can also be added to your pom for the idea plugin (or Eclipse and Netbeans - just look up the plugin docs).

<project>
[...]
<build>
[...]
<plugins>
<plugin>
<groupid>org.apache.maven.plugins</groupid>
<artifactid>maven-idea-plugin</artifactid>
<configuration>

<downloadsources>true</downloadsources>
<downloadjavadocs>true</downloadjavadocs>
<dependenciesaslibraries>true</dependenciesaslibraries>
</configuration> </plugin>
</plugins>
[...]
</build>
[...]
</project>

Maven tips - compiler plugin

2007-08-03T11:19:00.000+02:00

Usually you get off by using target and source for a java 1.4 or java 1.5 compiler to produce 1.3 class files. However I bumped into a problem with a class extending an abstract class in the javax.sql package, which in java 1.4 had a new abstract method. I could not get the 1.5 compiler using maven to build that class with target and source set to 1.3. The compiler complained that I had to implement that method even though I should not have to.

To solve this I configured the compiler plug in according to below snippet:

  [...]
  <build>
    [...]
    <plugins>
      <plugin>

          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-compiler-plugin</artifactId>
          <configuration>
            <compilerVersion>1.3</compilerVersion>
            <executable>${jdk1.3.home}/bin/javac.exe</executable>
            <encoding>ISO-8859-1</encoding>
            <verbose>true</verbose>
            <target>1.1</target>
            <source>1.3</source>
            <fork>true</fork>
          </configuration>
        </plugin>
    </plugins>
    [...]
  </build>
  [...]
</project>

This worked fine. It forks of a new process using the command line to do the compilation. Command lines in windows are - well - not good. There is a size limitation of 2K (win2k) and 4K (WinXP). At some point I had another problem where maven just spit out that it could not parse the output of the error log. This had to do with exactly that - the command line being to long - in my case it was ~14000 characters. The project had a lot of dependencies. Fortunately for that maven project I could switch back to using jdk1.5 with target and source 1.3.

More info at maven-compiler-plugin

Why I do no like ClearCase

2007-06-18T14:57:00.000+02:00

1 It is slow... no matter what the Rational salesguy tells you - those 1-4 secs of delays are making me crazy.
2 Their client applications are far from good AND they do not run on my Intel Mac :-( How about making the ClearCase Explorer handle a flat view? The tools needs some consilidation since some functionality can be found in one tool and other in another.
3 Clearcase do not integrate well with other open source projects. Continous integration - like Continuum - does not handle Clearcase. How about beeing able to work against ClearCase from Eclipse? I see guys in my project switching to a ClearCase tool from Eclipse just to be able to edit the read only file.
4 Try working off line! How do you do that? Nobody at my client site seem to do this.
5 It is expensive, very expensive. And you need someone who can administer their server/s.

Design Pattern - Strategy

2007-04-23T21:54:00.000+02:00

Background
On a project I worked a couple of years back we used JMS to decouple subsystems and also provide distribution of workload - worked great. Now this system was relatively big with over 100 working nodes. Every time when the application server was restarted all clients (a lot of them) lost contact with the application servers message broker (JBoss) and we started to get those lovely org.jboss.mq.SpyJMSException. The basic problem was understanding of how JMS works and that you should always register an exception listener on the JMS connection. In an effort to solve the the reconnect problem and also hide some of the JMS plumbing code I developed a small set of classes. The classes have been refactored completely and can now be found at my open source message router ( see source code - grouter ) along with unit tests.

Enter the pattern
Definition: A strategy pattern is an encapsulation of an exchangeable algorithm using an interface or an abstract class.

Consider a use case where a client wants to use JMS destination - a queue or a topic. The client does not want to know how to reconnect to a message broker if this message broker is restarted and a connection is lost. How to reconnect should not be the concern of the client - it should be hidden to the client who only needs to tell the Destination what "Strategy" to use. In the below class diagram a Destination class has an association to a RebindBehavior - in this case an abstract class but it could also be an interface. Upon instantiation (or through a setter) on the Destination a client can specify which strategy it wants to use for a rebind scenario if/when a message broker is restarted. How - the algorithm - is hidden by the abstract class and its subclasses. The strategy can be changed in run time if needed and is not statically compiled into the Destination class.

Full size (new window)

A client wanting to listen to messages from a JMS queue could look like below (code is from a JMSReader in the open source grouter project I am working on)

    private AbstractDestination initConnection(Node node) throws Exception 

    { 
        Map endPointContext = node.getInBound().getEndPointContext(); 
        String destinationName = (String) endPointContext.get(JMS_DESTINATIONNAME); 
        String queueConnectionFactory = (String) endPointContext.get(JMS_CONNECTIONFACTORY); 
        String contextFactory = (String) endPointContext.get(JMS_CONTEXTFACTORY); 
        String urlPkgPrefixes = (String) endPointContext.get(JMS_URLPKGPREFIXES); 
        String providerUrl = node.getInBound().getUri();    
        boolean isQueue = Boolean.parseBoolean((String) endPointContext.get(JMS_ISQUEUE)); 
        AbstractDestination destination; 
        if (isQueue) 
        { 
            // Queue destination 
            destination = new QueueListenerDestination(destinationName, queueConnectionFactory, 
                    new NeverRebind(), getInitialContext(providerUrl, contextFactory, urlPkgPrefixes), 

                    null, AcknowledgeMode.CLIENT_ACKNOWLEDGE); 
        } else 
        { 
            // Topic destination 
            destination = new TopicListenerDestination(destinationName, queueConnectionFactory, 
                    new NeverRebind(), getInitialContext(providerUrl, contextFactory, urlPkgPrefixes), 

                    null, AcknowledgeMode.CLIENT_ACKNOWLEDGE); 
        } 
        try 
        { 
            destination.bind(); 
        } catch (Exception e) 
        { 

            throw new Exception(e.getMessage(), e); 
        } 
        return destination; 
    }

Hope this helps clarify a very simple but strong pattern. Another use case could be to define a logging behavior - log to file, log to database etc.

JavaOne 2007 - Creating Amazing Web Interfaces with Ajax

2007-04-20T12:21:00.001+02:00

This guy was a rockstar last year and well worth watching - Ben Galbraith

"...In this session, Dion Almaer and Ben Galbraith of Ajaxian.com
demonstrate exactly how to create rich web interfaces by using modern
techniques pioneered by the Ajax community. They do this by presenting
a variety of specific interface scenarios and case studies."

Windows app in MacOS - Parallel

2007-04-17T21:30:00.000+02:00

I am quite happy with everything on my Mac. It is easy, looks good, has the power of Unix shells coupled with a GUI that looks great. One thing missing though on my Mac is an application I use for UML modeling - Enterprise Architect. EA is easy and fun to use - and fast. Never ever do modeling in Visio - demand to use a real UML tool like EA. I have use TogetherJ in the past but Borland has managed to mess that tool up and moved it to the messy and sluggish Eclipse platform.
To use EA on my Mac I have installed - Parallel Desktop. I have tried PD previously (and VMVare back when I used a windowz pc) but since their latest release when they introduced something they called coherent mode it really looks and behavs great. Coherent mode makes a win app behave like an MacOS application frame. You can minimize it, tab through to it etc. Have a look on the attached pic:

Access your company mail i gmail

2007-03-16T12:06:00.000+01:00

I love the gmail interface and thought it would be nice to have the same interface for my company's mail also. To do so login to gmail and go to settings and click tab "Accounts" then add a new account "Get mail from other accounts:"

Next in the pop up add the company email address and the url to the pop server at your company:

If everything works out fine gmail will start importing you old mail into gmail and you will get something like this:

Good luck

Quotes

2007-02-19T21:30:00.000+01:00

"...premature optimization is the root of all evil." - Donald Knuth

"With great powers come great responsibilities" - Spiderman

"premature generalization is the root of all evil" - ?

Reducing I/O on a central CVS/Cruisecontrol server

2007-02-19T21:29:00.000+01:00

I know of two ways to reduce I/O on a cvs/build machine. One concerns the CruiseControl builds and one concerns how CVS handles updates from developers/users.

Normally you schedule CruiseControl to run every x min and poll cvs if there are any modifications made. This is today done using the a modification set with type cvs. By changing this to use a modification set of type compound and altering CVSROOT commitinfo to produce a loginfo file on user/developer commits one could reduce I/O to actually happen only when somebody has comitted anything into the CVS repository. The commitinfo file can be used to produce commit files for different cvs modules - thereby one could let the compound modicification set in CruisControl for one build use one file as a trigger to actually do a cvs update (and giving us I/O on io).

The second way of reducing this I/O traffic has to do with how CVS works. Normally when a user or developer does a CVS update all the local entries are sent of to the central cvs server which does an update locally -> producing a lot of I/O for every update. One way of getting rid of this I/O traffic is to reconfigure cvs (something like /etc/init.d -> cvs -T /cvstmp -f --allow-root=/opt/utm/pse/src pserver) so that instead of writing temp files to disk is writes it to a mounted internal memory location. This will speed up updates (~10times) and make the I/O traffic a lot less.

Maven tips - deploy dependency internally

2007-02-19T21:19:00.000+01:00

Sometimes you may not find a 3rd party dependency on the Internet - or you need to upload some company library to you corporate internal Maven2 repository. Define this in your local settings.xml file:

xxxxxxxx

Then issue this command :
$ mvn deploy:deploy-file -DgroupId=test -DartifactId=test -Dversion=1.0.0 -DgeneratePom=true -Dpackaging=jar -Dfile=test-1.0.0.jar -DrepositoryId=buildserver -Durl=scp://theipaddresss:/home/bob/public_html/maven2

Maven tips - skip tests

2007-02-19T21:17:00.000+01:00

Sometimes somebody checks in something that breaks - compilation error. Or even something that breaks the Unit tests run with the Maven build.
In Maven a compile error or a unit test failure both breaks the build - and sometimes we need to continue working even though the unit tests are not working (perhaps bob the mighty Java guru is fixing it...). Anyway, to skip tests do:

$mvn -Dmaven.test.failure.ignore=true clean install

Maven tips - downloadSources

2007-02-19T20:11:00.000+01:00

Maven build failure - old xerces jar

2007-01-22T15:29:00.000+01:00

Today I suddenly started to get test failures - which showed up as failures when trying to parse a Spring xml file for DAO beans. A nice little stack trace showed up (in the Surefire report produced by Maven):

-------------------------------------------------------------------------------
Test set: com.xyz.classifieds.data.dao.CategoryDAOTest
-------------------------------------------------------------------------------
Tests run: 5, Failures: 0, Errors: 5, Skipped: 0, Time elapsed: 0.077 sec <<< FAILURE!
testClass(com.xyz.classifieds.data.dao.CategoryDAOTest) Time elapsed: 0.009 sec <<< ERROR!
org.springframework.beans.factory.BeanDefinitionStoreException: Unable to validate using XSD: Your JAXP provider [org.apache.xerces.jaxp.DocumentBuilderFactoryImpl@533bdf] does not support XML Schema. Are you running on Java 1.4 or below with Apache Crimson? Upgrade to Apache Xerces (or Java 1.5) for full XSD support.
at org.springframework.beans.factory.xml.DefaultDocumentLoader.createDocumentBuilderFactory(DefaultDocumentLoader.java:102)
...

Struggling to find why suddenly my maven build test started to fail took me a couple of hours. Using mvn test -X is a good way of finding out what actually is put on the Surefire plugin (unit test) classpath. However doing so did not reveal some old Xerces jar laying around before the actual Xercer jar I was trying ot use. The actual version of Xerces used was indicating the latest - but probably some of the other dependencies held an Implementation conflicting with the Xercer jar. If I find time I will investigate (probably never) more on which dependecny pulled in some classs already present in Xerces. I finally had to resort to putting the latest and greatest Xerces jar in the $JAVA_HOME/lib/endorsed folder to get it first in my classpath.

First cup

2007-01-20T09:10:00.000+01:00

Just testing