Apache Solr Install Notes

I started working on full text searching solution for our map application. No surprise, Apache Solr is the selected solution and here are some quick notes for you to get started with it.

Part 1 - Basic Setup

1. Download Solr from: http://www.apache.org/dyn/closer.cgi/lucene/solr/

2. Unzip to: C:\Program Files\Apache Software Foundation\Solr (ensure that you rename the solr download to Solr)

3. Stop Tomcat

4. Copy C:\Program Files\Apache Software Foundation\Solr\dist\apache-solr-1.4.1.war to: C:\Program Files\Apache Software Foundation\Tomcat 6.0\webapps

5 Rename: C:\Program Files\Apache Software Foundation\Tomcat 6.0\webapps\apache-solr-1.4.1.war to Solr.war

6. Open: C:\Program Files\Apache Software Foundation\Tomcat 6.0\bin\tomcat6w.exe goto Java tab and add Java options: -Dsolr.solr.home=C:\Program Files\Apache Software Foundation\Solr\example\solr

7. Start Tomcat and go to: http://localhost:8080/Solr/admin/

Part 2 - Configuring and running the DataImportHandler 

1. Added the following to file: C:\Program Files\Apache Software Foundation\Solr\example\solr\conf\solrconfig.xml

<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">

    <lst name="defaults">

      <str name="config">data-config.xml</str>

    </lst>

</requestHandler>

2. Downloaded sqljdbc.jar and sqljdbc4.jar into C:\Program Files\Apache Software Foundation\Tomcat 6.0\webapps\solr\WEB-INF\lib

3.Create data-config.xml in C:\Program Files\Apache Software Foundation\Solr\example\solr\conf\

   Add to data-config.xml: 

<dataConfig>

 <dataSource type="JdbcDataSource"

  driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"

  url="jdbc:sqlserver://localhost\\sqlserver08;databaseName=Business Objects"

  user="sa" password="sqlserver08" />

 <document>

  <entity name="address"

   query="select OBJECTID,HOUSENO,StreetName,Suburb from Address">

   <field column="OBJECTID" name="id" />

   <field column="HOUSENO" name="house_number" />

   <field column="StreetName" name="street" />

   <field column="Suburb" name="Suburb" />

  </entity>

 </document>

</dataConfig> 

4. Added to schema.xml (C:\Program Files\Apache Software Foundation\Solr\example\solr\conf) in:

   <field name="house_number" type="string" indexed="true" stored="true"/>

   <field name="street" type="string" indexed="true" stored="true"/>

   <field name="Suburb" type="string" indexed="true" stored="true"/>

5. Run: http://localhost:8080/Solr/dataimport?command=full-import 

    You can use the interface at :  http://localhost:8080/Solr/admin/dataimport.jsp?handler=/dataimport

6. Test: http://localhost:8080/Solr/select?q=TYTUS or use http://localhost:8080/Solr/admin/form.jsp

Part 3 - Querying 

http://localhost:8088/solr/db/select/?q=house_number:41&version=2.2&start=0&rows=50&indent=on

Faceting:  http://localhost:8080/Solr/select?indent=on&version=2.2&q=house_number:41&facet=true&facet.field=Suburb

You can facet on multiple fields by appending &facet.field=<field name> The key to faceting is giving all your documents a category or type.

To be or not to be GWT?

With the current project that I’m working on, I had to work out some GWT stuff to get specific UI level requirements done. In fact I’m not 100% convinced the way it was done that started pinching me to explore more about it.

This is the first time I touched GWT in real live project development. Being java developer for a long time, I have never come across this much of complexity at the UI level especially when you have been dealing with latest technologies and tools. Basically this system has several layers of wrappers before it really feed with back end calls. Also it contains an in-house written configurable UI Binder that uses xml file to configure the UI. According to my understanding this was done to make reusable and scalable UI level components. But I’m not sure if a software house should invest this much of resources to make front end reusable and scalable as these front end tools can be easily replaced by a new technology that is yet to be arrived in near future. As of now, with GWT 2.0 there is a UI Binder feature which allows you to configure your UI in xml files. In fact I found out that smart-client offers you same kind of configurable UI. So all effort they put to make UI configurable is in vain with this kind of releases. Another disadvantage I see is the expertise required to develop the front end. Front end development supposed to be something different so that it requires different mindset with creativity. Obviously GWT development requires experienced java developers, not the creative designers :). So are we doing Swing development for the web? I really feel someone should have considered writing a tool to convert swing applications in to GWT type of front end applications which could have added more value, considering the power it brings to the web applications. There are number of cool features that you can enjoy with GWT based front end, providing an end to the all time headache "browser compatibility".

In my readings, I came across Flex which has similar kind of UI dynamic binding scripts but something easier to deal with. There are so many pros and cons with both GWT and Flex. There is a good comparison here
- More readings.
- Loading speed comparison

Caching Service

While I was working on my last assignment, there was a need for implementing good caching solution for the company. This has been a long time headache as we did not utilize proper caching mechanism to cache our frequently used data. Only single component that we were using had many draw backs and it was so much restricted cache reservation details only. The highly coupled nature of that component sometime made trouble shooting a nightmare for developers. As I was assigned to work on new searching component, with my refactoring cycles I had to remove most of the highly coupled back-end logics ultimately existing component became obsolete. Therefore I started working on new component and designed it in such a way where we can use it as a service.

This Caching Service has been developed to facilitate data caching needs required not only by our system, but also any JEE based system. When talking about Caching service, a framework is essential to build and govern the infrastructure of it. Therefore I came up with a framework that accommodates existing caching needs with extensibility in mind.

This is a very powerful framework that can be used to implement new caching mechanisms from the scratch or it can be used as an interface for 3rd party caching engine (MemCache, JbossCache, OSCache … etc). It has a user friendly monitoring interface with functionality rich including the flushing results, flush contains by systems connected, view particular requests, delete particular request … etc.

The beauty of this is it has a central control over all systems connected to caching engine and you can govern them individually. You can have multiple systems connected to multiple caching engines, but you can govern and monitor them in a central location. You are free to add new caching solutions written by your own or you can plug new caching engines that use the same set of operations.

I have an idea of sharing something same as this framework to the open source community let’s see what kind of improved version I can come up with.

Heart of the system, Reservation search reengineered

I’ve concluded the new search component for Base system. This has been done in fully scalable manner which aligns with most of our future technical goals. It is important for everyone to understand the concept so we can make sure the new structure would not be affected with new requirements. In order to place the new component with existing structure, several re-factoring iterations were occupied within the same set of objects.

What did I do there?
This is actually very vital question to be answered. Because to the front interface this could be nothing more than retrieving results, may be in much stable manner. But there have been lots of exercises done to ensure the application structure, transaction management, concurrency and most importantly application extensibility. It is important to have an idea about the drawbacks we faced with previous structure.

• Highly coupled front end business logic

There were bulks of logics identified inside web application tier. These business logic have to be separated and reside inside server application in order to be aligned with the meaning of a layered architecture.

• Caching mechanism is tightly coupled with the web application

We had our caching component coupled so much with the web application. This kills the reusability of that component and there were no centralized controlling or monitoring capability.

• Server was not serving properly

We have used server application just to run particular application logic only. Sever had no control or idea about the application flow, it was only managed by the front tier. For example calling thread x and thread Y runs as two different processes. Finally it’s managed by the front tier application which we should have used to process UI tier functions.

I heavily used some refactoring methods to bring this structure to level where I can start re-designing.

PGP encryption

I've been configuring PGP /SMIME to encryption for one of our email application. I found some useful links for anyone who needs to workout this. You can check samples to work out this.

http://www.bouncycastle.org/

http://www.bouncycastle.org/java.html

http://javamail-crypto.sourceforge.net

Performance issues with java

Performance issues with java have been a real hazard for application deployment over the years. I was involved in many of the projects as performance engineer where I had to resolve some real time server crashing issues. Many of problems occurred due to improper memory handling with application development. The bottom line of this is application developer has no sense of what its look like during the deployment which apparently leads to many of the problems.

Memory leaks

If your application keeping reference to objects that are no longer used, it leads to memory leaks.

High Memory usage / Insufficient object creations

Modernized applications keep loads of information on user events in order to comfort user in turn it can be an overhead to your application memory space if you don’t manage it properly. Also some applications create very high amount of objects for use of keeping large set of information.

Garbage Collector

GC too can be a problem if you don’t configure it properly. You will need to run lots of profiling tests, loading tests and find optimum configuration for your system. Also if your application creates series of objects in very short period of time, it would be a very costly task for GC to perform its GC operations accordingly. The more frequently it creates, it demands the CPU a big time.

Refactoring legacy system...

Re-factor a legacy system is any passionate developer’s nightmare (I said re-factor NOT re-engineer). Job is really hard to go over chunk of code and try to make them in a correct order most importantly with out touching the flow of the program and with out touching the base design. As I go over it, I happen to find so much of buggy codes...not really bugs… I would say “not properly written” codes. There are number of places I can see where I need to optimize but I’m limited by the time and scope I can change over. On the other hand there is a saying “if something works don’t try to change". Sometimes I really finding it interesting to do things in different ways but other time its very boarded to move codes here and there, change variable names… etc. I work on the heart of the company base system and it really needs concentration to finalize better design.

This is kind of exercise where you get to change your wheels while you are on ride. The most difficult thing is to think through all possible issues with out practice TDD.

Working on Rules Engine ...

Started to design a rules engine that can execute simple rules describe in a XML config. Obviously this will look simpler but idea is to cater dynamic change of requirements on operational flow of an entire business process...
In my mind i have what i needs to achieve but it’s too early to estimate how far this will turn good... Anyway i will do more readings on SOA, ESB and other rules engine standards soon...