Bye bye blogger!

It's been a long time since I blogged last. As I endeavor to blog more often, I've switched myself to wordpress site. Please follow me here. Thank you!.

Getting around Tomcat out-of-memory issues

“Server running out of memory” is one of the common causes for webserver crashing experienced by most of the Web Application Admins, yet it’s really easy for anyone to determine it happens due to java heap space limitation (JVM memory allocation).  This could lead to false conclusions such as application needing more heap space or application should be clustered/load balanced to provide high availability. On the other hand, when the service is running out of memory it could appear as a memory leak within the application and eventually it can end up with development team testing it.

This post covers simple steps to fine-tune your system memory according to your application requirement, prior to deciding on any infrastructure changes that would be considered.

OutOfMemoryError

JVM throws this error when there is no space in memory for JVM to continue working. This happens mostly while you try to create more objects when there is no space in the heap to allocate as needed.

There are two main causes for this error.

1.  Java heap space limitation.
2.  Java perm gen limitation

Java heap limitation

This is the most commonly known fact related to the out-of-memory issue. You can tweak heap space settings using –Xmx (Maximum heap size) and –Xms (Initial heap size) parameters. In general, if you have more memory the less frequent it needs to be garbage collected. So you could expect applications to run faster when there is more memory available. But from GC’s perspective it will have to work on a larger memory than usual. It’s just like for a floor cleaner you have a bigger floor to clean than he usually does. This in turn can decrease overall performance of your application so you should tweak these settings consciously.

Example :
    -Xmx1536m
    -Xms1024m

Java Permanent Generation limitation

This is where JVM store meta data information about classes, methods (on Eden heap and Tenured heap) and other java primitives. By default this value is set to 64m. Most of the time it’s not sufficient for many applications. So it’s really important to change this setting and provide some space for perm gen to work on. Unlike xmx and xms, you have to use “=” to define this setting. Also you can use additional JVM flags “-XX:+CMSClassUnloadingEnabled” and “-XX:+CMSPermGenSweepingEnabled”. This will enforce sweeping through the permanent generation to release unused references to some extent.

Example :
    -XX:PermSize=64M
    -XX:MaxPermSize=256m
    -XX:+CMSPermGenSweepingEnabled
    -XX:+CMSClassUnloadingEnabled

Generally many Application Admins are aware of heap space settings. Yet there are many instances where perm gen settings have gone unnoticed. It’s really important to fine-tune your application for permanent generation heap as it could be a good reason for your out-of-memory errors

Apache Solr Install Notes

I started working on full text searching solution for our map application. No surprise, Apache Solr is the selected solution and here are some quick notes for you to get started with it.

Part 1 - Basic Setup

1. Download Solr from: http://www.apache.org/dyn/closer.cgi/lucene/solr/

2. Unzip to: C:\Program Files\Apache Software Foundation\Solr (ensure that you rename the solr download to Solr)

3. Stop Tomcat

4. Copy C:\Program Files\Apache Software Foundation\Solr\dist\apache-solr-1.4.1.war to: C:\Program Files\Apache Software Foundation\Tomcat 6.0\webapps

5 Rename: C:\Program Files\Apache Software Foundation\Tomcat 6.0\webapps\apache-solr-1.4.1.war to Solr.war

6. Open: C:\Program Files\Apache Software Foundation\Tomcat 6.0\bin\tomcat6w.exe goto Java tab and add Java options: -Dsolr.solr.home=C:\Program Files\Apache Software Foundation\Solr\example\solr

7. Start Tomcat and go to: http://localhost:8080/Solr/admin/

Part 2 - Configuring and running the DataImportHandler 

1. Added the following to file: C:\Program Files\Apache Software Foundation\Solr\example\solr\conf\solrconfig.xml

<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">

    <lst name="defaults">

      <str name="config">data-config.xml</str>

    </lst>

</requestHandler>

2. Downloaded sqljdbc.jar and sqljdbc4.jar into C:\Program Files\Apache Software Foundation\Tomcat 6.0\webapps\solr\WEB-INF\lib

3.Create data-config.xml in C:\Program Files\Apache Software Foundation\Solr\example\solr\conf\

   Add to data-config.xml: 

<dataConfig>

 <dataSource type="JdbcDataSource"

  driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"

  url="jdbc:sqlserver://localhost\\sqlserver08;databaseName=Business Objects"

  user="sa" password="sqlserver08" />

 <document>

  <entity name="address"

   query="select OBJECTID,HOUSENO,StreetName,Suburb from Address">

   <field column="OBJECTID" name="id" />

   <field column="HOUSENO" name="house_number" />

   <field column="StreetName" name="street" />

   <field column="Suburb" name="Suburb" />

  </entity>

 </document>

</dataConfig> 

4. Added to schema.xml (C:\Program Files\Apache Software Foundation\Solr\example\solr\conf) in:

   <field name="house_number" type="string" indexed="true" stored="true"/>

   <field name="street" type="string" indexed="true" stored="true"/>

   <field name="Suburb" type="string" indexed="true" stored="true"/>

5. Run: http://localhost:8080/Solr/dataimport?command=full-import 

    You can use the interface at :  http://localhost:8080/Solr/admin/dataimport.jsp?handler=/dataimport

6. Test: http://localhost:8080/Solr/select?q=TYTUS or use http://localhost:8080/Solr/admin/form.jsp

Part 3 - Querying 

http://localhost:8088/solr/db/select/?q=house_number:41&version=2.2&start=0&rows=50&indent=on

Faceting:  http://localhost:8080/Solr/select?indent=on&version=2.2&q=house_number:41&facet=true&facet.field=Suburb

You can facet on multiple fields by appending &facet.field=<field name> The key to faceting is giving all your documents a category or type.

Test Drive ...

I just posted some of the new pics taken with my new DSLR. Follow me on flicker. Watch out for more actions ;)

To be or not to be GWT?

With the current project that I’m working on, I had to work out some GWT stuff to get specific UI level requirements done. In fact I’m not 100% convinced the way it was done that started pinching me to explore more about it.

This is the first time I touched GWT in real live project development. Being java developer for a long time, I have never come across this much of complexity at the UI level especially when you have been dealing with latest technologies and tools. Basically this system has several layers of wrappers before it really feed with back end calls. Also it contains an in-house written configurable UI Binder that uses xml file to configure the UI. According to my understanding this was done to make reusable and scalable UI level components. But I’m not sure if a software house should invest this much of resources to make front end reusable and scalable as these front end tools can be easily replaced by a new technology that is yet to be arrived in near future. As of now, with GWT 2.0 there is a UI Binder feature which allows you to configure your UI in xml files. In fact I found out that smart-client offers you same kind of configurable UI. So all effort they put to make UI configurable is in vain with this kind of releases. Another disadvantage I see is the expertise required to develop the front end. Front end development supposed to be something different so that it requires different mindset with creativity. Obviously GWT development requires experienced java developers, not the creative designers :). So are we doing Swing development for the web? I really feel someone should have considered writing a tool to convert swing applications in to GWT type of front end applications which could have added more value, considering the power it brings to the web applications. There are number of cool features that you can enjoy with GWT based front end, providing an end to the all time headache "browser compatibility".

In my readings, I came across Flex which has similar kind of UI dynamic binding scripts but something easier to deal with. There are so many pros and cons with both GWT and Flex. There is a good comparison here
- More readings.
- Loading speed comparison

ESB competition,This is how it began...

"Ultra ESB", another class product from a Sri Lankan has been released. They've published the results of the Round 4 benchmark testing. Its interesting to wait and see how vendors such as Mule, BEA, WSO2 reacts on this. http://esbperformance.org/wiki/ESB_Performance_Testing

Another update from Chrome

http://www.infoq.com/news/2010/02/Web-SQL-Database

Lets make the web a better place

Speed of the web matters everyone who browse over the internet. Internet has come a long way to reach the level of speed that possess today. Still it demands a better and better speed to make possible many great things that are restricted due to slow internet speed. Google has put a step forward to solve this for the world. It’s a great effort, as people who build the web we all should contribute. Please refer: http://code.google.com/speed/articles/

Caching Service

While I was working on my last assignment, there was a need for implementing good caching solution for the company. This has been a long time headache as we did not utilize proper caching mechanism to cache our frequently used data. Only single component that we were using had many draw backs and it was so much restricted cache reservation details only. The highly coupled nature of that component sometime made trouble shooting a nightmare for developers. As I was assigned to work on new searching component, with my refactoring cycles I had to remove most of the highly coupled back-end logics ultimately existing component became obsolete. Therefore I started working on new component and designed it in such a way where we can use it as a service.

This Caching Service has been developed to facilitate data caching needs required not only by our system, but also any JEE based system. When talking about Caching service, a framework is essential to build and govern the infrastructure of it. Therefore I came up with a framework that accommodates existing caching needs with extensibility in mind.

This is a very powerful framework that can be used to implement new caching mechanisms from the scratch or it can be used as an interface for 3rd party caching engine (MemCache, JbossCache, OSCache … etc). It has a user friendly monitoring interface with functionality rich including the flushing results, flush contains by systems connected, view particular requests, delete particular request … etc.

The beauty of this is it has a central control over all systems connected to caching engine and you can govern them individually. You can have multiple systems connected to multiple caching engines, but you can govern and monitor them in a central location. You are free to add new caching solutions written by your own or you can plug new caching engines that use the same set of operations.

I have an idea of sharing something same as this framework to the open source community let’s see what kind of improved version I can come up with.

Heart of the system, Reservation search reengineered

I’ve concluded the new search component for Base system. This has been done in fully scalable manner which aligns with most of our future technical goals. It is important for everyone to understand the concept so we can make sure the new structure would not be affected with new requirements. In order to place the new component with existing structure, several re-factoring iterations were occupied within the same set of objects.

What did I do there?
This is actually very vital question to be answered. Because to the front interface this could be nothing more than retrieving results, may be in much stable manner. But there have been lots of exercises done to ensure the application structure, transaction management, concurrency and most importantly application extensibility. It is important to have an idea about the drawbacks we faced with previous structure.

• Highly coupled front end business logic

There were bulks of logics identified inside web application tier. These business logic have to be separated and reside inside server application in order to be aligned with the meaning of a layered architecture.

• Caching mechanism is tightly coupled with the web application

We had our caching component coupled so much with the web application. This kills the reusability of that component and there were no centralized controlling or monitoring capability.

• Server was not serving properly

We have used server application just to run particular application logic only. Sever had no control or idea about the application flow, it was only managed by the front tier. For example calling thread x and thread Y runs as two different processes. Finally it’s managed by the front tier application which we should have used to process UI tier functions.

I heavily used some refactoring methods to bring this structure to level where I can start re-designing.

It's a place to be ... The Apache Asia Roadshow 2009

The Apache Asia Roadshow 2009 is scheduled to hit Colombo from the 3rd - 5th of December. The layout of the Roadshow would be a two day conference and a one-day hackathon/ unconference at the beach. If you are interested in anything Apache related then this is the place to be

http://www.apacheasia09.foss.lk

PGP encryption

I've been configuring PGP /SMIME to encryption for one of our email application. I found some useful links for anyone who needs to workout this. You can check samples to work out this.

http://www.bouncycastle.org/

http://www.bouncycastle.org/java.html

http://javamail-crypto.sourceforge.net

Performance issues with java

Performance issues with java have been a real hazard for application deployment over the years. I was involved in many of the projects as performance engineer where I had to resolve some real time server crashing issues. Many of problems occurred due to improper memory handling with application development. The bottom line of this is application developer has no sense of what its look like during the deployment which apparently leads to many of the problems.

Memory leaks

If your application keeping reference to objects that are no longer used, it leads to memory leaks.

High Memory usage / Insufficient object creations

Modernized applications keep loads of information on user events in order to comfort user in turn it can be an overhead to your application memory space if you don’t manage it properly. Also some applications create very high amount of objects for use of keeping large set of information.

Garbage Collector

GC too can be a problem if you don’t configure it properly. You will need to run lots of profiling tests, loading tests and find optimum configuration for your system. Also if your application creates series of objects in very short period of time, it would be a very costly task for GC to perform its GC operations accordingly. The more frequently it creates, it demands the CPU a big time.

tech-chat-rich-sharples

http://java.dzone.com/videos/tech-chat-rich-sharples

How Do You Convince Your Boss to TDD?

Really good article.

http://java.dzone.com/articles/how-do-you-convince-your-boss

Refactoring legacy system...

Re-factor a legacy system is any passionate developer’s nightmare (I said re-factor NOT re-engineer). Job is really hard to go over chunk of code and try to make them in a correct order most importantly with out touching the flow of the program and with out touching the base design. As I go over it, I happen to find so much of buggy codes...not really bugs… I would say “not properly written” codes. There are number of places I can see where I need to optimize but I’m limited by the time and scope I can change over. On the other hand there is a saying “if something works don’t try to change". Sometimes I really finding it interesting to do things in different ways but other time its very boarded to move codes here and there, change variable names… etc. I work on the heart of the company base system and it really needs concentration to finalize better design.

This is kind of exercise where you get to change your wheels while you are on ride. The most difficult thing is to think through all possible issues with out practice TDD.

Working on Rules Engine ...

Started to design a rules engine that can execute simple rules describe in a XML config. Obviously this will look simpler but idea is to cater dynamic change of requirements on operational flow of an entire business process...
In my mind i have what i needs to achieve but it’s too early to estimate how far this will turn good... Anyway i will do more readings on SOA, ESB and other rules engine standards soon...

Is nothing sacred on the racetrack?

By Himal Kotelawala, The Sunday Times, June 1, 2008

Colombo, Sri Lanka -- Sri Lanka is considering some diplomatic action regarding cases where a horse named ‘Buddhist Monk’ and another named ‘Buddah’ had run in two separate horse races in Yarmouth, England and Tramore, Ireland respectively, last week.
www.racingpost.co.uk. The latter website also carried a summary of ‘Buddhist Monk’s’ performance. It said “ ‘Buddhist Monk’ had a rug for stalls entry but got away well and was soon tracking the leader. He edged left in the closing stages and was only shaken up by his jockey without resorting to the whip and can repay the kindness in a similar contest,”
‘Buddhist Monk’ finished the race in second place on Thursday. Its rider was D O’Donohoe and the horse is owned by Lord Derby. The betting was stated as 10:1, meaning if the horse had won, a prospective bet winner would have made Rs. 10 for a Rs. 1 he bet on it. Meanwhile ‘Buddah’ owned by Joseph A. Shannon and ridden by R.Walsh had come ninth in a field of 14 in a two mile hurdle race on Friday.
When contacted by The Sunday Times, British High Commission media spokesman
http://www.buddhistchannel.tv/index.php?id=43

Lamport's one-time password algorithm (or, don't talk to complete strangers!)

There's a subtle beauty in simple things that present great value. To paraphrase Albert Einstein, a solution to a problem should be as simple as it can be, but no simpler. Applying a one-time password (OTP) scheme between distributed systems makes it more difficult for a would-be intruder to access and gain unauthorized control of restricted resources such as data, physical devices, or service end points. An OTP scheme is obviously a step up from completely open access, or access limited only by physical network barriers. But a solution based on an OTP challenge also has some advantages over static, infrequently changing passwords, because the window of opportunity to gain access to credentials is much smaller. There's a practical place for either type of authentication, or even both used in concert.

Refer : http://www.javaworld.com/javaworld/jw-03-2009/jw-03-lamport-otp.html