Java Garbage Collection : Unexpected Full GC

I've been digging garbage collection logs these days. We have a production server which suffers long pauses of garbage collection. According to garbage collection log file, there are three different cases that force the JVM to do a StopTheWorld collection:

1. Full GC after a promotion failure: This is the evidence of not enough contiguous space in the old (tenured) generation. The simplest solution to this is increasing heap size. (I am glad it worked for us.)

2. Full GC after a concurrent mode failure (in other words; full promotion guarantee failure) : CMS collector can not catch up with the object allocation speed of the application. If this is the case try setting CMSInitiatingOccupancyFraction to a lower value. CMS will start early but will finish its job on-time.

3. The third one was very difficult for me to figure out. Here is what it looks like: Everything seems to work fine but suddenly Full GC kicks in and 37 seconds of break for application threads.

216066.711: [GC 216066.713: [ParNew: 911554K->71428K(943744K), 0.0860545 secs] 2346321K->1508264K(4089472K), 0.0878449 secs] [Times: user=3.63 sys=0.32, real=0.09 secs]
216066.896: [Full GC 216066.897: [CMS: 1436836K->1003062K(3145728K), 37.8691646 secs] 1527124K->1003062K(4089472K), [CMS Perm : 86016K->31249K(86016K)], 37.8707796 secs] [Times: u
ser=37.64 sys=0.35, real=37.87 secs]
216106.398: [GC 216106.399: [ParNew: 838912K->46337K(943744K), 0.0555625 secs] 1841974K->1049399K(4089472K), 0.0573404 secs] [Times: user=1.33 sys=0.45, real=0.06 secs]

The above log says that in order to clean permanent generation a full collection occured and (86016 - 31249) kilobytes of space freed in permanent generation.

What I've learned today is : The garbage of the permanent generation is only collected by a Full GC. If you see unexpected full collections try expanding permanent generation.

Load Test Whatever You Want With Apache JMeter

This is the second post about load testing with Apache JMeter, read the first post here: A step by step tutorial about load testing relational databases.

JMeter has lots of Samplers. If you need a sampler that is not provided by JMeter you can write your custom sampler. (custom samplers are called "Java Request" in JMeter terminology)
This post will show you, step by step, how to write a JMeter Java Request.

Step 1: Preparing the development environment

Add these two jar files to the java classpath.

1. $JMETER_HOME/lib/ext/ApacheJMeter_core.jar
   
2. $JMETER_HOME/lib/ext/ApacheJMeter_java.jar
   

(If you are using Eclipse, add these files as external jar files to the java build path.)

Step 2: Extending AbstractJavaSamplerClient

After setting up the classpath, create a custom sampler by extending AbstractJavaSamplerClient and override the following methods.

public Arguments getDefaultParameters() {...}
public void setupTest(JavaSamplerContext context) {...}
public void teardownTest(JavaSamplerContext context) {...}
public SampleResult runTest(JavaSamplerContext context) {...}

getDefaultParameters
Implement getDefaultParameters if you want initial values for test paramters. JMeter will display the parameters in its Java Request configuration GUI. (See the contents of the red rectangle in the picture below.) Here's an example implementation:

public Arguments getDefaultParameters() {
    Arguments defaultParameters = new Arguments();
    defaultParameters.addArgument("memcached_servers", "localhost:11211");
    defaultParameters.addArgument("username", "testuser");
    defaultParameters.addArgument("password", "testpasswd");
    return defaultParameters;
}

setupTest
This is where you read test parameters and initialize your test client. JMeter calls this method only once for each test thread.

teardownTest
Clean up the mess.

runTest
Write your test logic in this method. JMeter will call runTest method for every execution of test threads. Here is a typical runTest implementation:

@Override
public SampleResult runTest(JavaSamplerContext context) {
    SampleResult result = new SampleResult();
    boolean success = true;
    result.sampleStart();
    //
    // Write your test code here.
    //
    result.sampleEnd();
    result.setSuccessful(success);
    return result;
}

The time elapsed betweed result.sampleStart() and result.sampleEnd() is used to calculate average response time of the application under test.

Step 3: Deploy your custom sampler

When you are done create a jar file (containing your custom sampler) in the $JMETER_HOME/lib/ext/ directory. JMeter will display your java request in the java request configuration page.

You can see the results of your test by adding listeners to your test plan. "A step by step tutorial about load testing relational databases" post shows how to add listeners to test plans.

Load Testing Relational Databases With JMeter

Apache JMeter is a performance testing tool which is entirely written in Java. Any application that works on request/response model can be load tested with JMeter. A relational database is not an exception: receives sql queries, executes them and returns the results of the execution.

I'am going to show you how easy it is to set up test scenarios with the graphical user interface of JMeter. But before diving into details let's give a shot to basic terms:

Test plan : describes a test scenario
Thread Group : represents users running your test scenario.
Samples : a way of sending request and waiting response. HTTP request, JDBC request, SOAP/XML-RPC request and java object request are examples of samples.
Logic Controller : used to customize the logic that JMeter uses to decide when to send requests
Listeners : receives test results and displays reports.
Timers : cause JMeter to delay a certain amount of time before each request that a thread makes.
Assertions : test that application returns expected responses

Note : This post is not meant to be an alternative documentation for JMeter. JMeter has a great documentation. You can find the details in its User's Manual (http://jakarta.apache.org/jmeter/usermanual/index.html

Suppose we have an application that logs every transaction into a relational database. We are going to create a test plan - step by step - in order to answer the questions below.

• How many transaction records can be inserted to transaction table in a second?
• How much time does it take to insert a single transaction record to transaction table?
• How does number of concurrent threads (users) affects the inserts/secs and average response times?
• How does number of records affects the insert/secs and average response times?

Step 1

Copy mysql jdbc driver into the lib folder of your JMeter installation. JMeter needs a suitable jdbc driver in the classpath to connect to the database.

Example ~/tools/jakarta-jmeter-2.3.4/lib/mysql-connector-java-5.0.5.jar

We are going to store orders of the customers and the result of the order in the transactions table.

CREATE TABLE transactions (
    id INT NOT NULL AUTO_INCREMENT,
    customer_id INT NOT NULL,
    order_id INT NOT NULL,
    result INT,
    PRIMARY KEY (id)
);

Step 2

Create a test plan and name it "Test MYSQL DB". Then add the following jmeter components to the test plan.

1. Thread group named 'Database Users'
2. Sampler of type JDBC Request
3. Config element of type JDBC Connection Configuration
4. Three config elements of type Random Variable
5. Listener of type Summary Report

After adding these components JMeter test plan looks like the following picture.

Step 3

Configure database users. The thread group component simulates the database users.
1. Number of users (threads)
2. How many times a user will send request (loop count). If you select 'Forever', threads will run in a while(true) {...} loop until you decide to stop the test.

Step 4

Configure JDBC connection pool. JDBC Connection Configuration component is used to create jdbc connection pools. Database url, jdbc driver, database user and password are configured with this component. Connection pools are identified by "Variable Name". JDBC Samplers (requests) use this variable name (connection pool name) to pop and push connections. I named the test connection pool as "my db pool"

Step 5

Define random variables that will be used in INSERT statements. In this test I am using three random variables : user id, order id and result. Following picture shows the a random number configuration for user id. Random number generator will give us a random integers between 1 and 1000000. We can refer to generated random number with the name user_id.

Step 6

JDBC Request component is the place where we tell our users (threads) what to do. The name of the pool that was configured in Step 3 "my db pool" will be used as the "variable name bound to pool". All threads will execute prepared statements. User id, order id and result will be generated by the random number configurator (described in Step 5)

Step 7

Now we have our threads configured to insert transaction records to the transactions table. In this last step we will add a Listener of type Summary Report in order to view test results.

The results tells us that 10 concurrent users (threads) working in an infinite loop can insert nearly 3300 rows in our transactions table. And the average time spent for inserting a row is 2 ms. You can also choose "Graph Results" listener to view visual representation of the results.
I created and run a simple DB test plan. I hope you'll find this post helpful. Keep this motto in mind

if you can’t measure it, you can neither manage it nor improve it

Happy testing...

2010 Winter Olympics - Bahçeköy/İSTANBUL

Hi everybody, I've been busy training hard for the winter olympics. You can watch me practicing bobsleigh with my two assistants Johny and Karsaf :)

Python API for Flickr Services - Part I

These days I am working on a facebook application that mashes up servises from Facebook and Flickr. The application is hosted by Google Appengine and I am developing on Django web application framework.

Flickr has an easy to use API which is called Flickr Services. Here are some interesting things you can do with the Flickr Services:

• You can upload photos from anywhere (web application, desktop application or mobile application)
  
• You can search for images. Flickr Services lets you search images by
  - tags
  - geo location
  - photo taken time / photo upload time
  - and more...
  
• execute methods that require authentication
  - create, edit, remove tags
  - create, edit, remove photo sets
  - replace, remove photos
  

There are two Python APIs listed in the Flickr Services page:

1. http://www.stuvel.eu/projects/flickrapi
   
2. http://code.google.com/p/flickrpy/
   

I have implemented Flickr services in Python. Don't think this as "reinventing the wheel" because it was a coding practice for me. Plus, i have my own, customized, shortcut methods ;)

You can download the library from http://code.google.com/p/ilkinulassandbox/. There are also some other javascript and python code snippets there.

Here is a basic usage of the API:

>>> import flickr
>>> api = flickr.FlickrAPI('API_KEY','SECRET_KEY')
>>> api.search({'tags':'python', 'per_page':'3'})
{'photos': {'page': 1,
            'pages': 8326,
            'perpage': 3,
            'photo': [{'farm': 4,
                       'id': '3911963732',
                       'isfamily': 0,
                       'isfriend': 0,
                       'ispublic': 1,
                       'owner': '30041312@N03',
                       'secret': '44aae23e98',
                       'server': '3090',
                       'title': 'Centralian Carpet Python (Apophis)'},
                      {'farm': 3,
                       'id': '3911184515',
                       'isfamily': 0,
                       'isfriend': 0,
                       'ispublic': 1,
                       'owner': '30041312@N03',
                       'secret': '0aae3c8127',
                       'server': '2513',
                       'title': 'Centralian Carpet Python (Apophis)'},
                      {'farm': 3,
                       'id': '3911127131',
                       'isfamily': 0,
                       'isfriend': 0,
                       'ispublic': 1,
                       'owner': '31355375@N02',
                       'secret': '2921fea364',
                       'server': '2564',
                       'title': 'Sendra Python Nails 122 01'}],
            'total': '24977'},
 'stat': 'ok'}

Methods of the API returns a dictionary instance. Caller must handle the returned result.

>>> result = api.search({'tags':'python', 'per_page':'3'})
>>> result['photos']['photo'][0]['id']
'3911751961'

Flickr photo url construction is described here in details. You can easily get the url of a returned photo search result by using build_photo_url. Default image size is 'm' which means 'normal'

>>> api.build_photo_url(result['photos']['photo'][0])
'http://farm4.static.flickr.com/3462/3911751961_43d409cf7c_m.jpg'
>>> api.build_photo_url(result['photos']['photo'][1], size='s')
'http://farm3.static.flickr.com/2613/3911666353_ac0cc98871_s.jpg'

Lets search for pictures tagged 'fun' around point (lat=41.070293, lon=28.24894)

>>> pictures = api.search({'lat':'41.034046', 'lon':'28.980217', 'radius':'10', 'radius_units':'km','tags':'fun'})
>>> api.build_photo_url(fun_pictures['photos']['photo'][20])
'http://farm1.static.flickr.com/116/293251976_ce5337908d_m.jpg'

The next post of "Python API for Flickr Services" will show how you can call methods that require authentication. For the curious readers the api is ready http://code.google.com/p/ilkinulassandbox/. But it lacks documentation :(


NOTE : I use this python API in my facebook application "Türkiye'yi ne kadar iyi tanıyorsun?". Application is a simple guessing game. You see a geo tagged picture from flickr and you answer which city this picture belongs.

UPDATE : This library returns native Python datatypes rather than XML trees or unparsed JSON, which the other Flickr APIs have a tendency to do. (Thanks martian)

Div Roller : jQuery plugin for rotating html div elements

In this post I want to introduce a jQuery plugin that i have written for one of my projects.
jquery.divroller plugin is written to rotate messages displayed in the home page of www.bahele.com. bahele is a facebook connect site where users can login with their facebook accounts and write messages about what they like or dislike. (Bahele is only available in Turkish for now.)

For the ones who have not heard of this library here's the motto from jquery.com

"jQuery is a fast and concise JavaScript Library that simplifies HTML document 
traversing, event handling, animating, and Ajax interactions for rapid web 
development. jQuery is designed to change the way that you write JavaScript."

jQuery is a widely used javascript library. There are tons of javascript libraries in the wild, each has its own clever implementations. jQuery differs from other frameworks by its chaining property. Most of the methods in the library returns a jQuery object. You can call a jQuery function on a return value of another jQuery function. Here is what i mean:

       $("a[href$='.pdf']")
           .addClass("external_link")
           .append("<img src="http://www.blogger.com/images/pdf.png" />")
           .click(pdfLinkClicked);

One line of javascript is splitted into four lines for better readability.

1. The first line selects all the links in the page whose 'href' attribute ends with '.pdf'
   
2. The second line adds 'external_link' css class to selected link elements.
   
3. The third line places an image after the link element in order to specify that is a link to download a pdf file.
   
4. The last line binds 'pdfLinkClicked' function to the click event of link element.
   

Enough for a short introduction, now lets look how we can take advantage of jquery.divroller. Below is a simple usage scenario. Plugin is applied to the html div element whose id attrubute is equal to "divroller_container". There are two configuration parameters:

• pause : Sleep duration between two rotations. This parameter determines the speed of the rolling divs.
  
• visible : Number of visible html elements inside the container. Plugin shows only "visible" number of elements, the others are hidden.
  

<div id="divroller_container">
<div class="box_dark box_size" id="item1">
<a href="http://ilkinbalkanay.blogspot.com/2007/11/my-favorite-shell-commands.html">
My favorite shell commands</a>
</div>
<div class="box_light box_size" id="item2">
<a href="http://ilkinbalkanay.blogspot.com/2007/10/volatile-variables-in-java.html">
Volatile Variables In JAVA</a>
</div>
<div class="box_dark box_size" id="item3">
<a href="http://ilkinbalkanay.blogspot.com/2007/09/power-off-shell.html">
The power off shell</a>
</div>
<div class="box_light box_size" id="item4">
<a href="http://ilkinbalkanay.blogspot.com/2007/09/seach-java-class-files-in-jars.html">
 Search java class files in jars. </a>
</div>
<div class="box_dark box_size" id="item5">
<a href="http://ilkinbalkanay.blogspot.com/2008/01/readwritelock-example-in-java.html">
ReadWriteLock example in Java</a>
</div>
<div class="box_light box_size" id="item6">
<a href="http://ilkinbalkanay.blogspot.com/2008/01/how-to-change-gnome-desktop-wallpaper.html">
How to change GNOME desktop wallpaper programmatically</a>
</div>
</div>

<script type="text/javascript">
$("#divroller_container").divroller({pause:2000, visible:3});
</script>

You can see the plugin in action : Live demonstration
You can also download the javascript source.

Note : There are countless jQuery plugins available on the internet. Before writing your own plugin you should check http://plugins.jquery.com/, you might be reinventing the wheel.