OutOfMemoryError (A.K.A OOME)

November 6, 2009

Recently, I was called by one of the groups in my company to help analyze a problem of an alleged memory leak in a Java application. When I asked to get the printout of the OOME, I was told that no one had ever seen it. Mysteriously.

To avoid such cases, there are some tools at our disposal:

  1. Write the OOME to a file instead of the standard error. This can be accomplished by the following code snippet:

    File file = new File("err.txt");
    FileOutputStream fos = new FileOutputStream(file);
    PrintStream ps = new PrintStream(fos);
    System.setErr(ps);

    This way, in case of system restart, the file will be there to analyze.

  2. The information of the OOME can also shed some light on the origin of the problem. The following are some of the errors that the OOME expose:
    1. Java Heap Space – objects cannot be allocated in the heap space
    2. Permgen Space – permgen is where classes are loaded. This can be fixed by setting -XX:MaxPermSize to a larger size
    3. <reason><stack trace> (native method) – Native methods can be C++ methods accessed by JNI or just methods in the Java core (which is written in C++). This means that native method encountered an allocation error. If this is the case, enlarging the heap size will not help and you will have to use OS specific diagnostic tools.

    For example, in the following OOME, we can see the first case:

    Exception in thread “main” java.lang.OutOfMemoryError: Java heap space
    at com.comverse.cpo.App.main(App.java:30)

  3. Create a heap dump, right before reaching the OOME. This can be accomplished by the following vm flag:java –XX:+HeapDumpOnOutOfMemoryError –cp <classpath> <class name>

    The heap dump can be viewed by many tools. I recommend mat – Eclipse memory analyzer  (Home Page) which comes as a standalone application or as an Eclipse plugin.

  4. Crash before OOME –  This occurs with native code that does not check for errors returned by memory allocation functions. This is a tricky one.
    1. You can look at the dump to find clues
    2. You can look at the fatal log. To configure the fatal log, use the following flag: java -XX:ErrorFile=/var/log/java/java_error%p.log. In the this example, the error log file will be written to the directory /var/log/java and will be named java_errorpid.log.

For more information, visit:
http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/memleaks.html
and other related information:
http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/docinfo.html

Assemble, Realease, Publish – Maven Application (Part 0)

September 27, 2010

One of the most striking things about Maven for a developer is how easy it is to get used to the Maven way – Once you go, you can never go back. A lot of developers in my company find it almost impossible when they are asked to revive an older application for bug fixes purposes. The development life cycle is very well defined, so as the directory structure, integration with the various IDE’s are quite satisfactory, all makes the setup and maintenance of an old application, a breeze.

But what about configuration management issues: assembling, releasing, perform various tests like sanity tests, publishing, etc’…

In this series of articles, I will demonstrate our efforts in this area in terms of the Maven toolbox, methodologies and conforming them all to an already existing non-Maven processes.

Stay tuned…

Maven Dependecy JAR Configuration

August 12, 2010

Maven is a great tool to manage your build system and it has become a leader in its domain. More than 3.5 million java developers have switched to Maven and counting. I am leading such continuing process of migrating projects to use Maven and I’ll try and share some of the things I learned in the process.
Maven pushes the java developer to think in modules. Maven itself is conceptually modularized as it separate the data (sources, resources, etc…) and the meta-data (pom.xml, although not fully as it contain information such as plugin information to be used at run time, but for the sake of argument…).
Maven has a declarative dependency management system, which in short means that the developer needs to declare (in the pom.xml) what he\she dependent on and Maven takes care of the rest (download it and put it in the classpath).

Here is a snippet pom.xml code that declare dependency in jUnit:

<dependencies>
    <dependency>
        <groupid>junit</groupid>
        <artifactid>junit</artifactid>
        <version>3.8.1</version>
        <scope>test</scope>
    </dependency>
</dependencies>

It’s that simple. by adding the above snippet, in case still absent in the classpath (Maven local repository) it will be retrieved and installed in the classpath.

But, what will happen, when, for example, your dependency has some configuration files? the standard Maven way is to put all the configuration files under src/main/resources. That way, they will be located in the resulting Jar at the right place, enabling your application to function properly.
But, what if you need those configuration files to be subject to changes? In case they are located inside the jar, you are stuck with the default configuration without the ability to edit and change the module behavior.

I will offer here a way to enable such Jar configuration usage.

Let us say we have a module, packed into Jar called jar-with-conf. This module contains a single class called ConfReader.
This class contains a single static function that retrieves a value from a configuration file, given a key:

public class ConfReader 
{
	public static String getValue(String key)
	{
		String val = null;
		Configuration config = null;
		try 
		{
			config = new PropertiesConfiguration("src/main/config/conf.properties");
		} 
		catch (ConfigurationException e) 
		{
			Logger.error(...)
		}
		
		if (config != null)
		{
			val = config.getString(key);
		}
		
		return val;
	}
}

Apache commons-configuration is used in this example.

This module has also, a configuration file called conf.properties and it is located at src/main/config:

key1=value1
key2=value2

Now, let us say there is an application that is dependent on jar-with-conf called app-using-jar-with-conf (forgive the names I named – one of the toughest task in computer science, as someone once said). This is most simple application:

public class UsingDepWithConf 
{
    public static void main( String[] args )
    {
        System.out.println("value of key1 is " + ConfReader.getValue("key1"));
        System.out.println("value of key2 is " + ConfReader.getValue("key2"));
    }
}

Of course, while running this simple main, the getValue() will fail as the configuration file is absent from the jar (we didn’t put it under the resources directory).

In order to achieve the above requirements we will create a zip, containing the config directory and deploy it to a Maven repository. The client application will retrieve the zip and extract it to its own directory structure.

  1. Creating The Zip

    We will use maven-assembly-plugin for the job. First let’s create an assembly descriptor and call it zip.xml:

    <assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
        <id>bin</id>
        <baseDirectory>/</baseDirectory>
        <formats>
            <format>zip</format>
        </formats>
        <fileSets>
            <fileSet>
                <directory>${basedir}/src/main/config</directory>
            </fileSet>
        </fileSets>
    </assembly>
    

    In plain English we say here we want to create a zip file out of config directory.

    next let’s configure the pom.xml of jar-with-conf to use this descriptor:

    <build>
      	<plugins>
      		<plugin>
    			<artifactId>maven-assembly-plugin</artifactId>
    			<version>2.2-beta-5</version>
    			<configuration>
    				<descriptors>
    					<descriptor>src/main/assembly/zip.xml</descriptor>
    				</descriptors>
    			</configuration>
    			<executions>
    				<execution>
    					<id>make-assembly</id> 
    					<phase>package</phase> 
    					<goals>
    					  	<goal>single</goal> 
    					</goals>
    				</execution>
    			</executions>
    		</plugin>
      	</plugins>
      </build>
    

    Which means, we want to create a single zip file during the package phase according to the zip.xml assembly file.

  2. Deploy The Zip
    Of course, packing the config directory into a zip file will not suffice. there is no way client application can access it. That is exactly the reason we have Maven repositories. We will deploy the zip file alongside the module jar artifact. For that, I’ll use maven-deploy-plugin.
    This plugin deploy goal is bound when creating a jar artifact, to the deploy phase in the default Maven lifecycle. Here, I am going to use a different goal – deploy-file which enables us the deployments of artifacts other than the default one.
    Here is a snippet of the deployment task in the pom.xml:

    <plugin>
      	<groupId>org.apache.maven.plugins</groupId>
      	<artifactId>maven-deploy-plugin</artifactId>
      	<version>2.4</version>
      	<executions>
      		<execution>
      			<id>deploy-conf-zip</id>
      			<phase>deploy</phase>
      			<goals>
      				<goal>deploy-file</goal>
      			</goals>
      		</execution>
      	</executions>
      	<configuration>
      		<repositoryId>snapshots</repositoryId>
      		<file>${project.build.directory}/${project.artifactId}-${project.version}-bin.zip</file>
      		<url>http://nexus:8081/nexus/content/repositories/snapshots</url>
      		<groupId>com.my-company.example</groupId>
                    <artifactId>jar-with-conf-config</artifactId>
                    <version>${project.version}</version>
                    <packaging>zip</packaging>
      	</configuration>
    </plugin>
    

    In the above snippet, the goal deploy-file is bound to the deploy phase, and in the configuration section, all the relevant information is supplied for successful deployment, like repositoryId (make sure it is configured in your settings.xml file and that you have permissions to deploy to the repository), repository URL and, of course, the new artifact identifier – the trinity of groupId, artifactId and version.

  3. Using The Jar with the Zip
    Now, let’s configure an application to be dependent on the jar-with-conf artifact and also retrieve the configuration folder and put it in the right place.

    The following are snippets from the pom.xml of the application app-using-jar-with-conf:

    The easy part is the artifact dependency:

    <dependency>
        	<groupId>com.my-company.example</groupId>
        	<artifactId>jar-with-conf</artifactId>
        	<version>0.0.1-SNAPSHOT</version>
        	<type>jar</type>
        	<scope>compile</scope>
    </dependency>
    

    Next, let’s retrieve the second artifact, that contains the configuration:

    <plugin>
         <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-dependency-plugin</artifactId>
         <executions>
             <execution>
                 <id>unpack</id>
                 <phase>package</phase>
                 <goals>
                     <goal>unpack</goal>
                 </goals>
                 <configuration>
                     <artifactItems>
                         <artifactItem>
                             <groupId>com.my-company.example</groupId>
                             <artifactId>jar-with-conf-config</artifactId>
                             <version>0.0.1-SNAPSHOT</version>
                             <type>zip</type>
                             <overWrite>false</overWrite>
                             <outputDirectory>${basedir}</outputDirectory>
                         </artifactItem>
                     </artifactItems>
                     <outputDirectory>${basedir}</outputDirectory>
                 </configuration>
             </execution>
        </executions>
    </plugin>
    

    The details of this plugin can be found here

    The result of this will be an unpacked config folder under src/main in the app-using-jar-with-conf project. the jar of jar-with-conf is in classpath and its configuration is available for change in the client application.

    This kind of project setup encourages modularity in big systems. Systems architects should break the design into self contained modules with simple and clear functionality and API. Those modules can serve whichever piece of software that is in need of such functionality.


Design a site like this with WordPress.com
Get started