JMX Monitoring of a CLM Tomcat Server only with the CLM Webinterface


Stefan is doing some amazing work here and I think you might be interested. This is an example for server monitoring in a dashboard.

Steve Blog

[Update: Source Code now with Garbacollector]
Sometimes it is necessary to tackle down CLM performance issues. To monitor the OS resources like CPU, IO, Memory etc. is not really helpful, because Java is like a black box for an OS like LINUX. With JMX you can look “behind the scene”.

You can monitor the Heap, Threads, Memory and other things from a CLM  Tomcat Server. With the help of some JMX settings you can access this information with jconsole. jconsole is include in every Java JDK (not JRE) since 1.5.

But JMX has some disadvantages:

  • Only Java
  • Problems with firewalls. Especially Productive CLM server are in a protected environment, so it is nearly impossible to get access with your jconsole.
  • “All or nothing” protection

Today we will create a Open Social Widget that will show the amount of running Threads, Heapsize, CCM Requests and Garbage Collector with the help of JMX…

View original post 105 more words

Advertisements

Boost Your Automation Performance Using an Automation Server


Can you speed up your automation? During testing the command line client for adding a comment to a work item I found the performance of my command line client not acceptable.

A quick measurement on my machine running the command line client and the server (Tomcat/Derby) shows that a single start of the automation and adding a comment takes around 6 seconds (stopped with a stopwatch). This is no great deal, if you just run the command line once or twice a day. But if it is called very often, especially if it is potentially called a lot of times by some external tool, every second counts.

I looked into the code and tried to understand where the time is spent, in order to find a way to make it faster. It turns out that the majority of the time is spent in the code

TeamPlatform.startup()

The time spent to start the Team Platform is around 5 seconds on my machine. This code however is essential as it is needed to start the framework that allows the Plain Java Client Library to run. So how can this be addressed?

Solution Options

I thought about approaches that came into mind.

  1. Test if it already started and then avoid doing it?
  2. Try to not shutdown the Team Platform?
  3. Have a daemon process running that starts it one time and keeps it open?

Unfortunately

  1. Does not work because the classes just vanish once the java runtime exits and the test says always it is down.
  2. Does not work, because the java runtime does not exit with the team platform still running in some threads.

So the only solution I could come up with, was to have a server running. The server would manage the commands. It would have to do the TeamPlatform.startup() only once and otherwise it would just run the commands. Each command would potentially deal with a different user and repository. It would log in, do what needs to be done and log out.

There are several options how to implement such a server. Some require managing sockets, threads, connections etc. These solutions would also require to serialize the parameters for the call and manage message lengths. I wanted it easy to do and test and finally settled with going with Java RMI.

I was able to get the run time of adding a comment to an existing work Item from around 6 seconds down to 1 second, a good 1/6th of the original time which is an improvement of more than 80%. Essentially the time spent in the TeamPlatform.startup() is gone for all except the first call. I found this quite amazing.

Code and Copyright

As always, our lawyers, reminded me to state that the code in this post is derived from examples from Jazz.net as well as the RTC SDK. The usage of code from that example source code is governed by this license. Therefore this code is governed by this license, which basically means you can use it for internal usage, but not sell. Please also remember, as stated in the disclaimer, that this code comes with the usual lack of promise or guarantee.

I hadn’t done RMI code, I looked into some book I have and I looked into the internet. Since the whole copyright and patent situation is now getting into the way, I won’t publish all the code this time. I think copyright and patents has gotten out of hands. It will effectively make development and even provide snippets of code impossible, soon.

So, I looked into GoTo Java 2, a dust-catcher I have on the shelf since 2000 (yes, these ancient things made from dead trees, where you have to flip the pages and not swipe; you can also actually drop them into water and they just look really ugly afterwards, but still work).

I also found the Java Tutorials for RMI that I did not look into, because I finally found the best source for my purpose to be the Wikipedia Entry for RMI. The text in Wikipedia is available under the Creative Commons Attribution-ShareAlike License; additional terms may apply. I could probably publish code from the last two sources, but since I don’t know, from where they got the code, I won’t. I will tell you what parts of the code you can redevelop or reuse from the Wikipedia Entry for RMI instead.

The Solution Steps
First we need an interface to be able to call the command or operation on the remote server. I basically used the same call I use in all my examples. The call is just not a static call any longer, to be able to fit to an interface. I called the interface IRemoteWorkItemOperationCall. This interface also contains the string to locate the remote service used by the RMI client and the server.

/*******************************************************************************
 * Licensed Materials - Property of IBM
 * (c) Copyright IBM Corporation 2013. All Rights Reserved. 
 *
 * Note to U.S. Government Users Restricted Rights:  Use, duplication or 
 * disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
 *******************************************************************************/
package com.ibm.js.team.workitem.automation.examples.remoteserver;

import java.rmi.Remote;
import java.rmi.RemoteException;

import com.ibm.team.repository.common.TeamRepositoryException;

public interface IRemoteWorkItemOperationCall extends Remote  {

	public static final String LOCALHOST_REMOTE_WORKITEM_OPERATION_SERVER = "//localhost/RemoteWorkitemOperationServer";

	public abstract boolean runOperation(String[] args)
			throws TeamRepositoryException, RemoteException;
}

To be able to run our command line client published in this post, we need to make it to implement the interface we just created. We need to implement the runOperation() call for the interface, which is basically only calling the static method run(). We also need the default constructor and an ID for the serialization. The code below shows the interesting changes from the original.

/*******************************************************************************
 * Licensed Materials - Property of IBM
 * (c) Copyright IBM Corporation 2013. All Rights Reserved. 
 *
 * Note to U.S. Government Users Restricted Rights:  Use, duplication or 
 * disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
 *******************************************************************************/
package com.ibm.js.team.workitem.automation.examples;

import java.rmi.RemoteException;
import java.rmi.server.UnicastRemoteObject;

import com.ibm.js.team.workitem.automation.examples.remoteserver.IRemoteWorkItemOperationCall;

/**
 * Modifies a work item, creating a comment. The user who is associated with the
 * comment can be different from the modifying user if the optional user ID is provided.
 * 
 * Example code, see
 * https://jazz.net/wiki/bin/view/Main/ProgrammaticWorkItemCreation.
 */
public class ModifyWorkItemAddCommentOperation extends UnicastRemoteObject implements IRemoteWorkItemOperationCall{

	/**
	 * To allow serialization
	 */
	private static final long serialVersionUID = -8791626071201186450L;

	public ModifyWorkItemAddCommentOperation() throws RemoteException {
		super();
	}

	@Override
	public boolean runOperation(String[] args) throws TeamRepositoryException {
		return run(args);
	}
}

Now we need the RMI Server that will run the code. According to the Wikipedia Entry for RMI this class needs to extend UnicastRemoteObject and it also needs to implement IRemoteWorkItemOperationCall. Create the class to implement a main() method you can use to call it later.

The interesting code you have to do is shown below. You basically create the standard constructor. You create a serialization ID and then you implement the interface IRemoteWorkItemOperationCall. The call just checks if the TeamPlatform is already started and starts it once, if not. You can also move this part of the code into the constructor if you want to pay once at startup and never again later.

Then you instantiate the ModifyWorkItemAddCommentOperation. You call runOperation(), passing the parameters and return the result.

This is basically all you need to do for this simple example.

/*******************************************************************************
 * Licensed Materials - Property of IBM
 * (c) Copyright IBM Corporation 2013. All Rights Reserved. 
 *
 * Note to U.S. Government Users Restricted Rights:  Use, duplication or 
 * disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
 *******************************************************************************/
package com.ibm.js.team.workitem.automation.examples.remoteserver;

import java.rmi.Naming;
import java.rmi.RMISecurityManager;
import java.rmi.RemoteException;
import java.rmi.registry.LocateRegistry;
import java.rmi.server.UnicastRemoteObject;

import com.ibm.js.team.workitem.automation.examples.ModifyWorkItemAddCommentOperation;
import com.ibm.team.repository.client.TeamPlatform;
import com.ibm.team.repository.common.TeamRepositoryException;

public class RemoteWorkitemOperationServer extends UnicastRemoteObject implements IRemoteWorkItemOperationCall  {

	/**
	 * To allow serialisation
	 */
	private static final long serialVersionUID = 5857170183485435303L;

	protected RemoteWorkitemOperationServer() throws RemoteException {
		super();
	}

	@Override
	public boolean runOperation(String[] args) throws TeamRepositoryException, RemoteException {
		if(!TeamPlatform.isStarted()){
			System.out.println("Starting");
			TeamPlatform.startup();
		}

		IRemoteWorkItemOperationCall op = new ModifyWorkItemAddCommentOperation();
		return op.runOperation(args);
	}

	public static void main(String args[]) {
		// See server part of http://en.wikipedia.org/wiki/Java_remote_method_invocation
		// to implement the body
	}
}

Now implement the main() method similar to the Wikipedia Entry for RMI but use RemoteWorkitemOperationServer to instantiate your server and
LOCALHOST_REMOTE_WORKITEM_OPERATION_SERVER in the rebind statement like shown below:

RemoteWorkitemOperationServer obj = new RemoteWorkitemOperationServer();
 Naming.rebind(LOCALHOST_REMOTE_WORKITEM_OPERATION_SERVER, obj);

Finally implement a class RemoteWorkItemOperationClient similar to the client in the Wikipedia Entry for RMI. Modify the callOperation() method to pass the argument strings and pass back a boolean value. Use your interface IRemoteWorkItemOperationCall and IRemoteWorkItemOperationCall.LOCALHOST_REMOTE_WORKITEM_OPERATION_SERVER in the Naming.lookup call. Then call runOperation() passing the parameters.

    public boolean callOperation(String[] args) { 
        try {

        	IRemoteWorkItemOperationCall operation=(IRemoteWorkItemOperationCall) Naming.lookup(IRemoteWorkItemOperationCall.LOCALHOST_REMOTE_WORKITEM_OPERATION_SERVER);
            return operation.runOperation(args); 
        } catch (Exception e) { 
            return false;
        } 
    }

The last two statements in the main method after handling the SecurityManager look like below:

        RemoteWorkItemOperationClient remoteOperationClient= new RemoteWorkItemOperationClient(); 
        System.out.println(remoteOperationClient.callOperation(args));

If you have no compiler errors, you should be done.

Start the Server

To start the server, you need to follow the Wikipedia Entry for RMI and create a security policy. I used the no.policy because I had issues with the suggested client and server policy and did not want to solve the problem just yet.

You need a separate command line for the client and the server. I created a batch file based on the existing one, that can be called in the project folder.

RMIStartServer.bat looks as below:

%JAVA_HOME%/jre/bin/java -Djava.security.policy=no.policy -Djava.ext.dirs=C:\RTC403Dev\installs\PlainJavaAPI;%JAVA_HOME%\jre\lib\ext -cp ./bin/ com.ibm.js.team.workitem.automation.examples.remoteserver.RemoteWorkitemOperationServer

The client is started with RMIAddComment.bat which looks like below.

%JAVA_HOME%/jre/bin/java -Djava.security.policy=no.policy -Djava.ext.dirs=C:\RTC403Dev\installs\PlainJavaAPI;%JAVA_HOME%\jre\lib\ext -cp ./bin/ com.ibm.js.team.workitem.automation.examples.remoteserver.RemoteWorkItemOperationClient "https://clm.example.com:9443/ccm" "ralph" "ralph" "54" "Add a remote comment"

You should be able to start your server. Then start the client and see the performance of the first and subsequent calls. The subsequent calls should be a lot faster than the first one.

Summary

This post shows, how you can use some easy enhancements, to boost your automation performance, if you have to call the automation often.

  • It would obviously be possible to provide an additional parameter in the call that describes the desired command and to instantiate the automation related to the command with very few effort.
  • You could call the commands/operations from any machine, by providing a real network name; This could also make deployment easier, as you only have to maintain the server if you have new commands, the clients can stay as the are
  • If you want to call the commands from multiple machines, consider to synchronize the calls, to avoid racing conditions

You might also want to think about the interface and provide a better way for user feedback to the caller. However, I think the time saved for calls is clearly worth the effort. Please be aware, that the code above is just sample code and could use some enhancements for stability, useability and error handling.

As always, I hope this post is useful for others.