Registering Custom Resource Intensive Scenarios to CLM Applications

There is no such thing as limitless computing power. This is an unfortunate truth that can cause problems running the CLM and other tools, as the usage grows. To understand what systems actually do when getting under heavy load, more and more monitoring was introduced over the last years. Resource intensive scenarios where identified and the CLM tools have capabilities to record information about their frequency and duration. Plan loading and SCM compare workspace are examples in the product.

Custom Resource Intensive Scenarios

In addition to resource intensive scenarios that are built in, it is also possible to introduce custom resource intensive scenarios. Some examples are:

• Custom automation that execute long running operations on work items, SCM data, requirements, test artifacts. Typical scenarios are custom export/import, mass updates, custom analysis of source code, baselines, linked work items.
• Follow up actions
• Long running custom dashboards

This is by no means a comprehensive list. It is possible to bring your clients and servers to their knees with custom themes that do not scale, by work item attribute customization adding more and more custom attributes, JavaScript providers, value providers with thousands of values to choose from and other customization.

What is your server up to?

When users complain about performance problems, even if a server is getting overloaded, it is hard to find the root causes, because a typical server does so many things.

Monitoring that has been added over time has helped, but it is still hard. It is sometimes even hard to understand the situation. As an example for how complex this can become. Users complained about performance.

Our performance architect looked at the server load and the build load and a huge amount of calls that we were not able to account for. The server was unarguably under heavy load created by builds, but the build users and SCM users where not complaining. The developers we talked to had no real issues. Some users, at a different location, using work items and running work item queries, had.

Because we could not explain the inconsistent feedback, I finally went to the location where the users where complaining. I met the users followed their day to day work and found the work item performance unacceptable. The web browser was even locking up on them.

Knowing this, we were able to reproduce the use case, and look into what happened. We found that the work item load was slow, especially on slow laptops, because it had to load so many team areas and iterations. This was specific to how the project area was configured and used.

We also found that the browser flooded the server with requests that where definitely not part of what the product UI sent. This basically forced the Web Browser to process and cache thousands of calls, reserving more and more memory and exhausting the CPU capabilities of the relatively weak laptops used by the users that complained.

The final verdict was, that there was a custom extension to the theme that created all these calls. It took us weeks and was luck that we found this out. If we had known there was such an extension, we would have been able to find this a lot faster. The server was still under a heavy build load, but the performance issue reported was not related to that.

Needless to say that this extension was also deployed in other environments. If it had a detrimental impact, it was heavily depended on the timeline and iteration structure of a project area. The more and deeper the worse.

It would have helped if we could have seen the extension working, and see how long it worked would also have helped.

Registering Custom Resource Intensive Scenarios

The same mechanism that is used to register resource intensive scenarios in the product code can be used to register custom resource intensive scenarios. Unfortunately, we where lacking a good description and supporting code that we could provide customers to use it for their extensions.

This has now changed. Some colleagues and I, independently, started creating a customer usable description how to register resource intensive scenarios. A colleague wrote some cURL code to do this. I wrote Java code to do this and started creating a presentation. When we found out, we decided to combine the effort. Here the result.

The Deployment Wiki page Register Custom Scripts as a Resource Intensive Scenario, explains, using an example, how the API works in general. It also explains how to retrieve and monitor this information.

Then it provides example code to perform this using cURL, Eclipse Lyo OSLC4J based java code, and RTC Plain Java Client Libraries based Java Code.

The Java Code comes with main classes to run it. This is basically example code, but it can also be directly used in command line based automation.

Open Source Code

Disclaimer and Download

Any code downloadable or accessible in this post is provided as is, without support, and used at your own risk. Part of the code was developed in Java using Eclipse and is based on the Eclipse Lyo Client. This was published as open source, under
 Eclipse Public License – v 1.0, in the incredible (mostly German speaking) Jazz Community and can be found here: custom-expensive-scenario-notifier-oslc4j.

Another part of the code was developed in Java using Eclipse and is based on the Plain Java Client Libraries. This was published as open source, under MIT license, in the incredible (mostly German speaking) Jazz Community and can be found here: custom-expensive-scenario-notifier-plainjava.

See the other examples the Deployment Wiki page Register Custom Scripts As a Resource Intensive Scenario.

How does it work?

There is basically a REST API to register the start and the stop of a scenario. All there is to register the start of the scenario at the beginning and then register the stop, after you are done. See Register Custom Scripts as a Resource Intensive Scenario for more details on the code.

What should your automation do?

If you have written automation tools or extensions, you should use the methods described in Register Custom Scripts as a Resource Intensive Scenario, to register your extension as an resource intensive scenario. Add the code to register the start and stop in a way that allows for disabling it easily.

Monitor the various resource intensive scenarios over time. For a scenario that takes only a fraction of a second, you could temporarily disable the registration. Scenarios that take a second or longer should continue to be monitored.

Related

• Deployment Wiki – Monitoring
• Monitoring CLM
• Monitoring RTC Cluster
• Deployment Wiki Known Resource Expensive Scenarios
• Register Custom Scripts as a Resource Intensive Scenario

Feedback

If you have questions around the Custom Resource Intensive Scenario code, ask them in the Jazz.net forum instead of commenting on the article or this blog post. Tag the question as a clm question and add the tag: custom-resource-intensive-scenarios to mark it for the reader.

Summary

Please use the method above to enhance your automation and extensions to allow monitoring their duration, frequency and deviation.

As always I hope this helps users out there with the Jazz products.

JMX Monitoring of a CLM Tomcat Server only with the CLM Webinterface

Stefan is doing some amazing work here and I think you might be interested. This is an example for server monitoring in a dashboard.

Steve Blog

[Update: Source Code now with Garbacollector]

Sometimes it is necessary to tackle down CLM performance issues. To monitor the OS resources like CPU, IO, Memory etc. is not really helpful, because Java is like a black box for an OS like LINUX. With JMX you can look “behind the scene”.

You can monitor the Heap, Threads, Memory and other things from a CLM  Tomcat Server. With the help of some JMX settings you can access this information with jconsole. jconsole is include in every Java JDK (not JRE) since 1.5.

But JMX has some disadvantages:

• Only Java
• Problems with firewalls. Especially Productive CLM server are in a protected environment, so it is nearly impossible to get access with your jconsole.
• “All or nothing” protection

Today we will create a Open Social Widget that will show the amount of running Threads, Heapsize, CCM Requests and Garbage Collector with the help of JMX…

View original post 105 more words

Boost Your Automation Performance Using an Automation Server

Can you speed up your automation? During testing the command line client for adding a comment to a work item I found the performance of my command line client not acceptable.

A quick measurement on my machine running the command line client and the server (Tomcat/Derby) shows that a single start of the automation and adding a comment takes around 6 seconds (stopped with a stopwatch). This is no great deal, if you just run the command line once or twice a day. But if it is called very often, especially if it is potentially called a lot of times by some external tool, every second counts.

I looked into the code and tried to understand where the time is spent, in order to find a way to make it faster. It turns out that the majority of the time is spent in the code

TeamPlatform.startup()

The time spent to start the Team Platform is around 5 seconds on my machine. This code however is essential as it is needed to start the framework that allows the Plain Java Client Library to run. So how can this be addressed?

Solution Options

I thought about approaches that came into mind.

1. Test if it already started and then avoid doing it?
2. Try to not shutdown the Team Platform?
3. Have a daemon process running that starts it one time and keeps it open?

Unfortunately

1. Does not work because the classes just vanish once the java runtime exits and the test says always it is down.
2. Does not work, because the java runtime does not exit with the team platform still running in some threads.

So the only solution I could come up with, was to have a server running. The server would manage the commands. It would have to do the TeamPlatform.startup() only once and otherwise it would just run the commands. Each command would potentially deal with a different user and repository. It would log in, do what needs to be done and log out.

There are several options how to implement such a server. Some require managing sockets, threads, connections etc. These solutions would also require to serialize the parameters for the call and manage message lengths. I wanted it easy to do and test and finally settled with going with Java RMI.

I was able to get the run time of adding a comment to an existing work Item from around 6 seconds down to 1 second, a good 1/6th of the original time which is an improvement of more than 80%. Essentially the time spent in the TeamPlatform.startup() is gone for all except the first call. I found this quite amazing.

Code and Copyright

As always, our lawyers, reminded me to state that the code in this post is derived from examples from Jazz.net as well as the RTC SDK. The usage of code from that example source code is governed by this license. Therefore this code is governed by this license, which basically means you can use it for internal usage, but not sell. Please also remember, as stated in the disclaimer, that this code comes with the usual lack of promise or guarantee.

I hadn’t done RMI code, I looked into some book I have and I looked into the internet. Since the whole copyright and patent situation is now getting into the way, I won’t publish all the code this time. I think copyright and patents has gotten out of hands. It will effectively make development and even provide snippets of code impossible, soon.

So, I looked into GoTo Java 2, a dust-catcher I have on the shelf since 2000 (yes, these ancient things made from dead trees, where you have to flip the pages and not swipe; you can also actually drop them into water and they just look really ugly afterwards, but still work).

I also found the Java Tutorials for RMI that I did not look into, because I finally found the best source for my purpose to be the Wikipedia Entry for RMI. The text in Wikipedia is available under the Creative Commons Attribution-ShareAlike License; additional terms may apply. I could probably publish code from the last two sources, but since I don’t know, from where they got the code, I won’t. I will tell you what parts of the code you can redevelop or reuse from the Wikipedia Entry for RMI instead.

The Solution Steps

First we need an interface to be able to call the command or operation on the remote server. I basically used the same call I use in all my examples. The call is just not a static call any longer, to be able to fit to an interface. I called the interface IRemoteWorkItemOperationCall. This interface also contains the string to locate the remote service used by the RMI client and the server.

/*******************************************************************************
 * Licensed Materials - Property of IBM
 * (c) Copyright IBM Corporation 2013. All Rights Reserved. 
 *
 * Note to U.S. Government Users Restricted Rights:  Use, duplication or 
 * disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
 *******************************************************************************/
package com.ibm.js.team.workitem.automation.examples.remoteserver;

import java.rmi.Remote;
import java.rmi.RemoteException;

import com.ibm.team.repository.common.TeamRepositoryException;

public interface IRemoteWorkItemOperationCall extends Remote  {

	public static final String LOCALHOST_REMOTE_WORKITEM_OPERATION_SERVER = "//localhost/RemoteWorkitemOperationServer";

	public abstract boolean runOperation(String[] args)
			throws TeamRepositoryException, RemoteException;
}

To be able to run our command line client published in this post, we need to make it to implement the interface we just created. We need to implement the runOperation() call for the interface, which is basically only calling the static method run(). We also need the default constructor and an ID for the serialization. The code below shows the interesting changes from the original.

/*******************************************************************************
 * Licensed Materials - Property of IBM
 * (c) Copyright IBM Corporation 2013. All Rights Reserved. 
 *
 * Note to U.S. Government Users Restricted Rights:  Use, duplication or 
 * disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
 *******************************************************************************/
package com.ibm.js.team.workitem.automation.examples;

import java.rmi.RemoteException;
import java.rmi.server.UnicastRemoteObject;

import com.ibm.js.team.workitem.automation.examples.remoteserver.IRemoteWorkItemOperationCall;

/**
 * Modifies a work item, creating a comment. The user who is associated with the
 * comment can be different from the modifying user if the optional user ID is provided.
 * 
 * Example code, see
 * https://jazz.net/wiki/bin/view/Main/ProgrammaticWorkItemCreation.
 */
public class ModifyWorkItemAddCommentOperation extends UnicastRemoteObject implements IRemoteWorkItemOperationCall{

	/**
	 * To allow serialization
	 */
	private static final long serialVersionUID = -8791626071201186450L;

	public ModifyWorkItemAddCommentOperation() throws RemoteException {
		super();
	}

	@Override
	public boolean runOperation(String[] args) throws TeamRepositoryException {
		return run(args);
	}
}

Now we need the RMI Server that will run the code. According to the Wikipedia Entry for RMI this class needs to extend UnicastRemoteObject and it also needs to implement IRemoteWorkItemOperationCall. Create the class to implement a main() method you can use to call it later.

The interesting code you have to do is shown below. You basically create the standard constructor. You create a serialization ID and then you implement the interface IRemoteWorkItemOperationCall. The call just checks if the TeamPlatform is already started and starts it once, if not. You can also move this part of the code into the constructor if you want to pay once at startup and never again later.

Then you instantiate the ModifyWorkItemAddCommentOperation. You call runOperation(), passing the parameters and return the result.

This is basically all you need to do for this simple example.

/*******************************************************************************
 * Licensed Materials - Property of IBM
 * (c) Copyright IBM Corporation 2013. All Rights Reserved. 
 *
 * Note to U.S. Government Users Restricted Rights:  Use, duplication or 
 * disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
 *******************************************************************************/
package com.ibm.js.team.workitem.automation.examples.remoteserver;

import java.rmi.Naming;
import java.rmi.RMISecurityManager;
import java.rmi.RemoteException;
import java.rmi.registry.LocateRegistry;
import java.rmi.server.UnicastRemoteObject;

import com.ibm.js.team.workitem.automation.examples.ModifyWorkItemAddCommentOperation;
import com.ibm.team.repository.client.TeamPlatform;
import com.ibm.team.repository.common.TeamRepositoryException;

public class RemoteWorkitemOperationServer extends UnicastRemoteObject implements IRemoteWorkItemOperationCall  {

	/**
	 * To allow serialisation
	 */
	private static final long serialVersionUID = 5857170183485435303L;

	protected RemoteWorkitemOperationServer() throws RemoteException {
		super();
	}

	@Override
	public boolean runOperation(String[] args) throws TeamRepositoryException, RemoteException {
		if(!TeamPlatform.isStarted()){
			System.out.println("Starting");
			TeamPlatform.startup();
		}

		IRemoteWorkItemOperationCall op = new ModifyWorkItemAddCommentOperation();
		return op.runOperation(args);
	}

	public static void main(String args[]) {
		// See server part of http://en.wikipedia.org/wiki/Java_remote_method_invocation
		// to implement the body
	}
}

Now implement the main() method similar to the Wikipedia Entry for RMI but use RemoteWorkitemOperationServer to instantiate your server and

LOCALHOST_REMOTE_WORKITEM_OPERATION_SERVER in the rebind statement like shown below:

RemoteWorkitemOperationServer obj = new RemoteWorkitemOperationServer();
 Naming.rebind(LOCALHOST_REMOTE_WORKITEM_OPERATION_SERVER, obj);

Finally implement a class RemoteWorkItemOperationClient similar to the client in the Wikipedia Entry for RMI. Modify the callOperation() method to pass the argument strings and pass back a boolean value. Use your interface IRemoteWorkItemOperationCall and IRemoteWorkItemOperationCall.LOCALHOST_REMOTE_WORKITEM_OPERATION_SERVER in the Naming.lookup call. Then call runOperation() passing the parameters.

    public boolean callOperation(String[] args) { 
        try {

        	IRemoteWorkItemOperationCall operation=(IRemoteWorkItemOperationCall) Naming.lookup(IRemoteWorkItemOperationCall.LOCALHOST_REMOTE_WORKITEM_OPERATION_SERVER);
            return operation.runOperation(args); 
        } catch (Exception e) { 
            return false;
        } 
    }

The last two statements in the main method after handling the SecurityManager look like below:

        RemoteWorkItemOperationClient remoteOperationClient= new RemoteWorkItemOperationClient(); 
        System.out.println(remoteOperationClient.callOperation(args));

If you have no compiler errors, you should be done.

Start the Server

To start the server, you need to follow the Wikipedia Entry for RMI and create a security policy. I used the no.policy because I had issues with the suggested client and server policy and did not want to solve the problem just yet.

You need a separate command line for the client and the server. I created a batch file based on the existing one, that can be called in the project folder.

RMIStartServer.bat looks as below:

%JAVA_HOME%/jre/bin/java -Djava.security.policy=no.policy -Djava.ext.dirs=C:\RTC403Dev\installs\PlainJavaAPI;%JAVA_HOME%\jre\lib\ext -cp ./bin/ com.ibm.js.team.workitem.automation.examples.remoteserver.RemoteWorkitemOperationServer

The client is started with RMIAddComment.bat which looks like below.

%JAVA_HOME%/jre/bin/java -Djava.security.policy=no.policy -Djava.ext.dirs=C:\RTC403Dev\installs\PlainJavaAPI;%JAVA_HOME%\jre\lib\ext -cp ./bin/ com.ibm.js.team.workitem.automation.examples.remoteserver.RemoteWorkItemOperationClient "https://clm.example.com:9443/ccm" "ralph" "ralph" "54" "Add a remote comment"

You should be able to start your server. Then start the client and see the performance of the first and subsequent calls. The subsequent calls should be a lot faster than the first one.

Summary

This post shows, how you can use some easy enhancements, to boost your automation performance, if you have to call the automation often.

• It would obviously be possible to provide an additional parameter in the call that describes the desired command and to instantiate the automation related to the command with very few effort.
• You could call the commands/operations from any machine, by providing a real network name; This could also make deployment easier, as you only have to maintain the server if you have new commands, the clients can stay as the are
• If you want to call the commands from multiple machines, consider to synchronize the calls, to avoid racing conditions

You might also want to think about the interface and provide a better way for user feedback to the caller. However, I think the time saved for calls is clearly worth the effort. Please be aware, that the code above is just sample code and could use some enhancements for stability, useability and error handling.

As always, I hope this post is useful for others.