The Day The JBE Stood Still


Last week I realized my Jazz Build Engines for my test environments where not working any longer. Since I need them to work, I tried to fix this. It turned out that this was a major effort and I thought I should share what I found and how I was able to fix it – hopefully forever.

The Problem

The initial symptom was that the JBE process shells I usually see running, terminated immediately.

My initial thought was, that recently there were a lot of updates running on my machine to remove all Java versions with security issues and installing new versions. I suspected an issue with the Java version running.

Finally this proved to be true, however, during the attempt to come up with a working fix, I ran into a variety of other issues, I will summarize below.

The Solution

The final solution for me was to provide a working JRE to the Jazz Build Engine. To avoid it getting removed during security upgrades and to make sure it is compatible, I used a JRE that comes with the Jazz products.

One important detail is, that the JBE is currently a 32 bit application. If you use a 64 bit JRE with it, a variety of error symptoms crop up.

So the solution for me is, to

  1. Download the Plain ZIP package for the Eclipse client – make sure to download the 32 bit version
  2. Extract the download to some temporary location
  3. Locate the folder jazz\client\eclipse\jdk in the extracted folder
  4. Copy the folder jdk and all of its content into some folder you want to keep; for example: C:\CLMJava32
  5. Edit the file jbe.ini in \jazz\buildsystem\buildengine\eclipse and add the following lines where indicated
    -vm
    C:\CLMJava32\jdk\jre\bin

Your jbe.ini file should now look similar to the image below:

jbe.ini

You can probably get away with just the jre/bin folder, but I wanted the full JDK for other uses.

If you restart the JBE processes, everything should work.

Problems With the JBE and Wrong JRE’s

During my attempts to fix it I ran into various other issues, which I will try to list below.

  • The JBE shell just terminates
    This seems to happen due to incompatible JRE e.g. a newer version and for 64 bit JRE
  • The build process finishes with errors in the build result like
    !ENTRY org.eclipse.core.filesystem 1 1 2015-04-07 14:18:52.303
    !MESSAGE Could not load library: localfile_1_0_0.dll.  This library provides platform-specific optimizations for certain file system operations.  This library is not present on all platforms, so this may not be an error.  The resources plug-in will safely fall back to using java.io.File functionality.
    !STACK 0
    java.lang.UnsatisfiedLinkError: C:\CLM2014\5.0.2\jazz\buildsystem\buildengine\eclipse\configuration\org.eclipse.osgi\bundles\184\1\.cp\os\win32\x86\localfile_1_0_0.dll: Can't load IA 32-bit .dll on a AMD 64-bit platform
    

    This seems to happen due to using a 64 bit JRE; Errors can be found in the build logs or in the log of the build engine workspace; the log is usually located in \jazz\buildsystem\buildengine\eclipse\workspace\.metadata

  • The build process finishes with errors in the build result like
    2015-04-07 14:23:23 [Jazz build engine] Fetching files to fetch destination "C:\somefolder\JKEBuild\I20150407-1423" ... com.ibm.team.build.internal.scm.SourceControlUtility$2: Status ERROR: com.ibm.team.filesystem.client code=0 Cannot create sandbox at C:\somefolder\JKEBuild\I20150407-1423 because one already exists at C:\somefolder\ null

    This seems to happen due to using a 64 bit JRE

  • The build process finishes with errors in the build result like
    Failed to load the JNI shared library "C:\CLM2014\5.0.2\jazz\client\eclipse\jdk\jre\bin\j9vm\jvm.dll".

    This seems to happen due to using a 64 bit JRE

  • The build process is abandoned 

    This was the toughest issue; the builds where abandoned with the error message below

    2015-04-07 14:18:51 [Jazz build engine] Deleting fetch destination "C:\CLM2014\5.0.2\JazzTeamServer\server\JKEBuild\I20150407-1418" before fetching ...
    2015-04-07 14:18:51 [Jazz build engine] Fetching files to fetch destination "C:\CLM2014\5.0.2\JazzTeamServer\server\JKEBuild\I20150407-1418" ...
    com.ibm.team.build.common.TeamBuildStateException: Unable to "complete" build activity with label "_QCYgmd0gEeSrJ4S35NXZHw" because the build with ID "_P_aFEd0gEeSrJ4S35NXZHw", build definition ID "jke.dev", label "I20150407-1418" is in the "INCOMPLETE" state.
    	at com.ibm.team.build.internal.service.TeamBuildService.createIllegalBuildStateForActivityException(TeamBuildService.java:1599)
    	at com.ibm.team.build.internal.service.TeamBuildService$22.run(TeamBuildService.java:1471)
    	at com.ibm.team.build.internal.service.AbstractTeamBuildService.runAsRetryable(AbstractTeamBuildService.java:435)
    

    The reason for this finally was that the JBE processes that terminated left daemon Java processes, these processes tried to consume the build requests as well, which resulted in the build being abandoned

The abandoned builds cost me the most time, because it is completely hidden why this happens. Looking into the process list finally revealed the reason.

Summary

Providing a stable and compatible JRE is important to keep the build engines running. The solution above should work on all architectures. Searching the internet I did not find a lot of hints how to solve this, so I hope providing this post will come in handy and helps users that run into the same problem.

Advertisements

About rsjazz

Hi, my name is Ralph. I work for IBM and help colleagues and customers with adopting the Jazz technologies.
This entry was posted in Jazz, RTC, RTC Automation and tagged , , . Bookmark the permalink.

2 Responses to The Day The JBE Stood Still

  1. Nandini says:

    Hello Ralph ,

    can the above solution be used for RTC 6.0.1 version ? I did not find an eclipse client 32 bit for version 6. Could you tell me where i can find that?

    I am using Plain java client api 6 version and TeamPlatform.startup(); takes 20 mins when run through command prompt.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s