ELM Authentication

Authentication against a Jazz application is quite complex. When using the Plain Java Client Libraries, this is not a concern, as the ITeamRepository handles all that for you. It authenticates against the server and keeps the authentication valid over time. This is important, because application servers have timeouts that can void the the authentication of a session. A very common timeout that impacts the Jazz applications is the Liberty Profile default of 2 hours.

Running a client application against such a server requires detecting the need to authenticate and performing the authentication as well as to guarantee the current request is performed. Enterprise applications also require to work in complex setups supporting different ways to authenticate. Some examples are authentication based on Liberty Profile file based user management, LDAP based delegated authentication and user management, Jazz Authentication Server (JAS) based authentication, Kerberos and more. Implementing a client that uses the REST and OSLC APIs directly requires to develop a mechanism that handles this automatically.

Warning, this is complex!

I have tried to simplify this topic as good as possible, within my limited knowledge and experience, however this blog turned out to be very lengthy and complex. I also had to redo my screenshots multiple times to correct for changes during writing the post. Again, this ended up not being the casual blog post. You have been warned.

Approaching authentication

I have worked with the EWM Plain Java Client Libraries in the past. They provide a standard login mechanism to use and that keeps you logged in. I worked with Eclipse Lyo and used the mechanism provided with it, similar to plain Java API. I started just using an HTTP Client and I implemented a simple Login mechanism that was able to perform a Form based authentication. There was no code that would keep me logged in. When I started experimenting with Python, I just implemented a similar login mechanism that uses Form based authentication as explained in this post. In the first steps of the script, the authentication was executed, under the assumption that it was valid for the rest of the script.

That was sufficient for my experiments with my local test server. However it does not work with systems that have JAS enabled. This created questions, such as which authentication method was available and how to detect that, that required deeper knowledge about how the authentication was supposed to work. This approach also does not work for scripts that have to run longer durations. In many environments, the authentication expires after some timeout.

I started reading the Native Client Authentication article from the Team Wiki to understand how it was supposed to work. I got confused. I talked to some of my peers and hoped for someone having looked into this already. No luck. I cite the answer of a very skilled colleague that I hold in high regard: “Ah, this Native Client Authentication article is the most cryptic article I have ever come across.. at times it appears like everything has been told and at almost the same time, it appears like it is telling us nothing” and “I read the article now the 1001th time, and I still don’t understand”. Exactly.

I could not leave this as the answer and started coding while trying to follow the article. It makes no sense to me, to try to rewrite the article in this blog. Also just words will not bring any clarity. It looks as if language is not clear enough to describe what needs to be done.

The only way I am able to explain what goes on is showing code samples. I did not want these blogs to be about code, but it seems to be unavoidable to get clarity and to explain what goes on using the resulting code in my library class ELMCommLib. At this time I will use images to show the code. I will likely make the code available as open source later. Once done, I will link the download location to the blogs, as usual. The code is the result of a quite lengthy development and test process. It only implements parts of what the Team Wiki article explains. I have only so many test systems available and I can only test what these systems do. So here the disclaimer…

Disclaimer

Please note that so far I have only tested against a standard system supporting Form based authentication and a system supporting JAS. I used a trick to test the fallback to Basic authentication. My general disclaimer to the right of the page applies too. I’m not responsible for any damage done when following this article.

Context of the blog post is the series

This is the series of planned posts I intent to publish over time. Most of the examples will be EWM based, but quite a lot of the content applies to more ELM applications.

How the authentication code works

I started with creating a function internalJazzRequestWithAuthentication() that can be used for all HTTP requests by passing the method and all parameters needed. This function deals with the whole authentication requirements and makes sure the authentication dance is performed when needed. I provide more convenient functions that wrap this function and are easier to use.

Function internalJazzRequestWithAuthentication()

This function has become long and complicated, so I have to show it in smaller pieces below.

The function delegates the request to the function internalJazzRequest() which is performing the correct call using the requests library. The function looks like this.

Function internalJazzRequest()

The function returns the result for the request. The result is now evaluated in the first part of the authentication code of internalJazzRequestWithAuthentication().

The function internalJazzRequestWithAuthentication() checks the status code of the response to be able to determine if anything related to authentication needs to be done.

The first HTTP status code it evaluates is 200. Usually 200 means the call went OK and the result is the desired result. However, the result headers can contain information that the access to the resource would require authentication and the call actually did not return what was expected.

The function internalHandleFormChallenge() is used to handle this case. It checks the result headers to figure out if Form based authentication is required. The code to detect the Form authentication request is easy enough to understand. The code checks for the presence of a special custom header X-com-ibm-team-repository-web-auth-msg. If the header is present and contains the value authrequired a Form based authentication is required.

If Form based authentication is needed, the code performs the login and retries the initial call. Note that currently the code uses the URI that is provided when creating the communication library for authentication. This needs to be changed at some point. The code has to get the public URI root from the result. This is important for applications like DNG that delegate authentication to JTS.

Function internalHandleFormChallenge() handles Form based login challenges.

The function also handles the case that the initial request was done for an insecure URI. This is handled as a fail and a false is returned.

If no authentication was required, the current result is returned back.

If form based authentication is required, the function internalLoginForm() is used to perform the form based authentication. It is essentially the legacy code remaining from an initial login mechanism that I used for automation examples. There is a façade function loginForm in the code that keeps this code compatible to old scripts. The form based authentication sends the credentials to the URI publicURI/j_security_check and checks if the authentication worked by looking at the response location header.

Function internalLoginForm() performs the form authentication.

Back in the function internalHandleFormChallenge() the code tests if the login was successful. If so, the initial request for the resource is re-executed in the function internalRetry() and the result is returned. The retry code will be explained later.

There are several other return codes, typically for different methods like POST, PUT etc. that can require form based authentication in internalJazzRequestWithAuthentication(). One example is return status code 302 which is handled exactly like the code 200.

The wiki page Native Client Authentication mentions the return status codes 303 and 307. None of the test systems I worked against ever responded in any of those codes. At the time of writing they are detected in a check and the call will fail for these codes. Once I start sending unauthenticated PUT, POST, DELETE or PATCH requests, I might see these response codes. The handling would be the same. One of the next steps for me will be to setup tests that force the other situations.

The last possible response is 401. This is the most complex case. It handles all other cases for authentication, especially Jazz Authorization Server (JAS), OICD and some other authentication options. See the initial part of the handling for 401 in internalJazzRequestWithAuthentication() below.

401 Challenge, detection of the authentication method.

As explained in the wiki page it is necessary to check for the header WWW-Authenticate. If this header is present, it needs to be evaluated. This could be a Basic authentication challenge or a Bearer challenge for Jazz Authentication Server/OIDC/OAuth. The situation where the Bearer or the Basic challenge is missing is handled at the very end.

If the Basic challenge and the Bearer challenge is detected, the headers X-JSA-AUTHORIZATION-REDIRECT and X-JSA-AUTHORIZATION-URL are evaluated to determine the authentication strategy. For the test system I have available, I needed to follow the JAS authentication.

The first step to do JAS authenticate with Basic and Bearer challenge is to get the redirection URL from the header X-JSA-AUTHORIZATION-REDIRECT. Based on this the code constructs the first redirection URL stored in xjsaauthredirect with the additional parameter '&prompt=none'.

The code performs a GET on the no-prompt redirect URL. The result of this call determines the next steps. When the GET does not return 200, the authentication is still valid and the code runs a retry for the original request, the result of which is returned. This is shown here.

Retry code if no authentication is required.

If the GET returns status code 200 authentication is required. The code below shows the code to handle the authentication. Note, that there is some code that gets information that remain unused at this time.

To reauthenticate the code performs a GET on the redirection URI in xjsaauthredirect, this time without the additional no-prompt parameter. The request can result in a status code 401. This is indicating Kerberos or some other authentication method. I have not implemented this code path at the moment. I lack a test system that behaves this way.

The JAS/OAuth authentication dance

The request can result in a status code 200. In this case we are on the OAuth path. The code performs a request on the redirect URI in xjsaauthredirect, this time providing an additional authentication header conforming to the Basic authentication header. The header is computed in the call to HTTPBasicAuth(). If this succeeds with a status code 200, the client is authenticated and the code retries the initial request. This is not strictly necessary for all requests to do that, but for now, I wanted to do it anyway. The result is returned back to the outer code. Note that the Authentication header is only sent this one time. In contrast when using Basic authentication like below, the Authentication header is sent with all the subsequent calls.

In the case that the server only returned the Basic authentication header, the code below initializes the normal Basic authentication mechanism. The code is written in an way that now always sends the Basic authentication header with all requests in this case.

The case that Kerberos was detected is also handled here at the end. Lacking a test system the code returns unsupported and False for now. When the code was unable to identify the authentication method, e.g. the server was not a Jazz server, the code fails as well.

Handling of Kerberos and Basic authentication

This is what I was able to come up for handling authentication. There is no guarantee, that it is completely correct. I am lacking test systems that behave in different ways, so testing is very limited.

Please note, that the authentication is only valid for the session that is used and based on several cookies that are managed by the session. Also note, as already mentioned, the session can be invalidated by the server at any time and the authentication would have to (automatically) be done again.

Retry code

I ended up creating a function to handle the retry for the initial call that led to the authentication challenges. At the moment the code for GET is tested and works. The code currently basically reruns the initial request and checks if that was successful. Then it returns True and the result or False.

Then internalRetry() function performs a retry for the initial call.

While writing this code, it appears that the retry code might have to be changed later. When the first call requiring authentication is something other than GET the code might fail. As an example, I am not sure at the moment how a POST e.g. for creating a work item would be handled in detail. The key here is that POST of a work item may only be run once. This is going to be an interesting journey and there is much to be learned about redirects and retries. Once all cases are tested, I will try to re-upload the changes.

Example for authentication flows – Form authentication

As already mentioned, I am able to track the communication in my framework. The following sequence of messages is related to a GET request against a system that uses Form authentication. Please ignore the files with extension .nt, they are not relevant for the authentication.

Form authentication message flow.

The first call is to get the rootservices document. The rootservices document is not protected, so no authentication happens.

The next call is to get the service provider catalog which is discovered in the rootservices document. The service provider or workitems catalog is a protected resource, this means the request requires authentication. The request below shows that the call was redirected from the work items catalog to the web page showing the login prompt. The page code in my logs usually shows some mysterious message that “Javascript is either disabled or not available in your Browser”. This is based on the fact that there is no browser that sent the request and all JavaScript code in the HTML page fails.

The response header also contains the header X-com-ibm-team-repository-web-auth-msg authrequired indicating authentication is required.

Request returns with authentication challenge

The code detects that authentication is required. And executes the form based authentication, providing the username and password unencrypted. The latter is the reason why it is absolutely necessary to use HTTPS and not HTTP as protocol. HTTP sends the credential information unencrypted. The response of the login call contains the redirection location to the previous call.

Form based authentication is performed

At the moment, the code shown here does not use the redirect, it uses the original resource URI to retry the call. This is the retry the code performs. Please note that the destination URL and the Request URL are now the same. The response body contains the protected resource service provider catalog. This catalog contains the information about the visible available project areas.

Retry to get the service provider

All subsequent calls in this session are performed without any authentication challenge. The next call to get the work item service for the desired project area is omitted.

Example for authentication flows – JAS authentication

Finally lets have similar look at the sequence of events when running against a system with JAS Authentication. Note, I have replaced the original host name with a generic one, should there be any unclear references, in case I did not spot them. The following flow is the result of the same version of software running against a different URL that has JAS authentication configured.

JAS Authentication message flow.

The first call is to get the rootservices document again. The rootservices document is not protected, so no authentication happens.

The next call is to get the service provider catalog which is discovered in the rootservices document. The service provider/work item catalog is a protected resource, this means the request requires authentication. The request below shows that the call to the service provider catalog fails with a status 401. The response header also contains the header WWW-Authenticate with the form containing the Basic and Bearer challenge. The headers X-JSA-AUTHORIZATION-URL and X-JSA-AUTHORIZATION-REDIRECT are also present. This provides with the authentication and redirect URL for the authentication.

JAS authentication challenge with Basic and Bearer authentication header.

The next step is the GET request against the redirect URL with the additional parameter &prompt=none. This request collects additional information after some redirects. I don’t claim I understand the details here. Key is that the call can either determine that there is actually a valid authentication already, or a new authentication is required.

Request if logion is required

Here the previous call results in status code 200. This means that JAS authentication is required. So we GET the previous target URL, but without an extra parameter &prompt=none.

Note that the request gets redirected to the login URL that we have seen in the response to the initial call in the header. X-JSA-AUTHORIZATION-URL. The response header seems to indicate that Form based authentication is a valid fallback: X-com-ibm-team-integration-jazzop-auth-msg form-login. I am not sure and have not tried. I followed the wiki page Native Client Authentication implemented the JAS authentication. The response body here is the HTML code of the login page. If the client was a browser, it would be possible to show it to the user.

Determine the authentication mechanism

The previous request ended with a status 200 which means JAS/OAuth2 is the way to go. This is done in the next step. The request is executed again and an additional Basic authentication header is created and sent with the request. Note that the Authentication header is not shown below. If it was I would have had to redact it, because the data is only encoded, not encrypted. This is also the reason why Jazz Applications are by default all using HTTPs only. Do not send your credentials over HTTP.

Also note that the request gets successfully redirected back to the service catalog URI. The response body actually contains the service provider/work items catalog.

Authenticate to the system and continue

Because it is just a GET request, It would be possible to just return it as a result. At the moment I run a last request to retry the initial get. I will leave that request out in this example.

Summary

As always, I hope that this blog helps users out there that work with the Jazz products and their APIs. This was a very complex topic and I hope I was able to shed some light based on the experience I have gained over time. There are several cases missing, which I was unable to test lacking access to test systems and also test code that I have not yet developed. I will try to keep the post updated as always. Once I have finished this series, I will also try to publish the code for re-use.

If you have made your own experience with this topic and want to share, please do so in the comments.

Using the EWM REST and OSLC APIs

I have collected a lot of experience with the Java APIs for RTC/EWM over the years. Until 2020 I did not use the RTC/EWM REST and OSLC APIs at all. Luckily I got involved in several engagements where I had the opportunity to explore these APIs and learn how to use them.

I found the documentation for these APIs I was able to find underwhelming. The available documentation was often lacking complete working examples. There was usually some critical part missing or there where no examples at all, just the API specification. The latter seemed to be systematic to focus on the specification and not the specifics. However the lack of samples was confusing and left too much room for interpretation. I ended using search engines a lot. I had to experiment a lot to get things moving.

Obviously I learned a lot. I would like to spare others the hassle. So, as usual I want to share the experiences and lessons learned with the community. My intention is to provide with some relevant and working examples that are easy enough to perform on your own. I hope this can save people some time when trying to use these APIS as well.

This post will be the first in a series of posts and provide links to the other posts for easy navigation. In addition this post will discuss which development environment and tools were utilized to explore and use the APIs. I will share some of the code I have developed over the time to ease the exploration.

The planned blog posts for the series

Important links about the EWM/RTC APIs and the ELM APIs in general

I wrote Learning To Fly: Getting Started with the RTC Java API’s a couple of years back and it it still relevant. Read it to understand What API’s are Available for RTC and What Can You Extend? and how to get started with the RTC/EWM Java APIs. The post and the linked posts contain even more valuable links with respect to APIs.

Since back then, the ELM API Landing Page has been added to provide a more comprehensive overview about the available ELM APIs. If you are interested in ELM related APIs go over that page and find out what APIs are available. This page also points to other resources such as the OSLC standards and available workshops.

Finally the Interesting links page is a collection of, well, interesting links I found over the years.

Development environment choices

The first information I can share is how I explored and used the APIs and explain a little bit about the development environment options available and which development environments I used to explore the APIs.

I like developing with Java a lot. The EWM/RTC Java APIs are very rich and it is relatively easy to develop code for EWM/RTC, provided the development environment was set up by performing at least the complete Lab 1 of the Rational Team Concert Extensions Workshop. Eclipse, the RTC SDK and the Plain Java Client Libraries allow development of extensions and automation based on the EWM Java SDK and API. The same environment can also be used to develop code against the REST and OSLC APIs.

It is also possible to use Java or any other language supporting HTTP, to develop code for the EWM/RTC REST and OSLC APIs, just using libraries and available frameworks.

I have already used the Java based Eclipse Lyo framework to develop a client automation for Doors Next Generation (DNG) and I used the Eclipse Lyo Designer code generation framework to develop integration servers. My experience was that Lyo is a nice framework that helps a lot, if you know what you are doing. If I was not, I found it challenging, especially debugging and understanding what was going on in the HTTP requests.

I have looked into and used Postman and the Firefox addon RESTClient to experiment with REST and OSLC APIs. It is very useful for experimentation and I use it in parallel to the other development environments. A typical use case is to login and experiment with one call to get figured out how it works. If the call sequence and the amount of data becomes too big, it is not really efficient any more, and I would use a different approach.

I started using Python and Jupyter Notebook in 2020 when I had the need for some automation for importing manipulation, consolidating and querying a lot of CSV data for a customer. I was very impressed with the quality of the available libraries and the turn around times that were achievable. When I was asked to help one of our customer teams with information about the RTC/EWM APIs for the development of a prototype for a customer specific mobile client, I decided to use Python instead of Java. As mentioned above I also used RESTClient or Postman for experimentation with one or two API requests.

There are various Python development environments around. I do not think it matters which one you use. I used Spyder which comes with Anaconda. There is also PyCharm and kite. I am not opinionated. I just notice that the development environments are far away from the quality of Eclipse and the built in compiler and debugger. There are always tradeoffs, I guess.

Python – Libraries and Code Samples

The focus of the blog posts is more the APIs, how they work and how they can be used and not so much Python and how to use it. However, I figured that I want to share some code I developed over time, enabling easier data collection and debugging. So I will provide examples where I see fit.

The most important aspects of HTTP based APIs is to understand which method is used with which URI, which headers are used, which formats are sent and accepted and which request body (if any) is sent. The response data is also key, especially the status, response headers such as Location and obviously the response body. A mechanism that can log all this information is key in understanding the APIs and faster turn around times.

Python has a lot of libraries for various purposes. The Libraries that I used are shown below, loosely grouped by what they are used for. First the libraries used operating system and system specific purposes such as logging, files and execution.

import os
import sys
from datetime import datetime, timezone
from pathlib import Path

Then the requests library which is used for session handling and HTTP communication in my code.

import requests
from requests.auth import HTTPBasicAuth
from requests.packages.urllib3.exceptions import InsecureRequestWarning

This code below is necessary to suppress issues with certificates. This is a typical situation for me as I usually develop against some local test system.

# Disable warnings for self signed or invalid SSL certificates 
# to be able to talk to test systems
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

# Start a session
session = requests.Session()

The Libraries I used for RDF XML and JSON parsing and representation.

import json

from rdflib import Graph, URIRef, Namespace
from rdflib.namespace import CSVW, DC, DCAT, DCTERMS, DOAP, FOAF, ODRL2, ORG, OWL, PROF, PROV, RDF, RDFS, SDO, SH, SKOS, SOSA, SSN, TIME, VOID, XMLNS, XSD

Some miscellaneous library for encoding.

from base64 import b64encode

Python Logging and Reuse

I ended up creating a base library for the Communication with the ELM system that allowed better reuse. I will not share all the code at the moment, but I will share some basic learning and code that I found being key for me to be able to do my work. The library is referred to as:

from elmcommlib import ELMCommLib as elmcomm

The library is initialized with a session, the public URI and a name for the log folder to be created.

publicURI = 'https://elm.example.com:9443/ccm'
paName='JKE Banking (Change Management)'
user='ralph'
password='ralph'

# Disable warnings for self signed or invalid SSL certificates 
# to be able to talk to test systems
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

# Start a session
session = requests.Session()
comm=elmcomm(session, publicURI,'logCreateWiRDF')

The library also provides a mechanism to create and set log file folders using createLogFolder("FolderName") . In case the folder already exists it can alternatively set with setLogFolder("FolderName").

The log folder is used by the method writeResult() shown below, which dumps the complete communication in a text file, when a file name is provided. The file name should be constructed and numbered to better understand the flow of the sequences. Below shows such a sequence with file name numbering as an example.

The communication logs are always created in the current log folder. This allows to split the logs for the API usage into smaller sequences by switching the current log folder.

Content of a log folder.

A debug print dPrint() allows to avoid chatty logging. You can keep the logging entry and force it to show if you want. Printing a timestamp using timeStamp() is sometimes useful, especially when looking at performance of calls.

    # Folder for log files
    def createlogFolder(self,folderName):
        defaultLogFolder= 'commlogs'
        if(folderName==None):
            folderName=defaultLogFolder
        if(folderName==''):    
            folderName=defaultLogFolder
        script_dir = os.path.dirname(__file__) #<-- absolute dir the script is in
        logFolder=os.path.join(script_dir, folderName)
        Path(logFolder).mkdir(parents=True, exist_ok=True)
        return logFolder

    # Folder for log files
    def setlogFolder(self, folderName):
       self.logFolder=self.createlogFolder(folderName)


    # Log the HTTP communication for the request in a folder
    def writeResult(self, fileName, result, url=None):
        if(fileName!='fileName'):
            self.dPrint(f"Execute: '{fileName}'")
            logFileName=os.path.join(self.logFolder, fileName)
            with open(logFileName,'w') as f:
                if(url!=None):
                    f.write(f"Destination URL: {url}\n\n")
                reqMethod = result.request.method
                reqURL=result.request.url
                reqBody=result.request.body
                f.write(f"Request: {reqMethod} {reqURL}\n")
                f.write("\nRequest Headers:\n")    
                for header in result.request.headers:
                    value=result.request.headers[header]
                    f.write(f"\t{header} {value}\n")
                f.write(f"\nBody:\n{reqBody}\n") 
                f.write(f"\nResponse Status: {result.status_code}\n")
                f.write("\nResponse Headers:\n")
                for header in result.headers:
                    value=result.headers[header]
                    f.write(f"\t{header} {value}\n")
                cookies=result.cookies._cookies    
                f.write(f"\nResponse Cookies:\n {cookies}\n")
                f.write(f"\nResult Body:\n{result.text}\n")        
      
      
    # Debug print if debugging is on
    def dPrint(self, message=None, doPrint=True):
        # DebugPrint, switch off by sending doPrint=False
        if(message!=None):
            if(doPrint==True):
                print(message)
            else:
                pass
    

    # Print timestamp
    def timeStamp(self, message):
        now = datetime.now(timezone.utc)
        self.dPrint(f"{now}: {message}")
   

The image below shows how a log file for a message created using the method writeResult() looks like. Note that the log contains all the important pieces of the request, response pair. I used tooling in the editor Notepad++ to “pretty print” format the XML section in the response body. This makes it much easier to understand.

Logged http request – response

Help with RDF XML

The REST and OSLC APIs provide different serializations for the content that they accept and provide. One older one is XML based using the Resource Description Framework (RDF) specification. Newer standards such as JSON and more might be available. I have experience with RDF and JSON and I prefer JSON.

RDF is not for me. I always struggle to understand what and how I should be searching to get the data I want. Especially when the data is full blown with namespaces and what have you. This was one of the biggest struggles I had with Eclipse Lyo. The HTTP Client was hard to use for debugging, because the content was usually consumed when I tried to dump the response into a log file. So I could have a log entry for debugging and the call would not proceed or the call would work and I had no log data. Maybe I overlooked something.

In Python I was able to use the method writeResult() and continue processing the response data. I was able to use the function below to serialize RDF response bodies into a form that shows all the subject, predicate and object data and saves it into a file. That made it easier to work with RDF for me. I still prefer JSON format, if available. The OSLC discovery mechanism supported by RTC/EWM requires XML-RDF in the first steps, so you will have to deal with it.

    # Serializes a graph (based on RDF) in the nt format 
    # This format shows all graph nodes as Subject->Predicate->Object
    # This allows to better understand what to search for
    def debugSerialize(self, graph, fileName='fileName'):
        # Serializes a graph into the NT format. This provides 
        # a great source to look into RDF triples in the graph
        if(fileName!='fileName'):
            logFileName=os.path.join(self.logFolder, fileName)
            graph.serialize(logFileName, format="nt")

This requires the RDF definitions in Python. I used the ones below and defined them in my library.

    #RTC CM RDF definitions
    rtc_cm = 'rtc_cm'
    rtc_cm_URI = 'http://jazz.net/xmlns/prod/jazz/rtc/cm/1.0/'
    rtc_cm_ns = Namespace(rtc_cm_URI)
    
    oslc = 'oslc'
    oslc_URI = 'http://open-services.net/ns/core#'
    oslc_ns = Namespace(oslc_URI)
    
    oslc_cm = 'oslc_cm'
    oslc_cm_URI = 'http://open-services.net/ns/cm#'
    oslc_cm_ns = Namespace(oslc_cm_URI)
    
    oslc_xml_cm1 = 'oslc_cm1'
    oslc_xml_cm1_URI = 'http://open-services.net/xmlns/cm/1.0/'
    oslc_xml_cm1_ns = Namespace(oslc_xml_cm1_URI)
    
    jfs_process = 'jfs_proc'
    jfs_process_URI ='http://jazz.net/xmlns/prod/jazz/process/1.0/'
    jfs_process_ns = Namespace(jfs_process_URI)

    oslc_rm = 'oslc_rm'
    oslc_rm_URI = 'http://open-services.net/xmlns/rm/1.0/'
    oslc_rm_ns = Namespace(oslc_rm_URI)
    
    oslc_config = 'oslc_config'
    oslc_config_URI = 'http://open-services.net/ns/config#'
    oslc_config_ns = Namespace(oslc_config_URI)

This is how the parsed RDF graph data from above looks in the NT format. Every row (mind the word wrap) is a triple of subject, predicate and object. This provides with hints how to search for data.

The RDF-XML in NT format, providing the triples in the model.

The capabilities above where absolute key for me to be able to explore and understand the EWM/RTC APIs and document them for my colleagues.

Summary

This is the first of a series of posts, I hope to publish more soon. I will try to keep this post maintained and I am looking forward to the next posts. As always, I hope that my content, especially in this blog, helps someone in the ELM community out there. If it does, feedback would be awesome.

The RTC Extensions Workshop has been updated for EWM 7.0.x

I am very passionate about the RTC Extensions Workshop as you might be able to tell from the content of this blog. Performing it with EWM 7.0.x provided several challenges. It became apparent that an update to the workshop would be beneficial.

I spent a considerable amount of time in the past two months to update the workshop. As a summary the following items where addressed:

  1. Since the CCM server is shipped with WebSphere liberty profile, configuring the server for debugging needed to be changed. The old way to configure the server still worked in the 6.0.x versions, so it went unnoticed. With EWM 7.0.1 this is no longer the case and the workshop was updated to address this.
  2. The advanced capabilities introduced in the EWM SCM system in the 6.x and later caused a deviation of the screen shots showing the pending changes. The workshop setup tool was slightly changed to fix this.
  3. The workshop setup tool and its shell script has been tested with Linux and MAC OS.
  4. I wanted to add a section to Lab 1, explaining how to setup the existing Eclipse client/server development workspaces to better support development and debugging of the Plain Java Client Libraries forever. The new last optional section addresses this. For this reason Lab 1 of the workshop is a must for anyone intending to create Java based automation or extensions to RTC/EWM.
  5. I had an errata list with a number of small issues, typos, naming inconsistencies and the like that were fixed. During reviews a bunch more showed up and were fixed.
  6. A colleague ran the workshop on his MAC, so this works. Use whatever is available for MAC like Eclipse and where this is not specifically available, use the Linux versions.

The RTC Extensions Workshop has been published with an additional section for the new EWM versions and is now available for download. I will update recent posts around the workshop in the next few days.

As always, I hope that this blog post helps the users in the Jazz Community.

EWM Extensions Workshop remote debugging in 7.0.x

As explained in Issues with the EWM/RTC Extensions workshop in 7.0.x versions, I experienced issues with configuring the EWM Development server for remote debugging. The first EWM version I experienced this, is 7.0.1.

As mitigation I suggested to replace the JRE with an earlier one, or one that works. Unfortunately, this does not look sustainable. I have tried downloading the newest IBM Java SDK and the issue still happens.

I have looked a little bit deeper into it and discovered that the server.startup script provides a debug option. Using this option allows remote debugging the EWM server. I am looking into an update to the RTC/EWM Extensions Workshop for 7.0.x. until this is available here a procedure how to follow the workshop for 7.0.x versions.

Update for the Extensions Workshop now available

All the issues have been addressed in an update to the RTC Extensions Workshop for the 7.0.x and later versions. This should address all the issues.

Lab 1.2 – Do not add debug arguments.

In Lab 1.2, do not add the following lines to the server.startup script, remove these lines:

set JAVA_OPTS=%JAVA_OPTS% -Xdebug
set JAVA_OPTS=%JAVA_OPTS% -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=3388

Scroll to the bottom of the server.startup script and inspect the statements behind setting the JAVA_OPTS. Note that there are an action and an option supported.

Special parameters supported for starting Liberty Profile

Start server for remote debugging

The action -debug can be used to start the Liberty Profile Server in debug mode. When following the workshop, continue to Lab 1.5.

Follow the lab until 1.5__32 Add the following steps:

Open a console/cmd window and change directory into the server folder containing the server.startup script. Stop the development server by running server.shutdown.

>server.shutdown

Wait until the server is successfully stopped. Start the development server by running the following command.

>server.startup -debug

Note that the server does not start, it suspends listening for a remote connection on a dt_socket/port. The default socket/port address is 8000.

Server waits in debug console

The server will not continue starting unless a remote connection is opened on that port.

Connect Eclipse as remote debugger

Follow the workshop and open the [RTCExt] Debug Running Liberty Profile launch configuration. This launch is used to connect to a remote Java application. In our case the application is the RTS/EWM development server waiting on socket/port 8000.

Change the port in the launch to 8000 then click Apply and Debug.

Change debug port to 8000 apply and start debugging.

Switch to the command prompt. It takes a while and the server continues to start. Wait for the server application to successfully start.

Continue with the workshop and remote debug the server.

When disconnecting from the remote Java application, the RTC development server will show again, that it waits for a remote connection.

Reprovision the Jazz applications

The option -clean can be used to force to re-provision the Jazz Applications installed on the Liberty Profile server. This is necessary to force rereading custom server extensions during deployment of these extensions. The option can also used together with the -debug action.

Summary

Following the solution described above, allows to successfully perform the Lab 1.5 of the Rational Team Concert Extensions Workshop. I am currently trying to find out if there

  • Is an option to change the default port for this option
  • Is an option to avoid suspending the server startup until the remote debugger connection is initially done

I will update the workshop and publish the changes. I have already done several other changes to adjust the workshop for the newer EWM/RTC versions.

As always, I hope that the blog posts here help users in the Jazz community.

EWM/RTC Extensions Workshop works with EWM 7.0.x

As published in Issues with the EWM/RTC Extensions workshop in 7.0.x versions, there was an issue that caused the RTC Extensions Workshop to fail starting a Eclipse debug client in Lab 1.9. The problem and how it was caused has been found. I have just tested the RTC Extensions Workshop Lab 1 with the new EWM Client SDK (7.0.1 and 7.0.2) and it works now as it should. I will update my previous post.

Update for the Extensions Workshop now available

All the issues have been addressed in an update to the RTC Extensions Workshop for the 7.0.x and later versions. This should address all the issues.

While running the test, I found that I had to increase the maximum heap size for the launch [RTCExt] Jetty RTC Server, because the JUnit Test that is run to create the test database, ran out of heap memory. I doubled the value from -Xmx256M to -Xmx512M and the error was gone.

Increase the available heap

The fix is only in the EWM/RTC Client SDK, so it is only necessary to download that again.

Please note that due to the name changes and newer versions of Eclipse, there might be some small differences between the screen shots in the RTC Extensions Workshop and the newer versions. However, they are so minor, it should not be an issue. I am currently looking into updating the current Extensions Workshop material with small changes to help guide the user with these changes.

The issue with server debugging explained in Issues with the EWM/RTC Extensions workshop in 7.0.x versions, is still apparent in 7.0.2. The suggestion to change the JRE of the server in that blog post holds.

As always, I hope that my posts here help the Jazz user community and save them some time.

EWM/RTC Plain Java Client Libraries and Maven

I have worked with Maven applications using the RTC Plain Java Client libraries often recently. It has not always been fun, I have been struggling with Maven. I have been struggling with the Plain Java Client Libraries and Maven interacting. In this bog, I will try to share some things I have learned over the time, and recently, that you can use when working with Maven, especially in the context of EWM/RTC automation using the Plain Java Client Libraries.

Maven basics

How to install Maven is documented in this link. I do not intent to rewrite this documentation. I just want to emphasize some things that I always run into.

  1. Using Maven, requires a Java JDK. A JRE is not sufficient. Where to download a valid, supported, IBM JDK, is explained below.
  2. When using Maven in the command line, make sure to have the JAVA_HOME set to the correct JDK.
  3. When using Maven in the command line, make sure to have Maven added to the path.
  4. When using Maven with the Eclipse IDE, make sure to have an up to date Maven installed. I usually download the recent version, install it to work with a command line and then add that install to Eclipse.
  5. When using Maven with the Eclipse IDE, there is still the need to have a compatible JDK configured. Make sure to go to Preferences>Java>Installed JRE’s, add the JDK and set it as active. Then go to Preferences>Java>Installed JRE’s>Execution Environments configure an environment with the JDK. E.g. JavaSE-1.8 for EWM 7.0.x.
  6. When using Maven for a project in an Eclipse IDE right click on the project
    1. Use Maven>Update Project to make sure Eclipse is configured correctly.
    2. Use Run As>Maven clean on the Project or the pom.xml.
    3. Use Run As>Maven install on the Project or the pom.xml.
    4. Sometimes the Eclipse build mechanism seems to be in conflict with Maven, Project>Clean might help.
  7. Sometimes weird things happen. Run maven with -e or -X to see more details.
  8. Sometimes weird things happen, if all fails, consider to delete the local repository folder in the .m2 folder in your user directory.

Maven and the EWM Plain Java client libraries

When working with EWM/RTC automation based on the plain java client libraries, it is possible to download a valid, supported, IBM JDK on the same page where the Plain Java Client Libraries are hosted. When working with EWM/RTC and the Plain Java Client Libraries, the items below show the most important files to download to work with the Plain Java Client Libraries. To get them go to the All downloads tab for the version required. As an example for EWM 7.0.2 go to https://jazz.net/downloads/workflow-management/releases/7.0.2?p=allDownloads.

  1. Get the 10-Free Developers License Activation Kit. Scroll down to the section License Keys and download the 10-Free Developers License Activation Kit. It is very useful to run EWM test systems.
  2. Get the Client for Eclipse. Scroll to the Plain .zip Files section. Download the Client for Eclipse for your architecture.
  3. Get the plain Java Client Libraries. In the Plain .zip Files section download the Plain Java Client Libraries and the API documentation.
All Downloads page

The JDK

The download for the EWM Client for Eclipse contains the required JDK in the folder jazz/jdk. Unpack the EWM client and use the JDK for your development with the Eclipse Client Libraries and Maven.

The trouble with Maven and the Plain Java Client Libraries.

Unfortunately there is no Maven library for the Plain Java Client Libraries. This is a problem, because there is no real good solution (that I could find) to work with Maven and libraries that do not have a maven repository.

In some projects, I was able to get away with working with Maven and the Plain Java Client Libraries, using an Eclipse User Library with the Plain Java Client Libraries. This works, if at all, only in the Eclipse IDE.

In a recent project this solution was not manageable. I was unable to compile, or I was able to compile, but running the application failed with class not found exceptions. I got it working on my laptop, but we were unable to get it working for a colleague. We lost days.

My usual strategy of using <put your most popular internet search engine here> to find solutions did not really help. Solutions that I found where cryptic, not understandable and/or did not work.

Packaging the Plain Java Client Libraries as local Maven repository.

I finally found this script https://github.com/cokeSchlumpf/mvn-rtc-java-api that allows packaging the plain java client libraries as a maven repository and include that in the application POM file. This allows shipping the Plain Java Client Libraries as Maven repository with the application and its code to be able to run Maven in Eclipse as well as on the command line.

Although the script finally was a life saver, I had some tough learning to get things right. The main issue was that it is a Bash script and my Windows system does not Bash. I tried installing Cygwin64 but realized that I would have to install all kind of stuff to get that working. As I had recently done some other EWM and containerization related work on my NAS, I ended up setting up the required Maven and JDK on my NAS system. I was able to run the script from there 7Zip the result and upload it to my development environment.

In my first attempt, unfortunately, I did not pay enough attention to the format of the dependencies. The dependencies to external libraries has the format <groupId>:<artifactId>:<version>.

# 6.0.2
export DEPENDENCIES="org.apache.james:apache-mime4j:0.6 commons-io:commons-io:1.2 org.apache.httpcomponents:httpclient:4.5 org.apache.httpcomponents:httpclient-cache:4.5 org.apache.httpcomponents:httpclient-win:4.5 org.apache.httpcomponents:httpcore:4.4.1 org.apache.httpcomponents:httpcore-ab:4.4.1 org.apache.httpcomponents:httpcore-nio:4.4.1 org.apache.httpcomponents:httpmime:4.5"

For whatever reason I did not realize that and only provided the <groupId>. This creates an invalid repository. The symptom is the warning message ‘The POM for …. is invalid, transitive dependencies (if any) will not be available’. It took a while to figure out what the reason was. I ended up recreating the repository using the following dependencies values for 7.0.1:

# 7.0.1
export DEPENDENCIES="org.apache.james:apache-mime4j:0.6 commons-io:commons-io:1.2 org.apache.httpcomponents:httpclient:4.5.6 org.apache.httpcomponents:httpclient-cache:4.5.6 org.apache.httpcomponents:httpclient-win:4.5.6 org.apache.httpcomponents:httpcore:4.4.10 org.apache.httpcomponents:httpcore-ab:4.4.10 org.apache.httpcomponents:httpcore-nio:4.4.10 org.apache.httpcomponents:httpmime:4.5.6"

It is totally possible that some of my attempts to fix the above problem made things more miserable. I just realized the severity of the issue when I merged my projects to one and still had the problem.

Please note, if you follow the script above, provide the repository and change the pom of your application and run

mvn clean install

the repository is copied into the local maven repository on your machine. It is stored in the user directory in the folder

.m2/repository/com/ibm/rtc/rtc-java-api

If you build a new version of the repository, make sure to delete the folder above, to make sure your new version is actually installed. Maven does not recognize a difference and will not replace the local files on its own. The reason is that there is no version change visible in the data. This holds true also for the machines of your fellow developers. So when recreating a new Plain Java Client Libraries maven repository with the script, make sure to delete the old local maven repository for the library rtc-java-api in you local .m2 folder as explained above. Then run

mvn clean install

This will recreate the maven library.

I finally succeeded with running

# 7.0.1
export DEPENDENCIES="org.apache.james:apache-mime4j:0.6 commons-io:commons-io:1.2 org.apache.httpcomponents:httpclient:4.5.6 org.apache.httpcomponents:httpclient-cache:4.5.6 org.apache.httpcomponents:httpclient-win:4.5.6 org.apache.httpcomponents:httpcore:4.4.10 org.apache.httpcomponents:httpcore-ab:4.4.10 org.apache.httpcomponents:httpcore-nio:4.4.10 org.apache.httpcomponents:httpmime:4.5.6"

./setup-repo.sh -l ./libs-7.0.1 -r ./repo -d "${DEPENDENCIES}" -v 7.0.1

I followed the readme and added the following to my pom.xml

	<!-- The maven dependency for the EWM Plain Java Client Libraries provided in a local repository -->
		<dependency>
			<groupId>com.ibm.rtc</groupId>
			<artifactId>rtc-java-api</artifactId>
			<version>7.0.1</version>
			<type>pom</type>
		</dependency>

and the repository:

<!-- The maven repository for the EWM Plain Java Client Libraries -->
<repositories>
    <repository>
        <id>project-repo</id>
        <releases>
            <enabled>true</enabled>
            <checksumPolicy>ignore</checksumPolicy>
        </releases>
        <snapshots>
            <enabled>true</enabled>
        </snapshots>
        <url>file://${project.basedir}/repo</url>
    </repository>
</repositories>

The location is based on a project property:

		<project.basedir>${basedir}</project.basedir>

So the files of the repository where copied and located in my Eclipse project in the folder repo in the project root folder.

The repository is bundled in the folder repo

Now it is possible to run Maven from within Eclipse as well as on the shell/command line and my project cleans, installs and liberty:run and the warning mentioned above is gone.

I can only suggest to use the mvn-rtc-java-api – Simple setup RTC Java Client API for Maven project. I would like to thank Victor and Michael for their great contribution. I can only suggest to use it for your purposes.

Whats Next?

I am a bit hesitant to use my personal NAS system as an extended Linux system for development. I will see if I can find some time to understand what the application does in detail and maybe rewrite it in Python. That would make it more operating system independent and allow broader usage.

Summary

As always, I hope that this post helps users out there to achieve their goal and saves some hours of work.

Issues with the EWM/RTC Extensions workshop in 7.0.x versions

I have seen issues with the RTC Client SDK for the 7.0.x releases recently. The Server SDK seems not to be affected.

Client plug-in development

UPDATE: The issue with the Client SDK has been found and corrected. Just download and use the new Client SDK.

UPDATE: The issue with the server debugging can be solved as explained below.

If you are using the  Rational Team Concert Extensions Workshop for the EWM/RTC versions 7.0.0, 7.0.1, 7.0.2 and try to start an Eclipse client from the client launch as described in section 1.9 Test the RTC Eclipse client launch, you might run into a problem. I reported the issue in Defect 511733 (use your Jazz.net credentials).

This issue has been solved and you should not see it any longer. Make sure to download the latest Client SDK from Jazz.net.

If you need to develop client extensions and run into this problem, check the status of the defect. Worst case, use a 6.0.6.1 SDK to develop your extension. This is absolutely possible if you do not rely on a new client API feature from 7.0.x.

Server debugging

If you are using the  Rational Team Concert Extensions Workshop for the EWM/RTC versions 7.0.2, trying to launch the development server in debug mode that allows to attach an Eclipse debugger to that server, the server startup fails. In ‘1.2__15 start the server using server.startup.bat’ after adding the debug configuration, the server does not start. The log folder only contains the files start.log and console.log. The content indicates that the connection to the debug port failed because the port was already in use. When checking the pot usage, the port is not in use. This has not been seen in any version before. See Defect 527867 for work arounds and solutions.

Update: see http://EWM Extensions Workshop remote debugging in 7.0.x for how to deal with this.

Adding custom commands to the SCMUtils

The SCMUtils is based on a framework that I developed over time. The framework provides easy to reuse mechanisms to create custom commands that handle most of the essential requirements automatically. It handles calling a command based on a name string, the parameters the command needs and optional parameters. The command can define its own help content that is printed when either the command was called with missing parameters or if the command name is not supported. This blog post explains the steps needed.

What are the SCMUtils?

Please see SCM Utils – SCM data secure sharing, statistics and more for what they are and how to get them.

Enhancing the tool with own commands

It is easy to implement your own commands. The framework project provides abstract classes that can be used to implement own commands.

The framework allows easy implementation of new custom commands.

The class SampleCommandCmd is a simple example that can be used as a starting point. It implements all the basic capabilities that are needed.

The class AbstractCommand can be extended to a simple command that can execute your own code.

The class AbstractTeamrepositoryCommand is an abstract class that can be used to implement custom commands that interact with a Jazz application. It manages the parameter and behavior required for connecting to a Jazz server. In addition it automatically implements scenario logging. It can be extended to implement commands that work against a Jazz server.

It is not necessary to use one of the classes above. It is only required to implement the interface ICommand.

Tips and tricks for implementing custom commands

The framework comes with a sample command, that you can look to understand how the implementation works. If you want to create your own command, you should not do that in the project com.ibm.js.team.supporttools.framework. You should add it in a project area that refers to the com.ibm.js.team.supporttools.framework project. E.g. you can add the command in the project com.ibm.js.team.supporttools.scmutils.

Open the command /com.ibm.js.team.supporttools.framework.SampleCommandCmd and study the class.

Sample class part 1

The class SampleCommandCmd extends AbstractCommand, so it does not inherit logging into a jazz server. It also implements ICommand (which it inherits from AbstractCommand).

The next part is to define a logger. The framework uses Simple Logging Facade for Java (SLF4J) to provide logging. The log file will be written to disc. Use the logger to add custom logging.

The command must implement the default constructor and pass the command id to the framework. The super class implements this, so the command name is passed to the super class.

In addCommandOptions(), the class adds additional options (parameters). The first parameter in addOption() is the option/parameter name, the second defines if the option requires a value. The third parameter is the description of the option.

Sample class part 2

The command has to make sure it gets the options it needs. This is checked in the next step. The method checkParameters() is used to do just this. Test if the required options are available.

The method printSyntax() is called if there is some kind of issue with calling the SCMUtils. Implement your documentation here. The logger by default prints log level info.

Sample class part 3

The entry point for the custom code is the execute() method. This usually gets all the mandatory options and their values. The method getCmd().getOptionValue() is used to get the parameter or option value if the option has a value. The method getCmd().hasOption() is usually if the option does not have a value and is more like a flag.

Then it usually executes the payload code that does whatever the command is supposed to do. The return value should be true if the command succeeded and false otherwise.

Adding the custom command to the CommandFactory

After implementing a new command, it has to be added to the ScmSupportToolsFactory in the project com.ibm.js.team.supporttools.scmutils to make it available to be called..

The command factory

This is where the command needs to be added. See the available commands.

Add a command

Remove the comments before the put(new SampleCommandCmd()), to enable the sample command. You would add a put for your own command here. The SampleCommand is a good learning execise if you want to learn debugging.

Summary

It is really simple to add new commands, there is a set of commands already available, copy one of those if it looks similar to what you want to do.

Like always, I hope this helps someone out there.

SCM Utils – SCM data secure sharing, statistics and more

Last year, I was involved in a customer situation where we had issues with the EWM (RTC) Eclipse client. The customer reported client crashes and also performance/load issues. All these issues where intermittent and very hard to approach. It took a while and a lot of work to get them ironed out.

The performance or client load issue was apparently related to the customers data and its characteristics. I was able to reproduce it at the customer. We needed to be able to reproduce this issue in the development labs. The customer could not share their source code. The only way to share the data was to randomize the information that needed to be protected. Some kind of automation was also required, because we had to handle thousands of SCM components.

So I took some existing source code and wrote a small utility to help providing the data to enable our support reproducing the issue in the lab.

Initial Scope

Initially the tool provided commands supporting the following use cases:

  1. Export the components and their current content from a repository workspace to disc, randomizing the file content but keeping the names and structure intact.
  2. Importing the data into a repository workspace in a different repository recreating the components, their hierarchy and their content from the exported data.
  3. Converting an existing load rule for the original repository to match the component ID’s in the new repository to allow using the load rule.

This allowed us to share the data of the customer in a way that we were able to reproduce the issue in the lab.

To better understand the data, I began to use the tools capability to iterate the information in a repository workspace or stream to collect statistical information. This resulted in commands supporting the following use cases:

  1. Analyze a repository workspace or stream and provide statistics about its components and the contained folders and files.
  2. Analyze all streams in a repository to provide the statistics for each and the totals.
  3. Analyze a sandbox folder and its substructure for the same statistical information.

The information is printed on screen and provided as CSV file, allowing for drill down.

Download and License

The SCM Utils are provided as is, with no warranty or support, under MIT license. The source code is provided here. The repository also contains a comprehensive documentation of the commands and the parameters.

Working with the source code and building

Get the code either downloading the zip file or as local cloned GIT repository. Import all the projects into Eclipse e.g. in the GIT perspective.

Import all projects

There should be three projects as shown below in the Java perspective.

The three projects

The project com.ibm.js.team.supporttools.framework contains a small framework that is used by the tool. The project com.ibm.js.team.supporttools.scmutils contains the code for the SCM Utils. The project ewm-scm-utils contains the readme and all the other projects. I would suggest to close this project.

The projects show errors, because the projects are missing the Plain Java Client Libraries. In addition this tool uses Maven and Maven requires a JDK and an execution environment for it. Make sure to have one installed and activated. The tool was developed with Java 8.

Configure a JDK and set it in the preferences.

JDK is installed and selected

Make sure to activate an execution environment.

Execution Environment configured

Download the plain java client libraries from the All Downloads page of the EWM download. The Doc download for the Plain Java Client Libraries is not needed. Unzip the Plain Java client Libraries into a folder. Now create a user library for the Plain Java Client Libraries. In the preferences navigate to Java>Build Path>User Libraries. Click the New… button. Enter the name PlainJavaApi and click OK. Click on Add External JARs… Browse to the folder where you unpacked the Plain Java Client Libraries. Select all JAR files and click Open. Then click OK.

The PlainJavaApi user library.

There might be still errors, because Maven is missing dependencies.

Open the project com.ibm.js.team.supporttools.framework. Right click on the file pom.xml and select Run As > Maven clean. After that finished right click on the file pom.xml again and select Run As > Maven install. Do the same for com.ibm.js.team.supporttools.scmutils.

Run Maven Clean and Maven Install on both projects.

The code should now compile.

Working tool

The file README.md contains a help/description of the commands and parameters. The tool prints a help when it is run without or wrong parameters.

The code ships with example launches you can explore. To do that click on the triangle close to the bug symbol (or the triangle close to the run symbol). Open the Debug Configurations…. Search for the Java node and unfold it. Select a configuration and check the program arguments.

The available Debug/Run Configuration launches.

You can change the configurations to fit your data or create new ones. When you create new ones, consider storing the launches in a new project. Please do not push your changes to the GIT repository.

Build a runnable jar

The project com.ibm.js.team.supporttools.scmutils contains the file ReadMe – HowToRelease.txt. Follow the description in this file to create a runnable JAR. use the provided batch scripts to make starting the JAR easier.

Note, do not use the option to zip the libraries into the jar file. This results in terrible start times for the tool.

Exporting repository workspaces

The original use case that initiated writing this tool, requires a short explanation. The steps are in general:

  1. Run a exportScmWorkspace to export the data and structure.
  2. Run a importScmWorkspace to import the data where desired.
  3. If using loadrules based on component UUIDs, use convert convertLoadrule to replace the old UUID’s with the new UUID’s.

The command exportScmWorkspace exports the data of a repository workspace. It exports each component in its own zip file. The folder and file structure of the component is kept. The file and folder names are preserved. The file content is randomized by default. The file size is kept, but every character is replaced with a random number. The data is stored in a folder which can be set. In addition to the zip files a JSON file is created that contains the component names, ID’s and information about component hierarchy (sub component).

The optional parameter -exportmode can be used to change the behavior

-exportmode preserve keeps the file content as is.

-exportmode obfuscate replaces the content with random content from a text document containing code lines.

Example:

-command exportScmWorkspace -url "https://elm.example.com:9443/ccm/" -user myadmin -password myadmin -workspaceConnection "Hierarchical JKE Banking Integration Stream" -outputFolder "C:\aTemp\ScmExportRandomize"

This shows the following on the console

Executing Command exportScmWorkspace

The resulting export folder looks like

The JSON file contains information required for the import operation.

The structural and content information.

The information can now be used to import the data in any repository, including the repository.

Importing repository workspaces

The information that was exported can be imported into any Jazz SCM system. This is done using the command importScmWorkspace. The command requires the data of a previous export in a folder. In addition it requires a name for a repository workspace to import to, a project area name. The repository workspace is created during the import. The command has two special flags/switches

  1. reuseExistingWorkspace If the workspace to import to already exists, reuse it for import
  2. skipUploadingExistingComponents if a component with the same name already exists, skip the component upload.

These flags have been created to allow recovery from import errors. We ran into out of memory situations that we were unable to resolve. If an import fails with a component, rename the component that failed and remove it from the repository workspace. Then restart the import with these two switches and the import will skip over the components already imported and continue importing the missing components. This can be repeated until the import finally succeeded. Once the import of the components has succeeded, the component hierarchy is recreated.

The parameter componentNameModifier can be used to provide a prefix to be added to the name of each component that is created. This has several benefits. It makes the process repeatable with different prefixes. It also makes it easy to identify which components have been imported.

Example:

-command importScmWorkspace -url "https://elm.example.com:9443/ccm/" -user ralph -password ralph -inputFolder "C:\aTemp\ScmExportRandomize" -workspaceConnection "Imported Hierarchical Workspace" -projectarea "Formal" -componentNameModifier "IBMTestDefault_" -reuseExistingWorkspace -skipUploadingExistingComponents

This shows the following on the console

Importing components create and upload
Import components, recreate component structure

The resulting repository workspace contains all the components and the hierarchy:

Imported components in created repository workspace.

Note that the import stores mapping information between the component UUID’s of the component that are imported and the component that the data was imported in. The data is stored in a JSON file in the input folder.

The UUID mapping for the imported component.
The UUID mapping file

Load rule operations

The command convertLoadrule uses the folder containing the information for an export and an import and a load rule file as input and uses the mapping information to replace the source UUID’s in the load rule with the target UUID’s and saves the new load rule.

The command flattenLoadrule converts a load rule that has deep hierarchy removing the hierarchy. It Iterates a load rule and modifies pathPrefix entries for sandboxRelativePath. The modification replaces all / by _ except for the first /. This creates a flat load rule from a load rule that has hierarchy.

The command was created to try to understand if the hierarchy has impact on the load performance.

Analysis of streams and sandboxes

Another set of commands performs an analysis on the structure of repository workspaces, streams and sandboxes. They iterate the object, its content and substructure, and collect statistical information. The information is displayed and stored in CSV files. The CSV files allow to do more analysis later. The analysis currently collects obvious information such as file sizes and number of files per folder and less obvious information such as hierarchy depth, encoding and file extensions.

The following commands are available at the moment:

  1. analyzeScmRepository Iterates the whole repository and analyses the streams it has access to. Can be limited to scopes such as project or team areas. Collects the totals for the scope and uses the the capabilities of the commands below.
  2. analyzeScmWorkspace Analyses one repository workspace or stream.
  3. analyzeSandbox Analyses a folder on the local disk e.g. a sandbox. Excludes special folders such as .jazz3 or .metadata. The list of excluded folders is currently hardcoded.

Example:

-command analyzeScmRepository -url "https://elm.example.com:9443/ccm/" -user ralph -password ralph  -outputFolder="C:/aTemp/ScmAnalyzeRepository" -connectionOwnerScope "JKE Banking (Change Management)"

The analysis prints some of the data e.g. for each Stream and each component in the stream.

Collected data for each stream and each component in the stream

The aggregated data is printed for each stream.

Aggregated data for a stream

During the process, the analysis command generates csv files that are stored in the output folder.

CSV files as result

For each stream that is analyzed, a csv file is created. The csv file for the stream has all the data for the components in the stream and aggregated at stream level. The CSV file can be opened with Excel.

The final csv file created named _repository.csv contains the statistics for each stream that was analyzed and aggregated statistics across all analyzed streams. The individual csv files are accessible using a link.

Repository statistics

All csv files that are generated contain multiple sheets.the sheets to the right contain range statistics for the files and the extensions of this files for the analyzed context. The range metrix partitions the ranges in stripes with different top limits (and bottom limits). For each range it counts the the number of files that have a size that fits in a range. It also collects the file extensions associated to files in each range.

Stream operations

An additional set of operations has been recently added.

  1. uploadToStream Uploads a folder and its content as component to a stream and baselines the content. The folder name is used as the component name. The component is created if it does not yet exists. Ownership and visibility of the component is the project area. The component is added to the stream if it is not yet in it. All changes are contained in one change set. When a build result UUID is provided as optional parameter the command will publish the URIs for the stream, the baseline and the component as external links to the build result.
  2. downloadComponentBaseline Downloads the content of a component selected by a baseline into a local file system folder. The component name is created as folder and the content of the component is loaded into that folder.

Creating custom commands

It is possible to add custom commands to the SCMUtils. Adding custom commands to the SCMUtils explains how this in details.

Summary

The code is available, so it can be enhanced. The existing code covers several interesting areas in the RTC SCM API and can be used for inspiration. As always, I hope the SCM Utils will help someone out there.

Fixing the JKE Banking Sample Build

I ran into an issue related to the JKE Banking Sample application that ships with ELM beginning of this year. I ran into the same issue again. Here is what happens and how to fix it for 6.0.6.1 and for 7.0.

I use the JKE Banking Sample for presenting ELM and for a variety of other reasons that require to look deeper into ELM and how the Jazz tools work and interact. When I start looking into newer versions, I usually install a local test environment first and deploy the JKE Banking Sample. I also define build engines that allow me to test and demonstrate EWM/RTC build. I installed the 6.0.6.1 environment. When I tried to set up build, I ran into an issue. The build was failing with an error “The type java.util.Map$Entry cannot be resolved.” for unknown reasons. I found the issue and published how to solve it in this forum post. I created defect 500981 to address the issue.

The problem was the Eclipse JDT Core Batch Compiler (ECJ) that ships with the sample code was outdated and needed to be replaced with a newer version.

Recently I set up ELM 7.0.0 on my machine and I ran into the same issue.

As explained in the forum post above, I replace the ECJ Compiler with a current version. See Defect 500981 and  https://jazz.net/forum/questions/268299/jke-banking-sample-builds-failing-with-the-type-javautilmapentry-cannot-be-resolved-for-unknown-reasons-any-idea-how-to-fix-it .

Download the ECJ here: https://archive.eclipse.org/eclipse/downloads/drops4/R-4.6.3-201703010400/download.php?dropFile=ecj-4.6.3.jar. I chose that version because it is the same version the RTC Eclipse client is shipped with (zip version).

In the JKE Banking example load a repository workspace to disk.  Load the component Build. In the project JKEBuildScripts open the folder libs.

  1. Put the ecj-4.6.3.jar into the folder. Remove the file ecj-3.5.2.jar 
  2. Checkin the changes.
  3. In the Build definition go to the Ant tab. If the ANT argument refers to the old ecj-3.5.2.jar, rename the ant arguments to use the new version: -lib ${outputRootDirectory}\${buildLabel}\JKEBuildScripts\libs\ecj-4.6.3.jar
  4. Test the change fixes the problem by running a personal build against that repository workspace. 
New ECJ version uploaded.

See the fixed ANT Build Tab of the build definition to refer to the new compiler name in the ANT arguments section.

Ant arguments referring to the correct ECJ compiler.

But replacing the ECJ with a new version did not fix the issue entirely. After a while of trying to solve the issue, it became apparent that somehow the ANT build script was not matching the name for some of the libraries that ship with the sample code. The ANT file referenced file names that had been replaced by libraries that had different versions and different file names. Somehow this was overlooked.

So, in addition to changing the version of the ECJ as described in the link above, It is also necessary to change the line that creates the classpath for building in the build.xml highlighted above to

<property name="classpath" value="${applibsdir}/derby.jar;${applibsdir}/swt-win32-win32-x86/swt.jar;${applibsdir}/junit.jar;${applibsdir}/com.ibm.team.json_1.0.0.I200908182153.jar;${applibsdir}/jetty-server-9.4.19.v20190610.jar;${applibsdir}/jetty-util-9.4.19.v20190610.jar;${applibsdir}/jetty-servlet-9.4.19.v20190610.jar;"/>

Please note the name difference for the Jetty related files. Here the location in the build file.

Please make sure to check in and deliver all the changes to the component Build of the stream you are working on. Create a baseline for the component Build. Replace the component Build in the other streams with the new component baseline.

Please also make sure to correct the name of the ECJ in the ANT arguments on the ANT Tab for all ANT Build definitions.

Now the builds should work again.

There is work under way to fix this and potentially ship a backport. See defect 515939.