EWM OSLC Query API

After looking into how to create and update work items the question becomes, how to find work items to begin with. OSLC provides a query mechanism to allow querying for items. This post intents to show how OSLC queries work including some examples that work for me. The techniques explained in the previous posts in this series are important. If necessary, go back to the previous posts to understand the details. As usual the focus in this blog is EWM/RTC, however, the OSLC Query mechanism works for all product supporting it. So what is explained here based on EWM will work with ETM, DNG etc.

Context of the blog post is the series

This is the series of planned posts I intent to publish over time. Most of the examples will be EWM based, but quite a lot of the content applies to more ELM applications. The examples where performed with versions 6.0.6.1 and 7.0.x.

External Links

I used at least the following links for exploring the OSLC Query mechanism. The latest version of the OSLC Query V3 document contains several examples which are very helpful. I still found it a challenge to get some of the queries working.

RDF XML or JSON?

The examples in this blog are using the content type application/rdf+xml. The same examples work with the content type application/json, provided the function to parse the files are switched to using a JSON parser. I have the same examples using JSON. However the posts in this blog series are mostly using RDF+XML and I decided to stay with it, to make it easier to follow.

Query Base

Before being able to execute a query, it is necessary to discover the query base. The query base is the URI that defines the root for the OSLC query. The first step is to get the service provider catalog from the rootservices document as described in the blog post about EWM OSLC Discovery. Get the rootservices document

GET https://elm.example.com:9443/ccm/rootservices

Accept application/rdf+xml; charset=utf-8
OSLC-Core-Version 2.0

Get the service provider catalog like below. Pass the OSLC-Core-Version and the Accept content type header. This step requires authentication to be done.

https://elm.example.com:9443/ccm/oslc/workitems/catalog

The catalog lists the services available for each project area. To narrow down to a project area perform a GET on the work item service for the desired project area, providing the headers mentioned above. As an example

GET https://elm.example.com:9443/ccm/oslc/contexts/_8e5qfFpmEeukW7cqqDjAuA/workitems/services

The resulting response body contains various services that are provided for the various work item types of he project area.

  • Various dialogs such as OSLC selection, creation dialogs and pickers
  • The OSLC creation factories for all the work item types
  • The OSLC resource shapes for all the work item types
  • The OSLC query capabilities

To be able to query for work items, it is necessary to analyze the query capabilities provided by the service provider. The query capabilities can be found by searching for the oslc:queryCapability nodes.

There are usually at least two query capabilities in the work item service provider for an EWM project area. One is the query capability for deliverables. A deliverable is also referred to as a release and is a work item attribute type. The query capabilities for deliverables can be found using the resourceType http://open-services.net/ns/cm#Deliverable.

The second query capability is for work items (in OSLC referred to as change requests). The image below shows the oslc query capabilities for work items. This can be identified by the resourceType http://open-services.net/ns/cm#ChangeRequest

Query capability for work items.

The oslc:queryBase is the URI for the work item OSLC query mechanism for the selected project area. It will be used in constructing the OSLC Query URI. The query base has the form:

https://elm.example.com:9443/ccm/oslc/contexts/_8e5qfFpmEeukW7cqqDjAuA/workitems

where the second last segment is the internal unique ID of the project area.

The query capability also provides a resource shape for the Tracked Resource Set (TRS) provider. The resource shape has the form

https://elm.example.com:9443/ccm/oslc/context/_8e5qfFpmEeukW7cqqDjAuA/trs/shapes/workitems/query

The resource shape provides the information about the attributes provided by the TRS. The resource shape contains a oslc:valueShape for work items based on the work item type defect.

Value shape for work items

This provides the value shape that defines the common work item attributes, their types and the allowed values for these work item attribute value types. It is possible to GET the oslc:valueShape value shape (provide the OSLC related headers) like below:

GET https://elm.example.com:9443/ccm/oslc/context/_8e5qfFpmEeukW7cqqDjAuA/shapes/workitems/defect 

This value shape provides with all the common attribute and link types. The work item of a specific type might still have additional custom attributes. The post EWM Work Item OSLC CM API explains how to get that information using the work item type specific resource shape associated with the creation factory.

The image below shows the code that gets the query capabilities out of the project area work item service provider document.

Extract the query capabilities from the service provider document

This code shows how the query base is extracted from the service provider catalog. To do this, the code identifies the change request resource type and gets the associated query base.

Get the query base for the work items

Having the query base it is now possible to execute OSLC queries. The code below executes the most simple form of OSLC query:

Querying the query base.

The code performs the GET method on the query base.

GET https://elm.example.com:9443/ccm/oslc/contexts/_8e5qfFpmEeukW7cqqDjAuA/workitems

Accept application/rdf+xml; charset=utf-8
OSLC-Core-Version 2.0

Please note the OSLC headers Accept and OSLC-Core-Version above. The same headers are used in all calls in this blog.

Paging

The request is redirected to support paging like shown below. The paging mechanism is used to split queries with large result sets into smaller, more manageable chunks.

Paging redirect

The response body contains the work item URI’s for the first page of work items. Note, because there is no query parameter, the query only returns the work item URI’s and not any other property data. To get more properties of the resulting work items e.g. to display the summary would, in this case, require to get the work item using the URI from the response.

Response body contains work item URI’s and information about paging.

In addition the response contains information about the total amount of results and the URL to get the next page, in case the number of results exceeds the maximum items returned by this page. The image above shows the oslc:responseInfo which contains oslc:nextPage with the URI to get the next query result page and the oslc:totalCount with the total count of query results.

The code below shows how the query is executed. In the while loop a GET request to the query URI is executed. The result of the call is analyzed and this provides the next page of the query, if paging is needed. If no next page is available, the query is finished.

Execute an OSLC Query

The code below processes the result for a result page and gets the resulting work items to display them. It prints the work item URI. It also tries to get and print the identifier (Id) and the title (summary) of each work item in the query. If the identifier or the summary is not available nothing is printed. The code then analyzes the paging information. It returns the next page URL if there is one. It also returns the total count of the result.

Try to get at the result details of the query page

URL Encoding

OSLC Query provides a mechanism to create simple queries to retrieve items. These queries are sent as a URL in a GET HTTP request. There are limitations on which characters can be sent in the URL. Some characters have specific meanings in the URL. To avoid creating and sending the wrong or illegal URLs, parts of the OSLC query parameters need to be URL encoded. See some explanations about the URL encoding here:

In the code below url encoding is done using the function urlEncodeString as shown below.

comm.urlEncodeString('URL Encode me')

The code URL encodes the data in a way that has worked for me. I am not totally sure I understand the URL encoding in all its details and there might be issues in the code below. If I got something incorrect, please leave a comment with a suggestion to correct it.

More complex OSLC Queries

The code below uses the following constants (mind the = at the end of each term) which are used to compose more complex OSLC Queries:

select = 'oslc.select='
where = 'oslc.where='
searchTerms = 'oslc.searchTerms='
orderBy = 'oslc.orderBy='
prefix = 'oslc.prefix='

The statements represented by above constants and how they work are explained in the following sections.

The oslc.select Statement

OSLC queries support selecting the properties supposed to be returned in the query. The statement that is used for that is the oslc.select statement. The code below shows how the select statement is used. The first three lines are used to control running the queries and creating the file name for logging and can be ignored.

The interesting part is the selectString1. It defines the list of properties that should be returned for each query result. In the case below we want the type, the id, the title, the description, when it was last modified, who modified it, when it was created and who created it for each work item. This information is encoded in the selectString1 by creating a comma separated list of property identifiers. The select statement term is then created by concatenating the select statement ‘oslc.select=’ with the URL encoded version of the selectString1. This term is added to the query base as first query parameter using the separator ‘?’.

Select statement is provided to the OSLC query

When this query is run, the query result contains the properties for the work items that are returned in the query result. The image below shows the resulting data in the response.

The query result for a query using a select statement

The select statement can be nested to provide nested data. It is possible to use ‘*’ to select all available properties. The amount of data that is transferred can impact performance of the communication. The specification mentions that the servers SHOULD accept rdf:nil as single property. Both extremes are commented out in the cod above.

The oslc.properties Statement

The oslc.properties statement can be used to limit the set of properties or attributes of a work item that are considered in the OSLC request. The oslc.properties can be used in a GET request. In this case it can be used to specify which properties the requestor is interested in. The server does not have to collect and transmit all properties, but only the ones of interest.

The oslc.properties can be also used when updating a work item. This allows to perform partial updates. There is no example provided in this blog, but the syntax and encoding of oslc.properties and oslc.select are the same.

The oslc.where Statement

The oslc.where statement can be used to specify which work items to query. The first example below shows a query for the work item with the identifier 1. The select statement uses * to select all work item properties.

Get the work item with the ID 1 and return all properties.

The URL that is created can be seen below. The encoding makes it hard to understand. The important request headers are the OSLC-Core-Version and the Accept header.

The GET request with headers and URL encoding

The image below shows parts of the resulting RDF-XML for the work item with all the data for its properties.

The response with all properties

The In statement can be used to search for properties being in a range. The code below searches for the work items with identifiers being 1, 3, 9, 50. The query would run successful, if there are IDs that are not found.

Find the work items out of a list of ID’s

Please note as documented here, especially in the syntax section, the oslc.where statement only supports to compose complex conditions using the boolean operation ‘and‘. The operation ‘not’, ‘or’ or other complex nesting is not supported.

Before looking into more examples lets introduce the next statement.

Sorting – The oslc.orderBy Statement

The statement oslc.orderBy can be used to define the order for the result set. The example below finds all work items with the type defect and requests the result set to be ordered by severity (descending) and then by id (ascending). The order is defined with a prefix for the property. The prefix is + for ascending and as descending order. Multiple order specification can be given in a comma separated list.

Search for all defects and sort by descending severity and ascending id

Unfortunately, the orderBy seems not to work for EWM 7.0.2 and the accept format application/rdf+xml. I have seen it working with application/json.

Full text Search – the oslc.searchTerms Statement

The statement oslc.searchTerms can be used for full text search. The query below finds all work items that can be fond by full text searching for the term ‘dividend‘ creates 20 hits with the JKE Banking Sample.

Full text search for ‘dividend’

Define a Namespace Prefix – oslc.prefix

The OSLC query mechanism allows to define new namespace prefixes that can be used in the oslc.where, oslc.orderBy, and oslc.select statement. My experimentation with this feature was quite problematic. What worked for me is the following:

To define a namespace create a variable with the namespace prefix, including the equals sign, and encode the URL. Then use only one namespace in the oslc.prefix definition Only define one prefix for one namespace in each oslc.prefix statement.

pdcterms = 'dcterms=' + comm.urlEncodeString('<http://purl.org/dc/terms/>')
prefixTerm = 'oslc.prefix=' + pdcterms 

The code below defines several namespaces (mind the encoding part). Based on these namespaces I create a prefix term for each of the namespaces based on ‘oslc.prefix=’ and the namespace URI. To define multiple namespaces in a query concatenate the prefix terms with ‘&’. Some discussions on Jazz.net indicate it might be possible to use a comma separated list. I could not get this to work.

Defining prefixes

To use the prefixes in a query, just include the prefixes as query parameter.

Using prefixes in queries

The query URI can become very long when providing many prefixes.

The prefixes can make the query quite lengthy.

Search For Items With a Specific Enumeration Literal Value

The image below shows a query that selects only work items that have the severity set to critical. The enumeration literal to be provided for the selection is information that can be looked up by querying the resource shape and allowed values for each work item type associated to the creation factories. For common work item attributes the available attributes and allowed values can be found using the chain resourceShape to valueShape associated with the query base.

Queries for the severity literal representing the severity ‘Critical’

Search For Items Modified After a Certain Date

The image below shows a where term that selects all work items that are modified after a certain date.

Items modified after

Compound Where Terms

The OSLC Query mechanism supports compound where terms are built form simple terms. The only supported boolean operation is and. Or is not supported, neither is ‘not’. The code below shows a query that looks for work items of type ‘defect’ and severity ‘critical’.

Items of type ‘defect’ and severity ‘Critical’

Summary

This concludes my short blog about the OLSC Query mechanism. My intention, in addition to understand it better myself, was to provide some working examples and how to create the queries. I have tried this in the past and got some where, but it was quite challenging. The documentation I had available in the past was often lacking some small but important information. The documentation for OSLC CM 3.0 has improved a lot, but apparently there are still issues you can run into.

So as always I hope I was able to provide simple but relevant examples that help users of this technology to achieve their goals and save some time.

Using the EWM REST and OSLC APIs

I have collected a lot of experience with the Java APIs for RTC/EWM over the years. Until 2020 I did not use the RTC/EWM REST and OSLC APIs at all. Luckily I got involved in several engagements where I had the opportunity to explore these APIs and learn how to use them.

I found the documentation for these APIs I was able to find underwhelming. The available documentation was often lacking complete working examples. There was usually some critical part missing or there where no examples at all, just the API specification. The latter seemed to be systematic to focus on the specification and not the specifics. However the lack of samples was confusing and left too much room for interpretation. I ended using search engines a lot. I had to experiment a lot to get things moving.

Obviously I learned a lot. I would like to spare others the hassle. So, as usual I want to share the experiences and lessons learned with the community. My intention is to provide with some relevant and working examples that are easy enough to perform on your own. I hope this can save people some time when trying to use these APIS as well.

This post will be the first in a series of posts and provide links to the other posts for easy navigation. In addition this post will discuss which development environment and tools were utilized to explore and use the APIs. I will share some of the code I have developed over the time to ease the exploration.

The planned blog posts for the series

This is the series of planned posts I intent to publish over time. Most of the examples will be EWM based, but quite a lot of the content applies to more ELM applications. The examples where performed with versions 6.0.6.1 and 7.0.x.

I wrote Learning To Fly: Getting Started with the RTC Java API’s a couple of years back and it it still relevant. Read it to understand What API’s are Available for RTC and What Can You Extend? and how to get started with the RTC/EWM Java APIs. The post and the linked posts contain even more valuable links with respect to APIs.

Since back then, the ELM API Landing Page has been added to provide a more comprehensive overview about the available ELM APIs. If you are interested in ELM related APIs go over that page and find out what APIs are available. This page also points to other resources such as the OSLC standards and available workshops.

Finally the Interesting links page is a collection of, well, interesting links I found over the years.

Development environment choices

The first information I can share is how I explored and used the APIs and explain a little bit about the development environment options available and which development environments I used to explore the APIs.

I like developing with Java a lot. The EWM/RTC Java APIs are very rich and it is relatively easy to develop code for EWM/RTC, provided the development environment was set up by performing at least the complete Lab 1 of the Rational Team Concert Extensions Workshop. Eclipse, the RTC SDK and the Plain Java Client Libraries allow development of extensions and automation based on the EWM Java SDK and API. The same environment can also be used to develop code against the REST and OSLC APIs.

It is also possible to use Java or any other language supporting HTTP, to develop code for the EWM/RTC REST and OSLC APIs, just using libraries and available frameworks.

I have already used the Java based Eclipse Lyo framework to develop a client automation for Doors Next Generation (DNG) and I used the Eclipse Lyo Designer code generation framework to develop integration servers. My experience was that Lyo is a nice framework that helps a lot, if you know what you are doing. If I was not, I found it challenging, especially debugging and understanding what was going on in the HTTP requests.

I have looked into and used Postman and the Firefox addon RESTClient to experiment with REST and OSLC APIs. It is very useful for experimentation and I use it in parallel to the other development environments. A typical use case is to login and experiment with one call to get figured out how it works. If the call sequence and the amount of data becomes too big, it is not really efficient any more, and I would use a different approach.

I started using Python and Jupyter Notebook in 2020 when I had the need for some automation for importing manipulation, consolidating and querying a lot of CSV data for a customer. I was very impressed with the quality of the available libraries and the turn around times that were achievable. When I was asked to help one of our customer teams with information about the RTC/EWM APIs for the development of a prototype for a customer specific mobile client, I decided to use Python instead of Java. As mentioned above I also used RESTClient or Postman for experimentation with one or two API requests.

There are various Python development environments around. I do not think it matters which one you use. I used Spyder which comes with Anaconda. There is also PyCharm and kite. I am not opinionated. I just notice that the development environments are far away from the quality of Eclipse and the built in compiler and debugger. There are always tradeoffs, I guess.

Python – Libraries and Code Samples

The focus of the blog posts is more the APIs, how they work and how they can be used and not so much Python and how to use it. However, I figured that I want to share some code I developed over time, enabling easier data collection and debugging. So I will provide examples where I see fit.

The most important aspects of HTTP based APIs is to understand which method is used with which URI, which headers are used, which formats are sent and accepted and which request body (if any) is sent. The response data is also key, especially the status, response headers such as Location and obviously the response body. A mechanism that can log all this information is key in understanding the APIs and faster turn around times.

Python has a lot of libraries for various purposes. The Libraries that I used are shown below, loosely grouped by what they are used for. First the libraries used operating system and system specific purposes such as logging, files and execution.

import os
import sys
from datetime import datetime, timezone
from pathlib import Path

Then the requests library which is used for session handling and HTTP communication in my code.

import requests
from requests.auth import HTTPBasicAuth
from requests.packages.urllib3.exceptions import InsecureRequestWarning

This code below is necessary to suppress issues with certificates. This is a typical situation for me as I usually develop against some local test system.

# Disable warnings for self signed or invalid SSL certificates 
# to be able to talk to test systems
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

# Start a session
session = requests.Session()

The Libraries I used for RDF XML and JSON parsing and representation.

import json

from rdflib import Graph, URIRef, Namespace
from rdflib.namespace import CSVW, DC, DCAT, DCTERMS, DOAP, FOAF, ODRL2, ORG, OWL, PROF, PROV, RDF, RDFS, SDO, SH, SKOS, SOSA, SSN, TIME, VOID, XMLNS, XSD

Some miscellaneous library for encoding.

from base64 import b64encode

Python Logging and Reuse

I ended up creating a base library for the Communication with the ELM system that allowed better reuse. I will not share all the code at the moment, but I will share some basic learning and code that I found being key for me to be able to do my work. The library is referred to as:

from elmcommlib import ELMCommLib as elmcomm

The library is initialized with a session, the public URI and a name for the log folder to be created.

publicURI = 'https://elm.example.com:9443/ccm'
paName='JKE Banking (Change Management)'
user='ralph'
password='ralph'

# Disable warnings for self signed or invalid SSL certificates 
# to be able to talk to test systems
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

# Start a session
session = requests.Session()
comm=elmcomm(session, publicURI,'logCreateWiRDF')

The library also provides a mechanism to create and set log file folders using createLogFolder("FolderName") . In case the folder already exists it can alternatively set with setLogFolder("FolderName").

The log folder is used by the method writeResult() shown below, which dumps the complete communication in a text file, when a file name is provided. The file name should be constructed and numbered to better understand the flow of the sequences. Below shows such a sequence with file name numbering as an example.

The communication logs are always created in the current log folder. This allows to split the logs for the API usage into smaller sequences by switching the current log folder.

Content of a log folder.

A debug print dPrint() allows to avoid chatty logging. You can keep the logging entry and force it to show if you want. Printing a timestamp using timeStamp() is sometimes useful, especially when looking at performance of calls.

    # Folder for log files
    def createlogFolder(self,folderName):
        defaultLogFolder= 'commlogs'
        if(folderName==None):
            folderName=defaultLogFolder
        if(folderName==''):    
            folderName=defaultLogFolder
        script_dir = os.path.dirname(__file__) #<-- absolute dir the script is in
        logFolder=os.path.join(script_dir, folderName)
        Path(logFolder).mkdir(parents=True, exist_ok=True)
        return logFolder

    # Folder for log files
    def setlogFolder(self, folderName):
       self.logFolder=self.createlogFolder(folderName)


    # Log the HTTP communication for the request in a folder
    def writeResult(self, fileName, result, url=None):
        if(fileName!='fileName'):
            self.dPrint(f"Execute: '{fileName}'")
            logFileName=os.path.join(self.logFolder, fileName)
            with open(logFileName,'w') as f:
                if(url!=None):
                    f.write(f"Destination URL: {url}\n\n")
                reqMethod = result.request.method
                reqURL=result.request.url
                reqBody=result.request.body
                f.write(f"Request: {reqMethod} {reqURL}\n")
                f.write("\nRequest Headers:\n")    
                for header in result.request.headers:
                    value=result.request.headers[header]
                    f.write(f"\t{header} {value}\n")
                f.write(f"\nBody:\n{reqBody}\n") 
                f.write(f"\nResponse Status: {result.status_code}\n")
                f.write("\nResponse Headers:\n")
                for header in result.headers:
                    value=result.headers[header]
                    f.write(f"\t{header} {value}\n")
                cookies=result.cookies._cookies    
                f.write(f"\nResponse Cookies:\n {cookies}\n")
                f.write(f"\nResult Body:\n{result.text}\n")        
      
      
    # Debug print if debugging is on
    def dPrint(self, message=None, doPrint=True):
        # DebugPrint, switch off by sending doPrint=False
        if(message!=None):
            if(doPrint==True):
                print(message)
            else:
                pass
    

    # Print timestamp
    def timeStamp(self, message):
        now = datetime.now(timezone.utc)
        self.dPrint(f"{now}: {message}")
   

The image below shows how a log file for a message created using the method writeResult() looks like. Note that the log contains all the important pieces of the request, response pair. I used tooling in the editor Notepad++ to “pretty print” format the XML section in the response body. This makes it much easier to understand.

Logged http request – response

Help with RDF XML

The REST and OSLC APIs provide different serializations for the content that they accept and provide. One older one is XML based using the Resource Description Framework (RDF) specification. Newer standards such as JSON and more might be available. I have experience with RDF and JSON and I prefer JSON.

RDF is not for me. I always struggle to understand what and how I should be searching to get the data I want. Especially when the data is full blown with namespaces and what have you. This was one of the biggest struggles I had with Eclipse Lyo. The HTTP Client was hard to use for debugging, because the content was usually consumed when I tried to dump the response into a log file. So I could have a log entry for debugging and the call would not proceed or the call would work and I had no log data. Maybe I overlooked something.

In Python I was able to use the method writeResult() and continue processing the response data. I was able to use the function below to serialize RDF response bodies into a form that shows all the subject, predicate and object data and saves it into a file. That made it easier to work with RDF for me. I still prefer JSON format, if available. The OSLC discovery mechanism supported by RTC/EWM requires XML-RDF in the first steps, so you will have to deal with it.

    # Serializes a graph (based on RDF) in the nt format 
    # This format shows all graph nodes as Subject->Predicate->Object
    # This allows to better understand what to search for
    def debugSerialize(self, graph, fileName='fileName'):
        # Serializes a graph into the NT format. This provides 
        # a great source to look into RDF triples in the graph
        if(fileName!='fileName'):
            logFileName=os.path.join(self.logFolder, fileName)
            graph.serialize(logFileName, format="nt")

The serialization formats supported out of the box are “xml”, “n3”, “turtle”, “nt”, “pretty-xml”, “trix”, “trig” and “nquads”. For me NT and Turtle seem to be most useful, I built in capabilities to save the XML data as NT and Turtle format to help understanding how to be able to access the data later.

Update: The preferred option is to serialize the graph as Turtle format and look into that.

This is how the parsed RDF graph data from above looks in the NT format. Every row (mind the word wrap) is a triple of subject, predicate and object. This provides with hints how to search for data.

The RDF-XML in NT format, providing the triples in the model.

The capabilities above where absolute key for me to be able to explore and understand the EWM/RTC APIs and document them for my colleagues.

Operating on RDF requires the RDF definitions in Python. I used the ones below and defined them in my library.

    #RTC CM RDF definitions
    rtc_cm = 'rtc_cm'
    rtc_cm_URI = 'http://jazz.net/xmlns/prod/jazz/rtc/cm/1.0/'
    rtc_cm_ns = Namespace(rtc_cm_URI)
    
    oslc = 'oslc'
    oslc_URI = 'http://open-services.net/ns/core#'
    oslc_ns = Namespace(oslc_URI)
    
    oslc_cm = 'oslc_cm'
    oslc_cm_URI = 'http://open-services.net/ns/cm#'
    oslc_cm_ns = Namespace(oslc_cm_URI)
    
    oslc_xml_cm1 = 'oslc_cm1'
    oslc_xml_cm1_URI = 'http://open-services.net/xmlns/cm/1.0/'
    oslc_xml_cm1_ns = Namespace(oslc_xml_cm1_URI)
    
    jfs_process = 'jfs_proc'
    jfs_process_URI ='http://jazz.net/xmlns/prod/jazz/process/1.0/'
    jfs_process_ns = Namespace(jfs_process_URI)

    oslc_rm = 'oslc_rm'
    oslc_rm_URI = 'http://open-services.net/xmlns/rm/1.0/'
    oslc_rm_ns = Namespace(oslc_rm_URI)
    
    oslc_config = 'oslc_config'
    oslc_config_URI = 'http://open-services.net/ns/config#'
    oslc_config_ns = Namespace(oslc_config_URI)

Summary

This is the first of a series of posts, I hope to publish more soon. I will try to keep this post maintained and I am looking forward to the next posts. As always, I hope that my content, especially in this blog, helps someone in the ELM community out there. If it does, feedback would be awesome.

The RTC Extensions Workshop has been updated for EWM 7.0.x

I am very passionate about the RTC Extensions Workshop as you might be able to tell from the content of this blog. Performing it with EWM 7.0.x provided several challenges. It became apparent that an update to the workshop would be beneficial.

I spent a considerable amount of time in the past two months to update the workshop. As a summary the following items where addressed:

  1. Since the CCM server is shipped with WebSphere liberty profile, configuring the server for debugging needed to be changed. The old way to configure the server still worked in the 6.0.x versions, so it went unnoticed. With EWM 7.0.1 this is no longer the case and the workshop was updated to address this.
  2. The advanced capabilities introduced in the EWM SCM system in the 6.x and later caused a deviation of the screen shots showing the pending changes. The workshop setup tool was slightly changed to fix this.
  3. The workshop setup tool and its shell script has been tested with Linux and MAC OS.
  4. I wanted to add a section to Lab 1, explaining how to setup the existing Eclipse client/server development workspaces to better support development and debugging of the Plain Java Client Libraries forever. The new last optional section addresses this. For this reason Lab 1 of the workshop is a must for anyone intending to create Java based automation or extensions to RTC/EWM.
  5. I had an errata list with a number of small issues, typos, naming inconsistencies and the like that were fixed. During reviews a bunch more showed up and were fixed.
  6. A colleague ran the workshop on his MAC, so this works. Use whatever is available for MAC like Eclipse and where this is not specifically available, use the Linux versions.

The RTC Extensions Workshop has been published with an additional section for the new EWM versions and is now available for download. I will update recent posts around the workshop in the next few days.

As always, I hope that this blog post helps the users in the Jazz Community.

EWM Extensions Workshop remote debugging in 7.0.x

As explained in Issues with the EWM/RTC Extensions workshop in 7.0.x versions, I experienced issues with configuring the EWM Development server for remote debugging. The first EWM version I experienced this, is 7.0.1.

As mitigation I suggested to replace the JRE with an earlier one, or one that works. Unfortunately, this does not look sustainable. I have tried downloading the newest IBM Java SDK and the issue still happens.

I have looked a little bit deeper into it and discovered that the server.startup script provides a debug option. Using this option allows remote debugging the EWM server. I am looking into an update to the RTC/EWM Extensions Workshop for 7.0.x. until this is available here a procedure how to follow the workshop for 7.0.x versions.

Update for the Extensions Workshop now available

All the issues have been addressed in an update to the RTC Extensions Workshop for the 7.0.x and later versions. This should address all the issues.

Lab 1.2 – Do not add debug arguments.

In Lab 1.2, do not add the following lines to the server.startup script, remove these lines:

set JAVA_OPTS=%JAVA_OPTS% -Xdebug
set JAVA_OPTS=%JAVA_OPTS% -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=3388

Scroll to the bottom of the server.startup script and inspect the statements behind setting the JAVA_OPTS. Note that there are an action and an option supported.

Special parameters supported for starting Liberty Profile

Start server for remote debugging

The action -debug can be used to start the Liberty Profile Server in debug mode. When following the workshop, continue to Lab 1.5.

Follow the lab until 1.5__32 Add the following steps:

Open a console/cmd window and change directory into the server folder containing the server.startup script. Stop the development server by running server.shutdown.

>server.shutdown

Wait until the server is successfully stopped. Start the development server by running the following command.

>server.startup -debug

Note that the server does not start, it suspends listening for a remote connection on a dt_socket/port. The default socket/port address is 8000.

Server waits in debug console

The server will not continue starting unless a remote connection is opened on that port.

Connect Eclipse as remote debugger

Follow the workshop and open the [RTCExt] Debug Running Liberty Profile launch configuration. This launch is used to connect to a remote Java application. In our case the application is the RTS/EWM development server waiting on socket/port 8000.

Change the port in the launch to 8000 then click Apply and Debug.

Change debug port to 8000 apply and start debugging.

Switch to the command prompt. It takes a while and the server continues to start. Wait for the server application to successfully start.

Continue with the workshop and remote debug the server.

When disconnecting from the remote Java application, the RTC development server will show again, that it waits for a remote connection.

Reprovision the Jazz applications

The option -clean can be used to force to re-provision the Jazz Applications installed on the Liberty Profile server. This is necessary to force rereading custom server extensions during deployment of these extensions. The option can also used together with the -debug action.

Summary

Following the solution described above, allows to successfully perform the Lab 1.5 of the Rational Team Concert Extensions Workshop. I am currently trying to find out if there

  • Is an option to change the default port for this option
  • Is an option to avoid suspending the server startup until the remote debugger connection is initially done

I will update the workshop and publish the changes. I have already done several other changes to adjust the workshop for the newer EWM/RTC versions.

As always, I hope that the blog posts here help users in the Jazz community.

EWM/RTC Extensions Workshop works with EWM 7.0.x

As published in Issues with the EWM/RTC Extensions workshop in 7.0.x versions, there was an issue that caused the RTC Extensions Workshop to fail starting a Eclipse debug client in Lab 1.9. The problem and how it was caused has been found. I have just tested the RTC Extensions Workshop Lab 1 with the new EWM Client SDK (7.0.1 and 7.0.2) and it works now as it should. I will update my previous post.

Update for the Extensions Workshop now available

All the issues have been addressed in an update to the RTC Extensions Workshop for the 7.0.x and later versions. This should address all the issues.

While running the test, I found that I had to increase the maximum heap size for the launch [RTCExt] Jetty RTC Server, because the JUnit Test that is run to create the test database, ran out of heap memory. I doubled the value from -Xmx256M to -Xmx512M and the error was gone.

Increase the available heap

The fix is only in the EWM/RTC Client SDK, so it is only necessary to download that again.

Please note that due to the name changes and newer versions of Eclipse, there might be some small differences between the screen shots in the RTC Extensions Workshop and the newer versions. However, they are so minor, it should not be an issue. I am currently looking into updating the current Extensions Workshop material with small changes to help guide the user with these changes.

The issue with server debugging explained in Issues with the EWM/RTC Extensions workshop in 7.0.x versions, is still apparent in 7.0.2. The suggestion to change the JRE of the server in that blog post holds.

As always, I hope that my posts here help the Jazz user community and save them some time.

Issues with the EWM/RTC Extensions workshop in 7.0.x versions

I have seen issues with the RTC Client SDK for the 7.0.x releases recently. The Server SDK seems not to be affected.

Client plug-in development

UPDATE: The issue with the Client SDK has been found and corrected. Just download and use the new Client SDK.

UPDATE: The issue with the server debugging can be solved as explained below.

If you are using the  Rational Team Concert Extensions Workshop for the EWM/RTC versions 7.0.0, 7.0.1, 7.0.2 and try to start an Eclipse client from the client launch as described in section 1.9 Test the RTC Eclipse client launch, you might run into a problem. I reported the issue in Defect 511733 (use your Jazz.net credentials).

This issue has been solved and you should not see it any longer. Make sure to download the latest Client SDK from Jazz.net.

If you need to develop client extensions and run into this problem, check the status of the defect. Worst case, use a 6.0.6.1 SDK to develop your extension. This is absolutely possible if you do not rely on a new client API feature from 7.0.x.

Server debugging

If you are using the  Rational Team Concert Extensions Workshop for the EWM/RTC versions 7.0.2, trying to launch the development server in debug mode that allows to attach an Eclipse debugger to that server, the server startup fails. In ‘1.2__15 start the server using server.startup.bat’ after adding the debug configuration, the server does not start. The log folder only contains the files start.log and console.log. The content indicates that the connection to the debug port failed because the port was already in use. When checking the pot usage, the port is not in use. This has not been seen in any version before. See Defect 527867 for work arounds and solutions.

Update: see http://EWM Extensions Workshop remote debugging in 7.0.x for how to deal with this.

Adding custom commands to the SCMUtils

The SCMUtils is based on a framework that I developed over time. The framework provides easy to reuse mechanisms to create custom commands that handle most of the essential requirements automatically. It handles calling a command based on a name string, the parameters the command needs and optional parameters. The command can define its own help content that is printed when either the command was called with missing parameters or if the command name is not supported. This blog post explains the steps needed.

What are the SCMUtils?

Please see SCM Utils – SCM data secure sharing, statistics and more for what they are and how to get them.

Enhancing the tool with own commands

It is easy to implement your own commands. The framework project provides abstract classes that can be used to implement own commands.

The framework allows easy implementation of new custom commands.

The class SampleCommandCmd is a simple example that can be used as a starting point. It implements all the basic capabilities that are needed.

The class AbstractCommand can be extended to a simple command that can execute your own code.

The class AbstractTeamrepositoryCommand is an abstract class that can be used to implement custom commands that interact with a Jazz application. It manages the parameter and behavior required for connecting to a Jazz server. In addition it automatically implements scenario logging. It can be extended to implement commands that work against a Jazz server.

It is not necessary to use one of the classes above. It is only required to implement the interface ICommand.

Tips and tricks for implementing custom commands

The framework comes with a sample command, that you can look to understand how the implementation works. If you want to create your own command, you should not do that in the project com.ibm.js.team.supporttools.framework. You should add it in a project area that refers to the com.ibm.js.team.supporttools.framework project. E.g. you can add the command in the project com.ibm.js.team.supporttools.scmutils.

Open the command /com.ibm.js.team.supporttools.framework.SampleCommandCmd and study the class.

Sample class part 1

The class SampleCommandCmd extends AbstractCommand, so it does not inherit logging into a jazz server. It also implements ICommand (which it inherits from AbstractCommand).

The next part is to define a logger. The framework uses Simple Logging Facade for Java (SLF4J) to provide logging. The log file will be written to disc. Use the logger to add custom logging.

The command must implement the default constructor and pass the command id to the framework. The super class implements this, so the command name is passed to the super class.

In addCommandOptions(), the class adds additional options (parameters). The first parameter in addOption() is the option/parameter name, the second defines if the option requires a value. The third parameter is the description of the option.

Sample class part 2

The command has to make sure it gets the options it needs. This is checked in the next step. The method checkParameters() is used to do just this. Test if the required options are available.

The method printSyntax() is called if there is some kind of issue with calling the SCMUtils. Implement your documentation here. The logger by default prints log level info.

Sample class part 3

The entry point for the custom code is the execute() method. This usually gets all the mandatory options and their values. The method getCmd().getOptionValue() is used to get the parameter or option value if the option has a value. The method getCmd().hasOption() is usually if the option does not have a value and is more like a flag.

Then it usually executes the payload code that does whatever the command is supposed to do. The return value should be true if the command succeeded and false otherwise.

Adding the custom command to the CommandFactory

After implementing a new command, it has to be added to the ScmSupportToolsFactory in the project com.ibm.js.team.supporttools.scmutils to make it available to be called..

The command factory

This is where the command needs to be added. See the available commands.

Add a command

Remove the comments before the put(new SampleCommandCmd()), to enable the sample command. You would add a put for your own command here. The SampleCommand is a good learning execise if you want to learn debugging.

Summary

It is really simple to add new commands, there is a set of commands already available, copy one of those if it looks similar to what you want to do.

Like always, I hope this helps someone out there.

Searching and Exploring the RTC SDK

I have answered many customer questions about the RTC Java APIs in the past. Many of the posts in this blog are the result of such questions. I have no privileged access to the RTC source code. As far as I can tell, there is no secret library of RTC API documentation that the IBM development team is hiding from everybody else. Even if there was such documentation, I do not have access to it. So, how do I come up with those answers?

In short, by using the available capabilities to search for answers.

Prerequisites

The content below requires to know Learning To Fly: Getting Started with the RTC Java API’s and the Extensions Workshop.

Know where and how to search

The whole answer is, that it is not necessary to know everything. It is just necessary to know how and where to search for the answers. These days, there are basically two answers to where and how to search.

  1. The Internet using search engines
  2. The RTC SDK using the Eclipse IDE’s capabilities.

I have already explained how and where to search the internet in the post Learning To Fly: Getting Started with the RTC Java API’s in the section Where can I find Examples and Example Code?.

Search the internet

Search the internet and limit the search to sites that are likely to have relevant information. Examples are Jazz.net and stackoverflow.com. Jazz.net has the Forum and the development Wiki. Stackoverflow has a lot of questions and answers from many contributions.

I did contribute at stackoverflow as well, but gave up when they would not accept a link to external content as answer. The purpose of this blog is to be able to provide answers and share examples. For that very reason rsjazz.wordpress.com is also a good place to search. The search capability in the top right of this blog is also a valid place to search. The page Interesting Links is a collection of links to other sources I have come across over the years.

Very recently the Jazz community and Github have become a source as well.

Many questions in the context of RTC Extensions might be just related to Eclipse Plugin development. https://www.eclipse.org/ has a lot of examples and documentation around such questions.

Search the RTC SDK

If there are no ready examples and answers in the internet, there is a host of example code available in the RTC SDK. The RTC SDK contains at least the following:

  1. Unit test code for RTC the Java API. These contain especially examples for the administration API, but also for other parts of the client API provided by the Plain Java Client Libraries.
  2. The Java Code for the client API provided by the Plan Java Client Libraries, including the documentation that is used to create the JavaDoc for the Plain Java Client Libraries.
  3. RTC Rich Client Platform code used in the RTC Eclipse Client, including comments. This code uses the public client API but also uses internal API.
  4. RTC Client plug in code, including Client operation behavior such as advisors (pre-conditions), participants (follow up actions), aspect editors providing the Eclipse Admin UI and related information.
  5. RTC Server API and RTC Server code, including documentation for interfaces and methods.
  6. RTC Server plug in code, including server operation behavior such as advisors (pre-conditions), participants (follow up actions), aspect editors providing the Eclipse Admin UI and related information.
  7. Client and server extension points and related code.
  8. Code for asynchronous tasks.

All the code mentioned above is included in the RTC Server SDK that is shipped with RTC in the all download section. The RTC Client SDK that is shipped in addition since 6.0.3 only contains the client code which is kept compatible to earlier versions of the Eclipse client.

You can search the RTC SDK by using the capabilities provided by Eclipse. Especially the Eclipse Plugin Development (PDE) and Rich Client Platform (RCP) development tools provide a comprehensive set of features to search the RTC SDK. There are several approaches that can be used to search in Eclipse and the Eclipse PDE:

  1. Java Search for types, interfaces, methods and other Java Related properties. This includes the capability to use search pattern e.g. use an asterisk. It is possible to specify the relationships to search for e.g. search for classes that implement an interface.
  2. Search for Plugins, Extension points, references to extension points. This includes the capability to see the plugin.xml for other extensions, revealing all the implementation classes and relevant information that can be used again in 1 to narrow down the search to the relevant code.
  3. Eclipse RCP/SWT Inspectors such as Yari or tools like the built in Plugin Spy allow to use the declarative and reflective capabilities of Eclipse and the RCP to analyze the UI and other information provided in the plugins and features. This allows to find which classes are called in menus and to look at which data is used by views and how it is used.
  4. Other search capabilities such as File search and Text search can help in certain conditions.

These capabilities allow to pretty much find anything that is available in the RTC SDK. This does not make it trivial to understand what is found, especially the RCP UI code of RTC is sometimes very hard to understand, but it provides at least entry points, where to start.

Once something is found, there are various means available to

  1. Open or navigate to the element that is found
  2. Open the package containing the element, if applicable
  3. Open other items that are related to the found element

This is really all one needs to know. The rest is just using these capabilities to your advantage. Some examples how to use this are given below.

Finding Built-in Extensions in the SDK

The SDK contains all the client and server extensions and their source code. When developing a custom advisor or follow up action e.g. following the Extensions Workshop it is a good idea to get inspiration from the existing code. How does that code work? How does it use the API?

Any such scenario starts with finding examples that use one of the relevant extension points. The extension points are shown in the Extensions Workshop and other examples. Most likely there is an example using an extension point in the plugin.xml already, or a new plugin.xml has been just created.

In the open open editor of the plugin.xml on the Extensions tab, right click the extension point that is of interest. To understand where this is used select Find References and click to start the search.

Search for references to an existing extension point.
Search for references to an existing extension point.

The search window will open below and show the references found after a while. This can take some time, dependent on the performance of the client computer. The search result will show multiple hits. The image below shows a part of the hits in the current workspace. It especially shows the references from RTC SDK code to the extension point used by server operation advisors. All the references shown here have the same namespace prefix com.ibm.team. This hints that the origin of the code is in the RTC SDK.

Operation Advisors shipped with RTC.
Operation Advisors shipped with RTC.

Use the up and down arrows to browse the referencing plugin XML’s, or click one of the rows in the search that is of interest. Make yourself familiar with the name spaces that are used. This makes it easier to focus on the important examples you are looking for.

The namespaces refer to the following components of RTC

  • com.ibm.team.apt – Agile Planning
  • com.ibm.team.build – Build
  • com.ibm.team.filesystem – Source Control Filesystem
  • com.ibm.team.scm – Source Control
  • com.ibm.team.workitem – Work Items

When looking for an example for SCM related operations such as deliver, look at the examples in the namespaces filesystem and scm. When looking at work item related extensions look into the examples with the workitem namespace.

Click the search result to open the related plugin.xml. As an example the search result
com.ibm.team.workitem.service. Eclipse allows to find strings in open files, so it is possible to search for strings using CTRL+F. E.g. search for Required to find because you are interested in advisors for required attributes.

Required Attributes by Tape and State Advisor
Required Attributes by Tape and State Advisor

In the plugin.XML you can see the implementing class. You can use the full qualified class name e.g. com.ibm.team.workitem.common.internal.RequiredAttributesByTypeAndStateAdvisor to open the class using the Navigate>Open Type menu of Eclipse.

Open the implementation class of the advisor
Open the implementation class of the advisor

This repository workspace has the SDK set up as well as the RTC Plain Java Client Libraries. Select the class in the SDK. Browse the classes code and documentation. Open the Class in the package explorer.

Explore the package containing the class
Explore the package containing the class

In the package explorer view in Eclipse browse the package e.g. C:\RTC605Dev\installs\rtc-server-sdk\plugins\com.ibm.team.workitem.common_3.2.900.v20170928_1641.jar.

Scroll down to the lower area. There are some packages that do not contain Java classes. For example there is a package or rather folder, that contain files such as the schemas used by the aspect editors in the admin UI. You can open the files to look at the details.

Package explorer shows related content.
Package explorer shows related content.

Please note: the one I was really looking for is a bit elusive, and I have not been able to locate it until now. I will update the post as soon as I have a solution.

This is only one way of doing it. Dependent on what is available and what one knows there are many other ways to look into the source code. For example search for all extension points with a package name of com.ibm.team* to find all the extension points. Search for specific Plugins, in case the name is available.

Search for References to an extension point.

Try to explore different angles to find what you are looking for.

Search the Java Code

As already mentioned, it is possible to search for Java classes. This can also use search patter using an askerisk. Search for different aspects such as methods and interfaces and limit the search result set.

Once a class is open, it is easy to navigate within the class hierarchy, open the jar file in the package explorer etc..

Explore the UI to add Menus

Here a key point is knowing and using Eclipse RCP/SWT Inspectors such as Yari. How it is done is This has been explained in Adding Context Menus for Jazz Objects to the RTC Eclipse Client and Hiding UI Contributions in the RTC Eclipse Client.

Summary

As seen above, there are many ways to find stuff in the RTC SDK that can be used. As always I hope this helps users out there with their endeavors. I will try to add more examples.

Using a REST client to authenticate to ELM/CLM Applications

Recently the question how to authenticate to a ELM/CLM (aka Jazz) application came up in several occasions. I did not know the whole story either, so I had a look. Here is what I found.

Update history

2019/05/14 Added Basic Authentication

2021/01/22 Added Links

Important Links

Scenarios

There are several scenarios that could be of interest. Some of the scenarios will be provided below. A general documentation about how the authentication of a client to a Jazz application works is provided here in the development wiki.

Common scenarios are as follows:

  1. Using a REST client to explore the REST/OSLC APIs
  2. Creating custom automation using REST/OSLC APIs

Scenario 1 is also used in the OSLC Workshop. However, as explained below, things have changed since the workshop was initially written and the latest changes are not addressed in the workshop.

REST clients

The main changes that could be recognized are in the browsers and the integration of REST clients. These changes heavily impact how easy or hard it is to login. The first REST client used in the OSLC workshop was the Firefox browser addon REST Client . Using this addon It is easy to login to the Jazz application. All that needs to be done is

  1. Open a browser window
  2. Provide the URL to the CLM application and open the URL
  3. The application prompts for a login
  4. Provide user name and password and perform the login
  5. Open the REST Client Addon

The REST client addon uses the login performed in the steps above for subsequent calls to the server. This makes it easy, because it is transparent and there is no reason to worry about the details. The browser basically handled them.

Similar addons are available for the Chrome Browser. Examples are

  1. The Advanced REST Client as shown in this post.
  2. Postman as a Chrome App as explained at the end of the post above, which is no longer available.
  3. Tabbed Postman as Chrome App.

The browser integrations above all allowed to use the browser to login and use the login information in subsequent calls. These browser addons or extensions used to be very popular, but security threats and changes in integration philosophies seem to make them more and more unavailable. Some of the addons mentioned above are discontinued and might not be available much longer.

The Postman as packaged app as explained at the end of this post is currently still available and allows to configure a Postman Interceptor, that intercepts the login information from the browser and uses it in later calls. So it works like the solutions 1-3 above and the REST clients available as Firefox addons or extensions.

Alternatives are standalone solutions like Postman as standalone App which seems to be the path forward for Postman. There are many other stand alone REST clients. The caveat is, that the simple option to just login to the Jazz application is no longer available as an option. So it is important to know the details,

When trying to write custom tooling that needs to access data on a CLM/ELM server, it is also necessary to understand the details.

Testing and special considerations

It is a good idea to have a test system. Ideally local on your own machine. You can follow Lab 1 of the Rational Team Concert Extensions Workshop to set up a small system with JTS and RTC. It is not necessary to install the plain Java client libraries, any of the SDKs or to run the WorkshopSetup application. It is possible to install the other applications as well sing the Web Installer or one of the many Installation Manager repositories.

Create the sample project in the lifecycle project administration section to have data to play with.

Postman Settings

This blog shows the examples using Postman as standalone App which is very useful and allows to save and organize calls as a benefit. Like any application there are usually settings that can be beneficial or get into the way. Here the settings of interest in the context of this blog.

Disable SSL Certificate Verification

Please be aware that the SSL certificates of a test system as described above will typically be considered invalid. It is very common that test systems have invalid SSL certificates. A common issue is that the certificate is self signed and often for a different or generic host name. In such cases SSL certificate validation throws an error. Browsers, REST clients and APIs often reject working with servers presenting invalid SSL certificates. It is usually necessary to create exceptions or to configure the systems to ignore the SSL validation error.

Make sure the REST client or API works with sites that have invalid SSL certificates, or provide valid SSL certificates.

Postman rejects communicating with servers using invalid SSL certificates and displays an error message. This can be changed by changing the settings, turning the setting ‘SSL certificate verification’ to off. Postman now allows communicating to servers using invalid SSL certificates.

Disable SSL certificate verification
Disable SSL certificate verification to be able to work with example projects

As an alternative, it is possible to provide valid certificates. It is also possible to use any other test system that might be available, especially when dedicated test systems with valid certificates are available. The screen shots in this blog refer to a test system that has been set up similar to the Extensions workshop, but with all applications installed and the sample lifecycle project being deployed.

Automatically Follow Redirects

By default Postman automatically follows redirects. This can be nice but it hides a lot of details. To better understand the details explained in the wiki page Authentication of a native client with a Jazz-based application, switch the setting automatically follow redirects to off.

Setting to switch automatically follow redirects off and on.
Setting to switch automatically follow redirects off and on.

With the redirects disabled it is possible to see the 302 response as explained in Authentication of a native client with a Jazz-based application.

Minimal Header settings

When following redirects automatically, postman also collects several cookies. To focus on the really important details, and understand which headers and cookies are really needed, the setting below is very helpful.

Minimal setting - disable automatic redirect and additional headers
Minimal setting – disable automatic redirect and additional headers

Form based authentication

As explained in the wiki page Authentication of a native client with a Jazz-based application, there are multiple scenarios, how a client login would be performed. Unfortunately the number of scenarios are getting more over time as new standards are becoming available.

The easiest to use method that has worked for me so far has been the Form Based authentication. This works in a simple set up of CLM with any application server in a standard configuration without Jazz Authorization Server (OIDC) installed.

Perform a Form based authentication login to an application

To perform the Form based authentication to the application use the context root of the JTS or the application and append /j_security_check to the context root to create the login URI.

With a given context root of e.g. https://clm.example.com:9443/jts for JTS, the URI to login is https://clm.example.com:9443/jts/j_security_check.

As explained in Authentication of a native client with a Jazz-based application, the authentication is performed with a POST to the URI https://clm.example.com:9443/jts/j_security_check with a request body that contains the x-www-form-urlencoded encoded username and password.

The image below shows the simplest way of performing the authentication using Postman.

Minimum information required to perform FORM authentication
Minimum information required to perform FORM based authentication

The POST to the secure resource URI provides the username and password using the form encoded key and value pairs for the j_username and j_password. password and username are both identical myadmin.

The code for this request is shown below.

POST /jts/j_security_check HTTP/1.1
 Host: clm.example.com:9443
 cache-control: no-cache
 j_username=myadmin&j_password=myadmin

Please note that there are no additional headers necessary. The request does not specify anything in the request for Parameters, Authorization, Headers, Pre-request Script or Tests.

The response shows status 302. There is no header 
 X-com-ibm-team-repository-web-auth-msg. There is no header location with value  
 /auth/authfailed. This indicates the authentication was successful.
Post response shows status 302 and headers show no authentication failure.

The response shows status 302. There is no header X-com-ibm-team-repository-web-auth-msg. There is no header Location with any value, let alone
/auth/authfailed. This indicates the authentication was successful.

The Cookies keeps the LtpaToken to authenticate.

The LTPA Token is kept in a cookie.
The Cookies keeps the LtpaToken.

If the setting for Automatically follows redirects is on, and other header options are also set to on, the request would be redirected and likely show a status 200. There would be a lot more headers and Cookies collected as well.

Once successfully authenticated, it is possible to send other requests passing the Cookies. For this OSLC request it is also necessary to pass an accept header and the OCLS-Core-version header.

Successful retrieval of the OSLC workitems catalog URI.
Successful retrieval of the OSLC workitems catalog URI.

Please note that the Jazz Product APIs might need other special handling. In the above example the communication returns an additional Cookie JSESSIONID. In subsequent calls provide a header JSESSIONID with the value of the returned Cookie.

The easiest way to ‘log out’ is to delete the Cookies for the server.

delete all Cookies to restart.
Delete the Cookies for the domain to start again.

The request above can also be created by using a special content header Content-Type:application/x-www-form-urlencoded and a raw post body with a special encoding.

j_username=myadmin&j_password=myadmin

The images below show how this looks like in Postman.

The request above can also be created by using a special content header Content-Type:application/x-www-form-urlencoded and a raw post body with a special encoding.
Content-Type header
Raw POST request body.

This code shows the whole POST information.

POST /jts/j_security_check HTTP/1.1
 Host: clm.example.com:9443
 Accept: /
 X-Requested-With: XMLHttpRequest
 Content-Type: application/x-www-form-urlencoded
 cache-control: no-cache
 j_username=myadmin&j_password=myadmin

Authentication Failure

The Following image shows an unsuccessful authentication due to a typo in the password. This is only one of the possible flows, but the other scenarios are handled the same way.

The response shows status 302. There is a header Location with value /auth/authfailed. This indicates the authentication was not successful.
The Location header with value /auth/authfailed indicates a failed authentication attempt.

The response shows status 302. There is however a header Location with value /auth/authfailed. This indicates the authentication was not successful.

Authentication failed with Automatically follows redirects enabled leads to a JavaScript error page.
Authentication failed with redirect.

If the setting for Automatically follows redirects is on, the request would be redirected and likely show a status 200. The response body contains a page that the client could show to allow the user to enter the user ID and the password to authenticate. This web page uses JavaScript. Because the REST client Postman does not support JavaScript execution the page shows an error.

Where to Authenticate

My understanding so far has been that the CCM and the QM server have their own authentication. The RM server delegates authentication to the JTS. The Lyo implementation uses the public URI of the applications to create the authentication request. For the RM application it uses JTS instead.

The document Authentication of a native client with a Jazz-based application, seems to imply that authenticating to the JTS should do the trick for all applications. Using this has, so far, not caused any problems, so my suggestion would be to use the JTS URI to authenticate.

If you experience any issues, fall back trying to use the application specific URI.

Basic Authentication

Although Form based authentication has been the preference, Jazz also supports Basic authentication. However, when using Jazz Authorization Server/OIDC, Basic authentication is not supported. See this section (since 6.0.6).

See this information to understand the details.

As a summary, for Basic authentication, a special header Autorization that contains username and password encrypted in Base64 is sent with each request. The request below shows the header

Authorization: Basic c29tZXVzZXI6c29tZXBhc3N3b3Jk 

Testing and special considerations

When using Basic Authentication in Postman it is useful to have the Automatically follow redirects setting to be on.

To configure Basic authentication in Postman, select the Authorization type Basic Auth and enter the username and the password. Postman generates the authentication header. When using the API or other tools that do not create the authentication header, it has to be created and added.

Configure Basic Authentication.
Configure Basic Authentication.

The call below gets the workitems catalog for a RTC server on Jazz.net. This requires sending a valid Accept header and the OCLS-Core-version header.

Request with Basic Authentication.
Request with Basic Authentication.

The code below is the important information.

GET /jazz/oslc/workitems/catalog HTTP/1.1
 Host: jazzdev.rtp.raleigh.ibm.com:9443
 Accept: application/xml
 OSLC-Core-version: 2.0
 Authorization: Basic c29tZXVzZXI6c29tZXBhc3N3b3Jk
 cache-control: no-cache

When sending the request, the final response should have a status 200.

Successful response.
Successful response.

Please be aware that basic auth does not have a session handling and you have to always send the authentication header.

Malicious website

I am currently working with a customer trying to authenticate to one of their CLM systems. This seems to work except with the CCM server. When they try to authenticate they get the following message.

You have followed a direct link to log in to a Jazz server. This page has been presented to ensure that a malicious website cannot use cleverly crafted content to circumvent security. Please log in if you would like to access the server. 

We are looking into what could cause this issue. The web does not provide any good explanation. I found some questions in the Forum and other places and think I noticed a pattern. Several requests I have seen used a wrong URI to authenticate.

When using URIs like https://clm.example.com:9443/ccm/authenticated/j_security_check or
https://clm.example.com:9443/ccm/j_security_check and GET them in a browser, I get the same error.

You have followed a direct link to log in to a Jazz server. This page has been presented to ensure that a malicious website cannot use cleverly crafted content to circumvent security. Please log in if you would like to access the server.
Directly accessing internal URIs is prevented.

It is currently unclear what is happening and how to resolve the issue the customer is seeing. I will update the post, if we can figure out what is going wrong.

Summary

This blog post tries to shed some light on how to log into a CLM System to be able to use the REST and OSLC APIs. This should work for manual work as well as for automation tools. This blog uses Postman, but any REST client should work. The details, especially covering redirects and different authentication methods can be found inn this document Authentication of a native client with a Jazz-based application.

As always, I hope that this post helps users out there with using their Jazz tools.

Registering Custom Resource Intensive Scenarios to CLM Applications

There is no such thing as limitless computing power. This is an unfortunate truth that can cause problems running the CLM and other tools, as the usage grows. To understand what systems actually do when getting under heavy load, more and more monitoring was introduced over the last years. Resource intensive scenarios where identified and the CLM tools have capabilities to record information about their frequency and duration. Plan loading and SCM compare workspace are examples in the product.

Custom Resource Intensive Scenarios

In addition to resource intensive scenarios that are built in, it is also possible to introduce custom resource intensive scenarios. Some examples are:

  • Custom automation that execute long running operations on work items, SCM data, requirements, test artifacts. Typical scenarios are custom export/import, mass updates, custom analysis of source code, baselines, linked work items.
  • Follow up actions
  • Long running custom dashboards

This is by no means a comprehensive list. It is possible to bring your clients and servers to their knees with custom themes that do not scale, by work item attribute customization adding more and more custom attributes, JavaScript providers, value providers with thousands of values to choose from and other customization.

What is your server up to?

When users complain about performance problems, even if a server is getting overloaded, it is hard to find the root causes, because a typical server does so many things.

Monitoring that has been added over time has helped, but it is still hard. It is sometimes even hard to understand the situation. As an example for how complex this can become. Users complained about performance.

Our performance architect looked at the server load and the build load and a huge amount of calls that we were not able to account for. The server was unarguably under heavy load created by builds, but the build users and SCM users where not complaining. The developers we talked to had no real issues. Some users, at a different location, using work items and running work item queries, had.

Because we could not explain the inconsistent feedback, I finally went to the location where the users where complaining. I met the users followed their day to day work and found the work item performance unacceptable. The web browser was even locking up on them.

Knowing this, we were able to reproduce the use case, and look into what happened. We found that the work item load was slow, especially on slow laptops, because it had to load so many team areas and iterations. This was specific to how the project area was configured and used.

We also found that the browser flooded the server with requests that where definitely not part of what the product UI sent. This basically forced the Web Browser to process and cache thousands of calls, reserving more and more memory and exhausting the CPU capabilities of the relatively weak laptops used by the users that complained.

The final verdict was, that there was a custom extension to the theme that created all these calls. It took us weeks and was luck that we found this out. If we had known there was such an extension, we would have been able to find this a lot faster. The server was still under a heavy build load, but the performance issue reported was not related to that.

Needless to say that this extension was also deployed in other environments. If it had a detrimental impact, it was heavily depended on the timeline and iteration structure of a project area. The more and deeper the worse.

It would have helped if we could have seen the extension working, and see how long it worked would also have helped.

Registering Custom Resource Intensive Scenarios

The same mechanism that is used to register resource intensive scenarios in the product code can be used to register custom resource intensive scenarios. Unfortunately, we where lacking a good description and supporting code that we could provide customers to use it for their extensions.

This has now changed. Some colleagues and I, independently, started creating a customer usable description how to register resource intensive scenarios. A colleague wrote some cURL code to do this. I wrote Java code to do this and started creating a presentation. When we found out, we decided to combine the effort. Here the result.

The Deployment Wiki page Register Custom Scripts as a Resource Intensive Scenario, explains, using an example, how the API works in general. It also explains how to retrieve and monitor this information.

Then it provides example code to perform this using cURL, Eclipse Lyo OSLC4J based java code, and RTC Plain Java Client Libraries based Java Code.

The Java Code comes with main classes to run it. This is basically example code, but it can also be directly used in command line based automation.

Open Source Code

Disclaimer and Download

Any code downloadable or accessible in this post is provided as is, without support, and used at your own risk. Part of the code was developed in Java using Eclipse and is based on the Eclipse Lyo Client. This was published as open source, under
 Eclipse Public License – v 1.0, in the incredible (mostly German speaking) Jazz Community and can be found here: custom-expensive-scenario-notifier-oslc4j.

Another part of the code was developed in Java using Eclipse and is based on the Plain Java Client Libraries. This was published as open source, under MIT license, in the incredible (mostly German speaking) Jazz Community and can be found here: custom-expensive-scenario-notifier-plainjava.

See the other examples the Deployment Wiki page Register Custom Scripts As a Resource Intensive Scenario.

How does it work?

There is basically a REST API to register the start and the stop of a scenario. All there is to register the start of the scenario at the beginning and then register the stop, after you are done. See Register Custom Scripts as a Resource Intensive Scenario for more details on the code.

What should your automation do?

If you have written automation tools or extensions, you should use the methods described in Register Custom Scripts as a Resource Intensive Scenario, to register your extension as an resource intensive scenario. Add the code to register the start and stop in a way that allows for disabling it easily.

Monitor the various resource intensive scenarios over time. For a scenario that takes only a fraction of a second, you could temporarily disable the registration. Scenarios that take a second or longer should continue to be monitored.

Related

Feedback

If you have questions around the Custom Resource Intensive Scenario code, ask them in the Jazz.net forum instead of commenting on the article or this blog post. Tag the question as a clm question and add the tag: custom-resource-intensive-scenarios to mark it for the reader.

Summary

Please use the method above to enhance your automation and extensions to allow monitoring their duration, frequency and deviation.

As always I hope this helps users out there with the Jazz products.