EWM REST API to Access Existing Work Item Queries

Posted on March 1, 2022 by rsjazz

After looking into the EWM OSLC CM API, including the OSLC Query API, a natural question that comes up is: Is it possible to take advantage of the existing EWM work item queries? Is it possible to use that query mechanism and run existing work item queries to get the result set back?

I had to look into that for a customer and an internal project and where experiencing some difficulties. So I think it is valuable to share my experiences and examples that finally worked for me.

Context of the blog post is the series

This is the series of planned posts I intent to publish over time. Most of the examples will be EWM based, but quite a lot of the content applies to more ELM applications.

External Links

I used at least the following links for exploring this mechanism.

Resource Oriented Work Item API with OSLC_CM 1.0 especially the examples for how to use built in query. For completeness it might be worthwhile to check the experimental API introduced in RTC 1.0.1
Be aware of some characteristics and limitations discussed in work item 522500

EWM Stored Query API Introduction

EWM provides its own REST API that supports finding and running predefined, shared or personal work item queries that are stored in the repository. The API allows to query for all the stored work item queries available. The API allows to run such a query and to access the result information to present the work items and their data. This API is EWM only as far as I can tell. It is not documented to be available for any of the other products. It is also important to note, this API is not an OSLC API, even if the query URL contains oslc as part of the path.

The examples for how to use built in query are nice to have, but I found the documentation was lacking essential information. Maybe it is jut me and it comes naturally to all the native cloud wizz kids. I struggled to get the examples to work. One pattern I have now seen several times in documentation and examples is a lack of information about required and supported headers and how to URL encode query parameters. I will try to explain what worked for me. The documentation has recently been updated and now contains more details compared to when I first explored it.

For this specific part of the API, my observation was that getting the query parameters wrong, often resulted in the query returning results, but not the expected results. Usually the result set way way bigger than expected in those cases.

One important behavior that is very sensitive to the Accept header is how the information is returned and presented in the responses. There are two completely different pattern.

Only the URLs to the results are available in the response. To get any details about the result referenced by the URL, it is necessary to query that URL. This corresponds to the situation for OSLC queries without or an empty oslc.select statement.
Additional information is provided inline in the response. It is not necessary to query the result URI to be able to access the most relevant information. This corresponds to the situation for OSLC queries with an oslc.select=* statement.

When I looked at the API first, I missed information about the valid headers. As far as I can tell, the valid headers are now updated in the wiki page. The supported and working headers are mentioned below as well.

Headers and performance

Which header to use seems to be a quite unimportant difference, but it can have a huge impact on the communication performance. Lets create a Gedankenexperiment. Lets assume we perform a request to the server. We want to search in the response data to get the URI to a query with specific property values like a project area, query name and creator. Lets assume the following characteristics for the request:

Result set size is 130 items
Paging size is 50 items

To process the response with the properties inline it is necessary to run 3 requests at most. If the desired information is on page 1, only one request would be necessary. Worst case all related data needs to be sent and received.

To process the data without the properties inline it is necessary to load a page. Then it is necessary to run another request for each of the URLs in the result set page and process the received data. In worst case this requires 3 + 130 = 133 requests. Worst case all related data needs to be sent and received as well.

Why is that a problem? Worst case, if the desired data is in the last link processed, both methods have required to transfer the same amount of data. In many cases one could argue the 1st approach would even require more data to be transferred on average than in the 2nd approach.

However, in my experience the information transfer between two network nodes is significantly slower than the time that would be required in a server to get the desired information. Each message that needs to be sent causes a considerable amount of overhead and delay. It is usually cheaper to send fewer messages than many small ones that transfer a similar amount of data, just because of the overhead required for each message. In addition all the subsequent requests for the details cause activity on the server which is likely to cause a lot more overhead compared to collecting all the information for the 3 initial request(s).

As an real world example, when I started to look into this, I did not know which headers where available. I used the headers I had often used e.g. ‘application/rdf+xml’. In addition, the query that I created had an issue. Instead of only returning the queries for a specific user in a specific project area, it returned all queries for the whole server. The responses only contained the query URLs. To get the name and creator of the query each URL had to be requested. It was also not obvious why the result set was so large and I had no code to recognize that results were related to project areas other than the expected. The server I ran against is on another continent and I was in my home office. I stopped the experiment after an hour or two.

At the end I was able to create solution that performs well and reduces overhead to a limit. But it was no means a simple process. I reached out to someone for hints. As a customer I would not have had the opportunity. This is why I try to finally share this information.

Stored Query API URL

I was not able to find a way to discover the URI to get the work item queries. The Wiki page mentions to find the simpleQuery tag in the project areas service provider, but I seem to be unable to find it, so I need to construct it based on the wiki page. From the Resource Oriented API page I can deduce two possible forms for the URL. Assuming PublicURI being the public URI of the CCM server e.g. ‘https://elm.example.com:9443/ccm’ the URL for the Stored Query Collection is either

PublicURI + '/oslc/queries'

PublicURI + '/oslc/queries.xml'

Both seem to work. Choose one you like and that works for you.

Query Stored Queries

The request below gets all the stored queries for a repository.

Get all stored queries for the repository

Note that the only Accept headers documented to be valid at the moment are:

Accept text/xml
Accept text/json

It is necessary to be authenticated to the ccm server to be able to perform the operation.

The image below shows a part of the response to the query above.

The response to the query for all stored queries

The first section shows the query for the next page in the attribute oslc_cm:next and the total result count in oslc_cm:toatlCount.

The rest of the information is one element rtc_cm:Query for each query descriptor. The element contains the query URI as rdf:resource, the identifier for the query, the Query name/title and description and the project area the query belongs to. The most important information is the rtc_cm:results element with the URL to execute the query. As an example the URL below:

https://elm.example.com:9443/ccm/oslc/queries/_87dqoFpmEeukW7cqqDjAuA/rtc_cm:results

Additional information is the date of the last modification and the user that created the query.

Note that the result is either XML or JSON. The tools for RDF that were used in some of the previous posts using the Python rdflib do not work in this context. Instead the Python code shown in the blog uses xml.dom.minidom to access the information in case of XML. The JSON code examples shown in previous posts using the json library can be used as a reference. This post will mostly use the Accept header text/xml. The code depends on the following libraries.

The main Libraries used to process the stored queries API.

Some example code for processing the data in Python will be shown below.

Narrow Down Stored Queries

It is usually not desirable to query for all stored queries for all project areas. A real production server can have hundreds of project areas, team areas and users that can have predefined or shared or personal queries defined. The result could be thousands to ten thousands of queries. To collect and transfer all that data is a challenge and will drain server resources as well.

It is possible to filter for all the properties that make up a query, identifier, name, creator, owning project area. The code below computes the query parameters from the parameters passed and creates a query section for the query URL.

Create the query parameters to narrow down the query results.

As an example the query that is created could look like below. The condition can narrow down to the project area, the user and the title, dependent on how much information is provided. If enough information is given, this would usually only return one hit. Note that this is a string terminated by the singe quote and the double quote characters inside are important. The query narrows down to one project area, the query name ‘All’ and the creating user.

'rtc_cm:projectArea="_8e5qfFpmEeukW7cqqDjAuA" and dc:title="All" and dc:creator="https://elm.example.com:9443/jts/users/ralph"'

How to discover the project area UUID has been explained in previous posts e.g. about discovery. The code that was used here is the same used in the other posts.

The code blow shows how the query URL is composed.

Python code to compose the query parameters and to create the query URL

The parameters provided are the same as above, the project area, the query name and the query creator. The creator can be provided in different ways. It is possible to provide a user URI like

'https://elm.example.com:9443/jts/users/ralph'

It is also possible to provide the variable string below.

'{currentUser}'

The query mechanism replaces this variable with the URI of the user that runs the request. For example, the query below would return the query named All created by the current user.

'rtc_cm:projectArea="_8e5qfFpmEeukW7cqqDjAuA" and dc:title="All" and dc:creator="{currentUser}"'

The last three lines in the image above compose the query URL. First the base query URL is composed. This is the part of the query that returns every stored query. Then the query parameter section is composed by adding ‘?oslc_cm.query=’ and then adding the URL encoded query parameters. The resulting query URL looks like below.

'https://elm.example.com:9443/ccm/oslc/queries.xml?oslc_cm.query=rtc_cm%3AprojectArea%3D%22_8e5qfFpmEeukW7cqqDjAuA%22+and+dc%3Atitle%3D%22All%22+and+dc%3Acreator%3D%22%7BcurrentUser%7D%22'

Execute the Query for Stored Queries

The code below executes the query to search for stored queries.

The first line runs the query that was just created using the function findQueryResources().

Execute the query to find stored queries

This function is shown in the image below. It uses the Accept header ‘text/xml’. While there is a valid query URL for a page, it executes that query.

The code that executes the query for the stored queries and collects the results

It calls qeryQueryInfoXML() to process the page result data. This code is shown below. The function builds up two arrays, one containing the stored query URL and one the associated query name. It also gets the query URL for the next page, if there is any, and returns that.

Get the information about the queries from the response

All query pages are iterated and the results added to the respective arrays.

Evaluate the result and

At the end of executing findQueryResources, there is a list of query names and an associated list of query URLs. Dependent on the query parameters given, it could return many or no result. The code here as shown below focuses on using the query name, but it would be easy enough to extract more information per stored query to use different criteria to look into the result set. The code tries to find the first stored query returned with the given name.

If a stored query has been found, that query is finally executed in the function execute_query_xml(). The code for this function is shown below. Again, this uses the Accept header ‘test/xml’ and the results are processed as XML.

Process each result page of the stored query

The code iterates the query pages and is similar to the code already shown above. The code passes each query result page to the function processQueryResultInlineXML(). The code is shown below. It also looks very similar to the code we have already seen. The difference is that the code below now works on work item resource information and not on query information.

Processing one page of the stored query with work item data inline.

The code analyzes the oslc_cm:Collection and gets the total result count and the next page (if there is one). Then it analyzes the inline XML data for all the work items. For each work item it gets some of the available information and prints that. It returns the work item URIs, the totals and the next page etc.

Executing Stored Query Using JSON

The code here shows a JSON based implementation of the functions to execute the stored query after finding it. The function execute_query_json() does the same the function execute_query_xml() does and looks very similar. It is only using JSON as format to parse the data. The Accept header used below is ‘text/json’.

The function getQueryResultDetailsInlineJson() is called to process each page. Like its XML twin it gets the total count and the next page (if available). Then it gets details about the embedded inline work item data and logs it. Like the XML based code, the function builds an array with the work item URIs for the page. It also returns additional information such as the link to the next page.

Process one query page with JSON work item data inline

More information

Please check the section Getting Resource Representations to understand additional ways to get work item data representations. Also check the section Getting Partial Representations to understand options to limit the data that needs to be transferred.

CSV Export

The query descriptor URL provided in the search by the tag rtc_cm:Query, can also be used in a browser to get the query displayed. As an example see the information below.

<rtc_cm:Query rdf:resource="https://elm.example.com:9443/ccm/resource/itemOid/com.ibm.team.workitem.query.QueryDescriptor/_pdT5AATBEeyiDIlAjMF45Q">

Opening the URL in the rdf:resource results in the following display in the browser.

The query descriptor URL is opened in the browser and executes the query.

It is also possible to use that URL to download the query result as CSV file. By adding the query parameter media type e.g.

?_mediaType=text/csv

to the query descriptor URL, the query can be changed to export the data as CSV file. As an example the URL below exports the query result to CSV.

https://elm.example.com:9443/ccm/resource/itemOid/com.ibm.team.workitem.query.QueryDescriptor/_pdT5AATBEeyiDIlAjMF45Q?_mediaType=text/csv

By pasting the URL in the browser, the CSV download is triggered. It is also possible to use the URL above in tools like cURL.

Summary

This blog post has described the steps that are necessary to find existing work item queries and how to run such queries to get the data. I hope that the examples here in my blog help users out there with their work.

EWM Discovery

Posted on December 9, 2021 by rsjazz

Discovery is the name for the method to locate the entry points for the OSLC API in EWM and other ELM tools. The mechanism is the same for all applications but there are differences in the details. This post aims to help understanding the discovery process with a focus on EWM and work items. Ultimately we want to be able to create a new work item and need to get all that is required to do that.

Context of the blog post is the series

This is the series of planned posts I intent to publish over time. Most of the examples will be EWM based, but quite a lot of the content applies to more ELM applications. The examples where performed with versions 6.0.6.1 and 7.0.x.

External Links

Here is another interesting article about discovery.

Rootservices

The main entry point into EWM and other Jazz applications is the rootservices document. The document is an XML document that is based on RDF. The document can be accessed using a HTTP GET on the rootservices in the context root of the ccm and other applications. For example:

GET https://elm.example.com:9443/ccm/rootservices

The document is not password protected and does not require special headers to be accessed. It is only available as RDF XML document, there is no other way or format to get it. It is possible to use the URI for the rootservices document directly in the browser to display it.

The rootservices document contains information about all the resources and services provided by the Jazz Server.

Service Provider Catalog

To create a work item we need to find the Work Item Service Provider Catalog. We are interested in the rdf:resource entry of the element oslc_cm:cmServiceProviders.

There are certainly many ways to achieve this. The best way to achieve this is to use RDF. RDF support exists in various languages. I am not the best person for the job to explain RDF. My summary would be that RDF defines subject, predicate, object relationships in a Graph based on XML. Once the graph is created, it can be queried. For example, if one has the subject and the predicate it is possible to locate the objects.

The trick is to understand what the subject, predicate and object could be. I have always struggled to figure that out on my own, just looking at the RDF XML document. I have found two possible solutions, that work for me.

Serialize the graph as N-Triples and look into these
Even better, serialize the graph as Turtle format and look into that

Here is a part of the rootservices document, that was parsed as RDF XML serialized into Turtle.

Turtle serialized rootservices document showing the subject

In this document, line 19 shows the subject. Please also note line 17 that shows a prefix definition we are interested in. The next part of the document of interest is shown below.

Turtle serialized rootservices document with predicate and object we are after

Line 100 shows the object we are after, the URI

https://elm.example.com:9443/ccm/oslc/workitems/catalog

The predicate to locate the object is

oslc_cm1:cmServiceProviders

With Line 17 above the predicate has the following namespace prefix

@prefix oslc_cm1: <http://open-services.net/xmlns/cm/1.0/>

The lines 104 to the end of the file show additional subjects and their predicates and objects.

In Python, it is possible to use the library rdflib to work on RDF XML. A core communication support library that I have developed for this define the URIs and namespaces for the used domains. Also see the blog post Using the EWM REST and OSLC APIs for more information about this. The documentation for rdflib can be found here.

The image below shows an example for defining the namespace used above.

The image below shows the import statements used for rdflib in my communication library that has all the RDF support infrastructure.

The image shows the code that is used to serialize the rootservices document and the function to get the service provider catalog. The last argument is the predicate that locates the work item service provider catalog we are interested in. In this example the predicate is

comm.oslc_xml_cm1_ns.cmServiceProviders

Which translates to

http://open-services.net/xmlns/cm/1.0/cmServiceProviders

oslc_cm1:cmServiceProviders

used here to search for the service providers.

The function below composes the subject and then gets the objects selected by the subject and the predicate.

Get the objects based on the subject and predicate

The function creates the RDF graph. This binds all the namespace definitions and aliases.

Then the rootservice document is parsed based on this RDF definition to create the graphs content. The code creates the URI for the rootservices document that represents the subject – it is basically the URI of the rootservices document. Finally the code gets all objects selected by the subject and predicate, and creates an array of these URIs.

The code segment shows how the work item service provider URI is then taken out of the result array. Please note that all these activities did not require a login, username, password or anything in addition to the rootservices document.

The image below shows how the RDF graph is created and the desired aliases and namespaces are bound to the graph.

Create the graph by binding the namespace aliases and URIs

Work Item Service

We just discovered the work item service provider catalog shown in the GET command below. The next step in the discovery chain is to look up the work item services for the project areas. This is done using the service provider catalog that was discovered above.

GET https://elm.example.com:9443/ccm/oslc/workitems/catalog

and the listed OSLC request headers.

Accept application/rdf+xml; charset=utf-8
OSLC-Core-Version 2.0

Note that this request is for a protected resource. The server will redirect to the authentication. The details are explained in the previous post ELM Authentication.

The resulting RDF XML document contains the information we are interested in. The image below shows the Turtle serialization of the service provider catalog. It shows a structure similar to below.

Turtle format of the service provider catalog

The section with the predicate oslc:ServiceProviderCatalog shows the project areas services documents. For some purposes this might be enough. It is possible to iterate all the service provider catalogs to get more information for each one performing a GET on the URI. However, the response we already got contains more information that can be used.

By searching for the objects that have the predicate oslc:ServiceProvider instead, it is possible to access the URI for the service provider, as well as additional information such as the project areas name using the dcterms:title. This can help reducing the number of subsequent calls to get these details.

The code below shows how to get that information from the serialized document. It builds arrays of project area names and URIs which is later used. E.g. find the index for a project area name and then get the related URI. There are likely better ways to store the information in Python.

The first step is to look up the subject for the predicate oslc:ServiceProvider. The resulting URI is the service provider for the project area. For example

https://elm.example.com:9443/ccm/oslc/contexts/_8e5qfFpmEeukW7cqqDjAuA/workitems/services.xml

As explained above, the code uses the found subject to get the project area URI in the oslc_cm1:details and the project area name in the dcterms:title attribute.

Project Area OSLC Services

In EWM, work item services are project area specific. The reason is that each project area can have a different process and the process defines the work item types, attributes, workflows and all that. The next steps in the discovery process would be to get the project areas services.xml.

GET https://elm.example.com:9443/ccm/oslc/contexts/_8e5qfFpmEeukW7cqqDjAuA/workitems/services.xml

This requires the OSLC headers:

Accept application/rdf+xml; charset=utf-8
OSLC-Core-Version 2.0

The resulting RDF XML document contains information about various OSLC related capabilities and services. By looking through the information, especially using the Turtle format, it is easy to identify.

The OSLC query capability
Various dialogs such as OSLC selection, creation dialogs and pickers
The OSLC creation factories for all the work item types
The OSLC resource shapes for all the work item types

Similar to the work item services above, it is possible to get the desired information. Here example code getting the creation factories and related resource shapes for all the work item types.

The pattern repeats. To create a work item of a specific type, get the creation factory for that type. To get information about the attributes to create, in the work item, get the resource shape for the type. For the attributes in the resource shape get the type, allowed values and multiplicities.

Other Formats

The rootservices document is only available in RDF XML. Dependent on the tool, other formats might be supported for subsequent documents. For example using the header

Accept application/json; charset=utf-8

it is possible to try to get the JSON representation. This works for EWM in various places. No guarantee. The discovery mechanism in this case is similar to the XML example above, the only difference is that JSON is a format that is easier to read for humans in my opinion. Instead of creating a graph, it is necessary to use a JSON library to access the data.

The code below shows how to get the service providers when using JSON format instead of RDF XML.

The discovery is analogue to what has been shown above with RDF. Even the structure of the code is similar, which is due to the fact that the data structure is similar. It is just in a different format that is easier to digest than RDF, at least for me.

Summary

I have struggled for a while with how to explain the discovery process. Especially how to get the data out of the RDF XML has been hard. Since I found the Turtle format, it makes a lot more sense to me. So, as always, I hope that this was of interest and helps the users and the Jazz community to make their life easier.

ELM Authentication

Posted on October 15, 2021 by rsjazz

Authentication against a Jazz application is quite complex. When using the Plain Java Client Libraries, this is not a concern, as the ITeamRepository handles all that for you. It authenticates against the server and keeps the authentication valid over time. This is important, because application servers have timeouts that can void the the authentication of a session. A very common timeout that impacts the Jazz applications is the Liberty Profile default of 2 hours.

Running a client application against such a server requires detecting the need to authenticate and performing the authentication as well as to guarantee the current request is performed. Enterprise applications also require to work in complex setups supporting different ways to authenticate. Some examples are authentication based on Liberty Profile file based user management, LDAP based delegated authentication and user management, Jazz Authentication Server (JAS) based authentication, Kerberos and more. Implementing a client that uses the REST and OSLC APIs directly requires to develop a mechanism that handles this automatically.

Warning, this is complex!

I have tried to simplify this topic as good as possible, within my limited knowledge and experience, however this blog turned out to be very lengthy and complex. I also had to redo my screenshots multiple times to correct for changes during writing the post. Again, this ended up not being the casual blog post. You have been warned.

Approaching authentication

I have worked with the EWM Plain Java Client Libraries in the past. They provide a standard login mechanism to use and that keeps you logged in. I worked with Eclipse Lyo and used the mechanism provided with it, similar to plain Java API. I started just using an HTTP Client and I implemented a simple Login mechanism that was able to perform a Form based authentication. There was no code that would keep me logged in. When I started experimenting with Python, I just implemented a similar login mechanism that uses Form based authentication as explained in this post. In the first steps of the script, the authentication was executed, under the assumption that it was valid for the rest of the script.

That was sufficient for my experiments with my local test server. However it does not work with systems that have JAS enabled. This created questions, such as which authentication method was available and how to detect that, that required deeper knowledge about how the authentication was supposed to work. This approach also does not work for scripts that have to run longer durations. In many environments, the authentication expires after some timeout.

I started reading the Native Client Authentication article from the Team Wiki to understand how it was supposed to work. I got confused. I talked to some of my peers and hoped for someone having looked into this already. No luck. I cite the answer of a very skilled colleague that I hold in high regard: “Ah, this Native Client Authentication article is the most cryptic article I have ever come across.. at times it appears like everything has been told and at almost the same time, it appears like it is telling us nothing” and “I read the article now the 1001th time, and I still don’t understand”. Exactly.

I could not leave this as the answer and started coding while trying to follow the article. It makes no sense to me, to try to rewrite the article in this blog. Also just words will not bring any clarity. It looks as if language is not clear enough to describe what needs to be done.

The only way I am able to explain what goes on is showing code samples. I did not want these blogs to be about code, but it seems to be unavoidable to get clarity and to explain what goes on using the resulting code in my library class ELMCommLib. At this time I will use images to show the code. I will likely make the code available as open source later. Once done, I will link the download location to the blogs, as usual. The code is the result of a quite lengthy development and test process. It only implements parts of what the Team Wiki article explains. I have only so many test systems available and I can only test what these systems do. So here the disclaimer…

Disclaimer

Please note that so far I have only tested against a standard system supporting Form based authentication and a system supporting JAS. I used a trick to test the fallback to Basic authentication. My general disclaimer to the right of the page applies too. I’m not responsible for any damage done when following this article.

Context of the blog post is the series

Where to authenticate

To authenticate to ETM and EWM, use the context root of the server e.g. use https://elm.example.com:9443/qm or https://elm.example.com:9443/ccm as the destination for the authentication.
For DNG/ERM and JTS use the context root of the JTS e.g. https://elm.example.com:9443/jts to authenticate.

OAuth and Jazz Authentication Server

There is another type of authentication called OAuth. Please see ELM Applications Authentication and Authorization via OAuth 2.0 Workflow and related topics for this.

Also see OAuth2.0 Access flow on ELM Configured with a SAML IDP or OIDC Provider for other options.

See Jazz Authorization Server – Landing page for more details.

How the authentication code works

I started with creating a function internalJazzRequestWithAuthentication() that can be used for all HTTP requests by passing the method and all parameters needed. This function deals with the whole authentication requirements and makes sure the authentication dance is performed when needed. I provide more convenient functions that wrap this function and are easier to use.

This function has become long and complicated, so I have to show it in smaller pieces below.

The function delegates the request to the function internalJazzRequest() which is performing the correct call using the requests library. The function looks like this.

The function returns the result for the request. The result is now evaluated in the first part of the authentication code of internalJazzRequestWithAuthentication().

The function internalJazzRequestWithAuthentication() checks the status code of the response to be able to determine if anything related to authentication needs to be done.

The first HTTP status code it evaluates is 200. Usually 200 means the call went OK and the result is the desired result. However, the result headers can contain information that the access to the resource would require authentication and the call actually did not return what was expected.

The function internalHandleFormChallenge() is used to handle this case. It checks the result headers to figure out if Form based authentication is required. The code to detect the Form authentication request is easy enough to understand. The code checks for the presence of a special custom header X-com-ibm-team-repository-web-auth-msg. If the header is present and contains the value authrequired a Form based authentication is required.

If Form based authentication is needed, the code performs the login and retries the initial call. Note that currently the code uses the URI that is provided when creating the communication library for authentication. This needs to be changed at some point. The code has to get the public URI root from the result. This is important for applications like DNG that delegate authentication to JTS.

Function internalHandleFormChallenge() handles Form based login challenges.

The function also handles the case that the initial request was done for an insecure URI. This is handled as a fail and a false is returned.

If no authentication was required, the current result is returned back.

If form based authentication is required, the function internalLoginForm() is used to perform the form based authentication. It is essentially the legacy code remaining from an initial login mechanism that I used for automation examples. There is a façade function loginForm in the code that keeps this code compatible to old scripts. The form based authentication sends the credentials to the URI publicURI/j_security_check and checks if the authentication worked by looking at the response location header.

Function internalLoginForm() performs the form authentication.

Back in the function internalHandleFormChallenge() the code tests if the login was successful. If so, the initial request for the resource is re-executed in the function internalRetry() and the result is returned. The retry code will be explained later.

There are several other return codes, typically for different methods like POST, PUT etc. that can require form based authentication in internalJazzRequestWithAuthentication(). One example is return status code 302 which is handled exactly like the code 200.

The wiki page Native Client Authentication mentions the return status codes 303 and 307. None of the test systems I worked against ever responded in any of those codes. At the time of writing they are detected in a check and the call will fail for these codes. Once I start sending unauthenticated PUT, POST, DELETE or PATCH requests, I might see these response codes. The handling would be the same. One of the next steps for me will be to setup tests that force the other situations.

The last possible response is 401. This is the most complex case. It handles all other cases for authentication, especially Jazz Authorization Server (JAS), OICD and some other authentication options. See the initial part of the handling for 401 in internalJazzRequestWithAuthentication() below.

401 Challenge, detection of the authentication method.

As explained in the wiki page it is necessary to check for the header WWW-Authenticate. If this header is present, it needs to be evaluated. This could be a Basic authentication challenge or a Bearer challenge for Jazz Authentication Server/OIDC/OAuth. The situation where the Bearer or the Basic challenge is missing is handled at the very end.

If the Basic challenge and the Bearer challenge is detected, the headers X-JSA-AUTHORIZATION-REDIRECT and X-JSA-AUTHORIZATION-URL are evaluated to determine the authentication strategy. For the test system I have available, I needed to follow the JAS authentication.

The first step to do JAS authenticate with Basic and Bearer challenge is to get the redirection URL from the header X-JSA-AUTHORIZATION-REDIRECT. Based on this the code constructs the first redirection URL stored in xjsaauthredirect with the additional parameter '&prompt=none'.

The code performs a GET on the no-prompt redirect URL. The result of this call determines the next steps. When the GET does not return 200, the authentication is still valid and the code runs a retry for the original request, the result of which is returned. This is shown here.

Retry code if no authentication is required.

If the GET returns status code 200 authentication is required. The code below shows the code to handle the authentication. Note, that there is some code that gets information that remain unused at this time.

To reauthenticate the code performs a GET on the redirection URI in xjsaauthredirect, this time without the additional no-prompt parameter. The request can result in a status code 401. This is indicating Kerberos or some other authentication method. I have not implemented this code path at the moment. I lack a test system that behaves this way.

The request can result in a status code 200. In this case we are on the OAuth path. The code performs a request on the redirect URI in xjsaauthredirect, this time providing an additional authentication header conforming to the Basic authentication header. The header is computed in the call to HTTPBasicAuth(). If this succeeds with a status code 200, the client is authenticated and the code retries the initial request. This is not strictly necessary for all requests to do that, but for now, I wanted to do it anyway. The result is returned back to the outer code. Note that the Authentication header is only sent this one time. In contrast when using Basic authentication like below, the Authentication header is sent with all the subsequent calls.

In the case that the server only returned the Basic authentication header, the code below initializes the normal Basic authentication mechanism. The code is written in an way that now always sends the Basic authentication header with all requests in this case.

The case that Kerberos was detected is also handled here at the end. Lacking a test system the code returns unsupported and False for now. When the code was unable to identify the authentication method, e.g. the server was not a Jazz server, the code fails as well.

Handling of Kerberos and Basic authentication

This is what I was able to come up for handling authentication. There is no guarantee, that it is completely correct. I am lacking test systems that behave in different ways, so testing is very limited.

Please note, that the authentication is only valid for the session that is used and based on several cookies that are managed by the session. Also note, as already mentioned, the session can be invalidated by the server at any time and the authentication would have to (automatically) be done again.

Retry code

I ended up creating a function to handle the retry for the initial call that led to the authentication challenges. At the moment the code for GET is tested and works. The code currently basically reruns the initial request and checks if that was successful. Then it returns True and the result or False.

Then internalRetry() function performs a retry for the initial call.

While writing this code, it appears that the retry code might have to be changed later. When the first call requiring authentication is something other than GET the code might fail. As an example, I am not sure at the moment how a POST e.g. for creating a work item would be handled in detail. The key here is that POST of a work item may only be run once. This is going to be an interesting journey and there is much to be learned about redirects and retries. Once all cases are tested, I will try to re-upload the changes.

Example for authentication flows – Form authentication

As already mentioned, I am able to track the communication in my framework. The following sequence of messages is related to a GET request against a system that uses Form authentication. Please ignore the files with extension .nt, they are not relevant for the authentication.

The first call is to get the rootservices document. The rootservices document is not protected, so no authentication happens.

The next call is to get the service provider catalog which is discovered in the rootservices document. The service provider or workitems catalog is a protected resource, this means the request requires authentication. The request below shows that the call was redirected from the work items catalog to the web page showing the login prompt. The page code in my logs usually shows some mysterious message that “Javascript is either disabled or not available in your Browser”. This is based on the fact that there is no browser that sent the request and all JavaScript code in the HTML page fails.

The response header also contains the header X-com-ibm-team-repository-web-auth-msg authrequired indicating authentication is required.

Request returns with authentication challenge

The code detects that authentication is required. And executes the form based authentication, providing the username and password unencrypted. The latter is the reason why it is absolutely necessary to use HTTPS and not HTTP as protocol. HTTP sends the credential information unencrypted. The response of the login call contains the redirection location to the previous call.

At the moment, the code shown here does not use the redirect, it uses the original resource URI to retry the call. This is the retry the code performs. Please note that the destination URL and the Request URL are now the same. The response body contains the protected resource service provider catalog. This catalog contains the information about the visible available project areas.

All subsequent calls in this session are performed without any authentication challenge. The next call to get the work item service for the desired project area is omitted.

Example for authentication flows – JAS authentication

Finally lets have similar look at the sequence of events when running against a system with JAS Authentication. Note, I have replaced the original host name with a generic one, should there be any unclear references, in case I did not spot them. The following flow is the result of the same version of software running against a different URL that has JAS authentication configured.

The first call is to get the rootservices document again. The rootservices document is not protected, so no authentication happens.

The next call is to get the service provider catalog which is discovered in the rootservices document. The service provider/work item catalog is a protected resource, this means the request requires authentication. The request below shows that the call to the service provider catalog fails with a status 401. The response header also contains the header WWW-Authenticate with the form containing the Basic and Bearer challenge. The headers X-JSA-AUTHORIZATION-URL and X-JSA-AUTHORIZATION-REDIRECT are also present. This provides with the authentication and redirect URL for the authentication.

JAS authentication challenge with Basic and Bearer authentication header.

The next step is the GET request against the redirect URL with the additional parameter &prompt=none. This request collects additional information after some redirects. I don’t claim I understand the details here. Key is that the call can either determine that there is actually a valid authentication already, or a new authentication is required.

Here the previous call results in status code 200. This means that JAS authentication is required. So we GET the previous target URL, but without an extra parameter &prompt=none.

Note that the request gets redirected to the login URL that we have seen in the response to the initial call in the header. X-JSA-AUTHORIZATION-URL. The response header seems to indicate that Form based authentication is a valid fallback: X-com-ibm-team-integration-jazzop-auth-msg form-login. I am not sure and have not tried. I followed the wiki page Native Client Authentication implemented the JAS authentication. The response body here is the HTML code of the login page. If the client was a browser, it would be possible to show it to the user.

The previous request ended with a status 200 which means JAS/OAuth2 is the way to go. This is done in the next step. The request is executed again and an additional Basic authentication header is created and sent with the request. Note that the Authentication header is not shown below. If it was I would have had to redact it, because the data is only encoded, not encrypted. This is also the reason why Jazz Applications are by default all using HTTPs only. Do not send your credentials over HTTP.

Also note that the request gets successfully redirected back to the service catalog URI. The response body actually contains the service provider/work items catalog.

Because it is just a GET request, It would be possible to just return it as a result. At the moment I run a last request to retry the initial get. I will leave that request out in this example.

Summary

As always, I hope that this blog helps users out there that work with the Jazz products and their APIs. This was a very complex topic and I hope I was able to shed some light based on the experience I have gained over time. There are several cases missing, which I was unable to test lacking access to test systems and also test code that I have not yet developed. I will try to keep the post updated as always. Once I have finished this series, I will also try to publish the code for re-use.

If you have made your own experience with this topic and want to share, please do so in the comments.

Using the EWM REST and OSLC APIs

Posted on September 29, 2021 by rsjazz

I have collected a lot of experience with the Java APIs for RTC/EWM over the years. Until 2020 I did not use the RTC/EWM REST and OSLC APIs at all. Luckily I got involved in several engagements where I had the opportunity to explore these APIs and learn how to use them.

I found the documentation for these APIs I was able to find underwhelming. The available documentation was often lacking complete working examples. There was usually some critical part missing or there where no examples at all, just the API specification. The latter seemed to be systematic to focus on the specification and not the specifics. However the lack of samples was confusing and left too much room for interpretation. I ended using search engines a lot. I had to experiment a lot to get things moving.

Obviously I learned a lot. I would like to spare others the hassle. So, as usual I want to share the experiences and lessons learned with the community. My intention is to provide with some relevant and working examples that are easy enough to perform on your own. I hope this can save people some time when trying to use these APIS as well.

This post will be the first in a series of posts and provide links to the other posts for easy navigation. In addition this post will discuss which development environment and tools were utilized to explore and use the APIs. I will share some of the code I have developed over the time to ease the exploration.

The planned blog posts for the series

Important links about the EWM/RTC APIs and the ELM APIs in general

I wrote Learning To Fly: Getting Started with the RTC Java API’s a couple of years back and it is still relevant. Read it to understand What API’s are Available for RTC and What Can You Extend? and how to get started with the RTC/EWM Java APIs. The post and the linked posts contain even more valuable links with respect to APIs.

Since back then, the ELM API Landing Page has been added to provide a more comprehensive overview about the available ELM APIs. If you are interested in ELM related APIs go over that page and find out what APIs are available. This page also points to other resources such as the OSLC standards and available workshops.

Finally the Interesting links page is a collection of, well, interesting links I found over the years.

Development environment choices

The first information I can share is how I explored and used the APIs and explain a little bit about the development environment options available and which development environments I used to explore the APIs.

I like developing with Java a lot. The EWM/RTC Java APIs are very rich and it is relatively easy to develop code for EWM/RTC, provided the development environment was set up by performing at least the complete Lab 1 of the Rational Team Concert Extensions Workshop. Eclipse, the RTC SDK and the Plain Java Client Libraries allow development of extensions and automation based on the EWM Java SDK and API. The same environment can also be used to develop code against the REST and OSLC APIs.

It is also possible to use Java or any other language supporting HTTP, to develop code for the EWM/RTC REST and OSLC APIs, just using libraries and available frameworks.

I have already used the Java based Eclipse Lyo framework to develop a client automation for Doors Next Generation (DNG) and I used the Eclipse Lyo Designer code generation framework to develop integration servers. My experience was that Lyo is a nice framework that helps a lot, if you know what you are doing. If I was not, I found it challenging, especially debugging and understanding what was going on in the HTTP requests.

I have looked into and used Postman and the Firefox addon RESTClient to experiment with REST and OSLC APIs. It is very useful for experimentation and I use it in parallel to the other development environments. A typical use case is to login and experiment with one call to figure out how it works. If the call sequence and the amount of data becomes too big, it is not really efficient any more, and I would use a different approach.

I started using Python and Jupyter Notebook in 2020 when I had the need for some automation for importing, manipulating, consolidating and querying a lot of CSV data for a customer. I was very impressed with the quality of the available libraries and the turn around times that were achievable. When I was asked to help one of our customer teams with information about the RTC/EWM APIs for the development of a prototype for a customer specific mobile client, I decided to use Python instead of Java. As mentioned above I also used RESTClient or Postman for experimentation with one or two API requests.

There are various Python development environments around. I do not think it matters which one you use. I used Spyder which comes with Anaconda. There is also PyCharm and kite. I am not opinionated. I just notice that the development environments are far away from the quality of Eclipse and the built in compiler and debugger. There are always tradeoffs, I guess.

Python – Libraries and Code Samples

The focus of the blog posts is more the APIs, how they work and how they can be used and not so much Python and how to use it. However, I figured that I want to share some code I developed over time, enabling easier data collection and debugging. So I will provide examples where I see fit.

The most important aspects of HTTP based APIs is to understand which method is used with which URI, which headers are used, which formats are sent and accepted and which request body (if any) is sent. The response data is also key, especially the status, response headers such as Location and obviously the response body. A mechanism that can log all this information is key in understanding the APIs and faster turn around times.

Python has a lot of libraries for various purposes. The Libraries that I used are shown below, loosely grouped by what they are used for. First the libraries used operating system and system specific purposes such as logging, files and execution.

import os
import sys
from datetime import datetime, timezone
from pathlib import Path

Then the requests library which is used for session handling and HTTP communication in my code.

import requests
from requests.auth import HTTPBasicAuth
from requests.packages.urllib3.exceptions import InsecureRequestWarning

This code below is necessary to suppress issues with certificates. This is a typical situation for me as I usually develop against some local test system.

# Disable warnings for self signed or invalid SSL certificates 
# to be able to talk to test systems
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

# Start a session
session = requests.Session()

The Libraries I used for RDF XML and JSON parsing and representation.

import json

from rdflib import Graph, URIRef, Namespace
from rdflib.namespace import CSVW, DC, DCAT, DCTERMS, DOAP, FOAF, ODRL2, ORG, OWL, PROF, PROV, RDF, RDFS, SDO, SH, SKOS, SOSA, SSN, TIME, VOID, XMLNS, XSD

Some miscellaneous library for encoding.

from base64 import b64encode

Python Logging and Reuse

I ended up creating a base library for the Communication with the ELM system that allowed better reuse. I will not share all the code at the moment, but I will share some basic learning and code that I found being key for me to be able to do my work. The library is referred to as:

from elmcommlib import ELMCommLib as elmcomm

The library is initialized with a session, the public URI and a name for the log folder to be created.

publicURI = 'https://elm.example.com:9443/ccm'
paName='JKE Banking (Change Management)'
user='ralph'
password='ralph'

# Disable warnings for self signed or invalid SSL certificates 
# to be able to talk to test systems
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

# Start a session
session = requests.Session()
comm=elmcomm(session, publicURI,'logCreateWiRDF')

The library also provides a mechanism to create and set log file folders using createLogFolder("FolderName") . In case the folder already exists it can alternatively set with setLogFolder("FolderName").

The log folder is used by the method writeResult() shown below, which dumps the complete communication in a text file, when a file name is provided. The file name should be constructed and numbered to better understand the flow of the sequences. Below shows such a sequence with file name numbering as an example.

The communication logs are always created in the current log folder. This allows to split the logs for the API usage into smaller sequences by switching the current log folder.

A debug print dPrint() allows to avoid chatty logging. You can keep the logging entry and force it to show if you want. Printing a timestamp using timeStamp() is sometimes useful, especially when looking at performance of calls.

    # Folder for log files
    def createlogFolder(self,folderName):
        defaultLogFolder= 'commlogs'
        if(folderName==None):
            folderName=defaultLogFolder
        if(folderName==''):    
            folderName=defaultLogFolder
        script_dir = os.path.dirname(__file__) #<-- absolute dir the script is in
        logFolder=os.path.join(script_dir, folderName)
        Path(logFolder).mkdir(parents=True, exist_ok=True)
        return logFolder

    # Folder for log files
    def setlogFolder(self, folderName):
       self.logFolder=self.createlogFolder(folderName)


    # Log the HTTP communication for the request in a folder
    def writeResult(self, fileName, result, url=None):
        if(fileName!='fileName'):
            self.dPrint(f"Execute: '{fileName}'")
            logFileName=os.path.join(self.logFolder, fileName)
            with open(logFileName,'w') as f:
                if(url!=None):
                    f.write(f"Destination URL: {url}\n\n")
                reqMethod = result.request.method
                reqURL=result.request.url
                reqBody=result.request.body
                f.write(f"Request: {reqMethod} {reqURL}\n")
                f.write("\nRequest Headers:\n")    
                for header in result.request.headers:
                    value=result.request.headers[header]
                    f.write(f"\t{header} {value}\n")
                f.write(f"\nBody:\n{reqBody}\n") 
                f.write(f"\nResponse Status: {result.status_code}\n")
                f.write("\nResponse Headers:\n")
                for header in result.headers:
                    value=result.headers[header]
                    f.write(f"\t{header} {value}\n")
                cookies=result.cookies._cookies    
                f.write(f"\nResponse Cookies:\n {cookies}\n")
                f.write(f"\nResult Body:\n{result.text}\n")        
      
      
    # Debug print if debugging is on
    def dPrint(self, message=None, doPrint=True):
        # DebugPrint, switch off by sending doPrint=False
        if(message!=None):
            if(doPrint==True):
                print(message)
            else:
                pass
    

    # Print timestamp
    def timeStamp(self, message):
        now = datetime.now(timezone.utc)
        self.dPrint(f"{now}: {message}")

The image below shows how a log file for a message created using the method writeResult() looks like. Note that the log contains all the important pieces of the request, response pair. I used tooling in the editor Notepad++ to “pretty print” format the XML section in the response body. This makes it much easier to understand.

Help with RDF XML

The REST and OSLC APIs provide different serializations for the content that they accept and provide. One older one is XML based using the Resource Description Framework (RDF) specification. Newer standards such as JSON and more might be available. I have experience with RDF and JSON and I prefer JSON.

RDF is not for me. I always struggle to understand what and how I should be searching to get the data I want. Especially when the data is full blown with namespaces and what have you. This was one of the biggest struggles I had with Eclipse Lyo. The HTTP Client was hard to use for debugging, because the content was usually consumed when I tried to dump the response into a log file. So I could have a log entry for debugging and the call would not proceed or the call would work and I had no log data. Maybe I overlooked something.

In Python I was able to use the method writeResult() and continue processing the response data. I was able to use the function below to serialize RDF response bodies into a form that shows all the subject, predicate and object data and saves it into a file. That made it easier to work with RDF for me. I still prefer JSON format, if available. The OSLC discovery mechanism supported by RTC/EWM requires XML-RDF in the first steps, so you will have to deal with it.

    # Serializes a graph (based on RDF) in the nt format 
    # This format shows all graph nodes as Subject->Predicate->Object
    # This allows to better understand what to search for
    def debugSerialize(self, graph, fileName='fileName'):
        # Serializes a graph into the NT format. This provides 
        # a great source to look into RDF triples in the graph
        if(fileName!='fileName'):
            logFileName=os.path.join(self.logFolder, fileName)
            graph.serialize(logFileName, format="nt")

The serialization formats supported out of the box are “xml”, “n3”, “turtle”, “nt”, “pretty-xml”, “trix”, “trig” and “nquads”. For me NT and Turtle seem to be most useful, I built in capabilities to save the XML data as NT and Turtle format to help understanding how to be able to access the data later.

Update: The preferred option is to serialize the graph as Turtle format and look into that.

This is how the parsed RDF graph data from above looks in the NT format. Every row (mind the word wrap) is a triple of subject, predicate and object. This provides with hints how to search for data.

The RDF-XML in NT format, providing the triples in the model.

The capabilities above where absolute key for me to be able to explore and understand the EWM/RTC APIs and document them for my colleagues.

Operating on RDF requires the RDF definitions in Python. I used the ones below and defined them in my library.

    #RTC CM RDF definitions
    rtc_cm = 'rtc_cm'
    rtc_cm_URI = 'http://jazz.net/xmlns/prod/jazz/rtc/cm/1.0/'
    rtc_cm_ns = Namespace(rtc_cm_URI)
    
    oslc = 'oslc'
    oslc_URI = 'http://open-services.net/ns/core#'
    oslc_ns = Namespace(oslc_URI)
    
    oslc_cm = 'oslc_cm'
    oslc_cm_URI = 'http://open-services.net/ns/cm#'
    oslc_cm_ns = Namespace(oslc_cm_URI)
    
    oslc_xml_cm1 = 'oslc_cm1'
    oslc_xml_cm1_URI = 'http://open-services.net/xmlns/cm/1.0/'
    oslc_xml_cm1_ns = Namespace(oslc_xml_cm1_URI)
    
    jfs_process = 'jfs_proc'
    jfs_process_URI ='http://jazz.net/xmlns/prod/jazz/process/1.0/'
    jfs_process_ns = Namespace(jfs_process_URI)

    oslc_rm = 'oslc_rm'
    oslc_rm_URI = 'http://open-services.net/xmlns/rm/1.0/'
    oslc_rm_ns = Namespace(oslc_rm_URI)
    
    oslc_config = 'oslc_config'
    oslc_config_URI = 'http://open-services.net/ns/config#'
    oslc_config_ns = Namespace(oslc_config_URI)

Summary

This is the first of a series of posts, I hope to publish more soon. I will try to keep this post maintained and I am looking forward to the next posts. As always, I hope that my content, especially in this blog, helps someone in the ELM community out there. If it does, feedback would be awesome.

EWM/RTC Extensions Workshop works with EWM 7.0.x

Posted on March 3, 2021 by rsjazz

As published in Issues with the EWM/RTC Extensions workshop in 7.0.x versions, there was an issue that caused the RTC Extensions Workshop to fail starting a Eclipse debug client in Lab 1.9. The problem and how it was caused has been found. I have just tested the RTC Extensions Workshop Lab 1 with the new EWM Client SDK (7.0.1 and 7.0.2) and it works now as it should. I will update my previous post.

Update for the Extensions Workshop now available

All the issues have been addressed in an update to the RTC Extensions Workshop for the 7.0.x and later versions. This should address all the issues.

While running the test, I found that I had to increase the maximum heap size for the launch [RTCExt] Jetty RTC Server, because the JUnit Test that is run to create the test database, ran out of heap memory. I doubled the value from -Xmx256M to -Xmx512M and the error was gone.

The fix is only in the EWM/RTC Client SDK, so it is only necessary to download that again.

Please note that due to the name changes and newer versions of Eclipse, there might be some small differences between the screen shots in the RTC Extensions Workshop and the newer versions. However, they are so minor, it should not be an issue. I am currently looking into updating the current Extensions Workshop material with small changes to help guide the user with these changes.

The issue with server debugging explained in Issues with the EWM/RTC Extensions workshop in 7.0.x versions, is still apparent in 7.0.2. The suggestion to change the JRE of the server in that blog post holds.

As always, I hope that my posts here help the Jazz user community and save them some time.

Using a REST client to authenticate to ELM/CLM Applications

Posted on May 13, 2019 by rsjazz

Recently the question how to authenticate to a ELM/CLM (aka Jazz) application came up in several occasions. I did not know the whole story either, so I had a look. Here is what I found.

Update history

2019/05/14 Added Basic Authentication

2021/01/22 Added Links

Important Links

The ELM API Landing Page note the entries for the OSLC Workshop and OSLC in general.
Especially the explanation about client authentication to ELM
The Interesting Links for examples
The Learning To Fly: Getting Started with the RTC Java API’s page with an API overview for RTC.
OSLC Frameworks and examples can be found in the Eclipse Lyo project.

Scenarios

There are several scenarios that could be of interest. Some of the scenarios will be provided below. A general documentation about how the authentication of a client to a Jazz application works is provided here in the development wiki.

Common scenarios are as follows:

Using a REST client to explore the REST/OSLC APIs
Creating custom automation using REST/OSLC APIs

Scenario 1 is also used in the OSLC Workshop. However, as explained below, things have changed since the workshop was initially written and the latest changes are not addressed in the workshop.

REST clients

The main changes that could be recognized are in the browsers and the integration of REST clients. These changes heavily impact how easy or hard it is to login. The first REST client used in the OSLC workshop was the Firefox browser addon REST Client . Using this addon It is easy to login to the Jazz application. All that needs to be done is

Open a browser window
Provide the URL to the CLM application and open the URL
The application prompts for a login
Provide user name and password and perform the login
Open the REST Client Addon

The REST client addon uses the login performed in the steps above for subsequent calls to the server. This makes it easy, because it is transparent and there is no reason to worry about the details. The browser basically handled them.

Similar addons are available for the Chrome Browser. Examples are

The Advanced REST Client as shown in this post.
Postman as a Chrome App as explained at the end of the post above, which is no longer available.
Tabbed Postman as Chrome App.

The browser integrations above all allowed to use the browser to login and use the login information in subsequent calls. These browser addons or extensions used to be very popular, but security threats and changes in integration philosophies seem to make them more and more unavailable. Some of the addons mentioned above are discontinued and might not be available much longer.

The Postman as packaged app as explained at the end of this post is currently still available and allows to configure a Postman Interceptor, that intercepts the login information from the browser and uses it in later calls. So it works like the solutions 1-3 above and the REST clients available as Firefox addons or extensions.

Alternatives are standalone solutions like Postman as standalone App which seems to be the path forward for Postman. There are many other stand alone REST clients. The caveat is, that the simple option to just login to the Jazz application is no longer available as an option. So it is important to know the details,

When trying to write custom tooling that needs to access data on a CLM/ELM server, it is also necessary to understand the details.

Testing and special considerations

It is a good idea to have a test system. Ideally local on your own machine. You can follow Lab 1 of the Rational Team Concert Extensions Workshop to set up a small system with JTS and RTC. It is not necessary to install the plain Java client libraries, any of the SDKs or to run the WorkshopSetup application. It is possible to install the other applications as well sing the Web Installer or one of the many Installation Manager repositories.

Create the sample project in the lifecycle project administration section to have data to play with.

Postman Settings

This blog shows the examples using Postman as standalone App which is very useful and allows to save and organize calls as a benefit. Like any application there are usually settings that can be beneficial or get into the way. Here the settings of interest in the context of this blog.

Disable SSL Certificate Verification

Please be aware that the SSL certificates of a test system as described above will typically be considered invalid. It is very common that test systems have invalid SSL certificates. A common issue is that the certificate is self signed and often for a different or generic host name. In such cases SSL certificate validation throws an error. Browsers, REST clients and APIs often reject working with servers presenting invalid SSL certificates. It is usually necessary to create exceptions or to configure the systems to ignore the SSL validation error.

Make sure the REST client or API works with sites that have invalid SSL certificates, or provide valid SSL certificates.

Postman rejects communicating with servers using invalid SSL certificates and displays an error message. This can be changed by changing the settings, turning the setting ‘SSL certificate verification’ to off. Postman now allows communicating to servers using invalid SSL certificates.

Disable SSL certificate verification to be able to work with example projects

As an alternative, it is possible to provide valid certificates. It is also possible to use any other test system that might be available, especially when dedicated test systems with valid certificates are available. The screen shots in this blog refer to a test system that has been set up similar to the Extensions workshop, but with all applications installed and the sample lifecycle project being deployed.

Automatically Follow Redirects

By default Postman automatically follows redirects. This can be nice but it hides a lot of details. To better understand the details explained in the wiki page Authentication of a native client with a Jazz-based application, switch the setting automatically follow redirects to off.

With the redirects disabled it is possible to see the 302 response as explained in Authentication of a native client with a Jazz-based application.

Minimal Header settings

When following redirects automatically, postman also collects several cookies. To focus on the really important details, and understand which headers and cookies are really needed, the setting below is very helpful.

Minimal setting - disable automatic redirect and additional headers — Minimal setting – disable automatic redirect and additional headers

Form based authentication

As explained in the wiki page Authentication of a native client with a Jazz-based application, there are multiple scenarios, how a client login would be performed. Unfortunately the number of scenarios are getting more over time as new standards are becoming available.

The easiest to use method that has worked for me so far has been the Form Based authentication. This works in a simple set up of CLM with any application server in a standard configuration without Jazz Authorization Server (OIDC) installed.

Perform a Form based authentication login to an application

To perform the Form based authentication to the application use the context root of the JTS or the application and append /j_security_check to the context root to create the login URI.

With a given context root of e.g. https://clm.example.com:9443/jts for JTS, the URI to login is https://clm.example.com:9443/jts/j_security_check.

As explained in Authentication of a native client with a Jazz-based application, the authentication is performed with a POST to the URI https://clm.example.com:9443/jts/j_security_check with a request body that contains the x-www-form-urlencoded encoded username and password.

The image below shows the simplest way of performing the authentication using Postman.

Minimum information required to perform FORM authentication — Minimum information required to perform FORM based authentication

The POST to the secure resource URI provides the username and password using the form encoded key and value pairs for the j_username and j_password. password and username are both identical myadmin.

The code for this request is shown below.

POST /jts/j_security_check HTTP/1.1
 Host: clm.example.com:9443
 cache-control: no-cache
 j_username=myadmin&j_password=myadmin

Please note that there are no additional headers necessary. The request does not specify anything in the request for Parameters, Authorization, Headers, Pre-request Script or Tests.

The response shows status 302. There is no header
X-com-ibm-team-repository-web-auth-msg. There is no header location with value
/auth/authfailed. This indicates the authentication was successful. — Post response shows status 302 and headers show no authentication failure.

The response shows status 302. There is no header X-com-ibm-team-repository-web-auth-msg. There is no header Location with any value, let alone
/auth/authfailed. This indicates the authentication was successful.

The Cookies keeps the LtpaToken to authenticate.

The LTPA Token is kept in a cookie. — The Cookies keeps the LtpaToken.

If the setting for Automatically follows redirects is on, and other header options are also set to on, the request would be redirected and likely show a status 200. There would be a lot more headers and Cookies collected as well.

Once successfully authenticated, it is possible to send other requests passing the Cookies. For this OSLC request it is also necessary to pass an accept header and the OCLS-Core-version header.

Successful retrieval of the OSLC workitems catalog URI.

Please note that the Jazz Product APIs might need other special handling. In the above example the communication returns an additional Cookie JSESSIONID. In subsequent calls provide a header JSESSIONID with the value of the returned Cookie.

The easiest way to ‘log out’ is to delete the Cookies for the server.

delete all Cookies to restart. — Delete the Cookies for the domain to start again.

The request above can also be created by using a special content header Content-Type:application/x-www-form-urlencoded and a raw post body with a special encoding.

j_username=myadmin&j_password=myadmin

The images below show how this looks like in Postman.

The request above can also be created by using a special content header Content-Type:application/x-www-form-urlencoded and a raw post body with a special encoding. — Content-Type header

This code shows the whole POST information.

POST /jts/j_security_check HTTP/1.1
 Host: clm.example.com:9443
 Accept: /
 X-Requested-With: XMLHttpRequest
 Content-Type: application/x-www-form-urlencoded
 cache-control: no-cache
 j_username=myadmin&j_password=myadmin

Authentication Failure

The Following image shows an unsuccessful authentication due to a typo in the password. This is only one of the possible flows, but the other scenarios are handled the same way.

The response shows status 302. There is a header Location with value /auth/authfailed. This indicates the authentication was not successful. — The Location header with value **/auth/authfailed** indicates a failed authentication attempt.

The response shows status 302. There is however a header Location with value /auth/authfailed. This indicates the authentication was not successful.

Authentication failed with Automatically follows redirects enabled leads to a JavaScript error page. — Authentication failed with redirect.

If the setting for Automatically follows redirects is on, the request would be redirected and likely show a status 200. The response body contains a page that the client could show to allow the user to enter the user ID and the password to authenticate. This web page uses JavaScript. Because the REST client Postman does not support JavaScript execution the page shows an error.

Where to Authenticate

My understanding so far has been that the CCM and the QM server have their own authentication. The RM server delegates authentication to the JTS. The Lyo implementation uses the public URI of the applications to create the authentication request. For the RM application it uses JTS instead.

The document Authentication of a native client with a Jazz-based application, seems to imply that authenticating to the JTS should do the trick for all applications. Using this has, so far, not caused any problems, so my suggestion would be to use the JTS URI to authenticate.

If you experience any issues, fall back trying to use the application specific URI.

Basic Authentication

Although Form based authentication has been the preference, Jazz also supports Basic authentication. However, when using Jazz Authorization Server/OIDC, Basic authentication is not supported. See this section (since 6.0.6).

See this information to understand the details.

As a summary, for Basic authentication, a special header Autorization that contains username and password encrypted in Base64 is sent with each request. The request below shows the header

Authorization: Basic c29tZXVzZXI6c29tZXBhc3N3b3Jk

Testing and special considerations

When using Basic Authentication in Postman it is useful to have the Automatically follow redirects setting to be on.

To configure Basic authentication in Postman, select the Authorization type Basic Auth and enter the username and the password. Postman generates the authentication header. When using the API or other tools that do not create the authentication header, it has to be created and added.

The call below gets the workitems catalog for a RTC server on Jazz.net. This requires sending a valid Accept header and the OCLS-Core-version header.

The code below is the important information.

GET /jazz/oslc/workitems/catalog HTTP/1.1
 Host: jazzdev.rtp.raleigh.ibm.com:9443
 Accept: application/xml
 OSLC-Core-version: 2.0
 Authorization: Basic c29tZXVzZXI6c29tZXBhc3N3b3Jk
 cache-control: no-cache

When sending the request, the final response should have a status 200.

Please be aware that basic auth does not have a session handling and you have to always send the authentication header.

Malicious website

I am currently working with a customer trying to authenticate to one of their CLM systems. This seems to work except with the CCM server. When they try to authenticate they get the following message.

You have followed a direct link to log in to a Jazz server. This page has been presented to ensure that a malicious website cannot use cleverly crafted content to circumvent security. Please log in if you would like to access the server.

We are looking into what could cause this issue. The web does not provide any good explanation. I found some questions in the Forum and other places and think I noticed a pattern. Several requests I have seen used a wrong URI to authenticate.

When using URIs like https://clm.example.com:9443/ccm/authenticated/j_security_check or
https://clm.example.com:9443/ccm/j_security_check and GET them in a browser, I get the same error.

You have followed a direct link to log in to a Jazz server. This page has been presented to ensure that a malicious website cannot use cleverly crafted content to circumvent security. Please log in if you would like to access the server. — Directly accessing internal URIs is prevented.

It is currently unclear what is happening and how to resolve the issue the customer is seeing. I will update the post, if we can figure out what is going wrong.

Summary

This blog post tries to shed some light on how to log into a CLM System to be able to use the REST and OSLC APIs. This should work for manual work as well as for automation tools. This blog uses Postman, but any REST client should work. The details, especially covering redirects and different authentication methods can be found inn this document Authentication of a native client with a Jazz-based application.

As always, I hope that this post helps users out there with using their Jazz tools.

Type System Manager Part 1

Posted on February 1, 2019 by rsjazz

It has taken some time since I published Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 1: Manual procedures which is about this Jazz.net article.

I was very excited about it, because I was contributing to the effort myself. We finally have made progress and just released the next part. You can now read the article about Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 2: Automation.

So, what is that all about?

Type System Manager

I did some prototyping, and we experimented with different possible approaches for this automation. What we came up with is explained in the article above.

In summary, I created a prototype, lets name it Type System Manager (TSM), that can be used to automate the type system management, supporting the best practices identified in Part 1.

The prototype uses available public Doors Next Generation OSLC/REST API’s to perform the necessary tasks.

So I have been lucky to be able to do some prototyping and learn a lot about OSLC and REST API’s. The result is a prototype, that has a useful application, and is also a demonstrator for how to create automation using OSLC and REST API’s.

Usage of OSLC and REST API’s has been missing on this blog and this prepares the foundation for hopefully more examples, here in this blog, in the future.

Disclaimer and Download

I should not have to write this, but as this is the internet 8), so here goes: Any code downloadable or accessible in this post is provided as is, without support, and used at your own risk.

The code was developed in Java using Eclipse and is based on the Eclipse Lyo Client.

Thanks to IBM approving, the code was published as open source, under
Eclipse Public License – v 1.0, in the incredible (mostly German speaking) Jazz Community and can be found here.

What is next?

The second part explains how the TSM prototype can be used and what it does and how. The next part is currently under review and will explain the details of the code, how it works, how to re-use and add to it. It also explains how to download and work with the code.

Feedback?

As mentioned in the article Comments, feedback, ideas, and experiences are greatly appreciated.

If you have questions, ask them in the Jazz.net forum instead of commenting on the article or this blog post. Tag the question as a Rational DOORS Next Generation question and add the tag: dng-type-system-management to mark it for the reader.

Summary

As always I hope that the artifacts created for this blog will useful for the Jazz user community out there. Feedback, also usage, is greatly appreciated.

A RTC WorkItem Command Line Version 3.0

Posted on November 3, 2015 by rsjazz

106

I was interested in how complex it might be to export work items and import them again. So I looked into this and enhanced the work item command line (WCL) to support this.

I found it quite challenging to develop this. There are a lot of things to consider, so I drove this to a point that was sufficient for my purposes. The tool is used by my team to import work items from a CSV file we receive every now and then.

I implemented another export/import mode with some more capabilities and it works in tests, but is by no means production ready. The amount of necessary tests and test automation to make this reliable, is just overwhelming. So this is not thoroughly tested.

So be careful if using these commands, and do a good amount of testing before actually using this. The problem is, there are so many possible use cases and dependencies that it is very hard to develop this kind of capability and to test it.

One special case is importing/creating links. Some links have constraints i.e. parent and especially child links. A work item can not have multiple parents. So setting child links can cause the save to fail if the new child has already a different parent.

Solution Overview

The work item command line WCL now supports two new commands

exportworkitems
importworkitems

To export work items to a CSV file and import work items from a CSV file.

I chose to use a CSV file, because RTC itself can export and import that format already. It would be ideal if export and import from XML would be supported as well, but this would require a substantial effort to abstract the export and import operations to be able to use a strategy (or some other useful pattern to support abstraction).

No Support or Maintenance

This is provided as-is with no support or guarantee. This is not a tool that is officially supported by IBM or any other organization.

Please note, that I have very little time to do this and testing is always lacking. So take the code published here with a grain of salt. On the positive side, you have the code, can debug and enhance it.

Compatibility

This code has been used with RTC 4.x and 5.x with no changes and it is pretty safe to assume, that the code will work with newer versions of RTC. The code requires two external libraries that need to be downloaded and installed separately.

License and Download

The post contains published code, so our lawyers reminded me to state that the code in this post is derived from examples from Jazz.net as well as the RTC SDK. The usage of code from that example source code is governed by this license. Therefore this code is governed by this license. I found a section relevant to source code at the and of the license. Please also remember, as stated in the disclaimer, that this code comes with the usual lack of promise or guarantee. Enjoy!

Update: Added switch to change the export and import formats for dates, see details below. I also added a switch to suppress attribute not found errors and other frequent errors during export.

Update: Fixed duration set problem. Version updated to 3.2

You can download the latest version from this post The RTC Work Item Command Line On Bluemix. The older version 3.0 can be found here:

Please note, there might be restrictions to access Dropbox and therefore the code in your company or download location.

Just Starting With Extending RTC?

If you just get started with extending Rational Team Concert, or create API based automation, start with the post Learning To Fly: Getting Started with the RTC Java API’s and follow the linked resources.

You should be able to use the following code in this environment and get your own automation or extension working. The code linked from this post contains Client API.

Setup and Usage

For the general setup follow the description in A RTC WorkItem Command Line Version 2 and look at the additional setup steps below.

For usage follow the description in A RTC WorkItem Command Line Version 2 and in A RTC WorkItem Command Line Version 2.1. Check the README.txt which is included in the downloads.

Export Work Items

The export work items command has the syntax

-exportworkitems {Switch} repository=”value” user=”value” password=”value” projectArea=”value” query=”value” exportFile=”value” [columns=value] [encoding=value] [delimiter=value] [querysource=value]

Required Parameters are

repository=”value” – the repository URI, for example repository=”https://clm.example.com:9443/ccm“
user=”value” – The user ID of the user executing the command, for example user=”ralph”
password=”value” – the password of the user, for example password=”password”
projectArea=”value” – The project area to export items from, for example projectArea=”JKE Banking (Change Management)”
query=”value” – the name of the query to use, for example query=”All WorkItems”
exportFile=”value” – The path of the export file, for example exportFile=”C:\aaTemp\Export\Test.csv”; the folder that contains the export file must exist

Optional Parameters are

columns=value – The names or ID’s of the work item attributes to export; example columns=”Type,Id,Planned For,Filed Against,Description,Found In”; To specify the colums it is possible to use the name or the ID of the attribute, the switch headerIDs specifies the output format to use the ID instead of the name in the output; It is possible to use the values from an RTC Eclipse client export
encoding=value – The encoding; default encoding=”UTF_16LE”; options see available charset names; if the encoding is chosen different for export and import, the values will not be recognizable
delimiter=value – The delimiter to be used between the columns. Default is comma delimiter=”,”
querysource=value – If the parameter is omitted the command searches a personal query with the given name; if the value is provided a query shared by the process area is searched, a complete path from the project area to the sharing process area must be provided, for example querysource=”JKE Banking(Change Management),JKE Banking(Change Management)/Business Recovery Matters”
timestampFormat=value – To specify the time stamp format to be used; default “MMM d, yyyy hh:mm a”; see SimpleDateFormat for the supported pattern

Available switches are:

/ignoreErrors – Ignore minor errors in mapping and value lookup
/asrtceclipse – Export in a format compatible to the RTC CSV export and import; if the switch is not provided, the data is exported in a format that is compatible with the syntax used by the work item command line WCL to identify elements; see A RTC WorkItem Command Line Version 2 for the supported value representations
/headerIDs – Export header values as attribute IDs and not as attribute names
/suppressAttributeExceptions – Suppresses exceptions thrown for attributes that are not available on the work item type of for attribute types that are not yet implemented

Example

-exportworkitems /ignoreErrors repository="https://clm.example.com:9443/ccm" user=ralph password=ralph projectArea="JKE Banking (Change Management)" exportFile="C:\aaTemp\Export\Test.csv" query="All" columns="workItemType,summary,Attachments"

Import Work Items

The import work items command has the syntax

-importworkitems{Switch} repository=”value” user=”value” password=”value” projectArea=”value” query=”value” importFile=”value” [columns=value] [encoding=value] [delimiter=value] [querysource=value]

Required Parameters are

repository=”value” – the repository URI, for example repository=”https://clm.example.com:9443/ccm“
user=”value” – The user ID of the user executing the command, for example user=”ralph”
password=”value” – the password of the user, for example password=”password”
projectArea=”value” – The project area to export items from, for example projectArea=”JKE Banking (Change Management)”
importFile=”value” – The path of the import file, for example importFile=”C:\aaTemp\Export\Test.csv”

Optional Parameters are

mappingFile=”value” – A RTC work item import mapping file, for example mappingFile=”C:\temp\mapping.xml”; the file must be generated by RTC and customized to match the value mapping
encoding=value – The encoding; default encoding=”UTF_16LE”; options see available charset names; if the encoding is chosen different for export and import, the values will not be recognizable
delimiter=value – The delimiter to be used between the columns. Default is comma delimiter=”,”
timestampFormat=value – To specify the time stamp format to be used; default “MMM d, yyyy hh:mm a”; see SimpleDateFormat for the supported pattern

Available switches are:

/ignoreErrors – Ignore minor errors in mapping and value lookup
/importmultipass – Import the work items from the CSV file in a first iteration and build up a mapping for the ID’s provided in the import file and the actual ID’s created and recreate the work item links between the new work items based on that mapping in a second pass; the old work item ID for a work item has to be provided in a special column with header name com.ibm.js.oldid
/forcelinkcreation – if no target work item can be found in the map, use the given ID to create the link
/importdebug – Print more information during import attempts to help with finding issues
/enforceSizeLimits – Attributes such as description and medium strings have size limits, if this switch is set, the importer tries to clip content to avoid exceptions due to the size limits

Example

-importworkitems  /enforceSizeLimits  /importmultipass  /ignoreErrors repository="https://clm.example.com:9443/ccm" user=ralph password=ralph projectArea="ImportTest1" importFile=""c:\aaTemp\ExportImport\TestExportAll.csv"

RTC Eclipse Compatible Export Mode

In mode asrtceclipse, all data is exported the way RTC would export them in Eclipse. This means that certain information for example links, team areas, iterations, attachments and other data is exported in a way that makes it hard to map to data in the repository.

The import command does its best to map based on names, but for complex hierarchical information such as iterations and team areas, there is currently no search implemented that will find the object successfully. It would be possible to implement such methods, with some effort.

Example: An iteration is part of a timeline. The timeline is needed to find the iteration within. If there is no information about the timeline, it would be required to iterate all timelines with a good chance of mismatch.

The import command will try to find things by name and ID, with the limitations above.

If the work item ID attribute is provided as a column the importer will try to find the work item and update it during the import.

Default Export Mode

In the default export mode, the RTC Work Item Command line export command exports the data in greater detail, which makes it easier for the importer to identify the item.

Attachments

In default mode, the attachment is exported as a file relative to the location of the generated csv file. The attachment is downloaded to a location ./attachments//. So for each exported work item with attachments, a separate folder is created. The attachments are stored in that folder and the export information in the csv file is created compatible to the WCL parameter format to allow later import of the attachments, including upload and applying the additional information.

Other Complex Items and Links

Other complex items such as iterations and team areas are also exported with a lot more details. An iteration is being exported as path, including the timeline and parent iterations. A team area is also exported as path containing the Project are name and parent team area names

Multi-Pass Import

Importing work items and recreating the link relationships between them is problematic, because while importing the work items the link target may not yet exist. To be able to import a set of work items and then recreate the linkage, it is necessary to do the import and then map the ID of the old work item to the ID of the new work item.

When using the RTC CSV importer in the Eclipse client, existing work items are provided with a # in front of the work item ID. To do an import and then recreate the links between the new work items (and not to the old ones in the import), a user would have to run the import without the links, then replace the work item ID’s in the import file by the new work item ID’s and update the work items with a second import. This is very manual and error prone.

The switch importmultipass enables an import mode, where the WCL tries to create the links between the imported work items, rather to the old ones. It imports the work items in two passes. It creates the work items in the first pass and ignores the link creation. In the second pass it tries to create the links. For links between work items WCL tries to find the work items that were created during the import and tries to match the links to the new work items, where possible.

Note: Only links between work items are handled this way. Links to objects other than work items are recreated using the values provided in the import file.

To be able to do this, the import file has to provide the old work item ID of the work items that are imported. The import requires a special ID for the columns containing the old ID’s. The column header for this column has to be specified with com.ibm.js.oldid.

The import file below has been created using an export that included the ID of the work item in the export. The old column header for example ID of the column has been replaced by com.ibm.js.oldid. The work item links show the ID’s of the linked work items with their old ID’s.

The import works as follows.

The WCL runs the first pass and imports the work items. It stores the mapping between the original work item ID from the column com.ibm.js.oldid and the ID of the newly created work item in a map. Links are not created in this pass.

In the second pass WCL reads the import file again and only handles the columns that represent links. It detects if the link target represents a work item. If not, it tries to recreate the link as it is. If the link is a work item link, WCL tries to calculate if a new work item was created for the target using the map. If the work item was imported and a new ID is available, the new work item ID is used to create the link.

If the ID of the link target can not be found in the mapping, WCL can either ignore the link or it can try to create the link to the original work item. WCL supports these two modes. By default, the link is not created. If the switch forcelinkcreation is specified, the original value of the target work item ID is used as target for the link, if no mapping to a newly imported item was found.

Creating links is not trivial. One special case is importing/creating links. Some links have constraints i.e. parent and especially child links. A work item can not have multiple parents. So setting child links can cause the save to fail if the new child has already a different parent. This can create issues in import scenarios, especially if an export from the same repository is imported and the import causes child links to be created that have already another parent. In this case the import will fail with an error.

Limitations

Approvals and comments are imported into one comment. The effort to recreate approvals is just too big and I can’t see the added value.

Special Notes On Setup

For the general setup follow the description in A RTC WorkItem Command Line Version 2 and look at the additional setup steps below.

The export and the import commands of WCL need two libraries that are not shipped the downloads.

If you use the packaged WCL and want to use the export/import capability follow the steps below to add the required libraries to the folder lib in the folder lib in the WCL folder.

If you use the Eclipse project for WCL and want to use the export/import capability follow the steps below to add the required libraries to the folder lib in the Eclipse project com.ibm.js.team.workitem.commandline.

The export and the import commands of WCL use the Open CSV Library. I had issues with the newer versions of Open CSV that I could not resolve, so this code assumes the version 2.3. Download the version 2.3 from here. Uncompress and untar the the file opencsv-2.3-src-with-libs.tar.gz you downloaded. Look for the folder opencsv-2.3\deploy\ copy the JAR file opencsv-2.3.jar and put it into the lib folder of your version of WCL.

The import commands of WCL can only provide the capability to use a mapping file by using a JAR file that only ships with the RTC Eclipse client and the SDK. The classes used for the mapping file capability are located in the library com.ibm.team.workitem.rcp.core. Open the Install location of the RTC Eclipse client and search for com.ibm.team.workitem.rcp.core*. You should find a file names similar to this one: com.ibm.team.workitem.rcp.core_3.1.900.v20141010_0124.jar. The version numbers at the end could be different. Copy the JAR file into the into the lib folder of your version of WCL.

The packaged version should look like below.

If you have imported the Eclipse project for WCL open Configure Build Path and create a user library named openCSV and add the Open CSV library opencsv-2.3.jar. Create a user library named rtcmapping and add the com.ibm.team.workitem.rcp.core library you just copied to it.

Your Eclipse Project should now look like below.

Code Changes

During the work on import and export, the code structure was left untouched. Some classes were added to be able to handle the column header and some additional mapping of id’s and names i.e. for link types. In addition some of the code that was piling up in the WorkItemUpdateHelper (formerly known as WorkItemHelper) was moved to utility classes. this makes it also easier to look for useful API in case you are interested in how things work in the RTC API. See the scree shot below.

Summary

This WorkItem Command Line should allow for most of the automation needs when creating work items. In addition it is a nice resource for the RTC work Item API.

As always, I hope the post is an inspiration and helps someone out there to save some time. If you are just starting to explore extending RTC, please have a look at the hints in the other posts in this blog on how to get started.

A RTC WorkItem Command Line Version 2.2

Posted on February 27, 2015 by rsjazz

Creating links is not easy. Many things can go wrong. Testing by a user showed that there was an issue with links between work items and build results. I found that I got the link direction wrong. I fixed that. Here is the updated source code.

Latest Version

See A RTC WorkItem Command Line Version 3.0 for the latest version.

Related posts

License

Please also remember, as stated in the disclaimer, that this code comes with the usual lack of promise or guarantee.

On the other hand, you have the code and are able to add your own code to it. It would be nice to know what you did and how, if you do so.

Just Starting With Extending RTC?

You should be able to use the code attached to this post in the development environment you set up in the Rational Team Concert Extensions Workshop and get your own extensions or automation working there as well.

Download

You can download the latest version here:

Setup and Usage

Follow the description in A RTC WorkItem Command Line Version 2 and in A RTC WorkItem Command Line Version 2.1. Check the README.txt which is included in the downloads.

The WorkItem Command Line Explained

Posted on February 13, 2015 by rsjazz

This post explains how the WorkItem Command Line works. It explains its structure and the main classes. This should allow users to extend the capabilities the code and add new commands or extend the current commands.

Please note, as this is work in progress, things might change slightly in future versions, however the general structure should persist.

Latest Version

See A RTC WorkItem Command Line Version 3.0 for the latest version.

Related posts

Installing and using the WorkItemCommandLine Version 2.0
The WorkItem Command Line Explained – this post
Extending the WorkItem Command Line With New Commands

License

Please also remember, as stated in the disclaimer, that this code comes with the usual lack of promise or guarantee.

On the other hand, you have the code and are able to add your own code to it. It would be nice to know what you did and how, if you do so.

Just Starting With Extending RTC?

Importing The Project

Download the code from the post A RTC WorkItem Command Line Version 2. The file with the source code is named WorkItemCommandLine_Project-Vx-YYYYMMDD.zip. The x represents the version number and is followed by the date it was created. The file is an exported Eclipse project.

The project expects the Eclipse workspace to be set up as described in the posts Setting up Rational Team Concert for API Development. It requires the SDK to be set up as well as the Plain Java Client Libraries. The SDK is needed, because the project is a Plugin Project. this is done to be able to use the Eclipse Plugin Development Environment (PDE) to look at the API source code. The Plain Java Client Libraries are needed to run the code a Java application.

Use the Eclipse import File>Import. In the wizard window select “Existing Projects into Workspace” in the section General. Click Next and chose the option “Select archive file”, browse to the file you downloaded and select it. Make sure you see the project com.ibm.js.team.workitem.commandline selected and press Finish to start the import.

After the import you should see the project in your workspace. You should see no errors in the project. If you see errors, the most likely reasons for that are:

The SDK is not set up correctly and the classes can not be resolved
The SDK version is prior to RTC 4.0.1
The Plain Java Client Libraries are not installed or the User Library has a different name

The first two will show in the plugin.xml and the manifest file. Setup the SDK correctly, or change the minimal versions needed in the dependencies.

The third will show as an error in the the build path. Define a user library named PlainJavaApi as explained in Setting up Rational Team Concert for API Development or remove the existing user library entry and add you own. Make sure the dependency order of SDK and user library are correct as explained in Understanding and Using the RTC Java Client API.

In case you have other errors you should search the internet for a solution.

Explore the Project

You can now explore the project. The folder structure is shown below.

There are the following files and folders

src – contains the source code files.
build – contains a jardesc file to build a jar file for packaging
Launches – contains launch files used for testing
License – contains the license files
scripts – contains the script files used to start WCL, as well as a file with help information
the root contains a readme file, explaining how to build a releasable version of WCL, scripts used to start WCL in the development setup and a test file for upload attachment tests.

The Source Files

The image below shows the structure of the source code.

The package com.ibm.js.team.workitem.commandline contains the class WirkitemCommandLine, which has the main method to call WCL. The class OperationResult is used to pass result information. This is necessary, since the code could run in RMI mode and the output needs to be transferred to the RMI client. This class needs to support serializing in order to pass the result back. IWorkItemCommandLineConstants contains various constants used by WCL.

The package com.ibm.js.team.workitem.commandline.commands contains the classes that implement the currently available commands. CreateWorkItemCommand creates a work item of a specific type in a specific project area and sets the attributes as provided. PrintTypeAttributesCommand prints the attributes of a specific work item type in a specific project area. UpdateWorkItemCommand finds a work item and updates its attribute values.

The package com.ibm.js.team.workitem.commandline.framework contains a basic framework that is used by commands that are implemented in WCL. The main class requires the interface IWorkItemCommand to run the command. I ended up using this kind of framework, because all commands required some kind of parameters. The command should be able to define the parameters needed. The commands also require to do error handling. To interact with the RTC repository commands also need to login. The framework handles all the common activities and allows to create new commands without having to redevelop all this.

The class AbstractCommand implements the interface IWorkItemCommand and leaves some methods abstract that extending classes need to implement.

The class AbstractTeamRepositoryCommand adds a login to the team repository and the class AbstractWorkItemModificationCommand adds a WorkItemOperation to perform the changes to the work item. WorkItemCommandLineException is the exception class that is used to wrap other exceptions and thrown in case of unrecoverable errors.

The package com.ibm.js.team.workitem.commandline.helper contains helper classes. The class DevelopmentLineHelper is from another blog post Handling Iterations – Automation for the “Planned For” Attribute. It allows to find development lines and iterations on a development line. WorkItemHelper implements modifying work item attribute modification. Most of the RTC API related code is in there. WorkItemTypeHelper helps with printing the attribute information for a work item type.

The package com.ibm.js.team.workitem.commandline.parameter contains classes that implement all the parameter handling needed. The class Parameter is used to describe a parameter, if it is required, if it was already consumed, if it is a switch and the like. ParameterIDMapper defines a list of aliases that can be used instead of an attribute ID. You can add your own aliases that can be used for convenience. ParameterList represents a list of parameters. The class ParameterManager manages a parameter list and provides the central access to the parameters. The class ParameterParser is used to parse the parameters passed from outside and store them in a parameter list.

The package com.ibm.js.team.workitem.commandline.remote contains the remote interface IRemoteWorkItemOperationCall that is used in RMI mode.

The package com.ibm.js.team.workitem.commandline.utils contains some utility classes (providing static methods as interfaces). The class ProcessAreaUtil allows to search process areas. The class SimpleDateFormatUtil helps with conversion of timstamps from and to a string representation.

How The Code Works

The main method of the WorkitemCommandLine basically instantiates the class and then calls the method run(). We will look at that method later.

/**
 * The main entry point into the work item commandline
 * 
 * @param args
 *            - the arguments to be used by the commandline
 * @throws RemoteException
 */
public static void main(String[] args) {

	OperationResult result = new OperationResult();
	System.out.println("WorkItemCommandLine Version "
			+ IWorkItemCommandLineConstants.VERSIONINFO + "\n");
	WorkitemCommandLine commandline;
	try {
		commandline = new WorkitemCommandLine();
		result = commandline.run(args);
	} catch (RemoteException e) {
		result.appendResultString("RemoteException: " + e.getMessage());
		result.appendResultString(e.getStackTrace().toString());
	}
	System.out.println(result.getResultString());
	if (TeamPlatform.isStarted()) {
		TeamPlatform.shutdown();
	}
	if (!isServer()) {
		// If I am not in server mode, I need to exit and return success or
		// failure
		if (result.isSuccess()) {
			// If the operation was unsuccessful, terminate with an error
			System.exit(0);
		}
		System.exit(1);
	}
}

The operation run() will return a result if it terminates. The information in this result is used to create the exit code to terminate the call.

In case this WCL is started as RMI server, the process can not terminate with System.exit(). It needs to persist registered to the RMI registry. The static method isServer() is used to communicate this information.

The method run() parses the parameters passed. It then checks if it is supposed to run as RMI server or as RMI client. If that is the case it starts the RMI server mode or, uses RMI to call the server as client. If this is a normal run, it calls runCommands() with the parameters that have been parsed.

If started as RMI server, the method startRMIServer() is used to initialize RMI and to register the class to the registry. The method runOperation() is basically the interface that is used to run the command on the server and is called by RMI clients. The method runOperation() parses the parameters and calls runCommands() as well.

The method runCommands() really executes the command requested in the parameters. The first steps it does is to initialize the data it needs. Then it runs addSupportedCommands() to add the commands that are available.

/**
 * Add the supported commands. If introducing a new command, add it here.
 * 
 * @param parameterManager
 */
private void addSupportedCommands(ParameterManager parameterManager) {
	addSupportedCommand(new PrintTypeAttributesCommand(
			new ParameterManager(parameterManager.getArguments())));
	addSupportedCommand(new CreateWorkItemCommand(new ParameterManager(
			parameterManager.getArguments())));
	addSupportedCommand(new UpdateWorkItemCommand(new ParameterManager(
			parameterManager.getArguments())));
}

Now the method runCommands() gets the command from the Parameter Manager. If there is a command string, it gets the class that implements the command. If there is a command registered for this command string, runCommand() calls the command to validate if the required parameters for it to run are available. If this is the case, runCommand() calls the command and returns the result back.

In all other cases runCommand() prepares a result error and also uses the method helpGeneralUsage() to print a help for the command.

Adding Commands to the WorkItemCommandLine

It is easy to add new commands to the WorkitemCommandLine. You need to implement a new command and add a new entry for it in the method addSupportedCommands().

How Commands Work

Commands have to implement the IWorkItemCommand interface. You should pick one of the abstract classes in the framework and extend them. This makes sure the basic workflow will work. If you command needs to create or modify a work item based on property values, use the class AbstractWorkItemModificationCommand. If you only need to have a repository connection, use the class AbstractTeamRepositoryCommand. In both cases all you need to do really is to override and implement the methods required. There are three things that need to be there.

In the method getCommandName() you need to return the name of the command you implement.

@Override
public String getCommandName() {
	return IWorkItemCommandLineConstants.COMMAND_CREATE;
}

If your command needs additional parameters, override the method setRequiredParameters(). Call the method of the superclass to have it add its required parameters and add your parameters. Here is an example

/*
 * (non-Javadoc)
 * 
 * @see com.ibm.js.team.workitem.commandline.framework.
 * AbstractWorkItemCommandLineCommand#setRequiredParameters()
 */
public void setRequiredParameters() {
	super.setRequiredParameters();
	// Add the parameters required to perform the operation
	// getParameterManager().syntaxCommand()
	getParameterManager()
			.syntaxAddRequiredParameter(
					IWorkItemCommandLineConstants.PARAMETER_PROJECT_AREA_NAME_PROPERTY,
					IWorkItemCommandLineConstants.PARAMETER_PROJECT_AREA_NAME_PROPERTY_EXAMPLE);
	getParameterManager()
			.syntaxAddRequiredParameter(
					IWorkItemCommandLineConstants.PARAMETER_WORKITEM_TYPE_PROPERTY,
					IWorkItemCommandLineConstants.PARAMETER_WORKITEM_TYPE_PROPERTY_EXAMPLE);
	getParameterManager().syntaxAddSwitch(
			IWorkItemCommandLineConstants.SWITCH_IGNOREERRORS);
	getParameterManager().syntaxAddSwitch(
			IWorkItemCommandLineConstants.SWITCH_ENABLE_DELETE_ATTACHMENTS);
	getParameterManager().syntaxAddSwitch(
			IWorkItemCommandLineConstants.SWITCH_ENABLE_DELETE_APPROVALS);
}

Parameters added with syntaxAddRequiredParameter() will be assumed to be required. If they are not available the command line will show an error during the parameter validation. The error message is automatically created from the parameter information provided here.

Finally you have to override and implement the method process() to implement the command.

/*
 * (non-Javadoc)
 * 
 * @see com.ibm.js.team.workitem.commandline.framework.
 * AbstractWorkItemCommandLineCommand#process()
 */
@Override
public OperationResult process() throws TeamRepositoryException {
	// Get the parameters such as project area name and Attribute Type and
	// run the operation
	String projectAreaName = getParameterManager()
			.consumeParameter(
					IWorkItemCommandLineConstants.PARAMETER_PROJECT_AREA_NAME_PROPERTY)
			.trim();
	// Find the project area
	IProjectArea projectArea = ProcessAreaUtil.findProjectArea(
			projectAreaName, getProcessClientService(), getMonitor());
	if (projectArea == null) {
		throw new WorkItemCommandLineException("Project Area not found: "
				+ projectAreaName);
	}

	String workItemTypeID = getParameterManager().consumeParameter(
			IWorkItemCommandLineConstants.PARAMETER_WORKITEM_TYPE_PROPERTY)
			.trim();
	// Find the work item type
	IWorkItemType workItemType = WorkItemHelper.findWorkItemType(
			workItemTypeID, projectArea.getProjectArea(),
			getWorkItemCommon(), getMonitor());
	// Create the work item
	createWorkItem(workItemType);
	return this.getResult();
}

To complete the code of this command, here is the method that creates the work item and uses the parameters to update the attributes.

/**
 * Create the work item and set the required attribute values.
 * 
 * @param workItemType
 * @return
 * @throws TeamRepositoryException
 */
private boolean createWorkItem(IWorkItemType workItemType)
		throws TeamRepositoryException {

	ModifyWorkItem operation = new ModifyWorkItem("Creating Work Item");
	this.setIgnoreErrors(getParameterManager().hasSwitch(
			IWorkItemCommandLineConstants.SWITCH_IGNOREERRORS));
	IWorkItemHandle handle;
	try {
		handle = operation.run(workItemType, getMonitor());
	} catch (TeamOperationCanceledException e) {
		throw new WorkItemCommandLineException("Work item not created. "
				+ e.getMessage(), e);
	}
	if (handle == null) {
		throw new WorkItemCommandLineException(
				"Work item not created, cause unknown.");
	} else {
		IWorkItem workItem = getAuditableCommon().resolveAuditable(handle,
				IWorkItem.SMALL_PROFILE, getMonitor());
		this.appendResultString("Created work item " + workItem.getId()
				+ ".");
		this.setSuccess();
	}
	return true;
}

In case you wonder where the actual work gets done – I wondered looking at it. The line

handle = operation.run(workItemType, getMonitor());

does all the work. By calling it this way, the WorkItemOperation creates the work item. The operation is based upon the code in the class AbstractWorkItemModificationCommand.

In that class, the execute() method is overwritten with this code:

/*
 * This is run by the framework
 * 
 * (non-Javadoc)
 * 
 * @see
 * com.ibm.team.workitem.client.WorkItemOperation#execute(com.ibm.team
 * .workitem.client.WorkItemWorkingCopy,
 * org.eclipse.core.runtime.IProgressMonitor)
 */
@Override
protected void execute(WorkItemWorkingCopy workingCopy,
		IProgressMonitor monitor) throws TeamRepositoryException,
		RuntimeException {
	// run the special method in the execute.
	// This is called by the framework.
	update(workingCopy);
}

The call to the method update() does the real work. It walks through all the unconsumed parameters in the parameter list – which should contain the attributes and values to be set and applies the changes to the work item.

/**
 * This operation does the main task of updating the work item
 * 
 * @param workingCopy
 *            the workingcopy of the workitem to be updated.
 * 
 * @throws RuntimeException
 * @throws TeamRepositoryException
 */
public void update(WorkItemWorkingCopy workingCopy)
		throws RuntimeException, TeamRepositoryException {

	ParameterList arguments = getParameterManager().getArguments();

	// We use a WorkItemHelper to do the real work
	WorkItemHelper workItemHelper = new WorkItemHelper(workingCopy,
			arguments, getMonitor());

	// Run through all properties not yet consumed and try to set the values
	// as provide
	for (Parameter parameter : arguments) {
		if (!(parameter.isConsumed() || parameter.isSwitch() || parameter
				.isCommand())) {
			// Get the property ID
			String propertyName = parameter.getName();
			// Get the property value
			String propertyValue = parameter.getValue();
			try {
				workItemHelper.updateProperty(propertyName, propertyValue);
			} catch (WorkItemCommandLineException e) {
				if (this.isIgnoreErrors()) {
					this.appendResultString("Exception! " + e.getMessage());
					this.appendResultString("Ignored....... ");
				} else {
					throw e;
				}
			} catch (RuntimeException e) {
				this.appendResultString("Runtime Exception: Property "
						+ propertyName + " Value " + propertyValue + " \n"
						+ e.getMessage());
				throw e;
			} catch (IOException e) {
				this.appendResultString("IO Exception: Property "
						+ propertyName + " Value " + propertyValue + " \n"
						+ e.getMessage());
				throw new RuntimeException(e.getMessage(), e);
			}
		}
	}
}

The Class WorkItemHelper

This class is basically doing all the work related to modifying work item data. The helper needs to be instantiated. Then the method updateProperty() can be called.

public void updateProperty(String propertyID, String value)
		throws TeamRepositoryException, WorkItemCommandLineException,
		IOException {
.
.
.
}

The method checks if the attribute is one of the special ones like the type, or complex attributes such as workflow or state changes, approvals or other pseudo attribute ID’s and handles these if detected. Otherwise it calls the method updateGeneralAttribute() to handle the update.

private void updateGeneralAttribute(ParameterValue parameter,
		List exceptions) throws TeamRepositoryException,
		WorkItemCommandLineException {
.
.
.
}

The method updateGeneralAttribute() checks if this attribute is actually available on the work item. If so it calls getRepresentation() to get a value that can be set for the attribute.

private Object getRepresentation(ParameterValue parameter,
		List exceptions) throws TeamRepositoryException,
		WorkItemCommandLineException {
.
.
.
}

The method getRepresentation() basically is a huge list of checks to narrow down what type the attribute to modify is. If the type is narrowed down, it calls a related methods to parse the input data and to create a value for the attribute, that can be returned and set.

Summary

This post explains how the code works and how you can utilize it to implement your own commands. As always, I hope that helps someone out there.

While creating this post, I realized, that I should have named some of the classes differently. This framework is not only good for a work item command line. This code could be used for any command line. Maybe I will adjust this a bit in later versions, should time permit.