Using a REST client to authenticate to ELM/CLM Applications


Recently the question how to authenticate to a ELM/CLM (aka Jazz) application came up in several occasions. I did not know the whole story either, so I had a look. Here is what I found.

Update history

2019/05/14 Added Basic Authentication

Scenarios

There are several scenarios that could be of interest. Some of the scenarios will be provided below. A general documentation about how the authentication of a client to a Jazz application works is provided here in the development wiki.

Common scenarios are as follows:

  1. Using a REST client to explore the REST/OSLC APIs
  2. Creating custom automation using REST/OSLC APIs

Scenario 1 is also used in the OSLC Workshop. However, as explained below, things have changed since the workshop was initially written and the latest changes are not addressed in the workshop.

REST clients

The main changes that could be recognized are in the browsers and the integration of REST clients. These changes heavily impact how easy or hard it is to login. The first REST client used in the OSLC workshop was the Firefox browser addon REST Client . Using this addon It is easy to login to the Jazz application. All that needs to be done is

  1. Open a browser window
  2. Provide the URL to the CLM application and open the URL
  3. The application prompts for a login
  4. Provide user name and password and perform the login
  5. Open the REST Client Addon

The REST client addon uses the login performed in the steps above for subsequent calls to the server. This makes it easy, because it is transparent and there is no reason to worry about the details. The browser basically handled them.

Similar addons are available for the Chrome Browser. Examples are

  1. The Advanced REST Client as shown in this post.
  2. Postman as a Chrome App as explained at the end of the post above, which is no longer available.
  3. Tabbed Postman as Chrome App.

The browser integrations above all allowed to use the browser to login and use the login information in subsequent calls. These browser addons or extensions used to be very popular, but security threats and changes in integration philosophies seem to make them more and more unavailable. Some of the addons mentioned above are discontinued and might not be available much longer.

The Postman as packaged app as explained at the end of this post is currently still available and allows to configure a Postman Interceptor, that intercepts the login information from the browser and uses it in later calls. So it works like the solutions 1-3 above and the REST clients available as Firefox addons or extensions.

Alternatives are standalone solutions like Postman as standalone App which seems to be the path forward for Postman. There are many other stand alone REST clients. The caveat is, that the simple option to just login to the Jazz application is no longer available as an option. So it is important to know the details,

When trying to write custom tooling that needs to access data on a CLM/ELM server, it is also necessary to understand the details.

Testing and special considerations

It is a good idea to have a test system. Ideally local on your own machine. You can follow Lab 1 of the Rational Team Concert Extensions Workshop to set up a small system with JTS and RTC. It is not necessary to install the plain Java client libraries, any of the SDKs or to run the WorkshopSetup application. It is possible to install the other applications as well sing the Web Installer or one of the many Installation Manager repositories.

Create the sample project in the lifecycle project administration section to have data to play with.

Postman Settings

This blog shows the examples using Postman as standalone App which is very useful and allows to save and organize calls as a benefit. Like any application there are usually settings that can be beneficial or get into the way. Here the settings of interest in the context of this blog.

Disable SSL Certificate Verification

Please be aware that the SSL certificates of a test system as described above will typically be considered invalid. It is very common that test systems have invalid SSL certificates. A common issue is that the certificate is self signed and often for a different or generic host name. In such cases SSL certificate validation throws an error. Browsers, REST clients and APIs often reject working with servers presenting invalid SSL certificates. It is usually necessary to create exceptions or to configure the systems to ignore the SSL validation error.

Make sure the REST client or API works with sites that have invalid SSL certificates, or provide valid SSL certificates.

Postman rejects communicating with servers using invalid SSL certificates and displays an error message. This can be changed by changing the settings, turning the setting ‘SSL certificate verification’ to off. Postman now allows communicating to servers using invalid SSL certificates.

Disable SSL certificate verification
Disable SSL certificate verification to be able to work with example projects

As an alternative, it is possible to provide valid certificates. It is also possible to use any other test system that might be available, especially when dedicated test systems with valid certificates are available. The screen shots in this blog refer to a test system that has been set up similar to the Extensions workshop, but with all applications installed and the sample lifecycle project being deployed.

Automatically Follow Redirects

By default Postman automatically follows redirects. This can be nice but it hides a lot of details. To better understand the details explained in the wiki page Authentication of a native client with a Jazz-based application, switch the setting automatically follow redirects to off.

Setting to switch automatically follow redirects off and on.
Setting to switch automatically follow redirects off and on.

With the redirects disabled it is possible to see the 302 response as explained in Authentication of a native client with a Jazz-based application.

Minimal Header settings

When following redirects automatically, postman also collects several cookies. To focus on the really important details, and understand which headers and cookies are really needed, the setting below is very helpful.

Minimal setting - disable automatic redirect and additional headers
Minimal setting – disable automatic redirect and additional headers

Form based authentication

As explained in the wiki page Authentication of a native client with a Jazz-based application, there are multiple scenarios, how a client login would be performed. Unfortunately the number of scenarios are getting more over time as new standards are becoming available.

The easiest to use method that has worked for me so far has been the Form Based authentication. This works in a simple set up of CLM with any application server in a standard configuration without Jazz Authorization Server (OIDC) installed.

Perform a Form based authentication login to an application

To perform the Form based authentication to the application use the context root of the JTS or the application and append /j_security_check to the context root to create the login URI.

With a given context root of e.g. https://clm.example.com:9443/jts for JTS, the URI to login is https://clm.example.com:9443/jts/j_security_check.

As explained in Authentication of a native client with a Jazz-based application, the authentication is performed with a POST to the URI https://clm.example.com:9443/jts/j_security_check with a request body that contains the x-www-form-urlencoded encoded username and password.

The image below shows the simplest way of performing the authentication using Postman.

Minimum information required to perform FORM authentication
Minimum information required to perform FORM based authentication

The POST to the secure resource URI provides the username and password using the form encoded key and value pairs for the j_username and j_password. password and username are both identical myadmin.

The code for this request is shown below.

POST /jts/j_security_check HTTP/1.1
 Host: clm.example.com:9443
 cache-control: no-cache
 j_username=myadmin&j_password=myadmin

Please note that there are no additional headers necessary. The request does not specify anything in the request for Parameters, Authorization, Headers, Pre-request Script or Tests.

The response shows status 302. There is no header 
 X-com-ibm-team-repository-web-auth-msg. There is no header location with value  
 /auth/authfailed. This indicates the authentication was successful.
Post response shows status 302 and headers show no authentication failure.

The response shows status 302. There is no header X-com-ibm-team-repository-web-auth-msg. There is no header Location with any value, let alone
/auth/authfailed. This indicates the authentication was successful.

The Cookies keeps the LtpaToken to authenticate.

The LTPA Token is kept in a cookie.
The Cookies keeps the LtpaToken.

If the setting for Automatically follows redirects is on, and other header options are also set to on, the request would be redirected and likely show a status 200. There would be a lot more headers and Cookies collected as well.

Once successfully authenticated, it is possible to send other requests passing the Cookies. For this OSLC request it is also necessary to pass an accept header and the OCLS-Core-version header.

Successful retrieval of the OSLC workitems catalog URI.
Successful retrieval of the OSLC workitems catalog URI.

Please note that the Jazz Product APIs might need other special handling. In the above example the communication returns an additional Cookie JSESSIONID. In subsequent calls provide a header JSESSIONID with the value of the returned Cookie.

The easiest way to ‘log out’ is to delete the Cookies for the server.

delete all Cookies to restart.
Delete the Cookies for the domain to start again.

The request above can also be created by using a special content header Content-Type:application/x-www-form-urlencoded and a raw post body with a special encoding.

j_username=myadmin&j_password=myadmin

The images below show how this looks like in Postman.

The request above can also be created by using a special content header Content-Type:application/x-www-form-urlencoded and a raw post body with a special encoding.
Content-Type header
Raw POST request body.

This code shows the whole POST information.

POST /jts/j_security_check HTTP/1.1
 Host: clm.example.com:9443
 Accept: /
 X-Requested-With: XMLHttpRequest
 Content-Type: application/x-www-form-urlencoded
 cache-control: no-cache
 j_username=myadmin&j_password=myadmin

Authentication Failure

The Following image shows an unsuccessful authentication due to a typo in the password. This is only one of the possible flows, but the other scenarios are handled the same way.

The response shows status 302. There is a header Location with value /auth/authfailed. This indicates the authentication was not successful.
The Location header with value /auth/authfailed indicates a failed authentication attempt.

The response shows status 302. There is however a header Location with value /auth/authfailed. This indicates the authentication was not successful.

Authentication failed with Automatically follows redirects enabled leads to a JavaScript error page.
Authentication failed with redirect.

If the setting for Automatically follows redirects is on, the request would be redirected and likely show a status 200. The response body contains a page that the client could show to allow the user to enter the user ID and the password to authenticate. This web page uses JavaScript. Because the REST client Postman does not support JavaScript execution the page shows an error.

Where to Authenticate

My understanding so far has been that the CCM and the QM server have their own authentication. The RM server delegates authentication to the JTS. The Lyo implementation uses the public URI of the applications to create the authentication request. For the RM application it uses JTS instead.

The document Authentication of a native client with a Jazz-based application, seems to imply that authenticating to the JTS should do the trick for all applications. Using this has, so far, not caused any problems, so my suggestion would be to use the JTS URI to authenticate.

If you experience any issues, fall back trying to use the application specific URI.

Basic Authentication

No system I worked with so far was configured to use Basic authentication.

However, this authentication method works as a fallback for systems that have Open ID Connect (OIDC/Jazz Authorization Server) enabled. This section explains how to use it with Postman. See this information to understand the details.

Please note that Basic Authentication does not work as fallback for systems that have Form based authentication configured.

As a summary, for Basic authentication, a special header Autorization that contains username and password encrypted in Base64 is sent with each request. The request below shows the header

Authorization: Basic c29tZXVzZXI6c29tZXBhc3N3b3Jk 

Testing and special considerations

When using Basic Authentication in Postman it is useful to have the Automatically follow redirects setting to be on.

To configure Basic authentication in Postman, select the Authorization type Basic Auth and enter the username and the password. Postman generates the authentication header. When using the API or other tools that do not create the authentication header, it has to be created and added.

Configure Basic Authentication.
Configure Basic Authentication.

The call below gets the workitems catalog for a RTC server on Jazz.net. This requires sending a valid Accept header and the OCLS-Core-version header.

Request with Basic Authentication.
Request with Basic Authentication.

The code below is the important information.

GET /jazz/oslc/workitems/catalog HTTP/1.1
 Host: jazzdev.rtp.raleigh.ibm.com:9443
 Accept: application/xml
 OSLC-Core-version: 2.0
 Authorization: Basic c29tZXVzZXI6c29tZXBhc3N3b3Jk
 cache-control: no-cache

When sending the request, the final response should have a status 200.

Successful response.
Successful response.


Malicious website

I am currently working with a customer trying to authenticate to one of their CLM systems. This seems to work except with the CCM server. When they try to authenticate they get the following message.

You have followed a direct link to log in to a Jazz server. This page has been presented to ensure that a malicious website cannot use cleverly crafted content to circumvent security. Please log in if you would like to access the server. 

We are looking into what could cause this issue. The web does not provide any good explanation. I found some questions in the Forum and other places and think I noticed a pattern. Several requests I have seen used a wrong URI to authenticate.

When using URIs like https://clm.example.com:9443/ccm/authenticated/j_security_check or
https://clm.example.com:9443/ccm/j_security_check and GET them in a browser, I get the same error.

You have followed a direct link to log in to a Jazz server. This page has been presented to ensure that a malicious website cannot use cleverly crafted content to circumvent security. Please log in if you would like to access the server.
Directly accessing internal URIs is prevented.

It is currently unclear what is happening and how to resolve the issue the customer is seeing. I will update the post, if we can figure out what is going wrong.

Summary

This blog post tries to shed some light on how to log into a CLM System to be able to use the REST and OSLC APIs. This should work for manual work as well as for automation tools. This blog uses Postman, but any REST client should work. The details, especially covering redirects and different authentication methods can be found inn this document Authentication of a native client with a Jazz-based application.

As always, I hope that this post helps users out there with using their Jazz tools.

Advertisements

Type System Manager Part 2


We finally published Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 3: Automation tool deep dive on Jazz.net.This was a major effort and took a long time to do. This article provides a closer look at the source code, what it does and how it does it. It also provides some insight in how OSLC4J works and can be used. The information in the article, especially for setup and deployment of the automation prototype is very reusable for other scenarios and I hope to be able to reuse it in later articles and blog posts.

Type System Manager

When this effort was planned and performed last year, we had no idea what would come out of this effort. When we finished the first iterations and I started to write Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 3: Automation tool deep dive, we called what we where working on an automation prototype in the articles. I decided to keep it that way.

Since then, I started to call the prototype Type System Manager and I will continue to use that name.

The article Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 3: Automation tool deep dive, basically explains the details of the code that was created until end of January 2019. This is Release 1 of Type System Manager.

New versions?

While working on the article, I continued to refine the Type System Manager. See the releases tab for the latest release. The latest releases support more automation It integrates finding editable configurations based on sub-strings in the description, create a source to target mapping and import/deliver type system changes based on this mapping. The new commands avoid the creation of a CSV file.

The code is also slightly refactored. Most of the content from Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 3: Automation tool deep dive, still applies, but don’t be surprised if there are slight changes in names and locations. There basically where inconsistencies that I found along the way and decided to fix.

The latest versions also register Custom Resource Intensive Scenarios.

TSM 1.1

In addition to all this, the code on this branch already supports Registering Custom Resource Intensive Scenarios to CLM Applications. Each command uses its own scenario name composed from the command name plus the postfix “Scenario” to register the scenario execution.

Disclaimer and Download

Any code downloadable or accessible in this post is provided as is, without support, and used at your own risk.

The code was developed in Java using Eclipse and is based on the Eclipse Lyo Client.

Thanks to IBM approving, the code was published as open source, under 
 Eclipse Public License – v 1.0, in the incredible (mostly German speaking) Jazz Community and can be found here.

Related

Feedback

If you have questions around the Type System Manager, ask them in the Jazz.net forum instead of commenting on the article or this blog post. Tag the question as a Rational DOORS Next Generation question and add the tag: dng-type-system-management to mark it for the reader.

Summary

As always I hope that the artifacts created for this blog and on Jazz.net will useful for the Jazz user community out there.

Give Me A REST


Recently I worked with OSLC/REST and discovered a nice REST client that I find really useful. I thought I would share it with you here.

If you have found tools that help you with this and want to share, please comment to the post. Please use English, describe the tool, what it does and why it is useful. If possible provide the URL in the comment as well. This will help me distinguishing the valuable information from occasional spam that slips through.

* Update * See Postman as another alternative. Links can be found below.

I have actually not done that much work with OSLC/REST in the past. I worked with the Java API’s most of the time. I have however done the Open Services for Lifecycle Collaboration Workshop in the past and also helped submitting it in classes.

The workshop suggests to use Firefox and one of the REST Client Addons available. I did exactly that and it was a pain in the, err.., neck. Why? The whole URL was maintained in one line and that, every time I changed the focus, snapped back to the position in the front. I found myself frantically scrolling back and forth trying to find the last edit location. In general the display did not give me a good way to understand what the request did and if there were issues with it. Not pretty. I tried out alternatives, but I couldn’t find anything satisfying.

I decided to try a different browser – despite the fact that I really like Firefox.

I looked into Chrome, which I recently started to use for JavaScript based Attribute Customization as described in this Wiki page and the Process Enactment Workshop (Lab 5). It suits me way better for JavaScript debugging, as I find it a lot easier to find the script, compared to Firefox Firebug.

The Nugget – Advanced REST Client

I discovered the Advanced REST client. And I it’s a nugget. A really big one actually, I think. You can simply download and install it in Chrome and have it in your Apps tile in the bookmarks bar.

AppBookMarksThis makes it easy to reach and does not take away a lot of real estate.

Now what are the things that distinguish this from other REST tools I have seen so far?

URL Management

The URL management is really nice as it basically allows you to work in two modes.

There is the traditional mode where you have the whole URL in the URL field. This is most efficient to use if one has a complete URL available and wants to copy/paste it.

Traditional URL ManagementThe first real great difference however is the small triangle in front of the URL field. It allows you to automatically decompose the URL into its interesting parts with respect to REST and also to easily manage the query parameters.

Note, I found some URL’s where this does not work as expected for the decomposition. You can still use the feature, but folding/unfolding needs some manual work during unfolding. I hope this gets corrected soon.

Decomposed URLThis is very useful, as it allows to focus on the parts that are really interesting and removes the need to parse the URL visually. It is really easy to edit, add and remove query parameters to create the needed content.

There are also several ways to help with encoding and decoding the request data.

The same is available for the request headers.

Request HeaderIt is possible to use the Raw mode to copy and paste complete headers and it is possible to switch to a Form view like above that works similar to the query parameter section.

 Manage Projects And Save Requests

The second really useful feature is the ability to manage requests and to save them in projects. This allows to store requests that work or are under construction for later use.

Manage And Save RequestsWhile I was looking into OSLC, I had issues with my request several times and I found myself maintaining my URL’s in a text file while I was exploring. Being able to save the request makes it a lot easier and it is no longer necessary to switch between applications.

If something works, it can be kept and changed to work towards the final goal.

You can have multiple projects to manage the request, for example for different projects.

Other REST Tools

I found another REST tool that has similar capabilities.

Postman is available as application (online usage) and as Packaged App. It provides similar capabilities, except I miss the encoding/decoding option. The advantage of Postman is that it keeps a request history. This is easier to use compared to saving the change every time.

Summary

The Advanced REST client together with Chrome made my life a lot easier and I can only recommend to use it. If you do, give it a big thumbs up and donate to help supporting it.

As always I hope this is useful to the Rational Team Concert and CLM users out there.