Using a REST client to authenticate to ELM/CLM Applications


Recently the question how to authenticate to a ELM/CLM (aka Jazz) application came up in several occasions. I did not know the whole story either, so I had a look. Here is what I found.

Update history

2019/05/14 Added Basic Authentication

Scenarios

There are several scenarios that could be of interest. Some of the scenarios will be provided below. A general documentation about how the authentication of a client to a Jazz application works is provided here in the development wiki.

Common scenarios are as follows:

  1. Using a REST client to explore the REST/OSLC APIs
  2. Creating custom automation using REST/OSLC APIs

Scenario 1 is also used in the OSLC Workshop. However, as explained below, things have changed since the workshop was initially written and the latest changes are not addressed in the workshop.

REST clients

The main changes that could be recognized are in the browsers and the integration of REST clients. These changes heavily impact how easy or hard it is to login. The first REST client used in the OSLC workshop was the Firefox browser addon REST Client . Using this addon It is easy to login to the Jazz application. All that needs to be done is

  1. Open a browser window
  2. Provide the URL to the CLM application and open the URL
  3. The application prompts for a login
  4. Provide user name and password and perform the login
  5. Open the REST Client Addon

The REST client addon uses the login performed in the steps above for subsequent calls to the server. This makes it easy, because it is transparent and there is no reason to worry about the details. The browser basically handled them.

Similar addons are available for the Chrome Browser. Examples are

  1. The Advanced REST Client as shown in this post.
  2. Postman as a Chrome App as explained at the end of the post above, which is no longer available.
  3. Tabbed Postman as Chrome App.

The browser integrations above all allowed to use the browser to login and use the login information in subsequent calls. These browser addons or extensions used to be very popular, but security threats and changes in integration philosophies seem to make them more and more unavailable. Some of the addons mentioned above are discontinued and might not be available much longer.

The Postman as packaged app as explained at the end of this post is currently still available and allows to configure a Postman Interceptor, that intercepts the login information from the browser and uses it in later calls. So it works like the solutions 1-3 above and the REST clients available as Firefox addons or extensions.

Alternatives are standalone solutions like Postman as standalone App which seems to be the path forward for Postman. There are many other stand alone REST clients. The caveat is, that the simple option to just login to the Jazz application is no longer available as an option. So it is important to know the details,

When trying to write custom tooling that needs to access data on a CLM/ELM server, it is also necessary to understand the details.

Testing and special considerations

It is a good idea to have a test system. Ideally local on your own machine. You can follow Lab 1 of the Rational Team Concert Extensions Workshop to set up a small system with JTS and RTC. It is not necessary to install the plain Java client libraries, any of the SDKs or to run the WorkshopSetup application. It is possible to install the other applications as well sing the Web Installer or one of the many Installation Manager repositories.

Create the sample project in the lifecycle project administration section to have data to play with.

Postman Settings

This blog shows the examples using Postman as standalone App which is very useful and allows to save and organize calls as a benefit. Like any application there are usually settings that can be beneficial or get into the way. Here the settings of interest in the context of this blog.

Disable SSL Certificate Verification

Please be aware that the SSL certificates of a test system as described above will typically be considered invalid. It is very common that test systems have invalid SSL certificates. A common issue is that the certificate is self signed and often for a different or generic host name. In such cases SSL certificate validation throws an error. Browsers, REST clients and APIs often reject working with servers presenting invalid SSL certificates. It is usually necessary to create exceptions or to configure the systems to ignore the SSL validation error.

Make sure the REST client or API works with sites that have invalid SSL certificates, or provide valid SSL certificates.

Postman rejects communicating with servers using invalid SSL certificates and displays an error message. This can be changed by changing the settings, turning the setting ‘SSL certificate verification’ to off. Postman now allows communicating to servers using invalid SSL certificates.

Disable SSL certificate verification
Disable SSL certificate verification to be able to work with example projects

As an alternative, it is possible to provide valid certificates. It is also possible to use any other test system that might be available, especially when dedicated test systems with valid certificates are available. The screen shots in this blog refer to a test system that has been set up similar to the Extensions workshop, but with all applications installed and the sample lifecycle project being deployed.

Automatically Follow Redirects

By default Postman automatically follows redirects. This can be nice but it hides a lot of details. To better understand the details explained in the wiki page Authentication of a native client with a Jazz-based application, switch the setting automatically follow redirects to off.

Setting to switch automatically follow redirects off and on.
Setting to switch automatically follow redirects off and on.

With the redirects disabled it is possible to see the 302 response as explained in Authentication of a native client with a Jazz-based application.

Minimal Header settings

When following redirects automatically, postman also collects several cookies. To focus on the really important details, and understand which headers and cookies are really needed, the setting below is very helpful.

Minimal setting - disable automatic redirect and additional headers
Minimal setting – disable automatic redirect and additional headers

Form based authentication

As explained in the wiki page Authentication of a native client with a Jazz-based application, there are multiple scenarios, how a client login would be performed. Unfortunately the number of scenarios are getting more over time as new standards are becoming available.

The easiest to use method that has worked for me so far has been the Form Based authentication. This works in a simple set up of CLM with any application server in a standard configuration without Jazz Authorization Server (OIDC) installed.

Perform a Form based authentication login to an application

To perform the Form based authentication to the application use the context root of the JTS or the application and append /j_security_check to the context root to create the login URI.

With a given context root of e.g. https://clm.example.com:9443/jts for JTS, the URI to login is https://clm.example.com:9443/jts/j_security_check.

As explained in Authentication of a native client with a Jazz-based application, the authentication is performed with a POST to the URI https://clm.example.com:9443/jts/j_security_check with a request body that contains the x-www-form-urlencoded encoded username and password.

The image below shows the simplest way of performing the authentication using Postman.

Minimum information required to perform FORM authentication
Minimum information required to perform FORM based authentication

The POST to the secure resource URI provides the username and password using the form encoded key and value pairs for the j_username and j_password. password and username are both identical myadmin.

The code for this request is shown below.

POST /jts/j_security_check HTTP/1.1
 Host: clm.example.com:9443
 cache-control: no-cache
 j_username=myadmin&j_password=myadmin

Please note that there are no additional headers necessary. The request does not specify anything in the request for Parameters, Authorization, Headers, Pre-request Script or Tests.

The response shows status 302. There is no header 
 X-com-ibm-team-repository-web-auth-msg. There is no header location with value  
 /auth/authfailed. This indicates the authentication was successful.
Post response shows status 302 and headers show no authentication failure.

The response shows status 302. There is no header X-com-ibm-team-repository-web-auth-msg. There is no header Location with any value, let alone
/auth/authfailed. This indicates the authentication was successful.

The Cookies keeps the LtpaToken to authenticate.

The LTPA Token is kept in a cookie.
The Cookies keeps the LtpaToken.

If the setting for Automatically follows redirects is on, and other header options are also set to on, the request would be redirected and likely show a status 200. There would be a lot more headers and Cookies collected as well.

Once successfully authenticated, it is possible to send other requests passing the Cookies. For this OSLC request it is also necessary to pass an accept header and the OCLS-Core-version header.

Successful retrieval of the OSLC workitems catalog URI.
Successful retrieval of the OSLC workitems catalog URI.

Please note that the Jazz Product APIs might need other special handling. In the above example the communication returns an additional Cookie JSESSIONID. In subsequent calls provide a header JSESSIONID with the value of the returned Cookie.

The easiest way to ‘log out’ is to delete the Cookies for the server.

delete all Cookies to restart.
Delete the Cookies for the domain to start again.

The request above can also be created by using a special content header Content-Type:application/x-www-form-urlencoded and a raw post body with a special encoding.

j_username=myadmin&j_password=myadmin

The images below show how this looks like in Postman.

The request above can also be created by using a special content header Content-Type:application/x-www-form-urlencoded and a raw post body with a special encoding.
Content-Type header
Raw POST request body.

This code shows the whole POST information.

POST /jts/j_security_check HTTP/1.1
 Host: clm.example.com:9443
 Accept: /
 X-Requested-With: XMLHttpRequest
 Content-Type: application/x-www-form-urlencoded
 cache-control: no-cache
 j_username=myadmin&j_password=myadmin

Authentication Failure

The Following image shows an unsuccessful authentication due to a typo in the password. This is only one of the possible flows, but the other scenarios are handled the same way.

The response shows status 302. There is a header Location with value /auth/authfailed. This indicates the authentication was not successful.
The Location header with value /auth/authfailed indicates a failed authentication attempt.

The response shows status 302. There is however a header Location with value /auth/authfailed. This indicates the authentication was not successful.

Authentication failed with Automatically follows redirects enabled leads to a JavaScript error page.
Authentication failed with redirect.

If the setting for Automatically follows redirects is on, the request would be redirected and likely show a status 200. The response body contains a page that the client could show to allow the user to enter the user ID and the password to authenticate. This web page uses JavaScript. Because the REST client Postman does not support JavaScript execution the page shows an error.

Where to Authenticate

My understanding so far has been that the CCM and the QM server have their own authentication. The RM server delegates authentication to the JTS. The Lyo implementation uses the public URI of the applications to create the authentication request. For the RM application it uses JTS instead.

The document Authentication of a native client with a Jazz-based application, seems to imply that authenticating to the JTS should do the trick for all applications. Using this has, so far, not caused any problems, so my suggestion would be to use the JTS URI to authenticate.

If you experience any issues, fall back trying to use the application specific URI.

Basic Authentication

No system I worked with so far was configured to use Basic authentication.

However, this authentication method works as a fallback for systems that have Open ID Connect (OIDC/Jazz Authorization Server) enabled. This section explains how to use it with Postman. See this information to understand the details.

Please note that Basic Authentication does not work as fallback for systems that have Form based authentication configured.

As a summary, for Basic authentication, a special header Autorization that contains username and password encrypted in Base64 is sent with each request. The request below shows the header

Authorization: Basic c29tZXVzZXI6c29tZXBhc3N3b3Jk 

Testing and special considerations

When using Basic Authentication in Postman it is useful to have the Automatically follow redirects setting to be on.

To configure Basic authentication in Postman, select the Authorization type Basic Auth and enter the username and the password. Postman generates the authentication header. When using the API or other tools that do not create the authentication header, it has to be created and added.

Configure Basic Authentication.
Configure Basic Authentication.

The call below gets the workitems catalog for a RTC server on Jazz.net. This requires sending a valid Accept header and the OCLS-Core-version header.

Request with Basic Authentication.
Request with Basic Authentication.

The code below is the important information.

GET /jazz/oslc/workitems/catalog HTTP/1.1
 Host: jazzdev.rtp.raleigh.ibm.com:9443
 Accept: application/xml
 OSLC-Core-version: 2.0
 Authorization: Basic c29tZXVzZXI6c29tZXBhc3N3b3Jk
 cache-control: no-cache

When sending the request, the final response should have a status 200.

Successful response.
Successful response.


Malicious website

I am currently working with a customer trying to authenticate to one of their CLM systems. This seems to work except with the CCM server. When they try to authenticate they get the following message.

You have followed a direct link to log in to a Jazz server. This page has been presented to ensure that a malicious website cannot use cleverly crafted content to circumvent security. Please log in if you would like to access the server. 

We are looking into what could cause this issue. The web does not provide any good explanation. I found some questions in the Forum and other places and think I noticed a pattern. Several requests I have seen used a wrong URI to authenticate.

When using URIs like https://clm.example.com:9443/ccm/authenticated/j_security_check or
https://clm.example.com:9443/ccm/j_security_check and GET them in a browser, I get the same error.

You have followed a direct link to log in to a Jazz server. This page has been presented to ensure that a malicious website cannot use cleverly crafted content to circumvent security. Please log in if you would like to access the server.
Directly accessing internal URIs is prevented.

It is currently unclear what is happening and how to resolve the issue the customer is seeing. I will update the post, if we can figure out what is going wrong.

Summary

This blog post tries to shed some light on how to log into a CLM System to be able to use the REST and OSLC APIs. This should work for manual work as well as for automation tools. This blog uses Postman, but any REST client should work. The details, especially covering redirects and different authentication methods can be found inn this document Authentication of a native client with a Jazz-based application.

As always, I hope that this post helps users out there with using their Jazz tools.

Advertisements

Registering Custom Resource Intensive Scenarios to CLM Applications


There is no such thing as limitless computing power. This is an unfortunate truth that can cause problems running the CLM and other tools, as the usage grows. To understand what systems actually do when getting under heavy load, more and more monitoring was introduced over the last years. Resource intensive scenarios where identified and the CLM tools have capabilities to record information about their frequency and duration. Plan loading and SCM compare workspace are examples in the product.

Custom Resource Intensive Scenarios

In addition to resource intensive scenarios that are built in, it is also possible to introduce custom resource intensive scenarios. Some examples are:

  • Custom automation that execute long running operations on work items, SCM data, requirements, test artifacts. Typical scenarios are custom export/import, mass updates, custom analysis of source code, baselines, linked work items.
  • Follow up actions
  • Long running custom dashboards

This is by no means a comprehensive list. It is possible to bring your clients and servers to their knees with custom themes that do not scale, by work item attribute customization adding more and more custom attributes, JavaScript providers, value providers with thousands of values to choose from and other customization.

What is your server up to?

When users complain about performance problems, even if a server is getting overloaded, it is hard to find the root causes, because a typical server does so many things.

Monitoring that has been added over time has helped, but it is still hard. It is sometimes even hard to understand the situation. As an example for how complex this can become. Users complained about performance.

Our performance architect looked at the server load and the build load and a huge amount of calls that we were not able to account for. The server was unarguably under heavy load created by builds, but the build users and SCM users where not complaining. The developers we talked to had no real issues. Some users, at a different location, using work items and running work item queries, had.

Because we could not explain the inconsistent feedback, I finally went to the location where the users where complaining. I met the users followed their day to day work and found the work item performance unacceptable. The web browser was even locking up on them.

Knowing this, we were able to reproduce the use case, and look into what happened. We found that the work item load was slow, especially on slow laptops, because it had to load so many team areas and iterations. This was specific to how the project area was configured and used.

We also found that the browser flooded the server with requests that where definitely not part of what the product UI sent. This basically forced the Web Browser to process and cache thousands of calls, reserving more and more memory and exhausting the CPU capabilities of the relatively weak laptops used by the users that complained.

The final verdict was, that there was a custom extension to the theme that created all these calls. It took us weeks and was luck that we found this out. If we had known there was such an extension, we would have been able to find this a lot faster. The server was still under a heavy build load, but the performance issue reported was not related to that.

Needless to say that this extension was also deployed in other environments. If it had a detrimental impact, it was heavily depended on the timeline and iteration structure of a project area. The more and deeper the worse.

It would have helped if we could have seen the extension working, and see how long it worked would also have helped.

Registering Custom Resource Intensive Scenarios

The same mechanism that is used to register resource intensive scenarios in the product code can be used to register custom resource intensive scenarios. Unfortunately, we where lacking a good description and supporting code that we could provide customers to use it for their extensions.

This has now changed. Some colleagues and I, independently, started creating a customer usable description how to register resource intensive scenarios. A colleague wrote some cURL code to do this. I wrote Java code to do this and started creating a presentation. When we found out, we decided to combine the effort. Here the result.

The Deployment Wiki page Register Custom Scripts as a Resource Intensive Scenario, explains, using an example, how the API works in general. It also explains how to retrieve and monitor this information.

Then it provides example code to perform this using cURL, Eclipse Lyo OSLC4J based java code, and RTC Plain Java Client Libraries based Java Code.

The Java Code comes with main classes to run it. This is basically example code, but it can also be directly used in command line based automation.

Open Source Code

Disclaimer and Download

Any code downloadable or accessible in this post is provided as is, without support, and used at your own risk. Part of the code was developed in Java using Eclipse and is based on the Eclipse Lyo Client. This was published as open source, under
 Eclipse Public License – v 1.0, in the incredible (mostly German speaking) Jazz Community and can be found here: custom-expensive-scenario-notifier-oslc4j.

Another part of the code was developed in Java using Eclipse and is based on the Plain Java Client Libraries. This was published as open source, under MIT license, in the incredible (mostly German speaking) Jazz Community and can be found here: custom-expensive-scenario-notifier-plainjava.

See the other examples the Deployment Wiki page Register Custom Scripts As a Resource Intensive Scenario.

How does it work?

There is basically a REST API to register the start and the stop of a scenario. All there is to register the start of the scenario at the beginning and then register the stop, after you are done. See Register Custom Scripts as a Resource Intensive Scenario for more details on the code.

What should your automation do?

If you have written automation tools or extensions, you should use the methods described in Register Custom Scripts as a Resource Intensive Scenario, to register your extension as an resource intensive scenario. Add the code to register the start and stop in a way that allows for disabling it easily.

Monitor the various resource intensive scenarios over time. For a scenario that takes only a fraction of a second, you could temporarily disable the registration. Scenarios that take a second or longer should continue to be monitored.

Related

Feedback

If you have questions around the Custom Resource Intensive Scenario code, ask them in the Jazz.net forum instead of commenting on the article or this blog post. Tag the question as a clm question and add the tag: custom-resource-intensive-scenarios to mark it for the reader.

Summary

Please use the method above to enhance your automation and extensions to allow monitoring their duration, frequency and deviation.

As always I hope this helps users out there with the Jazz products.

Type System Manager Part 2


We finally published Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 3: Automation tool deep dive on Jazz.net.This was a major effort and took a long time to do. This article provides a closer look at the source code, what it does and how it does it. It also provides some insight in how OSLC4J works and can be used. The information in the article, especially for setup and deployment of the automation prototype is very reusable for other scenarios and I hope to be able to reuse it in later articles and blog posts.

Type System Manager

When this effort was planned and performed last year, we had no idea what would come out of this effort. When we finished the first iterations and I started to write Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 3: Automation tool deep dive, we called what we where working on an automation prototype in the articles. I decided to keep it that way.

Since then, I started to call the prototype Type System Manager and I will continue to use that name.

The article Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 3: Automation tool deep dive, basically explains the details of the code that was created until end of January 2019. This is Release 1 of Type System Manager.

New versions?

While working on the article, I continued to refine the Type System Manager. See the releases tab for the latest release. The latest releases support more automation It integrates finding editable configurations based on sub-strings in the description, create a source to target mapping and import/deliver type system changes based on this mapping. The new commands avoid the creation of a CSV file.

The code is also slightly refactored. Most of the content from Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 3: Automation tool deep dive, still applies, but don’t be surprised if there are slight changes in names and locations. There basically where inconsistencies that I found along the way and decided to fix.

The latest versions also register Custom Resource Intensive Scenarios.

TSM 1.1

In addition to all this, the code on this branch already supports Registering Custom Resource Intensive Scenarios to CLM Applications. Each command uses its own scenario name composed from the command name plus the postfix “Scenario” to register the scenario execution.

Disclaimer and Download

Any code downloadable or accessible in this post is provided as is, without support, and used at your own risk.

The code was developed in Java using Eclipse and is based on the Eclipse Lyo Client.

Thanks to IBM approving, the code was published as open source, under 
 Eclipse Public License – v 1.0, in the incredible (mostly German speaking) Jazz Community and can be found here.

Related

Feedback

If you have questions around the Type System Manager, ask them in the Jazz.net forum instead of commenting on the article or this blog post. Tag the question as a Rational DOORS Next Generation question and add the tag: dng-type-system-management to mark it for the reader.

Summary

As always I hope that the artifacts created for this blog and on Jazz.net will useful for the Jazz user community out there.

Type System Manager Part 1


It has taken some time since I published Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 1: Manual procedures which is about this Jazz.net article.

I was very excited about it, because I was contributing to the effort myself. We finally have made progress and just released the next part. You can now read the article about Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 2: Automation.

So, what is that all about?

Type System Manager

I did some prototyping, and we experimented with different possible approaches for this automation. What we came up with is explained in the article above.

In summary, I created a prototype, lets name it Type System Manager (TSM), that can be used to automate the type system management, supporting the best practices identified in Part 1.

Example Execution Output

The prototype uses available public Doors Next Generation OSLC/REST API’s to perform the necessary tasks.

So I have been lucky to be able to do some prototyping and learn a lot about OSLC and REST API’s. The result is a prototype, that has a useful application, and is also a demonstrator for how to create automation using OSLC and REST API’s.

Usage of OSLC and REST API’s has been missing on this blog and this prepares the foundation for hopefully more examples, here in this blog, in the future.

Disclaimer and Download

I should not have to write this, but as this is the internet 8), so here goes: Any code downloadable or accessible in this post is provided as is, without support, and used at your own risk.

The code was developed in Java using Eclipse and is based on the Eclipse Lyo Client.

Thanks to IBM approving, the code was published as open source, under
 Eclipse Public License – v 1.0, in the incredible (mostly German speaking) Jazz Community and can be found here.

What is next?

The second part explains how the TSM prototype can be used and what it does and how. The next part is currently under review and will explain the details of the code, how it works, how to re-use and add to it. It also explains how to download and work with the code.

Related

Feedback?

As mentioned in the article Comments, feedback, ideas, and experiences are greatly appreciated.

If you have questions, ask them in the Jazz.net forum instead of commenting on the article or this blog post. Tag the question as a Rational DOORS Next Generation question and add the tag: dng-type-system-management to mark it for the reader.

Summary

As always I hope that the artifacts created for this blog will useful for the Jazz user community out there. Feedback, also usage, is greatly appreciated.

RTC Extensions Workshop – How to fix SSL protocol errors preventing connection to Jetty Debug server


Since some time now I started to run into a blocking issue with the Extensions workshop. I was not able to find a solution so far. Today a colleague saw the same and asked for help. Here what I found.

Problem: the browser prevents connecting to the Jetty server

When launching the Jetty Debug server, everything seems to be fine. You can even connect with an Eclipse client to the server. However, if you try to follow the description in the Extensions Workshop and you try to connect with a browser to https://localhost:7443/jazz/admin you see a screen like the one below.

SSL Error_1

In Chrome the error claims

This site can’t provide a secure connection localhost sent an invalid response.

Try running Windows Network Diagnostics.

ERR_SSL_PROTOCOL_ERROR

Firefox is even more alarming and says

Secure Connection Failed

An error occurred during a connection to localhost:7443. Peer reports it experienced an internal error. Error code: SSL_ERROR_INTERNAL_ERROR_ALERT

The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.

Please contact the website owners to inform them of this problem.

Learn more…

Report errors like this to help Mozilla identify and block malicious sites

Edge complains

Can’t connect securely to this page

This might be because the site uses outdated or unsafe TLS security settings. If this keeps happening, try contacting the website’s owner.

Try this:

Go back to the last page

It looks like the usual SSL issue with the standard certificate, but it actually is not. It does not allow you to proceed and add an exception to connect to the web site.

You are basically stuck.

I had the same issue yesterday with 6.0.5 and 6.0.6 and I knew it was working very recently – when I moved to my new laptop. I searched the internet and was not really sure about a solution still. The answers pointed at version issues and certification mismatches. So I decided to switch the Eclipse environment to a current JDK and that solved the problem. I have tried it with RTC 6.0.5 and 6.0.6 but I am pretty sure it would work with other versions as well.

Please note that the Extensions workshop suggests to use the same JDK the Server uses. I think since Browsers have recently increased their security measures, this is no longer true.

Solution

Here how to fix the problem.

  • Download and install a recent Java JDK/JRE that is compatible with the version of the RTC SDK you are working with.
    • 6.0.6 is compatible with Java 1.8
    • I use a JDK because there are other development tools that require a JDK
  • Open the Eclipse server development workspace
  • Open Window>preferences
  • Type JRE and navigate to Java>Installed JREs
    • Add the new JDK/JRE
    • Select the JDK/JRE as active click Apply and CloseNewJDK
    • Navigate to Java>Installed JREs>Execution Environments
    • Select the Execution Environment with the matching version
    • Select the new JDK/JRE as active click Apply and CloseExecEnvironment
  • Launch the Jetty debug serverLaunch

Now try to login to https://localhost:7443/jazz/admin again. You still get an error, however the browser provides the Advanced link to continue connection to the site.

SSL Error_2

The different browsers display it different, but the concept is the same for all of them.

Summary

Recent increases of browser security checks and invalidation of certificates requires to update to recent versions of the Java JDK/JRE to be used with the Extensions workshop. As always I hope that this helps people out there.

Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 1: Manual procedures


I have been involved in efforts around the configuration management features in the IBM Collaborative Lifecycle Management and Continuous Engineering Solution. These Configuration Management capabilities are a completely new approach to versioning and variant management across the different disciplines such as Requirements Management, Quality Management, Modeling, and Software Change and Configuration Management. We realize that there is a whole new area for best practices and our subject matter experts have been working on creating necessary information for some time. There is already some content available on Jazz.net. For example from Kathryn and Tim in the Deployment Wiki.

As we learn we try to create more content, involving more SMEs. Today we published the article Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 1: Manual procedures.

I am currently working on Part 2. If you want to know what that could be, check the article.

I am pretty excited, I hope to publish soon!

Jazz Community Contributions


The Jazz Community starts sharing their tools here: http://jazz-community.org/. The code for their tools can be found here.

There is a very active Jazz user community of members of several companies in Europe that are heavily using the Jazz products such as Rational Team Concert, Rational Quality Manager and Doors Next Generation.

The community members try to meet to share their experience with using, administrating and running the Jazz tools in their environments. It became clear that the different companies and community members face similar challenges and that it would be beneficial if they could share tools they created to make running such an environment easier.

The community has now started sharing their tools in this community project and in this code repository.

JazzCommunity2017-05-12_11-34-52

Some of the tools have already been shared on other sites. I have linked the ones I am aware of to the Interesting links page in the ‘Extensions Provided by the Community’ section. These are the ones I am aware of (and the code for some of them is already available in the community repository):

I am looking forward to see more community created tools soon. Visit the community to find out what they have to offer. The code for their tools can be found here.

Last but not least, a special thanks to Dani for getting this awesome user groups started and for the members of said community for their spirit, engagement and willingness to contribute and help each other. You know whom I address here!

The RTC SDK is about to change in 6.0.3


Since some time now, the RTC and Jazz Development teams are in discussion how to cope with the version compatibility requirements driven by Eclipse clients and the server API. In RTC 6.0.3 the SDK is about to be split into separate SDK’s for the Eclipse client and the Server. This will impact how the development environment needs to be set up and how extensions are developed. I will try to share a summary of what to expect here. I have so far only been able to experiment with development builds, there has not been an official release of the SDK for 6.0.3 yet.

Download the new workshop

Update * the new Extensions Workshop is finished for a while now. it can be found at the original Rational Team Concert Extensions Workshop location.

Why splitting the SDK?

The RTC Clients have been based on Eclipse 3.6 for a considerable amount of time now. This has been the case for the Jazz Servers as well. However, there is pressure on the server infrastructure for the need to support Eclipse 4.4.x and higher. On the other hand there are client applications that RTC needs to integrate with, that are lagging behind in adoption of new Eclipse versions.

As described in What API’s are Available for RTC and What Can You Extend? the RTC SDK currently contains the RTC Server API, the RTC Common API and the RTC Client API in one delivery. The RTC Common API is part of the RTC Server API as well as the RTC Client API. This is a potential problem when shipping the SDK and trying to keep the Server compatible to Eclipse 4.4.2 and above and being compatible with Eclipse 3.6 clients. As it looks, the RTC SDK will be split into two parts.

  1. A RTC Server SDK bundling the RTC Server API and the RTC Common API compatible with Eclipse 4.4.x and higher
  2. A RTC Client SDK bundling the RTC Client API and the RTC Common API compatible with Eclipse 3.6.x and higher

Impact of splitting the SDK

The split has various impacts on how extension development will now work. Please find below a short summary of changes that I have found necessary to perform the workshop.

Changes to section: 1.1 Download and Unzip the Required Files from jazz.net

The Server SDK Target Platform now requires an Eclipse 4.4.2 or higher. You can download the base Eclipse 4.4.2 client here. For example download the version Eclipse IDE for Java EE Developers. Install the client similar to described in the workshop. Then download the RTC Eclipse Client p2 install package and install this into your Eclipse 4.4.2.

In the Feature Based Launches download the new launcher442.zip. Unzip the zip file, browse to the enclosed JAR file and copy that into the dropins folder in the Eclipse client.

You might want to consider to do the following changes to the eclipse.ini file.

  • Add a section -vm with an additional line for the java virtual machine to use. If you run Eclipse with a different JVM, e.g. from Oracle, consider to specify a JRE or JDK that is compatible with the one that ships with RTC. This vm would also be used in the workspace setup section.
  • Add -showLocation in the org.eclipse.platform section; this shows the Eclipse workspace path in the upper border of the Eclipse client as below

    eclipse-workspace_2016-11-07_11-23-52

    This makes it possible to actually work with multiple workspaces and knowing which an Eclipse instance is responsible for.

  • A vmargs argument -Duser.language=en to make sure you get a consistent language in the menus if you want.

The image below shows the changes in my caseeclipse-ini-2016-11-07_11-09-02

Changes to section: 1.2 Setup for Development

Once the SDK is split into two parts the Rational Team Concert Extensions Workshop can no longer be performed using just one Eclipse Workspace. An SDK is set up as a Target Platform in the Plug-in Development section. Since the SDK’s are now split, it is necessary to have two target platforms. Since it is not possible to have more than one Target platform active in one Eclipse workspace it is not possible to launch a server for debugging while running an Eclipse client from the same workspace.

The RTC Extensions workshop will have to be changed to set up two separate workspaces.

  • One workspace will have to be set up with the RTC Server SDK as active Target Platform, for example using the path: C:\RTC603Dev\Workspaces\Dev1\Server
  • The other  workspace will have to be set up with the RTC Client SDK as active Target Platform, for example using the path: C:\RTC603Dev\Workspaces\Dev1\Client

Both workspaces will require to be set up as described in the RTC Extensions workshop document in section 1.2 Setup for Development.However, you will set up different target platforms in this step. Using the Server SDK for the server development workshop and the Client SDK for the Client development workspace.

Please note, it is a good idea to configure Eclipse to use an external browser as well in this step.

Changes to section: 1.3 Setup the RTC Tomcat Server

I am modifying the WorkshopSetup tool and data to setup the RTC project named RTC Extension Workshop to support an easier setup for the two workspaces. Basically two separate RTC Repository workspaces will be available. One will provide the launch, the configurations and the components needed to develop the RTC Eclipse server extension part of the workshop. The other one will provide the launch and the components needed to develop the RTC Eclipse client extension part of the workshop.

As long as this is not yet available it is possible to start with the existing setup tool and the related repository workspace and to load that into the two Eclipse workspaces. One workspace has to be set up with the RTC Client SDK will be used for development of the client part. The other with the RTC Server SDK set up is used to develop the server parts. When performing the workshop it will be necessary to work with the two workspaces and use one for all the server related tasks and the other one with the client related tasks. When Accepting changes into these workspaces it is necessary to understand what is part of the client and what is part of the server or what is shared. The image below shows what belongs to what.

  • The parts colored in blue are only related to server development
  • The parts colored in yellow are only related to client development
  • The uncolored parts are related to client and server development

client-server-common

Make sure to keep in mind which parts of the code are relevant for what. As an example, the project net.jazz.rtcext.workitem.extensions.ide.ui will not compile in the server development workspace. Similarly the net.jazz.rtcext.workitem.extensions.service project will not compile in the client development workspace.

Changes to section: 1.4 Complete Setup of Your RTC Eclipse Client

After Loading the repository workspace you have the choice to split the information into a sever part and a client part. For example you can duplicate RTC Extension Workshop Configuration and create one that only contains the client launches. Or you keep everything as it is and basically close the project areas you don’t need and ignore launches not needed. This is the easiest approach until a new Extension Workshop is available.

The initial step of copying the files services.xml and scr.xml is only needed in the server workspace. So when copying and importing, copy the files services.xml and scr.xml from your server’s ccm application in the installs\JazzTeamServer\server\conf\ccm folder into the RTC Extension Workshop Configuration project into the folder conf/jazz in the server development workspace.

When importing the plugins and features import the following into the server workspace:

  • com.ibm.team.common.tests.utils
  • com.ibm.team.jazz.foundation.server.licenses.enterprise-ea (or com.ibm.team.licensing.product.clm)
  • com.ibm.team.licensing.product.rtc-standalone

When importing the plugins and features import the following into the client workspace:

  • com.ibm.team.rtc.client.feature

Other considerations

As already mentioned, make sure to keep track which workspace you are working in and keep in mind that the server development part will not work in the client development workspace and vice versa.

Just Starting With Extending RTC?

If you just get started with extending Rational Team Concert, or create API based automation, start with the post Learning To Fly: Getting Started with the RTC Java API’s and follow the linked resources.

You should be able to use the following code in this environment and get your own automation or extension working.

Summary

We have major changes coming up in the RTC Extension development area. The RTC Extension workshop needs to be adjusted and parts of the workshop lab needs to be reorganized and rewritten. This post explains what to consider for experienced users. Once there is an update to the Extension workshop lab material this post will be updated.

As always, I hope this helps users out there and saves them some time.

Alien Skies II – JavaScript Yet Another Computer Language?


I tried to look into Web UI Development, especially in the context of RTC Extensions like dashboards and the like several times in the past. I couldn’t get it to fly. It was a total disaster so far. I couldn’t understand how all that works and what the fuzz was all about. The more I looked into it, the more confused I was.

Surprisingly, looking into Doors Next Extensions seems to have revealed the missing link(s) I needed to better understand the whole topic around JavaScript. I have talked to other people who had similar issues with JavaScript, so I hope it is worth sharing my thoughts here.

I am well aware that the topic and my approach to it could be very controversial. If you like to disagree with any of my assessment, please feel free to do so in the comments. I rarely dismiss comments, but please be aware that the comments in this blog are moderated xD.

I have a book about JavaScript, but I rarely see JavaScript like it is described in that book. So, how does this JavaScript stuff work?

Its Not The Language

As far as I can tell, the language itself does not really matter that much. The language syntax of JavaScript is very similar to other languages e.g. such as Java. The Language has inheritance mechanisms, the language can support event based, asynchronously processed and functional aspects. All in all, a collection of properties that are shared across several language families.

There are some issues with the language (at least from my, still uneducated, point of view). It is not type safe to get started. While this is very flexible it is also asking for trouble in my experience. JavaScript is very dynamic and does not get compiled. A lot of issues will be seen at run time and not during coding.

Anyway, to have a common language with a common syntax and semantic is probably useful if you have to switch the domains often. At least you don’t have to learn a new language over and over.

Its Not The Development Environment

I am missing a good library handling and context sensitive editor support. There might be some cool development environment out there, but it has eluded me so far. Any suggestions are welcome, as always.

I have seen the first computers making their appearance in schools. With enormous keyboards with two lines of 40 characters to display. Nothing one would associate with a computer today. I was lucky enough to be able to write my first own programs on Commodore PET, VIC-20, 64 and on the first IBM PC’s. Ignore the first program on a Texas Instruments calculator with 32 programmable steps. I have developed without debugger, because there were no debuggers available or not affordable e.g. you could buy in circuit debuggers for  embedded systems but that was sometimes unaffordable even for companies back then.

I have seen bad development environments and debugging by printf. It is not that bad with JavaScript, in browser based applications. It basically depends on what the use case is, if there is a good debugging environment available or not.

Feeding The Confusion

I am usually pretty good in spotting patterns. Looking into the JavaScript domain, I had a hard time finding the basic ones. A lot of the confusion I felt is also, in hind sight, probably related to the fact that there is not just one JavaScript. There are multiple. And they are used in different ways and using different patterns. In addition to that the expression JavaScript is often used in contexts, where it would probably be better to talk about Web or JavaScript libraries or frameworks instead.

  1. There is the JavaScript that runs in browsers. This JavaScript usually works together with CSS and HTML. It is used to create dynamic content and presentations in the Web to be able to provide more or less usable UI’s in a browser, designed to show more or less static HTML documents. The JavaScript engine is built into and shipped with the browser and there are different engines and development environments for different browsers. This type of JavaScript basically requires to understand how JavaScript, HTML and CSS work in combination and how to control their interaction.
  2. Browser based JavaScript is integrated in several different frameworks and tool kits such as Ajax/Dojo and others.
  3. There is NodeJS, which claims to be Java Script and actually is sever side JavaScript processing and a ton of libraries and frameworks that are available for it. The only common part I can identify here is the language that is used. I did not look into the details however, so maybe there is more e.g. the run time used could be one of the common ones.

So when I started to look at the Doors Next Extensions which said JavaScript, I realized quickly, that there were constructs in the examples that I had never seen before in any JavaScript example. The normal pattern again, every time I looked at something with JavaScript inside and thought I had some basic knowledge already, I saw something different. This time however I was able to figure out what I was missing and this seems to help with all the other cases as well.

The Doors Next Extension Mechanism

The Doors Next Extension mechanism is based on JavaScript, HTML and CSS. The API supports TypeScript which provides with a more type safe and compile like way of developing JavaScript. The Doors Next API itself comes with a Typescript interface definition. Some of the examples that are provided use jQuery a JavaScript based library and framework. The latter means that the extensions look a bit uncommon.

Uncommon Common

To some extend this seems to be fairly typical for my experience with JavaScript. Almost every solution I have seen has some common part that can be found in other solutions. But looking across all the examples there is only a tiny common core, which often is basic JavaScript functions. If the JavaScript based solution deals with UI’s, there is usually the common CSS and HTML, if it does not focus on UI’s this is missing. Since there are so many frameworks and libraries, Java Script based solutions can look very different and it is important to understand the detailed frameworks and resources that are used in the specific example.

Summary

Granted, it took a while for me to come to terms with JavaScript,  but looking into the Doors Next Generation Extension Mechanism provided me with some important insight. This will help me to understand these technologies in other contexts as well. I am looking forward to playing with Doors Next Extensions and add the results to this blog.

Alien Skies – Peeking Into Doors Next Extensibility


I started a recon mission into the alien territory of Doors Next extensions recently and almost lost my way.

Doors Next is one of the products I am responsible for. I have some experience with requirement management and I understand Doors Next usually good enough to find my way, but I am way more familiar with RTC, especially since I did a lot with its Java API and extensions. So I thought if this is similar to the situation with RTC, capabilities to extend Doors Next will be in high demand soon and I better get my hands dirty and into it. Since the language available to extend Doors Next is JavaScript, this looked also like a good opportunity to have some more exposure to JavaScript.

Update, a nice summary of RDNG API’s can be found here.

First Steps – Look into the documentation

So I browsed through the documentation a bit to learn what there is to learn! Other than me, please start with reading the most recent version of the documentation. This will save you detours like looking at WAS Liberty Profile and publishing data. I followed the guidance and played around with the simple examples and the more complex ones provided. I became adventurous and wanted to modify the Hello World open social gadget example and create one that should be published in the widget catalog along with the other examples. The example worked as open social gadget already and I thought, I could use it as widget as well.

That did not work so good. Why? If I used the same xml code the open social gadget showed and the Dashboard widget showed a blank content.

The Widget

This is how the basic widget looks like. All the rest such as CSS, JavaScript and the like can be added to it based on this structure.

DashboardWidget

The Open Social Gadget

This is how the open social gadget looks like. I tried to spot what the difference was for a long time. Until I finally noticed that the difference between a working gadget and a working widget is the tag that the open social gadget has and which is missing in the widget..

OpenSocialGadget

It took me quite some time to figure out what the problem was. Embarrassing, but on the other hand it shows again, that tool extensions, even the smallest, are complex business. Maybe there are better ways to debug these kinds of issues. Compared to RTC Java Extensions I felt like back in time, trying to debug C and Assembler with printf…..

See this site for some additional introduction into open social gadgets.

Deployment Structure

Similar to my description in Publishing XML and Other Data With the WAS Liberty Profile  the extensions where deployed in the apps folder like this:

WidgetDeployment

Compared to RTC Extensions, this is a relatively simple and accessible structure. As mentioned by Guido, in an enterprise context, it would make sense to have your own web server to publish the extensions. This would allow to have a more fine grained control about who can change what.

It would be possible to use a RTC Build workspace to publish these extensions. I think there is a need to come up with some naming pattern for the extensions, in order to avoid chaos and confusion over time.

This has been challenging in the past and I assume it will stay challenging. The structure above has a css folder, as well as a JavaScript folder just in case. There is also a need for some more documentation and the like.

Summary

Doors next has means to extend its capabilities. Provided some fundamental knowledge, this should allow for a lot of the automation needed in enterprise environment. My first steps into this alien territory where a bit shaky, but all in all, I think I now have the basic understanding where and how to get started with more complex extensions.