EWM Attribute Customization Introduction

While working on Debugging EWM Work Item Attribute Customization JavaScript I thought about all the questions I have seen being asked about Attribute Customization on the Jazz.net forum. There are some common questions I have seen a lot. There are some common things users would like to do and there are some limitations that might get into the way. I thought, I would create a blog that summarizes what I know about attribute customization and also provide the important links. I also want to provide a short summary about, for all I know, what can and can not be done with Attribute Customization and the provided API. I already create something like that in RTC Process Customization – What you can and cannot do. I will follow its structure and reuse some content but will try to be more specific in this blog. This blog will be limited to the work item attribute customization, with some few exceptions. The blog post will mention possible solutions that go beyond attribute customization, where this seems reasonable.

Important links

The following links are the most important references about EWM work item configuration and attribute customization.

• Customization of Work Items in Rational Team Concert
• Work Items Attribute Customization Wiki page
• Debugging EWM Work Item Attribute Customization JavaScript
• Work Item Customization in Rational Team Concert
• Work item editor presentations
• Defining URIs for artifact types, attributes, and other resources
• Read-only control for Work Item attributes in Rational Team Concert
• Changing your RTC Process Configuration – Best Practices
• RTC Process Customization – What you can and cannot do
• Process Enactment Workshop for the Rational solution for Collaborative Lifecycle Management has the last two labs explaining attribute customization.

About Jazz.net forum questions

A lot of the input that I take under consideration comes from questions and answers on the Jazz.net forum. Some of the examples originate from customer requests.

Questions on Jazz.net and even customer requests do often not provide good requirements. They often provide some implementation idea instead. The real problem or business case is usually hidden and not exposed. So it is important to question the request to drill down and expose why the question was asked as it is and what the background of the question is. If the real reason for the question is exposed, the answer can be completely different.

One example is: “Can I auto subscribe users to a work item?”. In pretty much all cases I can remember, the purpose of this was to send an e-mail notification to a user e.g. when the work item was created. Adding a user to the subscribers might or might not be a viable solution, but the example clearly exposes the problem.

So take the questions on Jazz.net with a grain on salt and try to dig deeper to understand what the question is really about.

What is Process Configuration and Attribute Customization?

EWM allows to configure the process of a project area. There are several options that can be used.

1. Work items, work item attributes, editor presentations, workflows can be configured.
2. Roles can be configured
3. Permissions can be configured
4. Operational behavior can be configured using out of the box preconditions and follow up actions.

Attribute Customization is a work item related capability provided by EWM. It provides built-in capabilities that can be configured and thus be considered process configuration as mentioned above.

The capability types available for configuration are:

1. Default value provider for built in attribute types.
2. Calculated value provider to support calculation with work item attributes to support certain processes.
3. Value Sets to provide customized sets of selectable values for work item attributes. Built in value sets allow to extract string values from an XML page in the Web, select contributors by role and allow to select enumeration vales based on other dependent enumerations.
4. Validators to validate values of attributes based on range or regular expressions. Validators can show a warning in the UI and be used with a built in advisor to prevent saving a work item, unless the offending attribute values are corrected.
5. Conditions can be used with built in advisors to make work item attributes read-only or required. There is no built in condition available at this time.

What is special about Attribute Customization?

What sets attribute customization apart from the other capabilities is, that it provides a JavaScript API that allows to create custom scripts for provider configurations. The JavaScript can be uploaded to the project areas process by any user with the required permission. This does not require write access to the application servers file system. For this reason it is possible to create and deploy custom JavaScript scripts even if the server is maintained by a cloud provider.

JavaScript is also reasonably easy to learn and a lot of practitioners have some JavaScript skills. This is a two edged sword.

Contrary to Eclipse extension plugins, another extension mechanism available in EWM, the JavaScript API has a relatively low experience threshold for beginners. It is way easier to develop than the other Eclipse plugin based customization options available in EWM that require great skills, understanding of the framework and familiarity with the available API. Since these options are based on Java and Eclipse plugin extension development, they require the plugins to be deployed which means they need to be copied into a file system, typically on the server where the CCM application is deployed. This is usually not possible in a cloud hosted environment. Except, some cloud services would perform such deployments for the customer, for a higher price tier.

The danger is that the low skill cap attracts very inexperienced users, which can eventually lead to issues, including server performance issues. Some of the capabilities can be overused. There is also a tendency to overcomplicate the work item workflow and the process.

The JavaScript Attribute Customization API constraints and limitations

Attribute Customization is configured on the attribute level. This means that any attribute customization will apply to all work item types that have the attribute configured.

JavaScript attribute customization scripts have to implement the required interface for their type. All the interfaces require to return some data. The data can be empty or undefined, dependent on the attribute type, e.g. an empty string or null. It has to return something that matches the attribute type it is configured for, nevertheless.

It is possible to detect the type of the work item the script is executed for, this allows to have different behavior for different work item types in the JavaScript attribute customization script. Based on the work item type attribute value, the script decides what to return, some custom value or the current attribute value to do nothing. This strategy allows to adjust for type specific attribute customization to some extend.

The API description in the Work Items Attribute Customization Wiki page explains which work item attribute value types are supported. Only very basic attribute types such as strings, numbers, enumerations, timestamps and dates are really supported. Especially note that the support for complex item based attribute types such as team areas, contributors etc. is very limited. For a those attributes, all you can access is the UUID and the display name. If you want to set such values, you must return the UUID of the item.

The documented API does not provide with methods to discover information beyond the work items current attribute values. There is no documented simple JavaScript API to discover details about not attribute based information. There is no documented simple API to follow or create links to work items or other artifacts. There is no documented API to access process information, such as users and roles or any other details. There is also no mechanism to detect a state change or workflow action. The workflow action is only available in conditions but the detection of a state change would be desirable in calculated values as well.

The return value of JavaScript attribute customization scripts is processed in the API. Dependent on the work item attribute type, the script is configured for, some kind of conversion is performed. As an example a string can be converted into a timestamp, provided the format is correct.

As far as I can tell, the conversion that is performed for html and string based attributes, including the subject and description, when returned from JavaScript converts the content into a string escaping any special content such as XML or HTML tags. This conversion happens outside of the JavaScript and there is no way to change the behavior. Because of this reason it seems to be impossible to return HTML with working links for HTML or string based attributes including the summary and description. Only the built in configurable default values work correctly. The only current attribute type that can be set by JavaScript attribute customization to contain links is the type Wiki. In this case the Wiki syntax has no XML/HTML tags involved that would have to be escaped.

Performance considerations

Some users seen to think there is such a thing as limitless size and performance. This does not apply to the real world. It is possible to heavily impact the performance of clients or servers with process configuration and especially customization. I am aware of a case, where JavaScript added to a Theme caused so much communication that the browsers on smaller laptops where drained of resources and almost came to a halt. Loading a work item took several minutes. This made working with the tools impossible for many users. So be careful and consider what you might do to the performance.

It is also very easy to annoy the users with overcomplicated processes, thousands of choices, required attributes nobody knows the values for etc. In general big numbers are bad and small numbers are good. When drop down boxes grow too big and the number of items ever increases, the users will be affected in their ability to successfully use the process. I have seen cases where the users got confused and did not realize that they had to enter a value and complained the UI had stopped. Overcomplicated processes will likely cause performance complaints and wear away the will to support the chosen tool and process.

There are limitations in the amount of custom attributes for a work item type (100). Having a lot of work item types is also not helping to make the process easier. Keep numbers low e.g. 10 would be a guidance for work item types.

Also keep in mind that there are size limits for all work item attribute types. Especially note that for list type attributes there is a size limit. How many selections can be stored is limited by the site of the list attributes and the size of the ID’s or item UUID’s that are stored.

Where does the JavaScript run?

The JavaScript runs in the Web Browser or in the EWM Eclipse client in a JavaScript interpreter..

Where and when is Attribute Customization not executing

Attribute customization including the JavaScript is not executed when the work item operations are created using an API such as OSLC or the SDK or Plain Java Client Libraries.

Work Item Proxy

When debugging the JavaScript, it is possible to see more objects accessible to the JavaScript in the debugger. It is possible to see an object called work Item proxy that looks very promising. Please note that this object is considered internal API. It should not be used by the JavaScript. The supported API is documented here. It is also important to understand, that this work item proxy is only available in the Web Browser and not if the script runs in the Eclipse client.

There are several posts on the Jazz.net that discuss how the proxy could be used. Again, it is internal API and should not be used, if you do, you are on your own. There might be unwanted side effects and the internal API could change at any time.

Why does my script not work?

This question comes up often on the forum. It is pretty much impossible to debug others scripts on Jazz.net. There are reasons for that:

1. Java Script Attribute Customization depends on the process used in the project area. The community does not have and, due to privacy, security and other concerns will not get this.
2. It is quite some effort to setup a test project area, even if one would have a process template.

So it is usually impossible to help on this level. What can be done is scanning the script and the description in the question for typical issues.

See Debugging EWM Work Item Attribute Customization JavaScript for how you can debug your own scripts.

Please note, there are different levels of issues with JavaScript. On the most fundamental level:

1. Attachment Scripts must be enabled in the advanced teamserver.properties to be working – see below.
2. The script must be accessible in the process.
3. The script files encoding must be UTF-8 and consistent so the file can be read, interpreted and executed.

If an error happens at this level the best you can hope for is an entry in the log file. Only if the JavaScript is successfully decoded and loaded, it can be debugged.

What you can and cannot do with JavaScript Attribute Customization

There are certain limitations in the JavaScript Attribute Customization API as far as I am aware. I provided a more comprehensive presentation here: Process Customization – What you can and cannot do.

I will try to provide some details here.

1. Attribute Customization JavaScript is always defined as a function. Regardless what type of Attribute Customization the script must return a value (can be null or empty). Otherwise this will be a potential error in the log. I am not sure what happens worst case.
2. Attribute customization is for the attribute. This means especially that the customization will be executed in all work item types that have the attribute configured.
3. Sure, in JavaScript, you can get the work item type, by reading the respective attribute. You can then decide what to return. As explained in 1. you cannot return nothing. You however can get the current value of the attribute and return that, this helps in some cases.
4. As far as I am aware, Attribute Customization only runs in the context of a users browser and the Eclipse client. As explained above, this also means that automation based on attribute customization does not work, if you access work items using any of the APIs. This is important, as pre-conditions/advisors and follow up actions/participants can be implemented as server extensions that always run, regardless who accesses the work item how. When the JavaScript is run in the Eclipse client, only the infrastructure of the Eclipse client is available.
5. Attribute customization scripts can run multiple times, for example the calculated values might execute multiple times, based on changes to the other attributes they depend on.
6. JavaScript Attribute Customization only has access to the data of the work item in which context it runs. It is not possible to follow work item links using the documented API.
7. Even the available data of a work item is limited. As an example, a workflow action for a state change is only available in conditions. It is not available in any other attribute customization types, especially not in calculated values. This essentially means, there is no simple way of tracking and detecting a workflow state change.

Java Eclipse plugin extension based Attribute Customization

Beyond JavaScript, it is possible to provide Java based Work Item Attribute Customization Plugin Extensions. This is possible for all available attribute customization types, default values, calculated values, value sets, conditions and validators as shown in these examples. These Java based plugin extensions have access to more complex EWM Java SDK API. This makes it more feasible to create complex custom attribute customization. It is also easier to control which return result types are supported and, based on these, create richer attribute customization. It would, for example, absolutely be possible to have a default value provider that returns an XML or HTML result, that shows working HTML links. The decision to return a result that escapes the HTML tags or return the correct format is an implementation detail. A more complex example with additional background information can be found here and in this post.

The Java based attribute customizations would show up in the Admin UI in the same way the built in Attribute customization does. To make this possible the Java based Eclipse plugins need to be deployed in the Eclipse client as well as the EWM Server (see here). The API that is available to the Eclipse client and the server is the EWM Common SDK API. Consider reading this blog post to better understand what this means. The requirement to deploy these extensions on the EWM server are a potential inhibitor for cloud based deployments.

EWM Eclipse Plugin Extensibility

Java base attribute customization is only a subset of the Java Eclipse plugin extensibility available in EWM. The SDK provided for EWM allows to develop all kinds of extensions, including pre-conditions, follow up actions and asynchronous tasks. It is possible to develop such extensions to run in the context of the Eclipse client or on the server. These extensions need to be deployed in the context they are developed for. Server extensions have access to the full capabilities of the EWM Common SDK API and the EWM Server SDK API.

Preconditions also called Advisors can act as process advisors that prevent saving if the data violates some standards. Preconditions can be used to implement attribute customization validators or conditions. Example Advisors can be found here.

Follow up actions, also called Participants can perform additional operations on the EWM data. As an example, follow up actions can collect data across linked artefacts, perform calculations on the data, set attributes and properties for items and save the changes. Example participants can be found here. This includes the capability to create new work items and link the new work items to other work items.

In some cases it is better to use these capabilities instead of attribute customization, because it provides more API than even the Java based attribute customization.

All plugin based solutions require to be deployed. Server extensions require to be deployed in the server, which might be a blocker in an environment hosted as a service in the cloud.

Popular requests

In the next sections, I will try to provide some examples of popular requests. I will describe the background these request have, as far as I can tell, and which implementation options might be viable.

Popular: Custom E-Mail Notification

This comes up in various flavors. Examples are

1. Can I automatically set an owner of a work item?
2. Can I automatically subscribe a user to a work item?

With respect to attribute customization, the answer would be yes, you can do this, to some extent. It is possible to set or add a subscriber, to set or add a work item owner. In the JavaScript API this needs to be done the contributors UUID. The UUID will have to be configured or hard coded when using the JavaScript API. There is no API to search for contributors in JavaScript.

A Java based attribute customization plugin would have more API access and can provide better control and automation.

A real custom mail notification is possible using the following approaches

1. Custom scheduled asynchronous tasks (see Due Date Notifier).
2. Custom follow up actions that trigger at work item save.

The last three options require to write custom Java extension plugins and their deployment on the application server, which makes it less attractive in a cloud environment.

As an alternative, EWM has other mechanism that might be better suited. For example there are RSS feeds for various change events, work item queries and dashboards. These do not send e-mail, but they might scale better than e-mail notifications.

Keep in mind that the e-mail notification options are controlled at user level by the user. A real Java Eclipse plugin that sends custom e-mail notifications could work without the need to be correctly configured at user level.

Popular: Calculation with work item attributes

This is often requested to calculate effort or something similar.

With respect to JavaScript based attribute customization, it is possible to perform calculations on attributes of the same work item. It is possible to perform numerical, boolean and other operations in calculated values. It is necessary to set the dependencies right, so that the chain of events executes until all the calculations have been performed. If multiple work item attributes need to be set, keep in mind, that the calculated value is a function that only works for one work item attribute. If multiple attributes need to be set, a calculation for each attribute is necessary. As already mentioned, It is not possible to calculate across values of multiple work items.

If it is necessary to calculate across multiple work items, JavaScript attribute customization does not work. It is necessary to use the Java plugin extension mechanisms such as follow up actions to implement this. An example is the RTC Update Parent Duration Estimation and Effort Participant.

It is also possible to use external tools based on API to collect data and then update work items e.g. in scheduled jobs.

Popular: Calculate how Often a work item was in a specific state

I have seen this several times. There is no easy way to detect work item changes. It might be possible to use an approach as shown below to detect this, but I am not sure if this idea below would really work.

With some tricks and several hidden custom attributes for state tracking ST1, ST2 and counting STC, it might be possible to cobble something together. E.g. initialize the hidden state tracking attributes st1 and st2 with a default based on the state ‘New’.

The calculated value of ST1 is triggered and calculated based on the current value of the work item state attribute. If the current state is not the value of ST1 return the new current value.

Calculate the counter based on a triggered change of ST1. If ST1 contains the desired state, return the current value of STC increased by one.

Calculate the value of ST2 when ST1 was changed. Return the value of ST1 as result to the calculated value.

This requires some intelligent usage of the dependent attributes. It might be necessary to use more artificial attributes to get this working. This attempt has however many downsides, especially the custom attributes are all showing up in the history and also create more changes in the work item.

It would be easier to develop a follow up action that actually works on the state change, if this is necessary. It would be important to see if reporting can be used instead.

Popular: Make work item attributes read-only or required

EWM has preconditions/Advisors to make work item attributes read only or required based on conditions. The question is, how far can you go with this? The JavaScript API is very limited in what data is available. It is obvious that conditions can work on this data. So you can make attributes read only or required on work item attribute information.

But since you have no access to any complex information such as team membership or roles or the current contributor, JavaScript can not be used to create more complex conditions. Any condition that needs to have access to more complex information, would require to be written as Java plugin. An example requested by a customer with additional background information can be found here and in this post. It is important to know that these condition based attribute customizations are the only way to customize rules to make attributes read-only or required. There is no other extension point available that you could use to hook into. This is one of the rare cases where custom preconditions/Advisors can not be used.

In my experience with requests like here and in this post, to make attributes required or read only, are often motivated by the desire to have more fine grained control. Consider to use permissions to prevent roles from changing attributes.

Requests like above are also often motivated by the desire to make EWM work like some legacy tool. My experience so far has always been, that trying to make a tool work against its design is usually fatal. It usually makes the process hard to understand and follow. It also often degrades usability and performance.

Since every user in EWM has at least the default role (Everyone), can have many roles, and the roles have an order, it is also debatable what ‘by role’ really means.

Popular: Dynamically Hiding Work Item Attributes

I have also seen several requests that want to make attributes hidden by some complex, usually role based rule. This is not possible today, regardless the approach.

There are very limited capabilities to control visibility built into EWM. Attributes can be hidden if they are empty, if the work item is in a workflow state, if some endpoint is empty, if there is no project link and if the work item is just created. An editor presentation tab can be hidden, if all contained attribute presentations are empty.

There is no extension point, and there is no attribute customization available, I am aware of, that would allow customize the behavior in any other way.

There are various enhancement requests for EWM in this context. I think it would be nice if there was a custom attribute customization option that could be used with a precondition e.g. ‘Hide for condition’ that could be used similar to ‘Read only for condition’. This would allow to hide attributes that are unimportant. The question is, if it is helpful when attributes show up and vanish for the same attribute type. This is at least debatable. As a user I would likely find this confusing. As always, keep in mind this could open the door for performance issues and abuse. The remark about roles and their usage in EWM also applies.

Please note, just because an attribute is not visible in the work item editor, presentation does not make it inaccessible by work item queries or reports. EWM was also designed for collaboration and not hiding.

Popular: Dynamic workflow

I don’t know where this comes from. I doubt this is a thing in other tools for change tracking, but it cam up more than once.

EWM/RTC does not support dynamically changing the workflow of an instance of a work item. The workflow is tied to the work item type. It would be possible to use a follow up action/Participant to change the work item type under certain circumstances. Due to the lack of support for detecting a state change and the lack of access to other data in attribute customization, it would be not feasible to use attribute customization. A participant is a far better option.

Popular: Dynamic creation of attributes

This came up several times.EWM does not officially support this. It is possible to create an attribute using the API, but this attribute does not surface in the admin UI. It is necessary to add statements to the process XML to make such an attribute configurable in the process and the editor presentations.

This is used by some integrations just once and not meant to be used constantly during operations. Also mind the limit of custom attributes and consider usability of the process.

Restrict Read Access To Data

Some customers want to have more control about who can access which data. EWM has the capability to restrict work item and SCM item read access permission based on access context. If an item can not be read, it can also not be written.

For work items, the restricted access context can be an access group, project area, or team area. It is possible to automate setting the work items restricted access based on categories. This sets the access context to the process area associated with the work item category. This is the only built in automation for category based work item access control. There is no other built in automation for work items e.g. enabling the automatic selection of an access group for the work item.

For SCM data there are more options to set the access context/visibility. Components can have access control set to a project area or an access groups. SCM content in a component, folders, files, can be configured more fine grained. It is possible to limit access to a contributor, a project or team area or an access group. There is no built in automation to set the access context for SCM items.

It is possible to create custom automation to set the access context for work items and SCM items. One implementation approach is using follow up actions. For work item restricted access this can be configured for the work item save operation. For SCM objects follow up actions for the check in and deliver operation could be developed to compute the desired access context. I have explained some of the options in the blog series Setting Access Control Permissions for SCM Versionables.

In contrast to my knowledge back then, I know today, that it is possible to elevate the operation context to an administrator, why this is important is explained in the second last paragraph.

The question would be, can JavaScript attribute customization be used for custom automation for the restricted access? Attribute customization is only available for work items. There is no such API for SCM content. It might be possible to set the restricted access attribute in JavaScript attribute customization by returning the UUID of the access context, but as already mentioned, because of the very limited amount of data and API access available, JavaScript would most likely not be feasible for anything beyond that.

Java Based attribute customization could be more feasible, but it is most likely not possible to elevate the operation to super user access due to the limitation of the available API to the common API. I would consider this approach to be unfeasible for any relevant complexity.

If it is necessary to develop a Java Eclipse plugin for attribute customization anyway, it looks to be more suitable to create a follow up action to automate setting the access context. This allows to use the full capabilities of the EWM SDK common and server API and it is possible to elevate the operation to run in an administrator/super user context.

The reason why it would it beneficial to run the operation in an administrator context is, that the operational behavior usually runs with the permissions and limitations of the user that performs the save operation. This also means the operation fails, if the permissions are preventing it. This limits what can be done. For example it is impossible to change the access context of an item in a way that makes it inaccessible to the user performing the operation. The operation would fail. Elevating the operation to run in an administrator context, would allow such an operation to be successfully executed, regardless of the repository access and permissions of the regular user triggering the operation.

I would suggest to not try to use attribute customization for these use cases.

Summary

As a summary, attribute customization, especially using JavaScript can be used for some interesting purposes. There are however a lot of limitations in the JavaScript API that prevent several use cases. The biggest benefit is, that it can be used without having to deploy anything on a server. This allows it to be used in cloud deployments, where the users have no access to the server file system.

For more complex problems, the Eclipse plugin Extensibility is a better option, regardless if it is the Java based attribute customization, or Eclipse plugin based EWM extensions, just because the APIs provided in the EWM SDK are so much richer.

This was quite some effort. As always, I hope that this is useful to someone out there.

Advertisement

Debugging EWM Work Item Attribute Customization JavaScript

I don’t know why I have delayed this post for so long. Anyway, I am working on some workshop material and part of the workshop is RTC/EWM Work Item Attribute Customization using some simple JavaScript scripts. I often see questions about this type of attribute customization on the Jazz.net forum. Since it is pretty much impossible to remote debug scripts for others for various reasons, I want to provide some short introduction how anyone can use the built in Developer Tools of a Firefox or Chrome browser to debug the scripts.

Open the Project Area

Pick a browser. The latest Chrome or Firefox will do. Open the project area in the browser.

Debug mode

The JavaScript code is compressed when it is executed in the browser. To be able to see the scripts code as it is executed, it is necessary to enable the debug mode. This is done by injecting

?debug=true

into the URL used to access the CCM application. For example access the Work Items menu of the project area that has the attribute customization. Into the resulting URI below

https://elm.example.com:9443/ccm/web/projects/JKE%20Banking%20(Change%20Management)#action=com.ibm.team.workitem.viewWelcome

Inject ?debug=true in front of the hash # like so:

https://elm.example.com:9443/ccm/web/projects/JKE%20Banking%20(Change%20Management)?debug=true#action=com.ibm.team.workitem.viewWelcome

Make sure the page loads and the injected code is still showing in the URL.

Note: do not confuse ?debug=true with ?internal=true, which is for a completely different purpose (see the EWM Extensions Workshop the deployment lab).

Find the Scripts in Chrome

Once you have done that, open a context where the scripts would be available. For example create a work item of the specific type of interest that has the attributes.

For Chrome use the three vertical dot menu close to the profile, navigate to the More tools menu and then select the Developer tools menu.

The shortcut is CTRL+Shift+I.

The Developer tools can dock in different locations. You can control where you want it to be docked using the Developer Tools Dock side settings.

I have found the scripts in different locations over the years, at least I believe so. I really used to struggle to find them in Firefox. In Chrome it was really easy to spot, at least in my environment which made it my go-to debugging browser. This location has recently changed for some reason unknown to me. But this time I was able to locate the scripts in the folder top/<publicURIroot>/ccm/web/projects.

To locate the scripts select Sources and select Page if necessary. Then expand the folders until the scripts become available in the folder top/<publicURIroot>/ccm/web/projects. Where <publicURIroot> is the public URI root of you ELM server, that is encoded in the first section of the URL you have open just now.

Click on the script you want to debug. The script opens in the editor and allows you to set breakpoints as shown in the image below.

This now allows you to step through the script and debug as well as inspect the code and the current values in the script instance. The next image shows the debugger pausing at a breakpoint of the script and the how to inspect the value of the parameter attributeId.

You can step over the script statement by statement and the debugger displays the values and call results in the debugger. This makes it as easy as it can get to understand what the script is doing and why (or why not).

Find the Scripts in Firefox

Open Firefox and open the project area you are interested in. Modify the URL like above with the ?debug=true statement. Navigate to the Work Item tab of the project area.

Open the application menu (three horizontal stripes on top of each other). Select More Tools and then select Web Developer Tools

The shortcut is also CTRL+Shift+I. You can control where the Web Developer Tools dock using the customization menu.

Dock the Web Developer Tools as you please. Then open the Debugger tab.

Unfold the nodes MainThread/<publicURIroot>/ccm/web/projects. Where <publicURIroot> is the same public URI root that we know and love.

Click at the script you are interested in to open it in the debugger. Place breakpoints as desired. This works exactly like it does in Chrome.

Create a new work item or open one with scripts you want to debug and use the Web Developer tools similar to the Chrome developer tools.

Possible Issues

You might run into issued if you try this. E.g. the projects folder does not contain any scripts. This can be the result of various reasons. Here what I have run into.

1. Did you enable Attchment Scripts in the Advanced Properties of the CCM server?
2. Check by using a default filled in example script. Maybe your script is too corrupted/wrong to be loaded. You might want to consider to check the log file if there are load issues.
3. If you just changed the scripts and still see the old version, make sure to force a reload e.g. Using STRG+F5 or otherwise reloading the page to pick up the new process.

Summary

I am at a loss, why I have not published this so far. Better late than never, this is going to be my URL for questions in the future.

I hope this helps users out there to smooth their development for attribute customization scripts.

The EWM / Rational Team Concert Extensions Workshop and recent Eclipse Versions

I updated the EWM / Rational Team Concert Extensions Workshop recently for 7.0.x and for 7.0.2SR1. There is an issue when using the P2 install explained in the Workshop with more recent Eclipse Clients. I think I have seen this with the 6.0.6.x as well as with the 7.0.x versions, dependent of the Eclipse version I used.

EWM P2 install with more modern Eclipse client versions

Eclipse client fails to connect with the EWM Server with the error message “Could not get value of field private transient sun.util.calendar.BaseCalendar$Date java.util.Date.cdate in 2022-07-31 09:37:08.301”. I was recently contacted by a colleague Thomas, who provided a solution for this issue. The solution is explained in this forum question. I added the following lines to the eclipse.ini file:

--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED

As an example in the image below.

This solved the issue for me for Eclipse 2022-06 and might be a solution for other versions as well.

I tried several other things such as different Java versions, but I did never find a really reliable solution that allowed to stay close to the workshop documentation.

Update

After discussing the findings with some of our developers, I found out that there was actually a setting in the eclipse.ini, that defined using a Java 17, which overrode my -vm setting to use a different Java version. The setting was buried somewhere in the middle of the ini file and looked like this:

-vm
plugins/org.eclipse.justj.openjdk.hotspot.jre.full.win32.x86_64_17.0.3.v20220515-1416/jre/bin

This specific setting also obscured that there is a Java shipped with Eclipse. At least after removing that and providing Java 11, the error does no longer happen. The settings above are only needed to allow the operation in Java 17 that is still allowed in Java 11. The defect that causes this is Illegal reflective access operation in EWM Eclipse Client (P2) (547430)) and fixed in 7.0.3. After removing the duplicate -vm with the Java 17, the issue went away.

UI Changes in newer Eclipse versions

When adding the plugins to the search path the UI has changed in more recent versions. The menu item “Add to Java Search”, used in in Lab 1.3 and 1.7, is renamed from to “Add to Java Workspace Scope” in newer Eclipse versions like the one above.

Caveat with the P2 Install

Please note that there is still an issue with the P2 install of EWM. After the install it is impossible to use the Help->Install New Software and the Marketplace. There is a plugin that causes an error as Thomas rightly pointed out to me.

Feature based Launches

Every now and then, I have seen issues with installing the Feature Based Launches in certain Eclipse versions. The Launch options added by the Feature Based Launches (such as OSGI2 and Eclipse2) sometimes do not appear to be working. They do not show up in the Debug Configurations.

If this is the case there is a reasonably safe approach that works. Install the latest Eclipse EWM Client that ships as a zip with EWM/RTC bundled. As an example unzip it into C:\RTC702SR1Dev\installs\TeamConcert\. Adding the dropins folder and the Feature Based Launches has always worked for me so far. The dropins folder needs to be the folder C:\RTC702SR1Dev\installs\TeamConcert\jazz\dropins if you unzip the client that way.

Import the RTC Client Feature

Unfortunately there is another caveat with this approach. In 1.8__53 Import a feature to make launching the RTC Eclipse client much easier the path to import the feature com.ibm.team.rtc.client.feature is C:\RTC702SR1Dev\installs\TeamConcert\jazz and not C:\RTC702SR1Dev\installs\TeamConcert\eclipse with the example install above. If you have done this successfully, you can use the standard EWM Eclipse client for development.

Using multiple versions of Eclipse

It is possible to use both approaches in parallel by installing into different folders such as TeamConcertVanilla and TeamConcert_2022_06. Please be aware that there will be warnings when using the same workspace with different Eclipse versions.

Summary

This blog should help with overcoming some new issues that show up when performing the EWM/RTC Extensions workshop for the latest versions of EWM/RTC. Thes solutions make the work easier for me and I hope sharing them helps others too.

WorkItem Command Line and RTC/EWM Extensions Workshop for 7.0.2 SR1

Work Item Command Line for 7.0.2 SR1

The Workitem Command Line had dependencies to the plain Java Client Libraries, including to Log4J1 in a few places. This breaks the Work Item Command Line for IBM Engineering Lifecycle Management 7.0.2 SR1 especially IBM Engineering Workflow Management 7.0.2 SR1 (EWM). WCL 5.3.1 and earlier will not work with the 7.0.2 SR1.

It was requested to support 7.0.2 SR1 with the WorkItem Command Line. I had a look and was able (I think) to update the code of the WCL to work with the new Plain Java Client Libraries. This removes the dependencies to Log4J1 and change the functionality to use Log4J2. The code is on a different branch named Log4j2, because it is incompatible with previous versions of EWM. A new pre-release was created and is available here as Work Item Command Line 6.0 prerelease. I have only done minimal testing. If you have an opportunity to run it, please report issues.

EWM/RTC Extensions Workshop for 7.0.2 SR1

To be able to perform the updates I had to perform (most parts of) Lab 1 of the Rational Team Concert Extensions Workshop. The workshop still works. There are some smaller issues and here is my erratum so far.

• I downloaded Eclipse IDE 2022-06 R and installed EWM into it. Unfortunately there is an issue when connecting to the EWM server. The error is “Could not get value of field private transient sun.util.calendar.BaseCalendar$Date java.util.Date.cdate in 2022-07-31 09:37:08.301”. Here is an approach how that can be solved.
• When adding the plugins to the search path the UI has changed and the menu item in the Eclipse version above is named “Add to Java Workspace Scope”.
• When trying to create the Test Database with the JUnit launch, the test ran out of memory. I increase the memory settings in the launch.

Summary

The changes to the ELM 7.0.2 SR1 break the Workitem Command Line and require a new setup of the Extensions workshop to develop extensions. The WorkItem Command Line is available as a prerelease for ELM 7.0.2 and the Extensions Workshop seems to be working with minor limitations.

EWM Reportable REST API

Finally, to close the series of posts, there is another public REST API available for EWM. The Reportable REST API provides information about resources that may be useful in reporting, and provide some features not available in a “normal” REST API. This API is intended to be consumed in a number of scenarios, including:

• For direct consumption by a reporting or document generation tool such as Rational Publishing Engine (RPE).
• The ETLs (data collection jobs) in RTC use these APIs as a source of data to deposit into the data warehouse.

Context of the blog post is the series

This is the series of planned posts I intent to publish over time. Most of the examples will be EWM based, but quite a lot of the content applies to more ELM applications.

• Using the EWM REST and OSLC APIs 
• ELM Authentication 
• EWM Discovery 
• EWM Work Item OSLC CM API 
• EWM OSLC Query API 
• EWM REST API to access existing Work Item Queries
• EWM Reportable REST API – this post

External Links

I used the following link for exploring this mechanism.

• EWM Reportable REST API
• DNG Reportable REST API – v6.0 or higher
• RQM Reportable REST API
• RMM Reportable REST API

Not OSLC

This series mentions OSLC a lot. The Reportable REST API is not an OSLC API. It uses different representations and no headers.

The Reportable REST API

I did completely miss the reportable REST API for years. Some 2 years ago, I helped a team working with a customer to access data that is not accessible in OSLC. The data is specifically the approval data for a work item.

I developed a solution prototype to access this data using a web microservice based on the plain Java Client Libraries API. When I did a presentation for the Engineering Community of Practice about that prototype, a colleague in the audience pointed out to me, the approval data would be available in the Reportable REST API. So I checked and they where right.

This made the microservice I had done part of the final solution unnecessary. The usage of the Reportable REST API had several additional advantages. It did not require its own application server, no updates for newer versions. Authentication, security, role based access was all supported. The solution was easier, cheaper, faster.

Although the documentation for the Reportable REST Service is quite good, I think some hints and examples might be useful.

Root URL

Assuming PublicURI being the public URI of the CCM server e.g. https://elm.example.com:9443/ccm the URL for the EWM Reportable REST API is

PublicURI + '/rpt/repository'

Supported Resources

The EWM Reportable REST API supports the following resource types.

• foundation: Common artifacts such as project areas, team areas, contributors, iterations and links
• scm: Source Control artifacts such as streams and components, as well as stream sizing deltas
• build: Build artifacts such as build results, build result contributions, build definitions, and build engines
• apt: Agile Planning artifacts such as team capacity and resource schedules and absences
• workitem: Work Item artifacts such as work items, categories, severities, and priorities

For each resource name ResourceType from the selection above, the URL is

PublicURI + '/rpt/repository/' + ResourceType

As an example for the workitem resource the URL is

https://elm.example.com:9443/ccm/rpt/repository/workitem

Trying to GET the URL above does not return any detailed data. The reason is, that the properties/attributes to display are not yet specified.

Resource Schema

For any of the supported resources a schema can be requested by a HTTP GET on the resource type URL of the resource type and the query parameter
?metadata=schema appended. As an example the GET below gets the schema for the work item resource type (no header required).

GET https://elm.example.com:9443/ccm/rpt/repository/workitem?metadata=schema

The schema information sent back looks like below.

Part of the work item resource type schema.

Please understand the Common Properties and the elements available for all the resource types by checking the wiki page Reportable REST API.

Please also check the Examples. The reportable REST API provides a query language that allows to select items based on certain criteria that are based on the schema.

As an example this query below finds the work item with the Id 1 and provides the id and the summary property for the work item.

https://elm.example.com:9443/ccm/rpt/repository/workitem?fields=workitem/workItem[id=1]/(id|summary)

The part workitem/workItem selects all work items. The part [id=1] selects the work item with id 1. Removing this term would query for all work items. The section /(id|summary) selects the work item properties/attributes id and summary to be returned. The image below shows this executed.

Execution of a query for the work item with id 1.

Paging

The reportable REST API supports paging. The documentation for EWM/RTC does not document how, but The DNG Reportable Rest API does.

The pagination supports these two query parameter to set the size and the size as well as the position (controlling the page).

size=10

size=10&pos=10

By default the pagination is set to 100, but other values are supported. The query parameter has to be added with a prefix ? or &, dependent on the parameters position in the URL.

The response data provides the URL for the next page. The reference is sent in the first block (see the blue box above). Please note that you can not use the URL as it is. It is necessary to url decode the URL first. The image below shows the URLs with paging information. The URL QueryURI is called. The code then gets the nextPage URL from the result, URL decodes it, and executes the next page.

Code to get the next page from the query. Page size is set to 10.

Please note that the URL for the next page contains a reference (id=_7dgCcJl2Eey2fKi1TcCeOA) to the previous query that can be used to call the next page. The URL in the call below, that shows this, was urldecoded from the response shown above.

GET https://elm.example.com:9443/ccm/rpt/repository/workitem?id=_7dgCcJl2Eey2fKi1TcCeOA&fields=workitem/workItem/(id)&size=10&pos=10

Nested Property/Attribute Data

The properties/attributes to be displayed can be nested. Below is a more complex query that gets approval data for the work item 1 and displays the details about the approval data:

GET https://elm.example.com:9443/ccm/rpt/repository/workitem?fields=workitem/workItem[id=1]/(id|summary|approvalDescriptors/(id|typeIdentifier|typeName|name|cumulativeStateIdentifier|cumulativeStateName|dueDate|approvals/(stateIdentifier|stateDate|stateName|approver/(name)|approvalDescriptor/(id))))

The query selects work item with id 1. For the selected work item(s) it returns the properties/attributes work item id, the summary and the approval descriptors. For the approval descriptors it returns the id, typeIdentifier, the typeName, name, cumulativeStateIdentifier, cumulativeStateName, dueDate and the approvals. For the approvals within, it returns the stateIdentifier, stateDate, stateName, the approvers and the approvalDescriptors. For the approvers it returns the user name. For the approvalDescriptors the ID is returned.

The / behind an item type refers to an instance of that item type. The term in brackets () specifies which attributes should be returned in the result. The attributes are separated by |. If an attribute is a complex type it can be referred to with a nested / and a specification which attributes to return in a nested () statement.

See the work item examples for more choices.

The image below shows a part of the complex values returned by the query above.

The complex nested structure of the query above.

The image below shows the work item approval data the query reads.

Work Item Approval data.

It is obvious you can query and create complex data sets that can be used for reporting, publishing and other uses.

Summary

This concludes my mini blog series about EWM OSLC and REST APIs. As usual I hope that I am able to help someone out there and make their work easier. IF you have suggestions and feedback, please leave a comment.

EWM REST API to Access Existing Work Item Queries

After looking into the EWM OSLC CM API, including the OSLC Query API, a natural question that comes up is: Is it possible to take advantage of the existing EWM work item queries? Is it possible to use that query mechanism and run existing work item queries to get the result set back?

I had to look into that for a customer and an internal project and where experiencing some difficulties. So I think it is valuable to share my experiences and examples that finally worked for me.

Context of the blog post is the series

This is the series of planned posts I intent to publish over time. Most of the examples will be EWM based, but quite a lot of the content applies to more ELM applications.

• Using the EWM REST and OSLC APIs 
• ELM Authentication 
• EWM Discovery 
• EWM Work Item OSLC CM API 
• EWM OSLC Query API 
• EWM REST API to access existing Work Item Queries – this post
• EWM Reportable REST API

External Links

I used at least the following links for exploring this mechanism.

• Resource Oriented Work Item API with OSLC_CM 1.0 especially the examples for how to use built in query. For completeness it might be worthwhile to check the experimental API introduced in RTC 1.0.1
• Be aware of some characteristics and limitations discussed in work item 522500

EWM Stored Query API Introduction

EWM provides its own REST API that supports finding and running predefined, shared or personal work item queries that are stored in the repository. The API allows to query for all the stored work item queries available. The API allows to run such a query and to access the result information to present the work items and their data. This API is EWM only as far as I can tell. It is not documented to be available for any of the other products. It is also important to note, this API is not an OSLC API, even if the query URL contains oslc as part of the path.

The examples for how to use built in query are nice to have, but I found the documentation was lacking essential information. Maybe it is jut me and it comes naturally to all the native cloud wizz kids. I struggled to get the examples to work. One pattern I have now seen several times in documentation and examples is a lack of information about required and supported headers and how to URL encode query parameters. I will try to explain what worked for me. The documentation has recently been updated and now contains more details compared to when I first explored it.

For this specific part of the API, my observation was that getting the query parameters wrong, often resulted in the query returning results, but not the expected results. Usually the result set way way bigger than expected in those cases.

One important behavior that is very sensitive to the Accept header is how the information is returned and presented in the responses. There are two completely different pattern.

• Only the URLs to the results are available in the response. To get any details about the result referenced by the URL, it is necessary to query that URL. This corresponds to the situation for OSLC queries without or an empty oslc.select statement.
• Additional information is provided inline in the response. It is not necessary to query the result URI to be able to access the most relevant information. This corresponds to the situation for OSLC queries with an oslc.select=* statement.

When I looked at the API first, I missed information about the valid headers. As far as I can tell, the valid headers are now updated in the wiki page. The supported and working headers are mentioned below as well.

Headers and performance

Which header to use seems to be a quite unimportant difference, but it can have a huge impact on the communication performance. Lets create a Gedankenexperiment. Lets assume we perform a request to the server. We want to search in the response data to get the URI to a query with specific property values like a project area, query name and creator. Lets assume the following characteristics for the request:

• Result set size is 130 items
• Paging size is 50 items

To process the response with the properties inline it is necessary to run 3 requests at most. If the desired information is on page 1, only one request would be necessary. Worst case all related data needs to be sent and received.

To process the data without the properties inline it is necessary to load a page. Then it is necessary to run another request for each of the URLs in the result set page and process the received data. In worst case this requires 3 + 130 = 133 requests. Worst case all related data needs to be sent and received as well.

Why is that a problem? Worst case, if the desired data is in the last link processed, both methods have required to transfer the same amount of data. In many cases one could argue the 1st approach would even require more data to be transferred on average than in the 2nd approach.

However, in my experience the information transfer between two network nodes is significantly slower than the time that would be required in a server to get the desired information. Each message that needs to be sent causes a considerable amount of overhead and delay. It is usually cheaper to send fewer messages than many small ones that transfer a similar amount of data, just because of the overhead required for each message. In addition all the subsequent requests for the details cause activity on the server which is likely to cause a lot more overhead compared to collecting all the information for the 3 initial request(s).

As an real world example, when I started to look into this, I did not know which headers where available. I used the headers I had often used e.g. ‘application/rdf+xml’. In addition, the query that I created had an issue. Instead of only returning the queries for a specific user in a specific project area, it returned all queries for the whole server. The responses only contained the query URLs. To get the name and creator of the query each URL had to be requested. It was also not obvious why the result set was so large and I had no code to recognize that results were related to project areas other than the expected. The server I ran against is on another continent and I was in my home office. I stopped the experiment after an hour or two.

At the end I was able to create solution that performs well and reduces overhead to a limit. But it was no means a simple process. I reached out to someone for hints. As a customer I would not have had the opportunity. This is why I try to finally share this information.

Stored Query API URL

I was not able to find a way to discover the URI to get the work item queries. The Wiki page mentions to find the simpleQuery tag in the project areas service provider, but I seem to be unable to find it, so I need to construct it based on the wiki page. From the Resource Oriented API page I can deduce two possible forms for the URL. Assuming PublicURI being the public URI of the CCM server e.g. ‘https://elm.example.com:9443/ccm’ the URL for the Stored Query Collection is either

PublicURI + '/oslc/queries'

or

PublicURI + '/oslc/queries.xml'

Both seem to work. Choose one you like and that works for you.

Query Stored Queries

The request below gets all the stored queries for a repository.

Get all stored queries for the repository

Note that the only Accept headers documented to be valid at the moment are:

Accept text/xml
Accept text/json

It is necessary to be authenticated to the ccm server to be able to perform the operation.

The image below shows a part of the response to the query above.

The response to the query for all stored queries

The first section shows the query for the next page in the attribute oslc_cm:next and the total result count in oslc_cm:toatlCount.

The rest of the information is one element rtc_cm:Query for each query descriptor. The element contains the query URI as rdf:resource, the identifier for the query, the Query name/title and description and the project area the query belongs to. The most important information is the rtc_cm:results element with the URL to execute the query. As an example the URL below:

https://elm.example.com:9443/ccm/oslc/queries/_87dqoFpmEeukW7cqqDjAuA/rtc_cm:results

Additional information is the date of the last modification and the user that created the query.

Note that the result is either XML or JSON. The tools for RDF that were used in some of the previous posts using the Python rdflib do not work in this context. Instead the Python code shown in the blog uses xml.dom.minidom to access the information in case of XML. The JSON code examples shown in previous posts using the json library can be used as a reference. This post will mostly use the Accept header text/xml. The code depends on the following libraries.

The main Libraries used to process the stored queries API.

Some example code for processing the data in Python will be shown below.

Narrow Down Stored Queries

It is usually not desirable to query for all stored queries for all project areas. A real production server can have hundreds of project areas, team areas and users that can have predefined or shared or personal queries defined. The result could be thousands to ten thousands of queries. To collect and transfer all that data is a challenge and will drain server resources as well.

It is possible to filter for all the properties that make up a query, identifier, name, creator, owning project area. The code below computes the query parameters from the parameters passed and creates a query section for the query URL.

Create the query parameters to narrow down the query results.

As an example the query that is created could look like below. The condition can narrow down to the project area, the user and the title, dependent on how much information is provided. If enough information is given, this would usually only return one hit. Note that this is a string terminated by the singe quote and the double quote characters inside are important. The query narrows down to one project area, the query name ‘All’ and the creating user.

'rtc_cm:projectArea="_8e5qfFpmEeukW7cqqDjAuA" and dc:title="All" and dc:creator="https://elm.example.com:9443/jts/users/ralph"'

How to discover the project area UUID has been explained in previous posts e.g. about discovery. The code that was used here is the same used in the other posts.

The code blow shows how the query URL is composed.

Python code to compose the query parameters and to create the query URL

The parameters provided are the same as above, the project area, the query name and the query creator. The creator can be provided in different ways. It is possible to provide a user URI like

'https://elm.example.com:9443/jts/users/ralph'

It is also possible to provide the variable string below.

'{currentUser}'

The query mechanism replaces this variable with the URI of the user that runs the request. For example, the query below would return the query named All created by the current user.

'rtc_cm:projectArea="_8e5qfFpmEeukW7cqqDjAuA" and dc:title="All" and dc:creator="{currentUser}"'

The last three lines in the image above compose the query URL. First the base query URL is composed. This is the part of the query that returns every stored query. Then the query parameter section is composed by adding ‘?oslc_cm.query=’ and then adding the URL encoded query parameters. The resulting query URL looks like below.

'https://elm.example.com:9443/ccm/oslc/queries.xml?oslc_cm.query=rtc_cm%3AprojectArea%3D%22_8e5qfFpmEeukW7cqqDjAuA%22+and+dc%3Atitle%3D%22All%22+and+dc%3Acreator%3D%22%7BcurrentUser%7D%22'

Execute the Query for Stored Queries

The code below executes the query to search for stored queries.

The first line runs the query that was just created using the function findQueryResources().

Execute the query to find stored queries

This function is shown in the image below. It uses the Accept header ‘text/xml’. While there is a valid query URL for a page, it executes that query.

The code that executes the query for the stored queries and collects the results

It calls qeryQueryInfoXML() to process the page result data. This code is shown below. The function builds up two arrays, one containing the stored query URL and one the associated query name. It also gets the query URL for the next page, if there is any, and returns that.

Get the information about the queries from the response

All query pages are iterated and the results added to the respective arrays.

Evaluate the result and

At the end of executing findQueryResources, there is a list of query names and an associated list of query URLs. Dependent on the query parameters given, it could return many or no result. The code here as shown below focuses on using the query name, but it would be easy enough to extract more information per stored query to use different criteria to look into the result set. The code tries to find the first stored query returned with the given name.

Find the query to run and execute it

If a stored query has been found, that query is finally executed in the function execute_query_xml(). The code for this function is shown below. Again, this uses the Accept header ‘test/xml’ and the results are processed as XML.

Process each result page of the stored query

The code iterates the query pages and is similar to the code already shown above. The code passes each query result page to the function processQueryResultInlineXML(). The code is shown below. It also looks very similar to the code we have already seen. The difference is that the code below now works on work item resource information and not on query information.

Processing one page of the stored query with work item data inline.

The code analyzes the oslc_cm:Collection and gets the total result count and the next page (if there is one). Then it analyzes the inline XML data for all the work items. For each work item it gets some of the available information and prints that. It returns the work item URIs, the totals and the next page etc.

Executing Stored Query Using JSON

The code here shows a JSON based implementation of the functions to execute the stored query after finding it. The function execute_query_json() does the same the function execute_query_xml() does and looks very similar. It is only using JSON as format to parse the data. The Accept header used below is ‘text/json’.

Process all query pages in JSON

The function getQueryResultDetailsInlineJson() is called to process each page. Like its XML twin it gets the total count and the next page (if available). Then it gets details about the embedded inline work item data and logs it. Like the XML based code, the function builds an array with the work item URIs for the page. It also returns additional information such as the link to the next page.

Process one query page with JSON work item data inline

More information

Please check the section Getting Resource Representations to understand additional ways to get work item data representations. Also check the section Getting Partial Representations to understand options to limit the data that needs to be transferred.

CSV Export

The query descriptor URL provided in the search by the tag rtc_cm:Query, can also be used in a browser to get the query displayed. As an example see the information below.

<rtc_cm:Query rdf:resource="https://elm.example.com:9443/ccm/resource/itemOid/com.ibm.team.workitem.query.QueryDescriptor/_pdT5AATBEeyiDIlAjMF45Q">

Opening the URL in the rdf:resource results in the following display in the browser.

The query descriptor URL is opened in the browser and executes the query.

It is also possible to use that URL to download the query result as CSV file. By adding the query parameter media type e.g.

?_mediaType=text/csv

to the query descriptor URL, the query can be changed to export the data as CSV file. As an example the URL below exports the query result to CSV.

https://elm.example.com:9443/ccm/resource/itemOid/com.ibm.team.workitem.query.QueryDescriptor/_pdT5AATBEeyiDIlAjMF45Q?_mediaType=text/csv

By pasting the URL in the browser, the CSV download is triggered. It is also possible to use the URL above in tools like cURL.

Summary

This blog post has described the steps that are necessary to find existing work item queries and how to run such queries to get the data. I hope that the examples here in my blog help users out there with their work.

EWM OSLC Query API

After looking into how to create and update work items the question becomes, how to find work items to begin with. OSLC provides a query mechanism to allow querying for items. This post intents to show how OSLC queries work including some examples that work for me. The techniques explained in the previous posts in this series are important. If necessary, go back to the previous posts to understand the details. As usual the focus in this blog is EWM/RTC, however, the OSLC Query mechanism works for all product supporting it. So what is explained here based on EWM will work with ETM, DNG etc.

Context of the blog post is the series

This is the series of planned posts I intent to publish over time. Most of the examples will be EWM based, but quite a lot of the content applies to more ELM applications. The examples where performed with versions 6.0.6.1 and 7.0.x.

• Using the EWM REST and OSLC APIs 
• ELM Authentication 
• EWM Discovery 
• EWM Work Item OSLC CM API 
• EWM OSLC Query API – this post
• EWM REST API to access existing Work Item Queries
• EWM Reportable REST API

External Links

I used at least the following links for exploring the OSLC Query mechanism. The latest version of the OSLC Query V3 document contains several examples which are very helpful. I still found it a challenge to get some of the queries working.

• OSLC Query V3
• OSLC CM Query Capabilities on the Jazz wiki
• OSLC Query V2 (archived)

RDF XML or JSON?

The examples in this blog are using the content type application/rdf+xml. The same examples work with the content type application/json, provided the function to parse the files are switched to using a JSON parser. I have the same examples using JSON. However the posts in this blog series are mostly using RDF+XML and I decided to stay with it, to make it easier to follow.

Query Base

Before being able to execute a query, it is necessary to discover the query base. The query base is the URI that defines the root for the OSLC query. The first step is to get the service provider catalog from the rootservices document as described in the blog post about EWM OSLC Discovery. Get the rootservices document

GET https://elm.example.com:9443/ccm/rootservices

Accept application/rdf+xml; charset=utf-8
OSLC-Core-Version 2.0

Get the service provider catalog like below. Pass the OSLC-Core-Version and the Accept content type header. This step requires authentication to be done.

https://elm.example.com:9443/ccm/oslc/workitems/catalog

The catalog lists the services available for each project area. To narrow down to a project area perform a GET on the work item service for the desired project area, providing the headers mentioned above. As an example

GET https://elm.example.com:9443/ccm/oslc/contexts/_8e5qfFpmEeukW7cqqDjAuA/workitems/services

The resulting response body contains various services that are provided for the various work item types of he project area.

• Various dialogs such as OSLC selection, creation dialogs and pickers
• The OSLC creation factories for all the work item types
• The OSLC resource shapes for all the work item types
• The OSLC query capabilities

To be able to query for work items, it is necessary to analyze the query capabilities provided by the service provider. The query capabilities can be found by searching for the oslc:queryCapability nodes.

There are usually at least two query capabilities in the work item service provider for an EWM project area. One is the query capability for deliverables. A deliverable is also referred to as a release and is a work item attribute type. The query capabilities for deliverables can be found using the resourceType http://open-services.net/ns/cm#Deliverable.

The second query capability is for work items (in OSLC referred to as change requests). The image below shows the oslc query capabilities for work items. This can be identified by the resourceType http://open-services.net/ns/cm#ChangeRequest

Query capability for work items.

The oslc:queryBase is the URI for the work item OSLC query mechanism for the selected project area. It will be used in constructing the OSLC Query URI. The query base has the form:

https://elm.example.com:9443/ccm/oslc/contexts/_8e5qfFpmEeukW7cqqDjAuA/workitems

where the second last segment is the internal unique ID of the project area.

The query capability also provides a resource shape for the Tracked Resource Set (TRS) provider. The resource shape has the form

https://elm.example.com:9443/ccm/oslc/context/_8e5qfFpmEeukW7cqqDjAuA/trs/shapes/workitems/query

The resource shape provides the information about the attributes provided by the TRS. The resource shape contains a oslc:valueShape for work items based on the work item type defect.

Value shape for work items

This provides the value shape that defines the common work item attributes, their types and the allowed values for these work item attribute value types. It is possible to GET the oslc:valueShape value shape (provide the OSLC related headers) like below:

GET https://elm.example.com:9443/ccm/oslc/context/_8e5qfFpmEeukW7cqqDjAuA/shapes/workitems/defect 

This value shape provides with all the common attribute and link types. The work item of a specific type might still have additional custom attributes. The post EWM Work Item OSLC CM API explains how to get that information using the work item type specific resource shape associated with the creation factory.

The image below shows the code that gets the query capabilities out of the project area work item service provider document.

Extract the query capabilities from the service provider document

This code shows how the query base is extracted from the service provider catalog. To do this, the code identifies the change request resource type and gets the associated query base.

Get the query base for the work items

Having the query base it is now possible to execute OSLC queries. The code below executes the most simple form of OSLC query:

Querying the query base.

The code performs the GET method on the query base.

GET https://elm.example.com:9443/ccm/oslc/contexts/_8e5qfFpmEeukW7cqqDjAuA/workitems

Accept application/rdf+xml; charset=utf-8
OSLC-Core-Version 2.0

Please note the OSLC headers Accept and OSLC-Core-Version above. The same headers are used in all calls in this blog.

Paging

The request is redirected to support paging like shown below. The paging mechanism is used to split queries with large result sets into smaller, more manageable chunks.

Paging redirect

The response body contains the work item URI’s for the first page of work items. Note, because there is no query parameter, the query only returns the work item URI’s and not any other property data. To get more properties of the resulting work items e.g. to display the summary would, in this case, require to get the work item using the URI from the response.

Response body contains work item URI’s and information about paging.

In addition the response contains information about the total amount of results and the URL to get the next page, in case the number of results exceeds the maximum items returned by this page. The image above shows the oslc:responseInfo which contains oslc:nextPage with the URI to get the next query result page and the oslc:totalCount with the total count of query results.

The code below shows how the query is executed. In the while loop a GET request to the query URI is executed. The result of the call is analyzed and this provides the next page of the query, if paging is needed. If no next page is available, the query is finished.

Execute an OSLC Query

The code below processes the result for a result page and gets the resulting work items to display them. It prints the work item URI. It also tries to get and print the identifier (Id) and the title (summary) of each work item in the query. If the identifier or the summary is not available nothing is printed. The code then analyzes the paging information. It returns the next page URL if there is one. It also returns the total count of the result.

Try to get at the result details of the query page

URL Encoding

OSLC Query provides a mechanism to create simple queries to retrieve items. These queries are sent as a URL in a GET HTTP request. There are limitations on which characters can be sent in the URL. Some characters have specific meanings in the URL. To avoid creating and sending the wrong or illegal URLs, parts of the OSLC query parameters need to be URL encoded. See some explanations about the URL encoding here:

• W3schools
• Wikipedia

In the code below url encoding is done using the function urlEncodeString as shown below.

comm.urlEncodeString('URL Encode me')

The code URL encodes the data in a way that has worked for me. I am not totally sure I understand the URL encoding in all its details and there might be issues in the code below. If I got something incorrect, please leave a comment with a suggestion to correct it.

More complex OSLC Queries

The code below uses the following constants (mind the = at the end of each term) which are used to compose more complex OSLC Queries:

select = 'oslc.select='
where = 'oslc.where='
searchTerms = 'oslc.searchTerms='
orderBy = 'oslc.orderBy='
prefix = 'oslc.prefix='

The statements represented by above constants and how they work are explained in the following sections.

The oslc.select Statement

OSLC queries support selecting the properties supposed to be returned in the query. The statement that is used for that is the oslc.select statement. The code below shows how the select statement is used. The first three lines are used to control running the queries and creating the file name for logging and can be ignored.

The interesting part is the selectString1. It defines the list of properties that should be returned for each query result. In the case below we want the type, the id, the title, the description, when it was last modified, who modified it, when it was created and who created it for each work item. This information is encoded in the selectString1 by creating a comma separated list of property identifiers. The select statement term is then created by concatenating the select statement ‘oslc.select=’ with the URL encoded version of the selectString1. This term is added to the query base as first query parameter using the separator ‘?’.

Select statement is provided to the OSLC query

When this query is run, the query result contains the properties for the work items that are returned in the query result. The image below shows the resulting data in the response.

The query result for a query using a select statement

The select statement can be nested to provide nested data. It is possible to use ‘*’ to select all available properties. The amount of data that is transferred can impact performance of the communication. The specification mentions that the servers SHOULD accept rdf:nil as single property. Both extremes are commented out in the cod above.

The oslc.properties Statement

The oslc.properties statement can be used to limit the set of properties or attributes of a work item that are considered in the OSLC request. The oslc.properties can be used in a GET request. In this case it can be used to specify which properties the requestor is interested in. The server does not have to collect and transmit all properties, but only the ones of interest.

The oslc.properties can be also used when updating a work item. This allows to perform partial updates. There is no example provided in this blog, but the syntax and encoding of oslc.properties and oslc.select are the same.

The oslc.where Statement

The oslc.where statement can be used to specify which work items to query. The first example below shows a query for the work item with the identifier 1. The select statement uses * to select all work item properties.

Get the work item with the ID 1 and return all properties.

The URL that is created can be seen below. The encoding makes it hard to understand. The important request headers are the OSLC-Core-Version and the Accept header.

The GET request with headers and URL encoding

The image below shows parts of the resulting RDF-XML for the work item with all the data for its properties.

The response with all properties

The In statement can be used to search for properties being in a range. The code below searches for the work items with identifiers being 1, 3, 9, 50. The query would run successful, if there are IDs that are not found.

Find the work items out of a list of ID’s

Please note as documented here, especially in the syntax section, the oslc.where statement only supports to compose complex conditions using the boolean operation ‘and‘. The operation ‘not’, ‘or’ or other complex nesting is not supported.

Before looking into more examples lets introduce the next statement.

Sorting – The oslc.orderBy Statement

The statement oslc.orderBy can be used to define the order for the result set. The example below finds all work items with the type defect and requests the result set to be ordered by severity (descending) and then by id (ascending). The order is defined with a prefix for the property. The prefix is + for ascending and – as descending order. Multiple order specification can be given in a comma separated list.

Search for all defects and sort by descending severity and ascending id

Unfortunately, the orderBy seems not to work for EWM 7.0.2 and the accept format application/rdf+xml. I have seen it working with application/json.

Full text Search – the oslc.searchTerms Statement

The statement oslc.searchTerms can be used for full text search. The query below finds all work items that can be fond by full text searching for the term ‘dividend‘ creates 20 hits with the JKE Banking Sample.

Full text search for ‘dividend’

Define a Namespace Prefix – oslc.prefix

The OSLC query mechanism allows to define new namespace prefixes that can be used in the oslc.where, oslc.orderBy, and oslc.select statement. My experimentation with this feature was quite problematic. What worked for me is the following:

To define a namespace create a variable with the namespace prefix, including the equals sign, and encode the URL. Then use only one namespace in the oslc.prefix definition Only define one prefix for one namespace in each oslc.prefix statement.

pdcterms = 'dcterms=' + comm.urlEncodeString('<http://purl.org/dc/terms/>')
prefixTerm = 'oslc.prefix=' + pdcterms 

The code below defines several namespaces (mind the encoding part). Based on these namespaces I create a prefix term for each of the namespaces based on ‘oslc.prefix=’ and the namespace URI. To define multiple namespaces in a query concatenate the prefix terms with ‘&’. Some discussions on Jazz.net indicate it might be possible to use a comma separated list. I could not get this to work.

Defining prefixes

To use the prefixes in a query, just include the prefixes as query parameter.

Using prefixes in queries

The query URI can become very long when providing many prefixes.

The prefixes can make the query quite lengthy.

Search For Items With a Specific Enumeration Literal Value

The image below shows a query that selects only work items that have the severity set to critical. The enumeration literal to be provided for the selection is information that can be looked up by querying the resource shape and allowed values for each work item type associated to the creation factories. For common work item attributes the available attributes and allowed values can be found using the chain resourceShape to valueShape associated with the query base.

Queries for the severity literal representing the severity ‘Critical’

Search For Items Modified After a Certain Date

The image below shows a where term that selects all work items that are modified after a certain date.

Items modified after

Compound Where Terms

The OSLC Query mechanism supports compound where terms are built form simple terms. The only supported boolean operation is and. Or is not supported, neither is ‘not’. The code below shows a query that looks for work items of type ‘defect’ and severity ‘critical’.

Items of type ‘defect’ and severity ‘Critical’

Summary

This concludes my short blog about the OLSC Query mechanism. My intention, in addition to understand it better myself, was to provide some working examples and how to create the queries. I have tried this in the past and got some where, but it was quite challenging. The documentation I had available in the past was often lacking some small but important information. The documentation for OSLC CM 3.0 has improved a lot, but apparently there are still issues you can run into.

So as always I hope I was able to provide simple but relevant examples that help users of this technology to achieve their goals and save some time.

EWM Work Item OSLC CM API

After looking into EWM Discovery, how can one create or update a work item using the techniques explained so far? This post will look into the steps that are required to create and update a work item. The techniques explained in the previous posts in this series are important. If necessary, go back to the previous posts to understand the details.

This blog was originally separated into two sections, creating a work item and then updating the work item. The first part mentioned programming and Python, the second used RESTClient to demonstrate the concepts. This was just due to the code I have written so far and due to complexity of the operations. The blog post was recently updated with Python code snippets for work item update and additional content.

Update: The blog post has also been rearranged and updated to separate getting a work item and updating a work item.

Update: The blog post has been rearranged and updated to show the usage of the oslc.properties query parameter for partial get and update.

Update: The blog post has been rearranged and updated to show how to resolve a work item and setting the resolution.

Update: The blog post has been updated with how to get the complete workflow information needed to navigate the workflow states and actions.

Context of the blog post is the series

This is the series of planned posts I intent to publish over time. Most of the examples will be EWM based, but quite a lot of the content applies to more ELM applications. The examples where performed with versions 6.0.6.1 and 7.0.x.

• Using the EWM REST and OSLC APIs 
• ELM Authentication 
• EWM Discovery 
• EWM Work Item OSLC CM API – this post
• EWM OSLC Query API
• EWM REST API to access existing Work Item Queries
• EWM Reportable REST API

External Links

I used at least the following links for exploring this mechanism.

• Resource Oriented Work Item API with OSLC_CM 1.0 especially the Updating Work Items section in the Resource Oriented Work Item API with OSLC_CM 1.0

Discover the Creation Factory

To create a work item using the OSLC API requires some information that is provided by the OSLC API and can be discovered. To create a work item first requires a URI/URL of a creation factory providing the capability to create the work item.

EWM supports customizable work item types with customizable attributes of customizable attribute types. Each work item type has its own creation factory URL that needs to be looked up to create a work item of that type.

Work item types have potentially different sets of attributes configured. Each attribute has a type from a number of available attribute types. EWM allows to create custom attribute types. Some of the attribute types are primitive and are easy to handle. As an example the summary of a work item is just a string and it is easy to provide data. Other attribute types are far more complex. Many attributes are enumeration types. Enumerations are a number of available literals with an ID and a display name for each literal. A work item usually has an owner or subscribers. These represent users that are available in the system referenced by a unique URI. Work items can be assigned to a project or team area, which are other attribute types available in a project areas process. Which process area a work item is assigned to is controlled by a Category that maps a user readable string to the process area represented by its URI.

Note, some data that shows up in the work item is not attribute based. As example the collection of reviews and approvals, the attachments and the comments are not attribute type based. Some attributes that show up are only presentations or are calculated and can not be changed. Some of the not attribute based data is provided as pseudo attribute e.g. to be able to export the data.

To create a work item it is necessary to find out which attributes of which types a work item can have and which values the attribute of such a type can have. Attribute types can be single value such as enumerations or strings or they can have multiple values such as enumeration lists or string lists. All this information is required when creating a work item.

The information which work item attributes with what types a work item type can have, can be discovered in OSLC APIs as part of discovering the creation factory. The information is accessible via a resource shape. How the discovery process works in general is described in EWM Discovery.

Reminder, for all requests use the mandatory OSLC header:

OSLC-Core-Version 2.0

To receive RDF encoded data use the Accept header and for sending RDF use the content Type header as shown below:

Accept application/rdf+xml; charset=utf-8
Content Type application/rdf+xml; charset=utf-8

The image below shows the RDF form of a part of the service provider catalog for a project area. Using the namespace

xmlns:oslc="http://open-services.net/ns/core#"

the oslc:CreationFactory entry contains the resourceType(s) and the resourceShape. The first resourceType http://open-services.net/ns/cm#ChangeRequest is used for OSLC integrations. The second resourceType provides the link to the work item type definition. The entry oslc:creation contains the creation factory URI.

RDF form of the creation factory for defect and task

To receive JSON encoded data use the Accept header and for sending JSON use the content Type header as shown below.

Accept application/json; charset=utf-8
Content Type application/json; charset=utf-8

The image below shows the JSON format of the creation factory entry for the work item type defect. The data is provided analogue to the RDF format and can also be used to access the data for the creation factories.

JSON form of the creation factory for defect

Discover Resource Type and Shape

Following the resourceType URI provides the details of the work item type. The image below shows the details for the defect. Getting the resourceType

GET https://elm.example.com:9443/ccm/oslc/types/_8e5qfFpmEeukW7cqqDjAuA/defect

with the headers introduced above provides more details. The display name is Defect and the work item type ID is defect. The data also contains the URI for the EWM project area that defines the work item type. The image below shows the result of such a GET request.

Type definition for defect

The Python code block below processes the RDF data and extracts the important data. The important take away are the predicates to select the information to be iterated.

Extract the creation factory information

The code operates on some simplifications. The code gets the work item type ID by filtering out the entry that starts with the public URI root. The code assumes that the end segment is equal to the work item type ID. This just saves the call to the resourceType URI which would contain the information. The code gets the creation factory URI and the resource shape URI.

To be able to create a work item, GET the resource shape using its URI and the headers described above. As an example:

GET https://elm.example.com:9443/ccm/oslc/context/_8e5qfFpmEeukW7cqqDjAuA/shapes/workitems/defect

The resource shape contains a number of attribute type definitions for the attributes (and link types) the work item type can have. The information contains the name (id) of the type, the title (display name) of the attribute type, the allowed values, the default value, if the attribute is read only etc.
The image below shows one example for the attribute Priority with ID priority, the default attribute literal and the link to the allowed values. The attribute is defined for values in an enumeration. The allowed values URI describes the available values defined in the enumeration.

Allowed Values

To get the allowed values for the attribute Priority perform a GET using the shape URI. As always provide the OSLC headers.

GET https://elm.example.com:9443/ccm/oslc/context/_8e5qfFpmEeukW7cqqDjAuA/shapes/workitems/defect/property/internalPriority/allowedValues

The result of the request is shown below.

Allowed values for the work item attribute Priority with ID internalPriority

Note that the allowed values, the priority enumeration literals, do not provide any information about the display name or any other details. To get the display name it is necessary to GET (OSLC headers!) each of the priority literal resource URIs and analyze the results.

To create a work item, the next steps would be to get allowed values for all required attributes and to create a request body containing this information.

Required Attributes

At the time of writing, OSLC does not provide a mechanism to get the list of attributes that are required to create a work item due to restrictions in the process. If the target system requires special attributes to be set e.g. in certain states, or expects special values, the work item creation or save will fail. To successfully create or update work items, it is necessary to know such limitations ore provide a configuration capability to adjust for such information.

Request Body

For the JKE Banking Example that is shipped with EWM, the process requires the following information to create a defect:

• Summary
• Category

To make the example a bit nicer we also want to also set the description of the work item. The summary and the description attributes are easy. This is just a string value. The category is the mapping to a process area. This is not a trivial attribute and it is necessary to get an allowed value. A valid approach is to get all allowed values into an array and to pick one value. If it is desired to find the correct allowed value based on a display value or some other characteristics, iterate all allowed value URIs and get the details.

The code below is the code that creates the payload for the request body needed to create the work item. It looks more complicated than it is, probably because of the RDF library. The code creates an empty graph, a node for the work item and then adds the predicate object configuration that represents the desired information e.g. the text for the summary and the description.

Create the payload and the work item

The payloadString serialized in createRDFWorkItem from the graph simply looks like the RDF XML below. It only contains the information about the summary, description and the category

Creating the Work Item

To create the work item, POST to the work item types creation factory URI like shown below:

POST https://elm.example.com:9443/ccm/oslc/contexts/_8e5qfFpmEeukW7cqqDjAuA/workitems/defect

Provide the required headers to declare OSLC, and the headers to control the content encoding that were already explained. The Content-Type defines the format of the request body. The request body is the payloadString created above.

OSLC-Core-Version 2.0
Accept application/rdf+xml; charset=utf-8
Content-type application/rdf+xml; charset=utf-8

The response status should be a 201. The response headers should contain the location header with the URI of the work item. The URI of the work item is the unique URI of the work item and can be used in subsequent operations to identify the work item. E.g. in GET, or PUT operations to retrieve or modify the work item. The URI is also used in the process of linking work items. The URI can be used in a redirect to open the work item editor in a web UI.

The response headers contain a new important header called ETag. The ETag value describes the state of the work item. Every time the work item is changed it gets a new ETag. When trying to update a work item it is necessary to send the ETag value from the previous GET in a special If-Match header. The server can use this value to evaluate the state of the work item the update is supposed to be applied to based on the current ETag. If the ETag in the request is not the current ETag of the work item state, the request is based on stale data. New changes where done to the work item. The server detects the mismatch and alerts the client that the change can not be executed. The client needs to refresh the work item information, GET the latest state, modify it accordingly and retry to update the work item based on this information.

The response body sent back by the server, contains the data for the new work item. The information is usually much more than the information that went into the creation of the work item. The server creates several special attributes such as the creation date, the creator and similar information. In addition attributes with default values are automatically set and will be returned.

Response body for the POST to the creation factory.

All subsequent operations on the work item should be based on this state, or a never version state that was retrieved by a GET on the work item URI.

The code below shows the creation of the work item and gets the work item URI from the results location header and the etag from the results ETag header.

The work item creation and the response analysis for RDF.

The same process can be used with the JSON format. The image below shows an example for a work item creation using the JSON encoding with the same attributes used with RDF above.

JSON payload to create a work item

JSON requires to use the headers shown below instead. The header Content-Type only needs to be set when a request body is sent.

OSLC-Core-Version 2.0
Accept application/json; charset=utf-8
Content-type application/json; charset=utf-8

Get a Work Item

The following sequence is done with the RESTClient extension installed in Firefox. RESTClient is a Firefox extension that runs in the browser. This allows you to log in to the server with the web UI in Firefox and then use the RESTClient without worrying about the authentication because RESTClient reuses the authentication done in the UI. Another Option would be Postman, but that requires to log into the CCM server as explained here. Install RESTClient and Firefox to do these steps yourself.

To get a work item we need the public URI of the work item. The public URI was returned in the Location header when creating the work item. It can also be located e.g. using an OSLC Query. The public URI has this form:

https://elm.example.com:9443/ccm/resource/itemName/com.ibm.team.workitem.WorkItem/242

where the number at the end is the work item unique ID for the repository. Note that the URI of the work item is not the URL for the work item editor you get when you copy the address URL from the browser. That URL contains information like the project area name and is not stable.

Open the Web UI of EWM. Open a work item e.g. the one created before. Get the work item URI by copying it out of the Web UI like shown below. You can also use the work item URI from the automation to create the work item.

Use copy link location on the icon to get the work item URI

To get the current state of the work item it is necessary to perform a HTTP GET operation on the public URI. Open a RESTClient and prepare to get a work item. As an example select GET as method and paste the work item URI into the URL address.

GET https://elm.example.com:9443/ccm/resource/itemName/com.ibm.team.workitem.WorkItem/242

Use the OSLC, Accept and Content Type headers as explained above.

Prepare the GET request

Push the Send button to execute the request.

The request should succeed with a status 200. Here the Request and the response headers in RESTClient in Firefox.

The successful GET request to read the work item 242

Browse the response Headers tab first. Note the ETag header. This is important for a later update.

Property Selection

Update: It is possible to use the oslc.properties query parameter to select which properties (attributes) are of interest in this OSLC request. Note that the oslc.properties require URL encoding to allow to send them in the request URL. The code shown below shows which part of the URI needs to be URL encoded. It is the the comma separated section of property definitions behind the

oslc.propertis=

The oslc.properties can be used in a GET request. In this case it can be used to specify which properties the requestor is interested in. The server does not have to collect and transmit all properties, but only the ones of interest. This saves computing resources and bandwidth.

The image below shows the result of a GET request on a work item URI, providing a oslc.properties selection. Note the response body only returns a limited amount of properties compared to the full GET above.

OSLC properties selection in a GET request.

The image below shows how the requested properties are selected using Python code.

Selection of the properties the request is interested in.

Update a Work Item

To update a work item it is necessary to get the current state of the work item. For this it is necessary to perform a HTTP GET operation on the public URI as shown above. Go back to the result of the previous section or open a RESTClient and prepare to get a work item. For the latter make sure to select GET as method have the work item URI into the URL address.

Make sure to use the OSLC, Accept and Content Type headers as explained above.

Perform the GET request

Push the Send button to execute the request.

The request should succeed with a status 200.

The ETAG in the successful GET request to read the work item 242

Browse the response Headers tab first to prepare for updating the work item. Copy the ETag header into a text editor, we need the value of the header later.

Switch over to the Response tab. Search the content for :title to locate the XML containing the work item summary attribute. Mark the response body and copy its content.

The work item summary attribute

Open a new RESTClient window. Paste the response body from the previous GET request into the request body of the new request. Copy the work item URI into the address URL. Change the method to PUT.

Search for the :title in the new request body. Modify the text enclosed in the tag e.g. to An updated summary like below.

<dcterms:title rdf:parseType="Literal">An updated summary</dcterms:title>

Add the OSLC and content headers:

OSLC-Core-Version 2.0
Accept application/json; charset=utf-8
Content-type application/json; charset=utf-8

Now add another header If-Match with the ETag value from the GET method.

If-Match "c6eebbc9-257c-335a-9726-287f39732012"

See the image below how the request looks like. Perform the send.

The update should succeed with a 200 status and contain a new ETag.

Successful update with a new ETAG.

Open the work item in a work item editor. You can do this by pasting the work item URI into the browser address field. If the work item editor is already open the editor should show the work item was changed. Refresh the editor if this is the case. The summary attribute was successfully updated:

The updated work item summary in the Web UI

Performing a subsequent GET in the initial RESTClient window provides the same information and the new ETag. The ETag will change if the work item is changed again.

Update: change a work item in Python code

The code below retrieves the work item based on its URI.

Getting a work item and the ETag for updating it.

After getting the work item and the etag, the response body is parsed into a graph that has the required namespaces bound to it. The response has the work item URI as subject in the response, this is retrieved and validated. The work item data is analyzed and some data is pulled out by querying the graph for objects with a specific predicate (e.g. DCTERMS.identifier) and the data is printed.

The code below shows the modification of the summary of the work item.

Updating the work item with If-Match header

The new summary is computed based on a timestamp. Then the value of the subject node is set using the code below.

# Set a RDF entry for a node, predicate and object.
    def setRDFAttribute(self, graph, node, rdfPredicate, rdfObject):
        graph.set((node, rdfPredicate, rdfObject))

The update header is created, including the If-Match header with the work items ETag from the get operation. The RDF graph is serialized to be sent. Then the PUT operation is performed. The change will have a new ETag, so that is received.

The response body is parsed into an RDF graph again and the information for some of the attributes is again analyzed and printed. The image below shows the console output that is created.

Successful update of a work item with the original and new ETag

Property Selection

Update: It is possible to use the oslc.properties query parameter to select which properties (attributes) are of interest in this OSLC request.

The oslc.properties can also be used in create and update requests. This can be used for partial updates. Please carefully read this section about updating work items and also check the examples that are provided. Especially for creating and removing links and custom attributes.

For using the oslc.properties query parameter, to limit the amount of properties to a subset, please check the sections oslc.select and oslc.properties in the post EWM OSLC Query API. Syntax and encoding for these query parameters are the same. You have to provide the oslc.properties as a query parameter URL encoded like shown below.

PUT https://elm.example.com:9443/ccm/resource/itemName/com.ibm.team.workitem.WorkItem/242?oslc.properties=rdf%3Atype%2Cdcterms%3Aidentifier%2Cdcterms%3Atitle%2Cdcterms%3Adescription%2Cdcterms%3Amodified%2Crtc_cm%3AmodifiedBy%2Cdcterms%3Acreated%2Cdcterms%3Acreator

As a summary, the usage of the oslc.properties query parameter allows to

• Limit the properties returned in a GET request to the selected properties. Instead of sending all properties, the server can just respond with the sub set of properties.
• Limit the properties that are considered to be received to the selected properties in PUT and POST requests. This can be used to just have to send a limited amount of attributes, and not all, to achieve a partial update. This can also used to remove properties by including a property in the oslc.properties but not sending data for it.

The link above mentions to use the PATCH method for partial updates a couple of times. I briefly tried to use PATCH to partially update a work item, but I got an “Error 501: Not Implemented”, so I assume PATCH is not available.

Work Item Workflow State change

One important work item attribute is the workflow state also abbreviated with state of the work item (not to be confused with the ETag). When checking the Resource Oriented Work Item API V2 it mentions that the state is read only. It mentions the capability to use an action to perform a state change, but there is no documentation provided that shows how that works. Digging deeper into the Jazz.net forum there are answers that hint on how it could work, but every one of the answers I found was lacking some detail. Here what I was able to come up with to make it work for me.

The work item workflow is defining the states of the workflow and actions that are valid to get from one state to another. States and actions have unique ID’s.

Analyzing the resource shape of the work item type, the state attribute with ID internalState can be located. The attribute has allowed values that can be discovered. The allowed values are the states of the workflow. As an example for one state there is a URI like below:

https://elm.example.com:9443/ccm/oslc/workflows/_8e5qfFpmEeukW7cqqDjAuA/states/com.ibm.team.workitem.defectWorkflow/com.ibm.team.workitem.defectWorkflow.state.s4

We are not interested in the state, we need the ID of the action to perform the workflow action. I have not found any simple way to discover the URI to get the workflow actions. There is however a pattern how to access the required information for workflows and actions and other information required to manipulate the workflow state of a work item.

Looking at the path of the URI for the workflow state reveals that the path section after the section workflows is the project area ID. The next path section is states and the section after that describes the workflow ID. The last section is the workflow state ID.

If we remove the workflow state ID section, and perform an GET on the resulting URI e.g.

GET https://elm.example.com:9443/ccm/oslc/workflows/_8e5qfFpmEeukW7cqqDjAuA/states/com.ibm.team.workitem.defectWorkflow

the resulting response is a list of the workflow states available for the workflow.

Please note, that I experienced issues with the headers to be used here, dependent on the headers used different information was returned in some cases. I would suggest to test using the OSLC headers. If there is a lack of information try to only use the header Accept application/xml (or application/json). The result could contain more information. Please note that in this case the format is not RDF.

If we replace the section states in the URI by actions we get an URI that allows to GET all the workflow actions.

GET https://elm.example.com:9443/ccm/oslc/workflows/_8e5qfFpmEeukW7cqqDjAuA/actions/com.ibm.team.workitem.defectWorkflow

Accept application/xml

Please note, to get the complete workflow information, use only the header Accept application/xml (or application/json). Do not use the Accept application/rdf+xml and do not use the OSLC-Core-Version or any other header. Run the call as shown above.

This way you receive the full workflow table. This contains the information of the workflow action including the resulting state. Given this information it is then possible to extract all workflow information necessary to navigate the workflow. I found this hint in this answer.

To get the information in JSON use only the header application/json.

The image below shows how the complete workflow information looks like, if you get it in the correct way.

The complete workflow information, including the result state of an action.

Once a desired workflow action ID is selected, it is possible to perform a workflow action. This can be achieved by adding a query parameter ?_action=<actionID>, to the work item URI, where <actionID> is replaced by the ID found as workflow action. The result is shown in the image below. Please note the familiar OSLC and content headers as well as the ETag for the last state of the work item are needed as well.

Updating the work item performing a workflow action with an empty resource representation.

It is also necessary to send the representation of the work item, otherwise the PUT call will fail. To get an ETag, it is necessary to do a GET, so all the information is already available. By default OSLC expects the complete representation and deletes data that is no longer available in the request.

I have tried to send an “empty” work item representation as hinted in one of the forum questions and the one shown above in the image and below as code worked for me. Note that there is no guarantee that this works with all versions and forever. An attempt to do a partial update using PATCH resulted in an not implemented response. In general see some hints on partial representations in Resource Oriented Work Item API V2.

Important Update about partial update: Please see the Updating Work Items section in the Resource Oriented Work Item API with OSLC_CM 1.0 wiki page and scan through the documentation, especially for partial update. Note the usage of the query parameter oslc_cm.properties. By providing a list of properties that is to be updated it is possible to send only part of the representation containing said properties. The oslc_cm.properties uses the same syntax used by the oslc_cm.select statement. See the post EWM OSLC Query API and check the section about oslc.select for how to create, compose and encode the parameter.

To get the minimal RDF shown here, I cut out all content and stripped it down to the bare minimum. RDF libraries should be able to serialize empty graphs as well and it should be easy enough to find a minimal empty body for JSON.

<rdf:RDF
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"> 
</rdf:RDF>

The URL below successfully updated the work item state without the need to pass additional attributes. The minimal request body above was used with the URL below and the work item state update was successfully performed.

PUT https://elm.example.com:9443/ccm/resource/itemName/com.ibm.team.workitem.WorkItem/241?_action=https://elm.example.com:9443/ccm/oslc/workflows/_8e5qfFpmEeukW7cqqDjAuA/actions/com.ibm.team.workitem.defectWorkflow/com.ibm.team.workitem.defectWorkflow.action.startWorking&oslc.properties=

With headers:

OSLC-Core-Version 2.0
Accept Type application/rdf+xml; charset=utf-8
Content Type application/rdf+xml; charset=utf-8
If-Match "fcd3b360-d152-367d-956e-d1f9f428ced4"

Note that the oslc.properties statement did not work with rdf.nil. But it was possible to omit a value as shown above to clarify that no property value would be included. IT would be possible to specify additional values in oslc.properties that would be sent in the request body. One example is the resolution – see below how that looks like.

Please note, that it is necessary to satisfy all required attributes when performing a workflow action. The new state might require additional values that you might have to provide, in which case you want to definitely use the full information from the resource.

Please also note that I am not seeing errors if the workflow action can not be done or does not exist. The response is 200, regardless of this fact.

Resolve Work Item With Resolution

One example for attributes that might become required with respect to a state change is the resolution attribute

rtc_cm:resolution 

that might be required in certain states when resolving/closing the work item. Following the pattern we discovered the resolutions can be found using the workflow URI and replacing the section states by resolutions. The resulting URI provides the available resolutions.

Please note, that I experienced issues with the headers to be used here, dependent on the headers used different information was returned in cases such as the actions above. I would suggest to test using the OSLC headers. If there is a lack of information try to only use the header Accept application/xml (or application/json). The result could contain more information. Please note that in the latter case the format is not RDF.

Since I had built all the logic to get the action information using XML (minidom) and not RDF in python, I used only Accept application/xml as header to get the resolution information and reused the code written to analyze the actions.

GET https://elm.example.com:9443/ccm/oslc/workflows/_8e5qfFpmEeukW7cqqDjAuA/resolutions/com.ibm.team.workitem.defectWorkflow

Accept application/xml

With the call above it is possible to GET all resolution information like ID’s and names. The image below shows an example. Mind the request header used.

Get the resolution information.

It is then possible to add or set the attribute value like below.

<rtc_cm:resolution rdf:resource="https://elm.example.com:9443/ccm/oslc/workflows/_8e5qfFpmEeukW7cqqDjAuA/resolutions/com.ibm.team.workitem.defectWorkflow/com.ibm.team.workitem.defectWorkflow.resolution.r4"/>

See the response from a GET response to an work item URI, containing the value below as the example.

Setting the resolution results in this additional attribute.

The example below shows all that is needed to close the work item and set the resolution updated in the same request. In addition to the workflow action as above use the the oslc.properties and define an additional property, the resolution attribute, to be provided.

PUT https://elm.example.com:9443/ccm/resource/itemName/com.ibm.team.workitem.WorkItem/241?_action=https://elm.example.com:9443/ccm/oslc/workflows/_8e5qfFpmEeukW7cqqDjAuA/actions/com.ibm.team.workitem.defectWorkflow/com.ibm.team.workitem.defectWorkflow.action.resolve&oslc.properties=rtc_cm%3Aresolution

The request URL contains the workflow action and the request body contains the data for the additional attribute to be set as follows:

<?xml version="1.0" encoding="UTF-8"?>
<rdf:RDF
   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
   xmlns:rtc_cm="http://jazz.net/xmlns/prod/jazz/rtc/cm/1.0/" >
	<rdf:Description rdf:about="https://elm.example.com:9443/ccm/resource/itemName/com.ibm.team.workitem.WorkItem/241">
	<rtc_cm:resolution rdf:resource="https://elm.example.com:9443/ccm/oslc/workflows/_8e5qfFpmEeukW7cqqDjAuA/resolutions/com.ibm.team.workitem.defectWorkflow/com.ibm.team.workitem.defectWorkflow.resolution.r3"/>
	</rdf:Description>   
</rdf:RDF>

The request headers, including the If-Match header, have been shown already and show up in the image below which shows all the important data in RESTClient:

Resolving the work item with setting the resolution attribute.

This is a complete example of a state change while changing the resolution attribute in addition. This can be expanded on to set more properties using oslc.properties.

I found numerous questions around this in the Jazz forum, but there was still some considerable amount of work to be done, to get the state change working.

The page Resource Oriented Work Item API V2 also describes other interesting concepts like query mechanisms and representations. Some of them will be presented in subsequent upcoming posts. Something I have not yet explored are the draft work items. Intriguing. Maybe I find time to look at it.

Summary

I am fully aware that I am quite late to the party, but despite trying, I was unable to find examples for the whole process that were consistent and simple enough, covered most of the ground or worked for me. There was quite some research that needed to be done to get all the concepts covered. All the information in the blog series I am currently writing has been collected to help customers as well as IBM internal resources with the ELM APIs.

So, as always I hope that the content of this blog (series) is of interest to users out there and helps to save some time achieving their goals.

EWM Discovery

Discovery is the name for the method to locate the entry points for the OSLC API in EWM and other ELM tools. The mechanism is the same for all applications but there are differences in the details. This post aims to help understanding the discovery process with a focus on EWM and work items. Ultimately we want to be able to create a new work item and need to get all that is required to do that.

Context of the blog post is the series

This is the series of planned posts I intent to publish over time. Most of the examples will be EWM based, but quite a lot of the content applies to more ELM applications. The examples where performed with versions 6.0.6.1 and 7.0.x.

• Using the EWM REST and OSLC APIs 
• ELM Authentication 
• EWM Discovery – this post
• EWM Work Item OSLC CM API
• EWM OSLC Query API
• EWM REST API to access existing Work Item Queries
• EWM Reportable REST API

External Links

• Here is another interesting article about discovery.

Rootservices

The main entry point into EWM and other Jazz applications is the rootservices document. The document is an XML document that is based on RDF. The document can be accessed using a HTTP GET on the rootservices in the context root of the ccm and other applications. For example:

GET https://elm.example.com:9443/ccm/rootservices

The document is not password protected and does not require special headers to be accessed. It is only available as RDF XML document, there is no other way or format to get it. It is possible to use the URI for the rootservices document directly in the browser to display it.

The rootservices document contains information about all the resources and services provided by the Jazz Server.

Service Provider Catalog

To create a work item we need to find the Work Item Service Provider Catalog. We are interested in the rdf:resource entry of the element oslc_cm:cmServiceProviders.

There are certainly many ways to achieve this. The best way to achieve this is to use RDF. RDF support exists in various languages. I am not the best person for the job to explain RDF. My summary would be that RDF defines subject, predicate, object relationships in a Graph based on XML. Once the graph is created, it can be queried. For example, if one has the subject and the predicate it is possible to locate the objects.

The trick is to understand what the subject, predicate and object could be. I have always struggled to figure that out on my own, just looking at the RDF XML document. I have found two possible solutions, that work for me.

1. Serialize the graph as N-Triples and look into these
2. Even better, serialize the graph as Turtle format and look into that

Here is a part of the rootservices document, that was parsed as RDF XML serialized into Turtle.

Turtle serialized rootservices document showing the subject

In this document, line 19 shows the subject. Please also note line 17 that shows a prefix definition we are interested in. The next part of the document of interest is shown below.

Turtle serialized rootservices document with predicate and object we are after

Line 100 shows the object we are after, the URI

https://elm.example.com:9443/ccm/oslc/workitems/catalog

The predicate to locate the object is

oslc_cm1:cmServiceProviders

With Line 17 above the predicate has the following namespace prefix

@prefix oslc_cm1: <http://open-services.net/xmlns/cm/1.0/>

The lines 104 to the end of the file show additional subjects and their predicates and objects.

In Python, it is possible to use the library rdflib to work on RDF XML. A core communication support library that I have developed for this define the URIs and namespaces for the used domains. Also see the blog post Using the EWM REST and OSLC APIs for more information about this. The documentation for rdflib can be found here.

The image below shows an example for defining the namespace used above.

Defining a new namespace.

The image below shows the import statements used for rdflib in my communication library that has all the RDF support infrastructure.

Imports for RDF XML rdflib

The image shows the code that is used to serialize the rootservices document and the function to get the service provider catalog. The last argument is the predicate that locates the work item service provider catalog we are interested in. In this example the predicate is

comm.oslc_xml_cm1_ns.cmServiceProviders

Which translates to

http://open-services.net/xmlns/cm/1.0/cmServiceProviders

or

oslc_cm1:cmServiceProviders

used here to search for the service providers.

Searching the service provider catalog

The function below composes the subject and then gets the objects selected by the subject and the predicate.

Get the objects based on the subject and predicate

The function creates the RDF graph. This binds all the namespace definitions and aliases.

Then the rootservice document is parsed based on this RDF definition to create the graphs content. The code creates the URI for the rootservices document that represents the subject – it is basically the URI of the rootservices document. Finally the code gets all objects selected by the subject and predicate, and creates an array of these URIs.

The code segment shows how the work item service provider URI is then taken out of the result array. Please note that all these activities did not require a login, username, password or anything in addition to the rootservices document.

The image below shows how the RDF graph is created and the desired aliases and namespaces are bound to the graph.

Create the graph by binding the namespace aliases and URIs

Work Item Service

We just discovered the work item service provider catalog shown in the GET command below. The next step in the discovery chain is to look up the work item services for the project areas. This is done using the service provider catalog that was discovered above.

GET https://elm.example.com:9443/ccm/oslc/workitems/catalog

and the listed OSLC request headers.

Accept application/rdf+xml; charset=utf-8
OSLC-Core-Version 2.0

Note that this request is for a protected resource. The server will redirect to the authentication. The details are explained in the previous post ELM Authentication.

The resulting RDF XML document contains the information we are interested in. The image below shows the Turtle serialization of the service provider catalog. It shows a structure similar to below.

Turtle format of the service provider catalog

The section with the predicate oslc:ServiceProviderCatalog shows the project areas services documents. For some purposes this might be enough. It is possible to iterate all the service provider catalogs to get more information for each one performing a GET on the URI. However, the response we already got contains more information that can be used.

Service Provider details

By searching for the objects that have the predicate oslc:ServiceProvider instead, it is possible to access the URI for the service provider, as well as additional information such as the project areas name using the dcterms:title. This can help reducing the number of subsequent calls to get these details.

The code below shows how to get that information from the serialized document. It builds arrays of project area names and URIs which is later used. E.g. find the index for a project area name and then get the related URI. There are likely better ways to store the information in Python.

Retrieve the service providers

The first step is to look up the subject for the predicate oslc:ServiceProvider. The resulting URI is the service provider for the project area. For example

https://elm.example.com:9443/ccm/oslc/contexts/_8e5qfFpmEeukW7cqqDjAuA/workitems/services.xml

As explained above, the code uses the found subject to get the project area URI in the oslc_cm1:details and the project area name in the dcterms:title attribute.

Project Area OSLC Services

In EWM, work item services are project area specific. The reason is that each project area can have a different process and the process defines the work item types, attributes, workflows and all that. The next steps in the discovery process would be to get the project areas services.xml.

GET https://elm.example.com:9443/ccm/oslc/contexts/_8e5qfFpmEeukW7cqqDjAuA/workitems/services.xml

This requires the OSLC headers:

Accept application/rdf+xml; charset=utf-8
OSLC-Core-Version 2.0

The resulting RDF XML document contains information about various OSLC related capabilities and services. By looking through the information, especially using the Turtle format, it is easy to identify.

• The OSLC query capability
• Various dialogs such as OSLC selection, creation dialogs and pickers
• The OSLC creation factories for all the work item types
• The OSLC resource shapes for all the work item types

Similar to the work item services above, it is possible to get the desired information. Here example code getting the creation factories and related resource shapes for all the work item types.

Getting the creation factories

The pattern repeats. To create a work item of a specific type, get the creation factory for that type. To get information about the attributes to create, in the work item, get the resource shape for the type. For the attributes in the resource shape get the type, allowed values and multiplicities.

Other Formats

The rootservices document is only available in RDF XML. Dependent on the tool, other formats might be supported for subsequent documents. For example using the header

Accept application/json; charset=utf-8

it is possible to try to get the JSON representation. This works for EWM in various places. No guarantee. The discovery mechanism in this case is similar to the XML example above, the only difference is that JSON is a format that is easier to read for humans in my opinion. Instead of creating a graph, it is necessary to use a JSON library to access the data.

The code below shows how to get the service providers when using JSON format instead of RDF XML.

Getting the service providers JSON

The discovery is analogue to what has been shown above with RDF. Even the structure of the code is similar, which is due to the fact that the data structure is similar. It is just in a different format that is easier to digest than RDF, at least for me.

Summary

I have struggled for a while with how to explain the discovery process. Especially how to get the data out of the RDF XML has been hard. Since I found the Turtle format, it makes a lot more sense to me. So, as always, I hope that this was of interest and helps the users and the Jazz community to make their life easier.

ELM Authentication

Authentication against a Jazz application is quite complex. When using the Plain Java Client Libraries, this is not a concern, as the ITeamRepository handles all that for you. It authenticates against the server and keeps the authentication valid over time. This is important, because application servers have timeouts that can void the the authentication of a session. A very common timeout that impacts the Jazz applications is the Liberty Profile default of 2 hours.

Running a client application against such a server requires detecting the need to authenticate and performing the authentication as well as to guarantee the current request is performed. Enterprise applications also require to work in complex setups supporting different ways to authenticate. Some examples are authentication based on Liberty Profile file based user management, LDAP based delegated authentication and user management, Jazz Authentication Server (JAS) based authentication, Kerberos and more. Implementing a client that uses the REST and OSLC APIs directly requires to develop a mechanism that handles this automatically.

Warning, this is complex!

I have tried to simplify this topic as good as possible, within my limited knowledge and experience, however this blog turned out to be very lengthy and complex. I also had to redo my screenshots multiple times to correct for changes during writing the post. Again, this ended up not being the casual blog post. You have been warned.

Approaching authentication

I have worked with the EWM Plain Java Client Libraries in the past. They provide a standard login mechanism to use and that keeps you logged in. I worked with Eclipse Lyo and used the mechanism provided with it, similar to plain Java API. I started just using an HTTP Client and I implemented a simple Login mechanism that was able to perform a Form based authentication. There was no code that would keep me logged in. When I started experimenting with Python, I just implemented a similar login mechanism that uses Form based authentication as explained in this post. In the first steps of the script, the authentication was executed, under the assumption that it was valid for the rest of the script.

That was sufficient for my experiments with my local test server. However it does not work with systems that have JAS enabled. This created questions, such as which authentication method was available and how to detect that, that required deeper knowledge about how the authentication was supposed to work. This approach also does not work for scripts that have to run longer durations. In many environments, the authentication expires after some timeout.

I started reading the Native Client Authentication article from the Team Wiki to understand how it was supposed to work. I got confused. I talked to some of my peers and hoped for someone having looked into this already. No luck. I cite the answer of a very skilled colleague that I hold in high regard: “Ah, this Native Client Authentication article is the most cryptic article I have ever come across.. at times it appears like everything has been told and at almost the same time, it appears like it is telling us nothing” and “I read the article now the 1001th time, and I still don’t understand”. Exactly.

I could not leave this as the answer and started coding while trying to follow the article. It makes no sense to me, to try to rewrite the article in this blog. Also just words will not bring any clarity. It looks as if language is not clear enough to describe what needs to be done.

The only way I am able to explain what goes on is showing code samples. I did not want these blogs to be about code, but it seems to be unavoidable to get clarity and to explain what goes on using the resulting code in my library class ELMCommLib. At this time I will use images to show the code. I will likely make the code available as open source later. Once done, I will link the download location to the blogs, as usual. The code is the result of a quite lengthy development and test process. It only implements parts of what the Team Wiki article explains. I have only so many test systems available and I can only test what these systems do. So here the disclaimer…

Disclaimer

Please note that so far I have only tested against a standard system supporting Form based authentication and a system supporting JAS. I used a trick to test the fallback to Basic authentication. My general disclaimer to the right of the page applies too. I’m not responsible for any damage done when following this article.

Context of the blog post is the series

This is the series of planned posts I intent to publish over time. Most of the examples will be EWM based, but quite a lot of the content applies to more ELM applications. The examples where performed with versions 6.0.6.1 and 7.0.x.

• Using the EWM REST and OSLC APIs 
• ELM Authentication – this post
• EWM Discovery
• EWM Work Item OSLC CM API
• EWM OSLC Query API
• EWM REST API to access existing Work Item Queries
• EWM Reportable REST API

Where to authenticate

To authenticate to ETM and EWM, use the context root of the server e.g. use https://elm.example.com:9443/qm or https://elm.example.com:9443/ccm as the destination for the authentication.
For DNG/ERM and JTS use the context root of the JTS e.g. https://elm.example.com:9443/jts to authenticate.

OAuth and Jazz Authentication Server

There is another type of authentication called OAuth. Please see ELM Applications Authentication and Authorization via OAuth 2.0 Workflow and related topics for this.

Also see Jazz Authorization Server – Landing page for more details.

How the authentication code works

I started with creating a function internalJazzRequestWithAuthentication() that can be used for all HTTP requests by passing the method and all parameters needed. This function deals with the whole authentication requirements and makes sure the authentication dance is performed when needed. I provide more convenient functions that wrap this function and are easier to use.

Function internalJazzRequestWithAuthentication()

This function has become long and complicated, so I have to show it in smaller pieces below.

The function delegates the request to the function internalJazzRequest() which is performing the correct call using the requests library. The function looks like this.

Function internalJazzRequest()

The function returns the result for the request. The result is now evaluated in the first part of the authentication code of internalJazzRequestWithAuthentication().

The function internalJazzRequestWithAuthentication() checks the status code of the response to be able to determine if anything related to authentication needs to be done.

The first HTTP status code it evaluates is 200. Usually 200 means the call went OK and the result is the desired result. However, the result headers can contain information that the access to the resource would require authentication and the call actually did not return what was expected.

The function internalHandleFormChallenge() is used to handle this case. It checks the result headers to figure out if Form based authentication is required. The code to detect the Form authentication request is easy enough to understand. The code checks for the presence of a special custom header X-com-ibm-team-repository-web-auth-msg. If the header is present and contains the value authrequired a Form based authentication is required.

If Form based authentication is needed, the code performs the login and retries the initial call. Note that currently the code uses the URI that is provided when creating the communication library for authentication. This needs to be changed at some point. The code has to get the public URI root from the result. This is important for applications like DNG that delegate authentication to JTS.

Function internalHandleFormChallenge() handles Form based login challenges.

The function also handles the case that the initial request was done for an insecure URI. This is handled as a fail and a false is returned.

If no authentication was required, the current result is returned back.

If form based authentication is required, the function internalLoginForm() is used to perform the form based authentication. It is essentially the legacy code remaining from an initial login mechanism that I used for automation examples. There is a façade function loginForm in the code that keeps this code compatible to old scripts. The form based authentication sends the credentials to the URI publicURI/j_security_check and checks if the authentication worked by looking at the response location header.

Function internalLoginForm() performs the form authentication.

Back in the function internalHandleFormChallenge() the code tests if the login was successful. If so, the initial request for the resource is re-executed in the function internalRetry() and the result is returned. The retry code will be explained later.

There are several other return codes, typically for different methods like POST, PUT etc. that can require form based authentication in internalJazzRequestWithAuthentication(). One example is return status code 302 which is handled exactly like the code 200.

The wiki page Native Client Authentication mentions the return status codes 303 and 307. None of the test systems I worked against ever responded in any of those codes. At the time of writing they are detected in a check and the call will fail for these codes. Once I start sending unauthenticated PUT, POST, DELETE or PATCH requests, I might see these response codes. The handling would be the same. One of the next steps for me will be to setup tests that force the other situations.

The last possible response is 401. This is the most complex case. It handles all other cases for authentication, especially Jazz Authorization Server (JAS), OICD and some other authentication options. See the initial part of the handling for 401 in internalJazzRequestWithAuthentication() below.

401 Challenge, detection of the authentication method.

As explained in the wiki page it is necessary to check for the header WWW-Authenticate. If this header is present, it needs to be evaluated. This could be a Basic authentication challenge or a Bearer challenge for Jazz Authentication Server/OIDC/OAuth. The situation where the Bearer or the Basic challenge is missing is handled at the very end.

If the Basic challenge and the Bearer challenge is detected, the headers X-JSA-AUTHORIZATION-REDIRECT and X-JSA-AUTHORIZATION-URL are evaluated to determine the authentication strategy. For the test system I have available, I needed to follow the JAS authentication.

The first step to do JAS authenticate with Basic and Bearer challenge is to get the redirection URL from the header X-JSA-AUTHORIZATION-REDIRECT. Based on this the code constructs the first redirection URL stored in xjsaauthredirect with the additional parameter '&prompt=none'.

The code performs a GET on the no-prompt redirect URL. The result of this call determines the next steps. When the GET does not return 200, the authentication is still valid and the code runs a retry for the original request, the result of which is returned. This is shown here.

Retry code if no authentication is required.

If the GET returns status code 200 authentication is required. The code below shows the code to handle the authentication. Note, that there is some code that gets information that remain unused at this time.

To reauthenticate the code performs a GET on the redirection URI in xjsaauthredirect, this time without the additional no-prompt parameter. The request can result in a status code 401. This is indicating Kerberos or some other authentication method. I have not implemented this code path at the moment. I lack a test system that behaves this way.

The JAS/OAuth authentication dance

The request can result in a status code 200. In this case we are on the OAuth path. The code performs a request on the redirect URI in xjsaauthredirect, this time providing an additional authentication header conforming to the Basic authentication header. The header is computed in the call to HTTPBasicAuth(). If this succeeds with a status code 200, the client is authenticated and the code retries the initial request. This is not strictly necessary for all requests to do that, but for now, I wanted to do it anyway. The result is returned back to the outer code. Note that the Authentication header is only sent this one time. In contrast when using Basic authentication like below, the Authentication header is sent with all the subsequent calls.

In the case that the server only returned the Basic authentication header, the code below initializes the normal Basic authentication mechanism. The code is written in an way that now always sends the Basic authentication header with all requests in this case.

The case that Kerberos was detected is also handled here at the end. Lacking a test system the code returns unsupported and False for now. When the code was unable to identify the authentication method, e.g. the server was not a Jazz server, the code fails as well.

Handling of Kerberos and Basic authentication

This is what I was able to come up for handling authentication. There is no guarantee, that it is completely correct. I am lacking test systems that behave in different ways, so testing is very limited.

Please note, that the authentication is only valid for the session that is used and based on several cookies that are managed by the session. Also note, as already mentioned, the session can be invalidated by the server at any time and the authentication would have to (automatically) be done again.

Retry code

I ended up creating a function to handle the retry for the initial call that led to the authentication challenges. At the moment the code for GET is tested and works. The code currently basically reruns the initial request and checks if that was successful. Then it returns True and the result or False.

Then internalRetry() function performs a retry for the initial call.

While writing this code, it appears that the retry code might have to be changed later. When the first call requiring authentication is something other than GET the code might fail. As an example, I am not sure at the moment how a POST e.g. for creating a work item would be handled in detail. The key here is that POST of a work item may only be run once. This is going to be an interesting journey and there is much to be learned about redirects and retries. Once all cases are tested, I will try to re-upload the changes.

Example for authentication flows – Form authentication

As already mentioned, I am able to track the communication in my framework. The following sequence of messages is related to a GET request against a system that uses Form authentication. Please ignore the files with extension .nt, they are not relevant for the authentication.

Form authentication message flow.

The first call is to get the rootservices document. The rootservices document is not protected, so no authentication happens.

The next call is to get the service provider catalog which is discovered in the rootservices document. The service provider or workitems catalog is a protected resource, this means the request requires authentication. The request below shows that the call was redirected from the work items catalog to the web page showing the login prompt. The page code in my logs usually shows some mysterious message that “Javascript is either disabled or not available in your Browser”. This is based on the fact that there is no browser that sent the request and all JavaScript code in the HTML page fails.

The response header also contains the header X-com-ibm-team-repository-web-auth-msg authrequired indicating authentication is required.

Request returns with authentication challenge

The code detects that authentication is required. And executes the form based authentication, providing the username and password unencrypted. The latter is the reason why it is absolutely necessary to use HTTPS and not HTTP as protocol. HTTP sends the credential information unencrypted. The response of the login call contains the redirection location to the previous call.

Form based authentication is performed

At the moment, the code shown here does not use the redirect, it uses the original resource URI to retry the call. This is the retry the code performs. Please note that the destination URL and the Request URL are now the same. The response body contains the protected resource service provider catalog. This catalog contains the information about the visible available project areas.

Retry to get the service provider

All subsequent calls in this session are performed without any authentication challenge. The next call to get the work item service for the desired project area is omitted.

Example for authentication flows – JAS authentication

Finally lets have similar look at the sequence of events when running against a system with JAS Authentication. Note, I have replaced the original host name with a generic one, should there be any unclear references, in case I did not spot them. The following flow is the result of the same version of software running against a different URL that has JAS authentication configured.

JAS Authentication message flow.

The first call is to get the rootservices document again. The rootservices document is not protected, so no authentication happens.

The next call is to get the service provider catalog which is discovered in the rootservices document. The service provider/work item catalog is a protected resource, this means the request requires authentication. The request below shows that the call to the service provider catalog fails with a status 401. The response header also contains the header WWW-Authenticate with the form containing the Basic and Bearer challenge. The headers X-JSA-AUTHORIZATION-URL and X-JSA-AUTHORIZATION-REDIRECT are also present. This provides with the authentication and redirect URL for the authentication.

JAS authentication challenge with Basic and Bearer authentication header.

The next step is the GET request against the redirect URL with the additional parameter &prompt=none. This request collects additional information after some redirects. I don’t claim I understand the details here. Key is that the call can either determine that there is actually a valid authentication already, or a new authentication is required.

Request if logion is required

Here the previous call results in status code 200. This means that JAS authentication is required. So we GET the previous target URL, but without an extra parameter &prompt=none.

Note that the request gets redirected to the login URL that we have seen in the response to the initial call in the header. X-JSA-AUTHORIZATION-URL. The response header seems to indicate that Form based authentication is a valid fallback: X-com-ibm-team-integration-jazzop-auth-msg form-login. I am not sure and have not tried. I followed the wiki page Native Client Authentication implemented the JAS authentication. The response body here is the HTML code of the login page. If the client was a browser, it would be possible to show it to the user.

Determine the authentication mechanism

The previous request ended with a status 200 which means JAS/OAuth2 is the way to go. This is done in the next step. The request is executed again and an additional Basic authentication header is created and sent with the request. Note that the Authentication header is not shown below. If it was I would have had to redact it, because the data is only encoded, not encrypted. This is also the reason why Jazz Applications are by default all using HTTPs only. Do not send your credentials over HTTP.

Also note that the request gets successfully redirected back to the service catalog URI. The response body actually contains the service provider/work items catalog.

Authenticate to the system and continue

Because it is just a GET request, It would be possible to just return it as a result. At the moment I run a last request to retry the initial get. I will leave that request out in this example.

Summary

As always, I hope that this blog helps users out there that work with the Jazz products and their APIs. This was a very complex topic and I hope I was able to shed some light based on the experience I have gained over time. There are several cases missing, which I was unable to test lacking access to test systems and also test code that I have not yet developed. I will try to keep the post updated as always. Once I have finished this series, I will also try to publish the code for re-use.

If you have made your own experience with this topic and want to share, please do so in the comments.