EWM Attribute Customization Introduction

While working on Debugging EWM Work Item Attribute Customization JavaScript I thought about all the questions I have seen being asked about Attribute Customization on the Jazz.net forum. There are some common questions I have seen a lot. There are some common things users would like to do and there are some limitations that might get into the way. I thought, I would create a blog that summarizes what I know about attribute customization and also provide the important links. I also want to provide a short summary about, for all I know, what can and can not be done with Attribute Customization and the provided API. I already create something like that in RTC Process Customization – What you can and cannot do. I will follow its structure and reuse some content but will try to be more specific in this blog. This blog will be limited to the work item attribute customization, with some few exceptions. The blog post will mention possible solutions that go beyond attribute customization, where this seems reasonable.

Important links

The following links are the most important references about EWM work item configuration and attribute customization.

About Jazz.net forum questions

A lot of the input that I take under consideration comes from questions and answers on the Jazz.net forum. Some of the examples originate from customer requests.

Questions on Jazz.net and even customer requests do often not provide good requirements. They often provide some implementation idea instead. The real problem or business case is usually hidden and not exposed. So it is important to question the request to drill down and expose why the question was asked as it is and what the background of the question is. If the real reason for the question is exposed, the answer can be completely different.

One example is: “Can I auto subscribe users to a work item?”. In pretty much all cases I can remember, the purpose of this was to send an e-mail notification to a user e.g. when the work item was created. Adding a user to the subscribers might or might not be a viable solution, but the example clearly exposes the problem.

So take the questions on Jazz.net with a grain on salt and try to dig deeper to understand what the question is really about.

What is Process Configuration and Attribute Customization?

EWM allows to configure the process of a project area. There are several options that can be used.

  1. Work items, work item attributes, editor presentations, workflows can be configured.
  2. Roles can be configured
  3. Permissions can be configured
  4. Operational behavior can be configured using out of the box preconditions and follow up actions.

Attribute Customization is a work item related capability provided by EWM. It provides built-in capabilities that can be configured and thus be considered process configuration as mentioned above.

The capability types available for configuration are:

  1. Default value provider for built in attribute types.
  2. Calculated value provider to support calculation with work item attributes to support certain processes.
  3. Value Sets to provide customized sets of selectable values for work item attributes. Built in value sets allow to extract string values from an XML page in the Web, select contributors by role and allow to select enumeration vales based on other dependent enumerations.
  4. Validators to validate values of attributes based on range or regular expressions. Validators can show a warning in the UI and be used with a built in advisor to prevent saving a work item, unless the offending attribute values are corrected.
  5. Conditions can be used with built in advisors to make work item attributes read-only or required. There is no built in condition available at this time.

What is special about Attribute Customization?

What sets attribute customization apart from the other capabilities is, that it provides a JavaScript API that allows to create custom scripts for provider configurations. The JavaScript can be uploaded to the project areas process by any user with the required permission. This does not require write access to the application servers file system. For this reason it is possible to create and deploy custom JavaScript scripts even if the server is maintained by a cloud provider.

JavaScript is also reasonably easy to learn and a lot of practitioners have some JavaScript skills. This is a two edged sword.

Contrary to Eclipse extension plugins, another extension mechanism available in EWM, the JavaScript API has a relatively low experience threshold for beginners. It is way easier to develop than the other Eclipse plugin based customization options available in EWM that require great skills, understanding of the framework and familiarity with the available API. Since these options are based on Java and Eclipse plugin extension development, they require the plugins to be deployed which means they need to be copied into a file system, typically on the server where the CCM application is deployed. This is usually not possible in a cloud hosted environment. Except, some cloud services would perform such deployments for the customer, for a higher price tier.

The danger is that the low skill cap attracts very inexperienced users, which can eventually lead to issues, including server performance issues. Some of the capabilities can be overused. There is also a tendency to overcomplicate the work item workflow and the process.

The JavaScript Attribute Customization API constraints and limitations

Attribute Customization is configured on the attribute level. This means that any attribute customization will apply to all work item types that have the attribute configured.

JavaScript attribute customization scripts have to implement the required interface for their type. All the interfaces require to return some data. The data can be empty or undefined, dependent on the attribute type, e.g. an empty string or null. It has to return something that matches the attribute type it is configured for, nevertheless.

It is possible to detect the type of the work item the script is executed for, this allows to have different behavior for different work item types in the JavaScript attribute customization script. Based on the work item type attribute value, the script decides what to return, some custom value or the current attribute value to do nothing. This strategy allows to adjust for type specific attribute customization to some extend.

The API description in the Work Items Attribute Customization Wiki page explains which work item attribute value types are supported. Only very basic attribute types such as strings, numbers, enumerations, timestamps and dates are really supported. Especially note that the support for complex item based attribute types such as team areas, contributors etc. is very limited. For a those attributes, all you can access is the UUID and the display name. If you want to set such values, you must return the UUID of the item.

The documented API does not provide with methods to discover information beyond the work items current attribute values. There is no documented simple JavaScript API to discover details about not attribute based information. There is no documented simple API to follow or create links to work items or other artifacts. There is no documented API to access process information, such as users and roles or any other details. There is also no mechanism to detect a state change or workflow action. The workflow action is only available in conditions but the detection of a state change would be desirable in calculated values as well.

The return value of JavaScript attribute customization scripts is processed in the API. Dependent on the work item attribute type, the script is configured for, some kind of conversion is performed. As an example a string can be converted into a timestamp, provided the format is correct.

As far as I can tell, the conversion that is performed for html and string based attributes, including the subject and description, when returned from JavaScript converts the content into a string escaping any special content such as XML or HTML tags. This conversion happens outside of the JavaScript and there is no way to change the behavior. Because of this reason it seems to be impossible to return HTML with working links for HTML or string based attributes including the summary and description. Only the built in configurable default values work correctly. The only current attribute type that can be set by JavaScript attribute customization to contain links is the type Wiki. In this case the Wiki syntax has no XML/HTML tags involved that would have to be escaped.

Performance considerations

Some users seen to think there is such a thing as limitless size and performance. This does not apply to the real world. It is possible to heavily impact the performance of clients or servers with process configuration and especially customization. I am aware of a case, where JavaScript added to a Theme caused so much communication that the browsers on smaller laptops where drained of resources and almost came to a halt. Loading a work item took several minutes. This made working with the tools impossible for many users. So be careful and consider what you might do to the performance.

It is also very easy to annoy the users with overcomplicated processes, thousands of choices, required attributes nobody knows the values for etc. In general big numbers are bad and small numbers are good. When drop down boxes grow too big and the number of items ever increases, the users will be affected in their ability to successfully use the process. I have seen cases where the users got confused and did not realize that they had to enter a value and complained the UI had stopped. Overcomplicated processes will likely cause performance complaints and wear away the will to support the chosen tool and process.

There are limitations in the amount of custom attributes for a work item type (100). Having a lot of work item types is also not helping to make the process easier. Keep numbers low e.g. 10 would be a guidance for work item types.

Also keep in mind that there are size limits for all work item attribute types. Especially note that for list type attributes there is a size limit. How many selections can be stored is limited by the site of the list attributes and the size of the ID’s or item UUID’s that are stored.

Where does the JavaScript run?

The JavaScript runs in the Web Browser or in the EWM Eclipse client in a JavaScript interpreter..

Where and when is Attribute Customization not executing

Attribute customization including the JavaScript is not executed when the work item operations are created using an API such as OSLC or the SDK or Plain Java Client Libraries.

Work Item Proxy

When debugging the JavaScript, it is possible to see more objects accessible to the JavaScript in the debugger. It is possible to see an object called work Item proxy that looks very promising. Please note that this object is considered internal API. It should not be used by the JavaScript. The supported API is documented here. It is also important to understand, that this work item proxy is only available in the Web Browser and not if the script runs in the Eclipse client.

There are several posts on the Jazz.net that discuss how the proxy could be used. Again, it is internal API and should not be used, if you do, you are on your own. There might be unwanted side effects and the internal API could change at any time.

Why does my script not work?

This question comes up often on the forum. It is pretty much impossible to debug others scripts on Jazz.net. There are reasons for that:

  1. Java Script Attribute Customization depends on the process used in the project area. The community does not have and, due to privacy, security and other concerns will not get this.
  2. It is quite some effort to setup a test project area, even if one would have a process template.

So it is usually impossible to help on this level. What can be done is scanning the script and the description in the question for typical issues.

See Debugging EWM Work Item Attribute Customization JavaScript for how you can debug your own scripts.

Please note, there are different levels of issues with JavaScript. On the most fundamental level:

  1. Attachment Scripts must be enabled in the advanced teamserver.properties to be working – see below.
  2. The script must be accessible in the process.
  3. The script files encoding must be UTF-8 and consistent so the file can be read, interpreted and executed.
Process Attachment Scripts enabled.

If an error happens at this level the best you can hope for is an entry in the log file. Only if the JavaScript is successfully decoded and loaded, it can be debugged.

What you can and cannot do with JavaScript Attribute Customization

There are certain limitations in the JavaScript Attribute Customization API as far as I am aware. I provided a more comprehensive presentation here: Process Customization – What you can and cannot do.

I will try to provide some details here.

  1. Attribute Customization JavaScript is always defined as a function. Regardless what type of Attribute Customization the script must return a value (can be null or empty). Otherwise this will be a potential error in the log. I am not sure what happens worst case.
  2. Attribute customization is for the attribute. This means especially that the customization will be executed in all work item types that have the attribute configured.
  3. Sure, in JavaScript, you can get the work item type, by reading the respective attribute. You can then decide what to return. As explained in 1. you cannot return nothing. You however can get the current value of the attribute and return that, this helps in some cases.
  4. As far as I am aware, Attribute Customization only runs in the context of a users browser and the Eclipse client. As explained above, this also means that automation based on attribute customization does not work, if you access work items using any of the APIs. This is important, as pre-conditions/advisors and follow up actions/participants can be implemented as server extensions that always run, regardless who accesses the work item how. When the JavaScript is run in the Eclipse client, only the infrastructure of the Eclipse client is available.
  5. Attribute customization scripts can run multiple times, for example the calculated values might execute multiple times, based on changes to the other attributes they depend on.
  6. JavaScript Attribute Customization only has access to the data of the work item in which context it runs. It is not possible to follow work item links using the documented API.
  7. Even the available data of a work item is limited. As an example, a workflow action for a state change is only available in conditions. It is not available in any other attribute customization types, especially not in calculated values. This essentially means, there is no simple way of tracking and detecting a workflow state change.

Java Eclipse plugin extension based Attribute Customization

Beyond JavaScript, it is possible to provide Java based Work Item Attribute Customization Plugin Extensions. This is possible for all available attribute customization types, default values, calculated values, value sets, conditions and validators as shown in these examples. These Java based plugin extensions have access to more complex EWM Java SDK API. This makes it more feasible to create complex custom attribute customization. It is also easier to control which return result types are supported and, based on these, create richer attribute customization. It would, for example, absolutely be possible to have a default value provider that returns an XML or HTML result, that shows working HTML links. The decision to return a result that escapes the HTML tags or return the correct format is an implementation detail. A more complex example with additional background information can be found here and in this post.

The Java based attribute customizations would show up in the Admin UI in the same way the built in Attribute customization does. To make this possible the Java based Eclipse plugins need to be deployed in the Eclipse client as well as the EWM Server (see here). The API that is available to the Eclipse client and the server is the EWM Common SDK API. Consider reading this blog post to better understand what this means. The requirement to deploy these extensions on the EWM server are a potential inhibitor for cloud based deployments.

EWM Eclipse Plugin Extensibility

Java base attribute customization is only a subset of the Java Eclipse plugin extensibility available in EWM. The SDK provided for EWM allows to develop all kinds of extensions, including pre-conditions, follow up actions and asynchronous tasks. It is possible to develop such extensions to run in the context of the Eclipse client or on the server. These extensions need to be deployed in the context they are developed for. Server extensions have access to the full capabilities of the EWM Common SDK API and the EWM Server SDK API.

Preconditions also called Advisors can act as process advisors that prevent saving if the data violates some standards. Preconditions can be used to implement attribute customization validators or conditions. Example Advisors can be found here.

Follow up actions, also called Participants can perform additional operations on the EWM data. As an example, follow up actions can collect data across linked artefacts, perform calculations on the data, set attributes and properties for items and save the changes. Example participants can be found here. This includes the capability to create new work items and link the new work items to other work items.

In some cases it is better to use these capabilities instead of attribute customization, because it provides more API than even the Java based attribute customization.

All plugin based solutions require to be deployed. Server extensions require to be deployed in the server, which might be a blocker in an environment hosted as a service in the cloud.

Popular requests

In the next sections, I will try to provide some examples of popular requests. I will describe the background these request have, as far as I can tell, and which implementation options might be viable.

Popular: Custom E-Mail Notification

This comes up in various flavors. Examples are

  1. Can I automatically set an owner of a work item?
  2. Can I automatically subscribe a user to a work item?

With respect to attribute customization, the answer would be yes, you can do this, to some extent. It is possible to set or add a subscriber, to set or add a work item owner. In the JavaScript API this needs to be done the contributors UUID. The UUID will have to be configured or hard coded when using the JavaScript API. There is no API to search for contributors in JavaScript.

A Java based attribute customization plugin would have more API access and can provide better control and automation.

A real custom mail notification is possible using the following approaches

  1. Custom scheduled asynchronous tasks (see Due Date Notifier).
  2. Custom follow up actions that trigger at work item save.

The last three options require to write custom Java extension plugins and their deployment on the application server, which makes it less attractive in a cloud environment.

As an alternative, EWM has other mechanism that might be better suited. For example there are RSS feeds for various change events, work item queries and dashboards. These do not send e-mail, but they might scale better than e-mail notifications.

Keep in mind that the e-mail notification options are controlled at user level by the user. A real Java Eclipse plugin that sends custom e-mail notifications could work without the need to be correctly configured at user level.

Popular: Calculation with work item attributes

This is often requested to calculate effort or something similar.

With respect to JavaScript based attribute customization, it is possible to perform calculations on attributes of the same work item. It is possible to perform numerical, boolean and other operations in calculated values. It is necessary to set the dependencies right, so that the chain of events executes until all the calculations have been performed. If multiple work item attributes need to be set, keep in mind, that the calculated value is a function that only works for one work item attribute. If multiple attributes need to be set, a calculation for each attribute is necessary. As already mentioned, It is not possible to calculate across values of multiple work items.

If it is necessary to calculate across multiple work items, JavaScript attribute customization does not work. It is necessary to use the Java plugin extension mechanisms such as follow up actions to implement this. An example is the RTC Update Parent Duration Estimation and Effort Participant.

It is also possible to use external tools based on API to collect data and then update work items e.g. in scheduled jobs.

Popular: Calculate how Often a work item was in a specific state

I have seen this several times. There is no easy way to detect work item changes. It might be possible to use an approach as shown below to detect this, but I am not sure if this idea below would really work.

With some tricks and several hidden custom attributes for state tracking ST1, ST2 and counting STC, it might be possible to cobble something together. E.g. initialize the hidden state tracking attributes st1 and st2 with a default based on the state ‘New’.

The calculated value of ST1 is triggered and calculated based on the current value of the work item state attribute. If the current state is not the value of ST1 return the new current value.

Calculate the counter based on a triggered change of ST1. If ST1 contains the desired state, return the current value of STC increased by one.

Calculate the value of ST2 when ST1 was changed. Return the value of ST1 as result to the calculated value.

This requires some intelligent usage of the dependent attributes. It might be necessary to use more artificial attributes to get this working. This attempt has however many downsides, especially the custom attributes are all showing up in the history and also create more changes in the work item.

It would be easier to develop a follow up action that actually works on the state change, if this is necessary. It would be important to see if reporting can be used instead.

Popular: Make work item attributes read-only or required

EWM has preconditions/Advisors to make work item attributes read only or required based on conditions. The question is, how far can you go with this? The JavaScript API is very limited in what data is available. It is obvious that conditions can work on this data. So you can make attributes read only or required on work item attribute information.

But since you have no access to any complex information such as team membership or roles or the current contributor, JavaScript can not be used to create more complex conditions. Any condition that needs to have access to more complex information, would require to be written as Java plugin. An example requested by a customer with additional background information can be found here and in this post. It is important to know that these condition based attribute customizations are the only way to customize rules to make attributes read-only or required. There is no other extension point available that you could use to hook into. This is one of the rare cases where custom preconditions/Advisors can not be used.

In my experience with requests like here and in this post, to make attributes required or read only, are often motivated by the desire to have more fine grained control. Consider to use permissions to prevent roles from changing attributes.

Requests like above are also often motivated by the desire to make EWM work like some legacy tool. My experience so far has always been, that trying to make a tool work against its design is usually fatal. It usually makes the process hard to understand and follow. It also often degrades usability and performance.

Since every user in EWM has at least the default role (Everyone), can have many roles, and the roles have an order, it is also debatable what ‘by role’ really means.

Popular: Dynamically Hiding Work Item Attributes

I have also seen several requests that want to make attributes hidden by some complex, usually role based rule. This is not possible today, regardless the approach.

There are very limited capabilities to control visibility built into EWM. Attributes can be hidden if they are empty, if the work item is in a workflow state, if some endpoint is empty, if there is no project link and if the work item is just created. An editor presentation tab can be hidden, if all contained attribute presentations are empty.

There is no extension point, and there is no attribute customization available, I am aware of, that would allow customize the behavior in any other way.

There are various enhancement requests for EWM in this context. I think it would be nice if there was a custom attribute customization option that could be used with a precondition e.g. ‘Hide for condition’ that could be used similar to ‘Read only for condition’. This would allow to hide attributes that are unimportant. The question is, if it is helpful when attributes show up and vanish for the same attribute type. This is at least debatable. As a user I would likely find this confusing. As always, keep in mind this could open the door for performance issues and abuse. The remark about roles and their usage in EWM also applies.

Please note, just because an attribute is not visible in the work item editor, presentation does not make it inaccessible by work item queries or reports. EWM was also designed for collaboration and not hiding.

Popular: Dynamic workflow

I don’t know where this comes from. I doubt this is a thing in other tools for change tracking, but it cam up more than once.

EWM/RTC does not support dynamically changing the workflow of an instance of a work item. The workflow is tied to the work item type. It would be possible to use a follow up action/Participant to change the work item type under certain circumstances. Due to the lack of support for detecting a state change and the lack of access to other data in attribute customization, it would be not feasible to use attribute customization. A participant is a far better option.

Popular: Dynamic creation of attributes

This came up several times.EWM does not officially support this. It is possible to create an attribute using the API, but this attribute does not surface in the admin UI. It is necessary to add statements to the process XML to make such an attribute configurable in the process and the editor presentations.

This is used by some integrations just once and not meant to be used constantly during operations. Also mind the limit of custom attributes and consider usability of the process.

Restrict Read Access To Data

Some customers want to have more control about who can access which data. EWM has the capability to restrict work item and SCM item read access permission based on access context. If an item can not be read, it can also not be written.

For work items, the restricted access context can be an access group, project area, or team area. It is possible to automate setting the work items restricted access based on categories. This sets the access context to the process area associated with the work item category. This is the only built in automation for category based work item access control. There is no other built in automation for work items e.g. enabling the automatic selection of an access group for the work item.

For SCM data there are more options to set the access context/visibility. Components can have access control set to a project area or an access groups. SCM content in a component, folders, files, can be configured more fine grained. It is possible to limit access to a contributor, a project or team area or an access group. There is no built in automation to set the access context for SCM items.

It is possible to create custom automation to set the access context for work items and SCM items. One implementation approach is using follow up actions. For work item restricted access this can be configured for the work item save operation. For SCM objects follow up actions for the check in and deliver operation could be developed to compute the desired access context. I have explained some of the options in the blog series Setting Access Control Permissions for SCM Versionables.

In contrast to my knowledge back then, I know today, that it is possible to elevate the operation context to an administrator, why this is important is explained in the second last paragraph.

The question would be, can JavaScript attribute customization be used for custom automation for the restricted access? Attribute customization is only available for work items. There is no such API for SCM content. It might be possible to set the restricted access attribute in JavaScript attribute customization by returning the UUID of the access context, but as already mentioned, because of the very limited amount of data and API access available, JavaScript would most likely not be feasible for anything beyond that.

Java Based attribute customization could be more feasible, but it is most likely not possible to elevate the operation to super user access due to the limitation of the available API to the common API. I would consider this approach to be unfeasible for any relevant complexity.

If it is necessary to develop a Java Eclipse plugin for attribute customization anyway, it looks to be more suitable to create a follow up action to automate setting the access context. This allows to use the full capabilities of the EWM SDK common and server API and it is possible to elevate the operation to run in an administrator/super user context.

The reason why it would it beneficial to run the operation in an administrator context is, that the operational behavior usually runs with the permissions and limitations of the user that performs the save operation. This also means the operation fails, if the permissions are preventing it. This limits what can be done. For example it is impossible to change the access context of an item in a way that makes it inaccessible to the user performing the operation. The operation would fail. Elevating the operation to run in an administrator context, would allow such an operation to be successfully executed, regardless of the repository access and permissions of the regular user triggering the operation.

I would suggest to not try to use attribute customization for these use cases.

Summary

As a summary, attribute customization, especially using JavaScript can be used for some interesting purposes. There are however a lot of limitations in the JavaScript API that prevent several use cases. The biggest benefit is, that it can be used without having to deploy anything on a server. This allows it to be used in cloud deployments, where the users have no access to the server file system.

For more complex problems, the Eclipse plugin Extensibility is a better option, regardless if it is the Java based attribute customization, or Eclipse plugin based EWM extensions, just because the APIs provided in the EWM SDK are so much richer.

This was quite some effort. As always, I hope that this is useful to someone out there.

Advertisement

EWM OSLC Query API

After looking into how to create and update work items the question becomes, how to find work items to begin with. OSLC provides a query mechanism to allow querying for items. This post intents to show how OSLC queries work including some examples that work for me. The techniques explained in the previous posts in this series are important. If necessary, go back to the previous posts to understand the details. As usual the focus in this blog is EWM/RTC, however, the OSLC Query mechanism works for all product supporting it. So what is explained here based on EWM will work with ETM, DNG etc.

Context of the blog post is the series

This is the series of planned posts I intent to publish over time. Most of the examples will be EWM based, but quite a lot of the content applies to more ELM applications. The examples where performed with versions 6.0.6.1 and 7.0.x.

External Links

I used at least the following links for exploring the OSLC Query mechanism. The latest version of the OSLC Query V3 document contains several examples which are very helpful. I still found it a challenge to get some of the queries working.

RDF XML or JSON?

The examples in this blog are using the content type application/rdf+xml. The same examples work with the content type application/json, provided the function to parse the files are switched to using a JSON parser. I have the same examples using JSON. However the posts in this blog series are mostly using RDF+XML and I decided to stay with it, to make it easier to follow.

Query Base

Before being able to execute a query, it is necessary to discover the query base. The query base is the URI that defines the root for the OSLC query. The first step is to get the service provider catalog from the rootservices document as described in the blog post about EWM OSLC Discovery. Get the rootservices document

GET https://elm.example.com:9443/ccm/rootservices

Accept application/rdf+xml; charset=utf-8
OSLC-Core-Version 2.0

Get the service provider catalog like below. Pass the OSLC-Core-Version and the Accept content type header. This step requires authentication to be done.

https://elm.example.com:9443/ccm/oslc/workitems/catalog

The catalog lists the services available for each project area. To narrow down to a project area perform a GET on the work item service for the desired project area, providing the headers mentioned above. As an example

GET https://elm.example.com:9443/ccm/oslc/contexts/_8e5qfFpmEeukW7cqqDjAuA/workitems/services

The resulting response body contains various services that are provided for the various work item types of he project area.

  • Various dialogs such as OSLC selection, creation dialogs and pickers
  • The OSLC creation factories for all the work item types
  • The OSLC resource shapes for all the work item types
  • The OSLC query capabilities

To be able to query for work items, it is necessary to analyze the query capabilities provided by the service provider. The query capabilities can be found by searching for the oslc:queryCapability nodes.

There are usually at least two query capabilities in the work item service provider for an EWM project area. One is the query capability for deliverables. A deliverable is also referred to as a release and is a work item attribute type. The query capabilities for deliverables can be found using the resourceType http://open-services.net/ns/cm#Deliverable.

The second query capability is for work items (in OSLC referred to as change requests). The image below shows the oslc query capabilities for work items. This can be identified by the resourceType http://open-services.net/ns/cm#ChangeRequest

Query capability for work items.

The oslc:queryBase is the URI for the work item OSLC query mechanism for the selected project area. It will be used in constructing the OSLC Query URI. The query base has the form:

https://elm.example.com:9443/ccm/oslc/contexts/_8e5qfFpmEeukW7cqqDjAuA/workitems

where the second last segment is the internal unique ID of the project area.

The query capability also provides a resource shape for the Tracked Resource Set (TRS) provider. The resource shape has the form

https://elm.example.com:9443/ccm/oslc/context/_8e5qfFpmEeukW7cqqDjAuA/trs/shapes/workitems/query

The resource shape provides the information about the attributes provided by the TRS. The resource shape contains a oslc:valueShape for work items based on the work item type defect.

Value shape for work items

This provides the value shape that defines the common work item attributes, their types and the allowed values for these work item attribute value types. It is possible to GET the oslc:valueShape value shape (provide the OSLC related headers) like below:

GET https://elm.example.com:9443/ccm/oslc/context/_8e5qfFpmEeukW7cqqDjAuA/shapes/workitems/defect 

This value shape provides with all the common attribute and link types. The work item of a specific type might still have additional custom attributes. The post EWM Work Item OSLC CM API explains how to get that information using the work item type specific resource shape associated with the creation factory.

The image below shows the code that gets the query capabilities out of the project area work item service provider document.

Extract the query capabilities from the service provider document

This code shows how the query base is extracted from the service provider catalog. To do this, the code identifies the change request resource type and gets the associated query base.

Get the query base for the work items

Having the query base it is now possible to execute OSLC queries. The code below executes the most simple form of OSLC query:

Querying the query base.

The code performs the GET method on the query base.

GET https://elm.example.com:9443/ccm/oslc/contexts/_8e5qfFpmEeukW7cqqDjAuA/workitems

Accept application/rdf+xml; charset=utf-8
OSLC-Core-Version 2.0

Please note the OSLC headers Accept and OSLC-Core-Version above. The same headers are used in all calls in this blog.

Paging

The request is redirected to support paging like shown below. The paging mechanism is used to split queries with large result sets into smaller, more manageable chunks.

Paging redirect

The response body contains the work item URI’s for the first page of work items. Note, because there is no query parameter, the query only returns the work item URI’s and not any other property data. To get more properties of the resulting work items e.g. to display the summary would, in this case, require to get the work item using the URI from the response.

Response body contains work item URI’s and information about paging.

In addition the response contains information about the total amount of results and the URL to get the next page, in case the number of results exceeds the maximum items returned by this page. The image above shows the oslc:responseInfo which contains oslc:nextPage with the URI to get the next query result page and the oslc:totalCount with the total count of query results.

The code below shows how the query is executed. In the while loop a GET request to the query URI is executed. The result of the call is analyzed and this provides the next page of the query, if paging is needed. If no next page is available, the query is finished.

Execute an OSLC Query

The code below processes the result for a result page and gets the resulting work items to display them. It prints the work item URI. It also tries to get and print the identifier (Id) and the title (summary) of each work item in the query. If the identifier or the summary is not available nothing is printed. The code then analyzes the paging information. It returns the next page URL if there is one. It also returns the total count of the result.

Try to get at the result details of the query page

URL Encoding

OSLC Query provides a mechanism to create simple queries to retrieve items. These queries are sent as a URL in a GET HTTP request. There are limitations on which characters can be sent in the URL. Some characters have specific meanings in the URL. To avoid creating and sending the wrong or illegal URLs, parts of the OSLC query parameters need to be URL encoded. See some explanations about the URL encoding here:

In the code below url encoding is done using the function urlEncodeString as shown below.

comm.urlEncodeString('URL Encode me')

The code URL encodes the data in a way that has worked for me. I am not totally sure I understand the URL encoding in all its details and there might be issues in the code below. If I got something incorrect, please leave a comment with a suggestion to correct it.

More complex OSLC Queries

The code below uses the following constants (mind the = at the end of each term) which are used to compose more complex OSLC Queries:

select = 'oslc.select='
where = 'oslc.where='
searchTerms = 'oslc.searchTerms='
orderBy = 'oslc.orderBy='
prefix = 'oslc.prefix='

The statements represented by above constants and how they work are explained in the following sections.

The oslc.select Statement

OSLC queries support selecting the properties supposed to be returned in the query. The statement that is used for that is the oslc.select statement. The code below shows how the select statement is used. The first three lines are used to control running the queries and creating the file name for logging and can be ignored.

The interesting part is the selectString1. It defines the list of properties that should be returned for each query result. In the case below we want the type, the id, the title, the description, when it was last modified, who modified it, when it was created and who created it for each work item. This information is encoded in the selectString1 by creating a comma separated list of property identifiers. The select statement term is then created by concatenating the select statement ‘oslc.select=’ with the URL encoded version of the selectString1. This term is added to the query base as first query parameter using the separator ‘?’.

Select statement is provided to the OSLC query

When this query is run, the query result contains the properties for the work items that are returned in the query result. The image below shows the resulting data in the response.

The query result for a query using a select statement

The select statement can be nested to provide nested data. It is possible to use ‘*’ to select all available properties. The amount of data that is transferred can impact performance of the communication. The specification mentions that the servers SHOULD accept rdf:nil as single property. Both extremes are commented out in the cod above.

The oslc.properties Statement

The oslc.properties statement can be used to limit the set of properties or attributes of a work item that are considered in the OSLC request. The oslc.properties can be used in a GET request. In this case it can be used to specify which properties the requestor is interested in. The server does not have to collect and transmit all properties, but only the ones of interest.

The oslc.properties can be also used when updating a work item. This allows to perform partial updates. There is no example provided in this blog, but the syntax and encoding of oslc.properties and oslc.select are the same.

The oslc.where Statement

The oslc.where statement can be used to specify which work items to query. The first example below shows a query for the work item with the identifier 1. The select statement uses * to select all work item properties.

Get the work item with the ID 1 and return all properties.

The URL that is created can be seen below. The encoding makes it hard to understand. The important request headers are the OSLC-Core-Version and the Accept header.

The GET request with headers and URL encoding

The image below shows parts of the resulting RDF-XML for the work item with all the data for its properties.

The response with all properties

The In statement can be used to search for properties being in a range. The code below searches for the work items with identifiers being 1, 3, 9, 50. The query would run successful, if there are IDs that are not found.

Find the work items out of a list of ID’s

Please note as documented here, especially in the syntax section, the oslc.where statement only supports to compose complex conditions using the boolean operation ‘and‘. The operation ‘not’, ‘or’ or other complex nesting is not supported.

Before looking into more examples lets introduce the next statement.

Sorting – The oslc.orderBy Statement

The statement oslc.orderBy can be used to define the order for the result set. The example below finds all work items with the type defect and requests the result set to be ordered by severity (descending) and then by id (ascending). The order is defined with a prefix for the property. The prefix is + for ascending and as descending order. Multiple order specification can be given in a comma separated list.

Search for all defects and sort by descending severity and ascending id

Unfortunately, the orderBy seems not to work for EWM 7.0.2 and the accept format application/rdf+xml. I have seen it working with application/json.

Full text Search – the oslc.searchTerms Statement

The statement oslc.searchTerms can be used for full text search. The query below finds all work items that can be fond by full text searching for the term ‘dividend‘ creates 20 hits with the JKE Banking Sample.

Full text search for ‘dividend’

Define a Namespace Prefix – oslc.prefix

The OSLC query mechanism allows to define new namespace prefixes that can be used in the oslc.where, oslc.orderBy, and oslc.select statement. My experimentation with this feature was quite problematic. What worked for me is the following:

To define a namespace create a variable with the namespace prefix, including the equals sign, and encode the URL. Then use only one namespace in the oslc.prefix definition Only define one prefix for one namespace in each oslc.prefix statement.

pdcterms = 'dcterms=' + comm.urlEncodeString('<http://purl.org/dc/terms/>')
prefixTerm = 'oslc.prefix=' + pdcterms 

The code below defines several namespaces (mind the encoding part). Based on these namespaces I create a prefix term for each of the namespaces based on ‘oslc.prefix=’ and the namespace URI. To define multiple namespaces in a query concatenate the prefix terms with ‘&’. Some discussions on Jazz.net indicate it might be possible to use a comma separated list. I could not get this to work.

Defining prefixes

To use the prefixes in a query, just include the prefixes as query parameter.

Using prefixes in queries

The query URI can become very long when providing many prefixes.

The prefixes can make the query quite lengthy.

Search For Items With a Specific Enumeration Literal Value

The image below shows a query that selects only work items that have the severity set to critical. The enumeration literal to be provided for the selection is information that can be looked up by querying the resource shape and allowed values for each work item type associated to the creation factories. For common work item attributes the available attributes and allowed values can be found using the chain resourceShape to valueShape associated with the query base.

Queries for the severity literal representing the severity ‘Critical’

Search For Items Modified After a Certain Date

The image below shows a where term that selects all work items that are modified after a certain date.

Items modified after

Compound Where Terms

The OSLC Query mechanism supports compound where terms are built form simple terms. The only supported boolean operation is and. Or is not supported, neither is ‘not’. The code below shows a query that looks for work items of type ‘defect’ and severity ‘critical’.

Items of type ‘defect’ and severity ‘Critical’

Summary

This concludes my short blog about the OLSC Query mechanism. My intention, in addition to understand it better myself, was to provide some working examples and how to create the queries. I have tried this in the past and got some where, but it was quite challenging. The documentation I had available in the past was often lacking some small but important information. The documentation for OSLC CM 3.0 has improved a lot, but apparently there are still issues you can run into.

So as always I hope I was able to provide simple but relevant examples that help users of this technology to achieve their goals and save some time.

The RTC Extensions Workshop has been updated for EWM 7.0.x

I am very passionate about the RTC Extensions Workshop as you might be able to tell from the content of this blog. Performing it with EWM 7.0.x provided several challenges. It became apparent that an update to the workshop would be beneficial.

I spent a considerable amount of time in the past two months to update the workshop. As a summary the following items where addressed:

  1. Since the CCM server is shipped with WebSphere liberty profile, configuring the server for debugging needed to be changed. The old way to configure the server still worked in the 6.0.x versions, so it went unnoticed. With EWM 7.0.1 this is no longer the case and the workshop was updated to address this.
  2. The advanced capabilities introduced in the EWM SCM system in the 6.x and later caused a deviation of the screen shots showing the pending changes. The workshop setup tool was slightly changed to fix this.
  3. The workshop setup tool and its shell script has been tested with Linux and MAC OS.
  4. I wanted to add a section to Lab 1, explaining how to setup the existing Eclipse client/server development workspaces to better support development and debugging of the Plain Java Client Libraries forever. The new last optional section addresses this. For this reason Lab 1 of the workshop is a must for anyone intending to create Java based automation or extensions to RTC/EWM.
  5. I had an errata list with a number of small issues, typos, naming inconsistencies and the like that were fixed. During reviews a bunch more showed up and were fixed.
  6. A colleague ran the workshop on his MAC, so this works. Use whatever is available for MAC like Eclipse and where this is not specifically available, use the Linux versions.

The RTC Extensions Workshop has been published with an additional section for the new EWM versions and is now available for download. I will update recent posts around the workshop in the next few days.

As always, I hope that this blog post helps the users in the Jazz Community.

Using a REST client to authenticate to ELM/CLM Applications

Recently the question how to authenticate to a ELM/CLM (aka Jazz) application came up in several occasions. I did not know the whole story either, so I had a look. Here is what I found.

Update history

2019/05/14 Added Basic Authentication

2021/01/22 Added Links

Important Links

Scenarios

There are several scenarios that could be of interest. Some of the scenarios will be provided below. A general documentation about how the authentication of a client to a Jazz application works is provided here in the development wiki.

Common scenarios are as follows:

  1. Using a REST client to explore the REST/OSLC APIs
  2. Creating custom automation using REST/OSLC APIs

Scenario 1 is also used in the OSLC Workshop. However, as explained below, things have changed since the workshop was initially written and the latest changes are not addressed in the workshop.

REST clients

The main changes that could be recognized are in the browsers and the integration of REST clients. These changes heavily impact how easy or hard it is to login. The first REST client used in the OSLC workshop was the Firefox browser addon REST Client . Using this addon It is easy to login to the Jazz application. All that needs to be done is

  1. Open a browser window
  2. Provide the URL to the CLM application and open the URL
  3. The application prompts for a login
  4. Provide user name and password and perform the login
  5. Open the REST Client Addon

The REST client addon uses the login performed in the steps above for subsequent calls to the server. This makes it easy, because it is transparent and there is no reason to worry about the details. The browser basically handled them.

Similar addons are available for the Chrome Browser. Examples are

  1. The Advanced REST Client as shown in this post.
  2. Postman as a Chrome App as explained at the end of the post above, which is no longer available.
  3. Tabbed Postman as Chrome App.

The browser integrations above all allowed to use the browser to login and use the login information in subsequent calls. These browser addons or extensions used to be very popular, but security threats and changes in integration philosophies seem to make them more and more unavailable. Some of the addons mentioned above are discontinued and might not be available much longer.

The Postman as packaged app as explained at the end of this post is currently still available and allows to configure a Postman Interceptor, that intercepts the login information from the browser and uses it in later calls. So it works like the solutions 1-3 above and the REST clients available as Firefox addons or extensions.

Alternatives are standalone solutions like Postman as standalone App which seems to be the path forward for Postman. There are many other stand alone REST clients. The caveat is, that the simple option to just login to the Jazz application is no longer available as an option. So it is important to know the details,

When trying to write custom tooling that needs to access data on a CLM/ELM server, it is also necessary to understand the details.

Testing and special considerations

It is a good idea to have a test system. Ideally local on your own machine. You can follow Lab 1 of the Rational Team Concert Extensions Workshop to set up a small system with JTS and RTC. It is not necessary to install the plain Java client libraries, any of the SDKs or to run the WorkshopSetup application. It is possible to install the other applications as well sing the Web Installer or one of the many Installation Manager repositories.

Create the sample project in the lifecycle project administration section to have data to play with.

Postman Settings

This blog shows the examples using Postman as standalone App which is very useful and allows to save and organize calls as a benefit. Like any application there are usually settings that can be beneficial or get into the way. Here the settings of interest in the context of this blog.

Disable SSL Certificate Verification

Please be aware that the SSL certificates of a test system as described above will typically be considered invalid. It is very common that test systems have invalid SSL certificates. A common issue is that the certificate is self signed and often for a different or generic host name. In such cases SSL certificate validation throws an error. Browsers, REST clients and APIs often reject working with servers presenting invalid SSL certificates. It is usually necessary to create exceptions or to configure the systems to ignore the SSL validation error.

Make sure the REST client or API works with sites that have invalid SSL certificates, or provide valid SSL certificates.

Postman rejects communicating with servers using invalid SSL certificates and displays an error message. This can be changed by changing the settings, turning the setting ‘SSL certificate verification’ to off. Postman now allows communicating to servers using invalid SSL certificates.

Disable SSL certificate verification
Disable SSL certificate verification to be able to work with example projects

As an alternative, it is possible to provide valid certificates. It is also possible to use any other test system that might be available, especially when dedicated test systems with valid certificates are available. The screen shots in this blog refer to a test system that has been set up similar to the Extensions workshop, but with all applications installed and the sample lifecycle project being deployed.

Automatically Follow Redirects

By default Postman automatically follows redirects. This can be nice but it hides a lot of details. To better understand the details explained in the wiki page Authentication of a native client with a Jazz-based application, switch the setting automatically follow redirects to off.

Setting to switch automatically follow redirects off and on.
Setting to switch automatically follow redirects off and on.

With the redirects disabled it is possible to see the 302 response as explained in Authentication of a native client with a Jazz-based application.

Minimal Header settings

When following redirects automatically, postman also collects several cookies. To focus on the really important details, and understand which headers and cookies are really needed, the setting below is very helpful.

Minimal setting - disable automatic redirect and additional headers
Minimal setting – disable automatic redirect and additional headers

Form based authentication

As explained in the wiki page Authentication of a native client with a Jazz-based application, there are multiple scenarios, how a client login would be performed. Unfortunately the number of scenarios are getting more over time as new standards are becoming available.

The easiest to use method that has worked for me so far has been the Form Based authentication. This works in a simple set up of CLM with any application server in a standard configuration without Jazz Authorization Server (OIDC) installed.

Perform a Form based authentication login to an application

To perform the Form based authentication to the application use the context root of the JTS or the application and append /j_security_check to the context root to create the login URI.

With a given context root of e.g. https://clm.example.com:9443/jts for JTS, the URI to login is https://clm.example.com:9443/jts/j_security_check.

As explained in Authentication of a native client with a Jazz-based application, the authentication is performed with a POST to the URI https://clm.example.com:9443/jts/j_security_check with a request body that contains the x-www-form-urlencoded encoded username and password.

The image below shows the simplest way of performing the authentication using Postman.

Minimum information required to perform FORM authentication
Minimum information required to perform FORM based authentication

The POST to the secure resource URI provides the username and password using the form encoded key and value pairs for the j_username and j_password. password and username are both identical myadmin.

The code for this request is shown below.

POST /jts/j_security_check HTTP/1.1
 Host: clm.example.com:9443
 cache-control: no-cache
 j_username=myadmin&j_password=myadmin

Please note that there are no additional headers necessary. The request does not specify anything in the request for Parameters, Authorization, Headers, Pre-request Script or Tests.

The response shows status 302. There is no header 
 X-com-ibm-team-repository-web-auth-msg. There is no header location with value  
 /auth/authfailed. This indicates the authentication was successful.
Post response shows status 302 and headers show no authentication failure.

The response shows status 302. There is no header X-com-ibm-team-repository-web-auth-msg. There is no header Location with any value, let alone
/auth/authfailed. This indicates the authentication was successful.

The Cookies keeps the LtpaToken to authenticate.

The LTPA Token is kept in a cookie.
The Cookies keeps the LtpaToken.

If the setting for Automatically follows redirects is on, and other header options are also set to on, the request would be redirected and likely show a status 200. There would be a lot more headers and Cookies collected as well.

Once successfully authenticated, it is possible to send other requests passing the Cookies. For this OSLC request it is also necessary to pass an accept header and the OCLS-Core-version header.

Successful retrieval of the OSLC workitems catalog URI.
Successful retrieval of the OSLC workitems catalog URI.

Please note that the Jazz Product APIs might need other special handling. In the above example the communication returns an additional Cookie JSESSIONID. In subsequent calls provide a header JSESSIONID with the value of the returned Cookie.

The easiest way to ‘log out’ is to delete the Cookies for the server.

delete all Cookies to restart.
Delete the Cookies for the domain to start again.

The request above can also be created by using a special content header Content-Type:application/x-www-form-urlencoded and a raw post body with a special encoding.

j_username=myadmin&j_password=myadmin

The images below show how this looks like in Postman.

The request above can also be created by using a special content header Content-Type:application/x-www-form-urlencoded and a raw post body with a special encoding.
Content-Type header
Raw POST request body.

This code shows the whole POST information.

POST /jts/j_security_check HTTP/1.1
 Host: clm.example.com:9443
 Accept: /
 X-Requested-With: XMLHttpRequest
 Content-Type: application/x-www-form-urlencoded
 cache-control: no-cache
 j_username=myadmin&j_password=myadmin

Authentication Failure

The Following image shows an unsuccessful authentication due to a typo in the password. This is only one of the possible flows, but the other scenarios are handled the same way.

The response shows status 302. There is a header Location with value /auth/authfailed. This indicates the authentication was not successful.
The Location header with value /auth/authfailed indicates a failed authentication attempt.

The response shows status 302. There is however a header Location with value /auth/authfailed. This indicates the authentication was not successful.

Authentication failed with Automatically follows redirects enabled leads to a JavaScript error page.
Authentication failed with redirect.

If the setting for Automatically follows redirects is on, the request would be redirected and likely show a status 200. The response body contains a page that the client could show to allow the user to enter the user ID and the password to authenticate. This web page uses JavaScript. Because the REST client Postman does not support JavaScript execution the page shows an error.

Where to Authenticate

My understanding so far has been that the CCM and the QM server have their own authentication. The RM server delegates authentication to the JTS. The Lyo implementation uses the public URI of the applications to create the authentication request. For the RM application it uses JTS instead.

The document Authentication of a native client with a Jazz-based application, seems to imply that authenticating to the JTS should do the trick for all applications. Using this has, so far, not caused any problems, so my suggestion would be to use the JTS URI to authenticate.

If you experience any issues, fall back trying to use the application specific URI.

Basic Authentication

Although Form based authentication has been the preference, Jazz also supports Basic authentication. However, when using Jazz Authorization Server/OIDC, Basic authentication is not supported. See this section (since 6.0.6).

See this information to understand the details.

As a summary, for Basic authentication, a special header Autorization that contains username and password encrypted in Base64 is sent with each request. The request below shows the header

Authorization: Basic c29tZXVzZXI6c29tZXBhc3N3b3Jk 

Testing and special considerations

When using Basic Authentication in Postman it is useful to have the Automatically follow redirects setting to be on.

To configure Basic authentication in Postman, select the Authorization type Basic Auth and enter the username and the password. Postman generates the authentication header. When using the API or other tools that do not create the authentication header, it has to be created and added.

Configure Basic Authentication.
Configure Basic Authentication.

The call below gets the workitems catalog for a RTC server on Jazz.net. This requires sending a valid Accept header and the OCLS-Core-version header.

Request with Basic Authentication.
Request with Basic Authentication.

The code below is the important information.

GET /jazz/oslc/workitems/catalog HTTP/1.1
 Host: jazzdev.rtp.raleigh.ibm.com:9443
 Accept: application/xml
 OSLC-Core-version: 2.0
 Authorization: Basic c29tZXVzZXI6c29tZXBhc3N3b3Jk
 cache-control: no-cache

When sending the request, the final response should have a status 200.

Successful response.
Successful response.

Please be aware that basic auth does not have a session handling and you have to always send the authentication header.

Malicious website

I am currently working with a customer trying to authenticate to one of their CLM systems. This seems to work except with the CCM server. When they try to authenticate they get the following message.

You have followed a direct link to log in to a Jazz server. This page has been presented to ensure that a malicious website cannot use cleverly crafted content to circumvent security. Please log in if you would like to access the server. 

We are looking into what could cause this issue. The web does not provide any good explanation. I found some questions in the Forum and other places and think I noticed a pattern. Several requests I have seen used a wrong URI to authenticate.

When using URIs like https://clm.example.com:9443/ccm/authenticated/j_security_check or
https://clm.example.com:9443/ccm/j_security_check and GET them in a browser, I get the same error.

You have followed a direct link to log in to a Jazz server. This page has been presented to ensure that a malicious website cannot use cleverly crafted content to circumvent security. Please log in if you would like to access the server.
Directly accessing internal URIs is prevented.

It is currently unclear what is happening and how to resolve the issue the customer is seeing. I will update the post, if we can figure out what is going wrong.

Summary

This blog post tries to shed some light on how to log into a CLM System to be able to use the REST and OSLC APIs. This should work for manual work as well as for automation tools. This blog uses Postman, but any REST client should work. The details, especially covering redirects and different authentication methods can be found inn this document Authentication of a native client with a Jazz-based application.

As always, I hope that this post helps users out there with using their Jazz tools.

Registering Custom Resource Intensive Scenarios to CLM Applications

There is no such thing as limitless computing power. This is an unfortunate truth that can cause problems running the CLM and other tools, as the usage grows. To understand what systems actually do when getting under heavy load, more and more monitoring was introduced over the last years. Resource intensive scenarios where identified and the CLM tools have capabilities to record information about their frequency and duration. Plan loading and SCM compare workspace are examples in the product.

Custom Resource Intensive Scenarios

In addition to resource intensive scenarios that are built in, it is also possible to introduce custom resource intensive scenarios. Some examples are:

  • Custom automation that execute long running operations on work items, SCM data, requirements, test artifacts. Typical scenarios are custom export/import, mass updates, custom analysis of source code, baselines, linked work items.
  • Follow up actions
  • Long running custom dashboards

This is by no means a comprehensive list. It is possible to bring your clients and servers to their knees with custom themes that do not scale, by work item attribute customization adding more and more custom attributes, JavaScript providers, value providers with thousands of values to choose from and other customization.

What is your server up to?

When users complain about performance problems, even if a server is getting overloaded, it is hard to find the root causes, because a typical server does so many things.

Monitoring that has been added over time has helped, but it is still hard. It is sometimes even hard to understand the situation. As an example for how complex this can become. Users complained about performance.

Our performance architect looked at the server load and the build load and a huge amount of calls that we were not able to account for. The server was unarguably under heavy load created by builds, but the build users and SCM users where not complaining. The developers we talked to had no real issues. Some users, at a different location, using work items and running work item queries, had.

Because we could not explain the inconsistent feedback, I finally went to the location where the users where complaining. I met the users followed their day to day work and found the work item performance unacceptable. The web browser was even locking up on them.

Knowing this, we were able to reproduce the use case, and look into what happened. We found that the work item load was slow, especially on slow laptops, because it had to load so many team areas and iterations. This was specific to how the project area was configured and used.

We also found that the browser flooded the server with requests that where definitely not part of what the product UI sent. This basically forced the Web Browser to process and cache thousands of calls, reserving more and more memory and exhausting the CPU capabilities of the relatively weak laptops used by the users that complained.

The final verdict was, that there was a custom extension to the theme that created all these calls. It took us weeks and was luck that we found this out. If we had known there was such an extension, we would have been able to find this a lot faster. The server was still under a heavy build load, but the performance issue reported was not related to that.

Needless to say that this extension was also deployed in other environments. If it had a detrimental impact, it was heavily depended on the timeline and iteration structure of a project area. The more and deeper the worse.

It would have helped if we could have seen the extension working, and see how long it worked would also have helped.

Registering Custom Resource Intensive Scenarios

The same mechanism that is used to register resource intensive scenarios in the product code can be used to register custom resource intensive scenarios. Unfortunately, we where lacking a good description and supporting code that we could provide customers to use it for their extensions.

This has now changed. Some colleagues and I, independently, started creating a customer usable description how to register resource intensive scenarios. A colleague wrote some cURL code to do this. I wrote Java code to do this and started creating a presentation. When we found out, we decided to combine the effort. Here the result.

The Deployment Wiki page Register Custom Scripts as a Resource Intensive Scenario, explains, using an example, how the API works in general. It also explains how to retrieve and monitor this information.

Then it provides example code to perform this using cURL, Eclipse Lyo OSLC4J based java code, and RTC Plain Java Client Libraries based Java Code.

The Java Code comes with main classes to run it. This is basically example code, but it can also be directly used in command line based automation.

Open Source Code

Disclaimer and Download

Any code downloadable or accessible in this post is provided as is, without support, and used at your own risk. Part of the code was developed in Java using Eclipse and is based on the Eclipse Lyo Client. This was published as open source, under
 Eclipse Public License – v 1.0, in the incredible (mostly German speaking) Jazz Community and can be found here: custom-expensive-scenario-notifier-oslc4j.

Another part of the code was developed in Java using Eclipse and is based on the Plain Java Client Libraries. This was published as open source, under MIT license, in the incredible (mostly German speaking) Jazz Community and can be found here: custom-expensive-scenario-notifier-plainjava.

See the other examples the Deployment Wiki page Register Custom Scripts As a Resource Intensive Scenario.

How does it work?

There is basically a REST API to register the start and the stop of a scenario. All there is to register the start of the scenario at the beginning and then register the stop, after you are done. See Register Custom Scripts as a Resource Intensive Scenario for more details on the code.

What should your automation do?

If you have written automation tools or extensions, you should use the methods described in Register Custom Scripts as a Resource Intensive Scenario, to register your extension as an resource intensive scenario. Add the code to register the start and stop in a way that allows for disabling it easily.

Monitor the various resource intensive scenarios over time. For a scenario that takes only a fraction of a second, you could temporarily disable the registration. Scenarios that take a second or longer should continue to be monitored.

Related

Feedback

If you have questions around the Custom Resource Intensive Scenario code, ask them in the Jazz.net forum instead of commenting on the article or this blog post. Tag the question as a clm question and add the tag: custom-resource-intensive-scenarios to mark it for the reader.

Summary

Please use the method above to enhance your automation and extensions to allow monitoring their duration, frequency and deviation.

As always I hope this helps users out there with the Jazz products.

Type System Manager Part 2

We finally published Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 3: Automation tool deep dive on Jazz.net.This was a major effort and took a long time to do. This article provides a closer look at the source code, what it does and how it does it. It also provides some insight in how OSLC4J works and can be used. The information in the article, especially for setup and deployment of the automation prototype is very reusable for other scenarios and I hope to be able to reuse it in later articles and blog posts.

Type System Manager

When this effort was planned and performed last year, we had no idea what would come out of this effort. When we finished the first iterations and I started to write Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 3: Automation tool deep dive, we called what we where working on an automation prototype in the articles. I decided to keep it that way.

Since then, I started to call the prototype Type System Manager and I will continue to use that name.

The article Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 3: Automation tool deep dive, basically explains the details of the code that was created until end of January 2019. This is Release 1 of Type System Manager.

New versions?

While working on the article, I continued to refine the Type System Manager. See the releases tab for the latest release. The latest releases support more automation It integrates finding editable configurations based on sub-strings in the description, create a source to target mapping and import/deliver type system changes based on this mapping. The new commands avoid the creation of a CSV file.

The code is also slightly refactored. Most of the content from Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 3: Automation tool deep dive, still applies, but don’t be surprised if there are slight changes in names and locations. There basically where inconsistencies that I found along the way and decided to fix.

The latest versions also register Custom Resource Intensive Scenarios.

TSM 1.1

In addition to all this, the code on this branch already supports Registering Custom Resource Intensive Scenarios to CLM Applications. Each command uses its own scenario name composed from the command name plus the postfix “Scenario” to register the scenario execution.

Disclaimer and Download

Any code downloadable or accessible in this post is provided as is, without support, and used at your own risk.

The code was developed in Java using Eclipse and is based on the Eclipse Lyo Client.

Thanks to IBM approving, the code was published as open source, under 
 Eclipse Public License – v 1.0, in the incredible (mostly German speaking) Jazz Community and can be found here.

Related

Feedback

If you have questions around the Type System Manager, ask them in the Jazz.net forum instead of commenting on the article or this blog post. Tag the question as a Rational DOORS Next Generation question and add the tag: dng-type-system-management to mark it for the reader.

Summary

As always I hope that the artifacts created for this blog and on Jazz.net will useful for the Jazz user community out there.

Type System Manager Part 1

It has taken some time since I published Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 1: Manual procedures which is about this Jazz.net article.

I was very excited about it, because I was contributing to the effort myself. We finally have made progress and just released the next part. You can now read the article about Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 2: Automation.

So, what is that all about?

Type System Manager

I did some prototyping, and we experimented with different possible approaches for this automation. What we came up with is explained in the article above.

In summary, I created a prototype, lets name it Type System Manager (TSM), that can be used to automate the type system management, supporting the best practices identified in Part 1.

Example Execution Output

The prototype uses available public Doors Next Generation OSLC/REST API’s to perform the necessary tasks.

So I have been lucky to be able to do some prototyping and learn a lot about OSLC and REST API’s. The result is a prototype, that has a useful application, and is also a demonstrator for how to create automation using OSLC and REST API’s.

Usage of OSLC and REST API’s has been missing on this blog and this prepares the foundation for hopefully more examples, here in this blog, in the future.

Disclaimer and Download

I should not have to write this, but as this is the internet 8), so here goes: Any code downloadable or accessible in this post is provided as is, without support, and used at your own risk.

The code was developed in Java using Eclipse and is based on the Eclipse Lyo Client.

Thanks to IBM approving, the code was published as open source, under
 Eclipse Public License – v 1.0, in the incredible (mostly German speaking) Jazz Community and can be found here.

What is next?

The second part explains how the TSM prototype can be used and what it does and how. The next part is currently under review and will explain the details of the code, how it works, how to re-use and add to it. It also explains how to download and work with the code.

Related

Feedback?

As mentioned in the article Comments, feedback, ideas, and experiences are greatly appreciated.

If you have questions, ask them in the Jazz.net forum instead of commenting on the article or this blog post. Tag the question as a Rational DOORS Next Generation question and add the tag: dng-type-system-management to mark it for the reader.

Summary

As always I hope that the artifacts created for this blog will useful for the Jazz user community out there. Feedback, also usage, is greatly appreciated.

RTC Extensions Workshop – How to fix SSL protocol errors preventing connection to Jetty Debug server

Since some time now I started to run into a blocking issue with the Extensions workshop. I was not able to find a solution so far. Today a colleague saw the same and asked for help. Here what I found.

Problem: the browser prevents connecting to the Jetty server

When launching the Jetty Debug server, everything seems to be fine. You can even connect with an Eclipse client to the server. However, if you try to follow the description in the Extensions Workshop and you try to connect with a browser to https://localhost:7443/jazz/admin you see a screen like the one below.

SSL Error_1

In Chrome the error claims

This site can’t provide a secure connection localhost sent an invalid response.

Try running Windows Network Diagnostics.

ERR_SSL_PROTOCOL_ERROR

Firefox is even more alarming and says

Secure Connection Failed

An error occurred during a connection to localhost:7443. Peer reports it experienced an internal error. Error code: SSL_ERROR_INTERNAL_ERROR_ALERT

The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.

Please contact the website owners to inform them of this problem.

Learn more…

Report errors like this to help Mozilla identify and block malicious sites

Edge complains

Can’t connect securely to this page

This might be because the site uses outdated or unsafe TLS security settings. If this keeps happening, try contacting the website’s owner.

Try this:

Go back to the last page

It looks like the usual SSL issue with the standard certificate, but it actually is not. It does not allow you to proceed and add an exception to connect to the web site.

You are basically stuck.

I had the same issue yesterday with 6.0.5 and 6.0.6 and I knew it was working very recently – when I moved to my new laptop. I searched the internet and was not really sure about a solution still. The answers pointed at version issues and certification mismatches. So I decided to switch the Eclipse environment to a current JDK and that solved the problem. I have tried it with RTC 6.0.5 and 6.0.6 but I am pretty sure it would work with other versions as well.

Please note that the Extensions workshop suggests to use the same JDK the Server uses. I think since Browsers have recently increased their security measures, this is no longer true.

Solution

Here how to fix the problem.

  • Download and install a recent Java JDK/JRE that is compatible with the version of the RTC SDK you are working with.
    • 6.0.6 is compatible with Java 1.8
    • I use a JDK because there are other development tools that require a JDK
  • Open the Eclipse server development workspace
  • Open Window>preferences
  • Type JRE and navigate to Java>Installed JREs
    • Add the new JDK/JRE
    • Select the JDK/JRE as active click Apply and CloseNewJDK
    • Navigate to Java>Installed JREs>Execution Environments
    • Select the Execution Environment with the matching version
    • Select the new JDK/JRE as active click Apply and CloseExecEnvironment
  • Launch the Jetty debug serverLaunch

Now try to login to https://localhost:7443/jazz/admin again. You still get an error, however the browser provides the Advanced link to continue connection to the site.

SSL Error_2

The different browsers display it different, but the concept is the same for all of them.

Summary

Recent increases of browser security checks and invalidation of certificates requires to update to recent versions of the Java JDK/JRE to be used with the Extensions workshop. As always I hope that this helps people out there.

Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 1: Manual procedures

I have been involved in efforts around the configuration management features in the IBM Collaborative Lifecycle Management and Continuous Engineering Solution. These Configuration Management capabilities are a completely new approach to versioning and variant management across the different disciplines such as Requirements Management, Quality Management, Modeling, and Software Change and Configuration Management. We realize that there is a whole new area for best practices and our subject matter experts have been working on creating necessary information for some time. There is already some content available on Jazz.net. For example from Kathryn and Tim in the Deployment Wiki.

As we learn we try to create more content, involving more SMEs. Today we published the article Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 1: Manual procedures.

I am currently working on Part 2. If you want to know what that could be, check the article.

I am pretty excited, I hope to publish soon!

Jazz Community Contributions

The Jazz Community starts sharing their tools here: http://jazz-community.org/. The code for their tools can be found here.

There is a very active Jazz user community of members of several companies in Europe that are heavily using the Jazz products such as Rational Team Concert, Rational Quality Manager and Doors Next Generation.

The community members try to meet to share their experience with using, administrating and running the Jazz tools in their environments. It became clear that the different companies and community members face similar challenges and that it would be beneficial if they could share tools they created to make running such an environment easier.

The community has now started sharing their tools in this community project and in this code repository.

JazzCommunity2017-05-12_11-34-52

Some of the tools have already been shared on other sites. I have linked the ones I am aware of to the Interesting links page in the ‘Extensions Provided by the Community’ section. These are the ones I am aware of (and the code for some of them is already available in the community repository):

I am looking forward to see more community created tools soon. Visit the community to find out what they have to offer. The code for their tools can be found here.

Last but not least, a special thanks to Dani for getting this awesome user groups started and for the members of said community for their spirit, engagement and willingness to contribute and help each other. You know whom I address here!