EWM Attribute Customization Introduction

While working on Debugging EWM Work Item Attribute Customization JavaScript I thought about all the questions I have seen being asked about Attribute Customization on the Jazz.net forum. There are some common questions I have seen a lot. There are some common things users would like to do and there are some limitations that might get into the way. I thought, I would create a blog that summarizes what I know about attribute customization and also provide the important links. I also want to provide a short summary about, for all I know, what can and can not be done with Attribute Customization and the provided API. I already create something like that in RTC Process Customization – What you can and cannot do. I will follow its structure and reuse some content but will try to be more specific in this blog. This blog will be limited to the work item attribute customization, with some few exceptions. The blog post will mention possible solutions that go beyond attribute customization, where this seems reasonable.

Important links

The following links are the most important references about EWM work item configuration and attribute customization.

About Jazz.net forum questions

A lot of the input that I take under consideration comes from questions and answers on the Jazz.net forum. Some of the examples originate from customer requests.

Questions on Jazz.net and even customer requests do often not provide good requirements. They often provide some implementation idea instead. The real problem or business case is usually hidden and not exposed. So it is important to question the request to drill down and expose why the question was asked as it is and what the background of the question is. If the real reason for the question is exposed, the answer can be completely different.

One example is: “Can I auto subscribe users to a work item?”. In pretty much all cases I can remember, the purpose of this was to send an e-mail notification to a user e.g. when the work item was created. Adding a user to the subscribers might or might not be a viable solution, but the example clearly exposes the problem.

So take the questions on Jazz.net with a grain on salt and try to dig deeper to understand what the question is really about.

What is Process Configuration and Attribute Customization?

EWM allows to configure the process of a project area. There are several options that can be used.

  1. Work items, work item attributes, editor presentations, workflows can be configured.
  2. Roles can be configured
  3. Permissions can be configured
  4. Operational behavior can be configured using out of the box preconditions and follow up actions.

Attribute Customization is a work item related capability provided by EWM. It provides built-in capabilities that can be configured and thus be considered process configuration as mentioned above.

The capability types available for configuration are:

  1. Default value provider for built in attribute types.
  2. Calculated value provider to support calculation with work item attributes to support certain processes.
  3. Value Sets to provide customized sets of selectable values for work item attributes. Built in value sets allow to extract string values from an XML page in the Web, select contributors by role and allow to select enumeration vales based on other dependent enumerations.
  4. Validators to validate values of attributes based on range or regular expressions. Validators can show a warning in the UI and be used with a built in advisor to prevent saving a work item, unless the offending attribute values are corrected.
  5. Conditions can be used with built in advisors to make work item attributes read-only or required. There is no built in condition available at this time.

What is special about Attribute Customization?

What sets attribute customization apart from the other capabilities is, that it provides a JavaScript API that allows to create custom scripts for provider configurations. The JavaScript can be uploaded to the project areas process by any user with the required permission. This does not require write access to the application servers file system. For this reason it is possible to create and deploy custom JavaScript scripts even if the server is maintained by a cloud provider.

JavaScript is also reasonably easy to learn and a lot of practitioners have some JavaScript skills. This is a two edged sword.

Contrary to Eclipse extension plugins, another extension mechanism available in EWM, the JavaScript API has a relatively low experience threshold for beginners. It is way easier to develop than the other Eclipse plugin based customization options available in EWM that require great skills, understanding of the framework and familiarity with the available API. Since these options are based on Java and Eclipse plugin extension development, they require the plugins to be deployed which means they need to be copied into a file system, typically on the server where the CCM application is deployed. This is usually not possible in a cloud hosted environment. Except, some cloud services would perform such deployments for the customer, for a higher price tier.

The danger is that the low skill cap attracts very inexperienced users, which can eventually lead to issues, including server performance issues. Some of the capabilities can be overused. There is also a tendency to overcomplicate the work item workflow and the process.

The JavaScript Attribute Customization API constraints and limitations

Attribute Customization is configured on the attribute level. This means that any attribute customization will apply to all work item types that have the attribute configured.

JavaScript attribute customization scripts have to implement the required interface for their type. All the interfaces require to return some data. The data can be empty or undefined, dependent on the attribute type, e.g. an empty string or null. It has to return something that matches the attribute type it is configured for, nevertheless.

It is possible to detect the type of the work item the script is executed for, this allows to have different behavior for different work item types in the JavaScript attribute customization script. Based on the work item type attribute value, the script decides what to return, some custom value or the current attribute value to do nothing. This strategy allows to adjust for type specific attribute customization to some extend.

The API description in the Work Items Attribute Customization Wiki page explains which work item attribute value types are supported. Only very basic attribute types such as strings, numbers, enumerations, timestamps and dates are really supported. Especially note that the support for complex item based attribute types such as team areas, contributors etc. is very limited. For a those attributes, all you can access is the UUID and the display name. If you want to set such values, you must return the UUID of the item.

The documented API does not provide with methods to discover information beyond the work items current attribute values. There is no documented simple JavaScript API to discover details about not attribute based information. There is no documented simple API to follow or create links to work items or other artifacts. There is no documented API to access process information, such as users and roles or any other details. There is also no mechanism to detect a state change or workflow action. The workflow action is only available in conditions but the detection of a state change would be desirable in calculated values as well.

The return value of JavaScript attribute customization scripts is processed in the API. Dependent on the work item attribute type, the script is configured for, some kind of conversion is performed. As an example a string can be converted into a timestamp, provided the format is correct.

As far as I can tell, the conversion that is performed for html and string based attributes, including the subject and description, when returned from JavaScript converts the content into a string escaping any special content such as XML or HTML tags. This conversion happens outside of the JavaScript and there is no way to change the behavior. Because of this reason it seems to be impossible to return HTML with working links for HTML or string based attributes including the summary and description. Only the built in configurable default values work correctly. The only current attribute type that can be set by JavaScript attribute customization to contain links is the type Wiki. In this case the Wiki syntax has no XML/HTML tags involved that would have to be escaped.

Performance considerations

Some users seen to think there is such a thing as limitless size and performance. This does not apply to the real world. It is possible to heavily impact the performance of clients or servers with process configuration and especially customization. I am aware of a case, where JavaScript added to a Theme caused so much communication that the browsers on smaller laptops where drained of resources and almost came to a halt. Loading a work item took several minutes. This made working with the tools impossible for many users. So be careful and consider what you might do to the performance.

It is also very easy to annoy the users with overcomplicated processes, thousands of choices, required attributes nobody knows the values for etc. In general big numbers are bad and small numbers are good. When drop down boxes grow too big and the number of items ever increases, the users will be affected in their ability to successfully use the process. I have seen cases where the users got confused and did not realize that they had to enter a value and complained the UI had stopped. Overcomplicated processes will likely cause performance complaints and wear away the will to support the chosen tool and process.

There are limitations in the amount of custom attributes for a work item type (100). Having a lot of work item types is also not helping to make the process easier. Keep numbers low e.g. 10 would be a guidance for work item types.

Also keep in mind that there are size limits for all work item attribute types. Especially note that for list type attributes there is a size limit. How many selections can be stored is limited by the site of the list attributes and the size of the ID’s or item UUID’s that are stored.

Where does the JavaScript run?

The JavaScript runs in the Web Browser or in the EWM Eclipse client in a JavaScript interpreter..

Where and when is Attribute Customization not executing

Attribute customization including the JavaScript is not executed when the work item operations are created using an API such as OSLC or the SDK or Plain Java Client Libraries.

Work Item Proxy

When debugging the JavaScript, it is possible to see more objects accessible to the JavaScript in the debugger. It is possible to see an object called work Item proxy that looks very promising. Please note that this object is considered internal API. It should not be used by the JavaScript. The supported API is documented here. It is also important to understand, that this work item proxy is only available in the Web Browser and not if the script runs in the Eclipse client.

There are several posts on the Jazz.net that discuss how the proxy could be used. Again, it is internal API and should not be used, if you do, you are on your own. There might be unwanted side effects and the internal API could change at any time.

Why does my script not work?

This question comes up often on the forum. It is pretty much impossible to debug others scripts on Jazz.net. There are reasons for that:

  1. Java Script Attribute Customization depends on the process used in the project area. The community does not have and, due to privacy, security and other concerns will not get this.
  2. It is quite some effort to setup a test project area, even if one would have a process template.

So it is usually impossible to help on this level. What can be done is scanning the script and the description in the question for typical issues.

See Debugging EWM Work Item Attribute Customization JavaScript for how you can debug your own scripts.

Please note, there are different levels of issues with JavaScript. On the most fundamental level:

  1. Attachment Scripts must be enabled in the advanced teamserver.properties to be working – see below.
  2. The script must be accessible in the process.
  3. The script files encoding must be UTF-8 and consistent so the file can be read, interpreted and executed.
Process Attachment Scripts enabled.

If an error happens at this level the best you can hope for is an entry in the log file. Only if the JavaScript is successfully decoded and loaded, it can be debugged.

What you can and cannot do with JavaScript Attribute Customization

There are certain limitations in the JavaScript Attribute Customization API as far as I am aware. I provided a more comprehensive presentation here: Process Customization – What you can and cannot do.

I will try to provide some details here.

  1. Attribute Customization JavaScript is always defined as a function. Regardless what type of Attribute Customization the script must return a value (can be null or empty). Otherwise this will be a potential error in the log. I am not sure what happens worst case.
  2. Attribute customization is for the attribute. This means especially that the customization will be executed in all work item types that have the attribute configured.
  3. Sure, in JavaScript, you can get the work item type, by reading the respective attribute. You can then decide what to return. As explained in 1. you cannot return nothing. You however can get the current value of the attribute and return that, this helps in some cases.
  4. As far as I am aware, Attribute Customization only runs in the context of a users browser and the Eclipse client. As explained above, this also means that automation based on attribute customization does not work, if you access work items using any of the APIs. This is important, as pre-conditions/advisors and follow up actions/participants can be implemented as server extensions that always run, regardless who accesses the work item how. When the JavaScript is run in the Eclipse client, only the infrastructure of the Eclipse client is available.
  5. Attribute customization scripts can run multiple times, for example the calculated values might execute multiple times, based on changes to the other attributes they depend on.
  6. JavaScript Attribute Customization only has access to the data of the work item in which context it runs. It is not possible to follow work item links using the documented API.
  7. Even the available data of a work item is limited. As an example, a workflow action for a state change is only available in conditions. It is not available in any other attribute customization types, especially not in calculated values. This essentially means, there is no simple way of tracking and detecting a workflow state change.

Java Eclipse plugin extension based Attribute Customization

Beyond JavaScript, it is possible to provide Java based Work Item Attribute Customization Plugin Extensions. This is possible for all available attribute customization types, default values, calculated values, value sets, conditions and validators as shown in these examples. These Java based plugin extensions have access to more complex EWM Java SDK API. This makes it more feasible to create complex custom attribute customization. It is also easier to control which return result types are supported and, based on these, create richer attribute customization. It would, for example, absolutely be possible to have a default value provider that returns an XML or HTML result, that shows working HTML links. The decision to return a result that escapes the HTML tags or return the correct format is an implementation detail. A more complex example with additional background information can be found here and in this post.

The Java based attribute customizations would show up in the Admin UI in the same way the built in Attribute customization does. To make this possible the Java based Eclipse plugins need to be deployed in the Eclipse client as well as the EWM Server (see here). The API that is available to the Eclipse client and the server is the EWM Common SDK API. Consider reading this blog post to better understand what this means. The requirement to deploy these extensions on the EWM server are a potential inhibitor for cloud based deployments.

EWM Eclipse Plugin Extensibility

Java base attribute customization is only a subset of the Java Eclipse plugin extensibility available in EWM. The SDK provided for EWM allows to develop all kinds of extensions, including pre-conditions, follow up actions and asynchronous tasks. It is possible to develop such extensions to run in the context of the Eclipse client or on the server. These extensions need to be deployed in the context they are developed for. Server extensions have access to the full capabilities of the EWM Common SDK API and the EWM Server SDK API.

Preconditions also called Advisors can act as process advisors that prevent saving if the data violates some standards. Preconditions can be used to implement attribute customization validators or conditions. Example Advisors can be found here.

Follow up actions, also called Participants can perform additional operations on the EWM data. As an example, follow up actions can collect data across linked artefacts, perform calculations on the data, set attributes and properties for items and save the changes. Example participants can be found here. This includes the capability to create new work items and link the new work items to other work items.

In some cases it is better to use these capabilities instead of attribute customization, because it provides more API than even the Java based attribute customization.

All plugin based solutions require to be deployed. Server extensions require to be deployed in the server, which might be a blocker in an environment hosted as a service in the cloud.

Popular requests

In the next sections, I will try to provide some examples of popular requests. I will describe the background these request have, as far as I can tell, and which implementation options might be viable.

Popular: Custom E-Mail Notification

This comes up in various flavors. Examples are

  1. Can I automatically set an owner of a work item?
  2. Can I automatically subscribe a user to a work item?

With respect to attribute customization, the answer would be yes, you can do this, to some extent. It is possible to set or add a subscriber, to set or add a work item owner. In the JavaScript API this needs to be done the contributors UUID. The UUID will have to be configured or hard coded when using the JavaScript API. There is no API to search for contributors in JavaScript.

A Java based attribute customization plugin would have more API access and can provide better control and automation.

A real custom mail notification is possible using the following approaches

  1. Custom scheduled asynchronous tasks (see Due Date Notifier).
  2. Custom follow up actions that trigger at work item save.

The last three options require to write custom Java extension plugins and their deployment on the application server, which makes it less attractive in a cloud environment.

As an alternative, EWM has other mechanism that might be better suited. For example there are RSS feeds for various change events, work item queries and dashboards. These do not send e-mail, but they might scale better than e-mail notifications.

Keep in mind that the e-mail notification options are controlled at user level by the user. A real Java Eclipse plugin that sends custom e-mail notifications could work without the need to be correctly configured at user level.

Popular: Calculation with work item attributes

This is often requested to calculate effort or something similar.

With respect to JavaScript based attribute customization, it is possible to perform calculations on attributes of the same work item. It is possible to perform numerical, boolean and other operations in calculated values. It is necessary to set the dependencies right, so that the chain of events executes until all the calculations have been performed. If multiple work item attributes need to be set, keep in mind, that the calculated value is a function that only works for one work item attribute. If multiple attributes need to be set, a calculation for each attribute is necessary. As already mentioned, It is not possible to calculate across values of multiple work items.

If it is necessary to calculate across multiple work items, JavaScript attribute customization does not work. It is necessary to use the Java plugin extension mechanisms such as follow up actions to implement this. An example is the RTC Update Parent Duration Estimation and Effort Participant.

It is also possible to use external tools based on API to collect data and then update work items e.g. in scheduled jobs.

Popular: Calculate how Often a work item was in a specific state

I have seen this several times. There is no easy way to detect work item changes. It might be possible to use an approach as shown below to detect this, but I am not sure if this idea below would really work.

With some tricks and several hidden custom attributes for state tracking ST1, ST2 and counting STC, it might be possible to cobble something together. E.g. initialize the hidden state tracking attributes st1 and st2 with a default based on the state ‘New’.

The calculated value of ST1 is triggered and calculated based on the current value of the work item state attribute. If the current state is not the value of ST1 return the new current value.

Calculate the counter based on a triggered change of ST1. If ST1 contains the desired state, return the current value of STC increased by one.

Calculate the value of ST2 when ST1 was changed. Return the value of ST1 as result to the calculated value.

This requires some intelligent usage of the dependent attributes. It might be necessary to use more artificial attributes to get this working. This attempt has however many downsides, especially the custom attributes are all showing up in the history and also create more changes in the work item.

It would be easier to develop a follow up action that actually works on the state change, if this is necessary. It would be important to see if reporting can be used instead.

Popular: Make work item attributes read-only or required

EWM has preconditions/Advisors to make work item attributes read only or required based on conditions. The question is, how far can you go with this? The JavaScript API is very limited in what data is available. It is obvious that conditions can work on this data. So you can make attributes read only or required on work item attribute information.

But since you have no access to any complex information such as team membership or roles or the current contributor, JavaScript can not be used to create more complex conditions. Any condition that needs to have access to more complex information, would require to be written as Java plugin. An example requested by a customer with additional background information can be found here and in this post. It is important to know that these condition based attribute customizations are the only way to customize rules to make attributes read-only or required. There is no other extension point available that you could use to hook into. This is one of the rare cases where custom preconditions/Advisors can not be used.

In my experience with requests like here and in this post, to make attributes required or read only, are often motivated by the desire to have more fine grained control. Consider to use permissions to prevent roles from changing attributes.

Requests like above are also often motivated by the desire to make EWM work like some legacy tool. My experience so far has always been, that trying to make a tool work against its design is usually fatal. It usually makes the process hard to understand and follow. It also often degrades usability and performance.

Since every user in EWM has at least the default role (Everyone), can have many roles, and the roles have an order, it is also debatable what ‘by role’ really means.

Popular: Dynamically Hiding Work Item Attributes

I have also seen several requests that want to make attributes hidden by some complex, usually role based rule. This is not possible today, regardless the approach.

There are very limited capabilities to control visibility built into EWM. Attributes can be hidden if they are empty, if the work item is in a workflow state, if some endpoint is empty, if there is no project link and if the work item is just created. An editor presentation tab can be hidden, if all contained attribute presentations are empty.

There is no extension point, and there is no attribute customization available, I am aware of, that would allow customize the behavior in any other way.

There are various enhancement requests for EWM in this context. I think it would be nice if there was a custom attribute customization option that could be used with a precondition e.g. ‘Hide for condition’ that could be used similar to ‘Read only for condition’. This would allow to hide attributes that are unimportant. The question is, if it is helpful when attributes show up and vanish for the same attribute type. This is at least debatable. As a user I would likely find this confusing. As always, keep in mind this could open the door for performance issues and abuse. The remark about roles and their usage in EWM also applies.

Please note, just because an attribute is not visible in the work item editor, presentation does not make it inaccessible by work item queries or reports. EWM was also designed for collaboration and not hiding.

Popular: Dynamic workflow

I don’t know where this comes from. I doubt this is a thing in other tools for change tracking, but it cam up more than once.

EWM/RTC does not support dynamically changing the workflow of an instance of a work item. The workflow is tied to the work item type. It would be possible to use a follow up action/Participant to change the work item type under certain circumstances. Due to the lack of support for detecting a state change and the lack of access to other data in attribute customization, it would be not feasible to use attribute customization. A participant is a far better option.

Popular: Dynamic creation of attributes

This came up several times.EWM does not officially support this. It is possible to create an attribute using the API, but this attribute does not surface in the admin UI. It is necessary to add statements to the process XML to make such an attribute configurable in the process and the editor presentations.

This is used by some integrations just once and not meant to be used constantly during operations. Also mind the limit of custom attributes and consider usability of the process.

Restrict Read Access To Data

Some customers want to have more control about who can access which data. EWM has the capability to restrict work item and SCM item read access permission based on access context. If an item can not be read, it can also not be written.

For work items, the restricted access context can be an access group, project area, or team area. It is possible to automate setting the work items restricted access based on categories. This sets the access context to the process area associated with the work item category. This is the only built in automation for category based work item access control. There is no other built in automation for work items e.g. enabling the automatic selection of an access group for the work item.

For SCM data there are more options to set the access context/visibility. Components can have access control set to a project area or an access groups. SCM content in a component, folders, files, can be configured more fine grained. It is possible to limit access to a contributor, a project or team area or an access group. There is no built in automation to set the access context for SCM items.

It is possible to create custom automation to set the access context for work items and SCM items. One implementation approach is using follow up actions. For work item restricted access this can be configured for the work item save operation. For SCM objects follow up actions for the check in and deliver operation could be developed to compute the desired access context. I have explained some of the options in the blog series Setting Access Control Permissions for SCM Versionables.

In contrast to my knowledge back then, I know today, that it is possible to elevate the operation context to an administrator, why this is important is explained in the second last paragraph.

The question would be, can JavaScript attribute customization be used for custom automation for the restricted access? Attribute customization is only available for work items. There is no such API for SCM content. It might be possible to set the restricted access attribute in JavaScript attribute customization by returning the UUID of the access context, but as already mentioned, because of the very limited amount of data and API access available, JavaScript would most likely not be feasible for anything beyond that.

Java Based attribute customization could be more feasible, but it is most likely not possible to elevate the operation to super user access due to the limitation of the available API to the common API. I would consider this approach to be unfeasible for any relevant complexity.

If it is necessary to develop a Java Eclipse plugin for attribute customization anyway, it looks to be more suitable to create a follow up action to automate setting the access context. This allows to use the full capabilities of the EWM SDK common and server API and it is possible to elevate the operation to run in an administrator/super user context.

The reason why it would it beneficial to run the operation in an administrator context is, that the operational behavior usually runs with the permissions and limitations of the user that performs the save operation. This also means the operation fails, if the permissions are preventing it. This limits what can be done. For example it is impossible to change the access context of an item in a way that makes it inaccessible to the user performing the operation. The operation would fail. Elevating the operation to run in an administrator context, would allow such an operation to be successfully executed, regardless of the repository access and permissions of the regular user triggering the operation.

I would suggest to not try to use attribute customization for these use cases.

Summary

As a summary, attribute customization, especially using JavaScript can be used for some interesting purposes. There are however a lot of limitations in the JavaScript API that prevent several use cases. The biggest benefit is, that it can be used without having to deploy anything on a server. This allows it to be used in cloud deployments, where the users have no access to the server file system.

For more complex problems, the Eclipse plugin Extensibility is a better option, regardless if it is the Java based attribute customization, or Eclipse plugin based EWM extensions, just because the APIs provided in the EWM SDK are so much richer.

This was quite some effort. As always, I hope that this is useful to someone out there.

Advertisement

The RTC Extensions Workshop has been updated for EWM 7.0.x

I am very passionate about the RTC Extensions Workshop as you might be able to tell from the content of this blog. Performing it with EWM 7.0.x provided several challenges. It became apparent that an update to the workshop would be beneficial.

I spent a considerable amount of time in the past two months to update the workshop. As a summary the following items where addressed:

  1. Since the CCM server is shipped with WebSphere liberty profile, configuring the server for debugging needed to be changed. The old way to configure the server still worked in the 6.0.x versions, so it went unnoticed. With EWM 7.0.1 this is no longer the case and the workshop was updated to address this.
  2. The advanced capabilities introduced in the EWM SCM system in the 6.x and later caused a deviation of the screen shots showing the pending changes. The workshop setup tool was slightly changed to fix this.
  3. The workshop setup tool and its shell script has been tested with Linux and MAC OS.
  4. I wanted to add a section to Lab 1, explaining how to setup the existing Eclipse client/server development workspaces to better support development and debugging of the Plain Java Client Libraries forever. The new last optional section addresses this. For this reason Lab 1 of the workshop is a must for anyone intending to create Java based automation or extensions to RTC/EWM.
  5. I had an errata list with a number of small issues, typos, naming inconsistencies and the like that were fixed. During reviews a bunch more showed up and were fixed.
  6. A colleague ran the workshop on his MAC, so this works. Use whatever is available for MAC like Eclipse and where this is not specifically available, use the Linux versions.

The RTC Extensions Workshop has been published with an additional section for the new EWM versions and is now available for download. I will update recent posts around the workshop in the next few days.

As always, I hope that this blog post helps the users in the Jazz Community.

Registering Custom Resource Intensive Scenarios to CLM Applications

There is no such thing as limitless computing power. This is an unfortunate truth that can cause problems running the CLM and other tools, as the usage grows. To understand what systems actually do when getting under heavy load, more and more monitoring was introduced over the last years. Resource intensive scenarios where identified and the CLM tools have capabilities to record information about their frequency and duration. Plan loading and SCM compare workspace are examples in the product.

Custom Resource Intensive Scenarios

In addition to resource intensive scenarios that are built in, it is also possible to introduce custom resource intensive scenarios. Some examples are:

  • Custom automation that execute long running operations on work items, SCM data, requirements, test artifacts. Typical scenarios are custom export/import, mass updates, custom analysis of source code, baselines, linked work items.
  • Follow up actions
  • Long running custom dashboards

This is by no means a comprehensive list. It is possible to bring your clients and servers to their knees with custom themes that do not scale, by work item attribute customization adding more and more custom attributes, JavaScript providers, value providers with thousands of values to choose from and other customization.

What is your server up to?

When users complain about performance problems, even if a server is getting overloaded, it is hard to find the root causes, because a typical server does so many things.

Monitoring that has been added over time has helped, but it is still hard. It is sometimes even hard to understand the situation. As an example for how complex this can become. Users complained about performance.

Our performance architect looked at the server load and the build load and a huge amount of calls that we were not able to account for. The server was unarguably under heavy load created by builds, but the build users and SCM users where not complaining. The developers we talked to had no real issues. Some users, at a different location, using work items and running work item queries, had.

Because we could not explain the inconsistent feedback, I finally went to the location where the users where complaining. I met the users followed their day to day work and found the work item performance unacceptable. The web browser was even locking up on them.

Knowing this, we were able to reproduce the use case, and look into what happened. We found that the work item load was slow, especially on slow laptops, because it had to load so many team areas and iterations. This was specific to how the project area was configured and used.

We also found that the browser flooded the server with requests that where definitely not part of what the product UI sent. This basically forced the Web Browser to process and cache thousands of calls, reserving more and more memory and exhausting the CPU capabilities of the relatively weak laptops used by the users that complained.

The final verdict was, that there was a custom extension to the theme that created all these calls. It took us weeks and was luck that we found this out. If we had known there was such an extension, we would have been able to find this a lot faster. The server was still under a heavy build load, but the performance issue reported was not related to that.

Needless to say that this extension was also deployed in other environments. If it had a detrimental impact, it was heavily depended on the timeline and iteration structure of a project area. The more and deeper the worse.

It would have helped if we could have seen the extension working, and see how long it worked would also have helped.

Registering Custom Resource Intensive Scenarios

The same mechanism that is used to register resource intensive scenarios in the product code can be used to register custom resource intensive scenarios. Unfortunately, we where lacking a good description and supporting code that we could provide customers to use it for their extensions.

This has now changed. Some colleagues and I, independently, started creating a customer usable description how to register resource intensive scenarios. A colleague wrote some cURL code to do this. I wrote Java code to do this and started creating a presentation. When we found out, we decided to combine the effort. Here the result.

The Deployment Wiki page Register Custom Scripts as a Resource Intensive Scenario, explains, using an example, how the API works in general. It also explains how to retrieve and monitor this information.

Then it provides example code to perform this using cURL, Eclipse Lyo OSLC4J based java code, and RTC Plain Java Client Libraries based Java Code.

The Java Code comes with main classes to run it. This is basically example code, but it can also be directly used in command line based automation.

Open Source Code

Disclaimer and Download

Any code downloadable or accessible in this post is provided as is, without support, and used at your own risk. Part of the code was developed in Java using Eclipse and is based on the Eclipse Lyo Client. This was published as open source, under
 Eclipse Public License – v 1.0, in the incredible (mostly German speaking) Jazz Community and can be found here: custom-expensive-scenario-notifier-oslc4j.

Another part of the code was developed in Java using Eclipse and is based on the Plain Java Client Libraries. This was published as open source, under MIT license, in the incredible (mostly German speaking) Jazz Community and can be found here: custom-expensive-scenario-notifier-plainjava.

See the other examples the Deployment Wiki page Register Custom Scripts As a Resource Intensive Scenario.

How does it work?

There is basically a REST API to register the start and the stop of a scenario. All there is to register the start of the scenario at the beginning and then register the stop, after you are done. See Register Custom Scripts as a Resource Intensive Scenario for more details on the code.

What should your automation do?

If you have written automation tools or extensions, you should use the methods described in Register Custom Scripts as a Resource Intensive Scenario, to register your extension as an resource intensive scenario. Add the code to register the start and stop in a way that allows for disabling it easily.

Monitor the various resource intensive scenarios over time. For a scenario that takes only a fraction of a second, you could temporarily disable the registration. Scenarios that take a second or longer should continue to be monitored.

Related

Feedback

If you have questions around the Custom Resource Intensive Scenario code, ask them in the Jazz.net forum instead of commenting on the article or this blog post. Tag the question as a clm question and add the tag: custom-resource-intensive-scenarios to mark it for the reader.

Summary

Please use the method above to enhance your automation and extensions to allow monitoring their duration, frequency and deviation.

As always I hope this helps users out there with the Jazz products.

RTC Extensions Workshop – How to fix SSL protocol errors preventing connection to Jetty Debug server

Since some time now I started to run into a blocking issue with the Extensions workshop. I was not able to find a solution so far. Today a colleague saw the same and asked for help. Here what I found.

Problem: the browser prevents connecting to the Jetty server

When launching the Jetty Debug server, everything seems to be fine. You can even connect with an Eclipse client to the server. However, if you try to follow the description in the Extensions Workshop and you try to connect with a browser to https://localhost:7443/jazz/admin you see a screen like the one below.

SSL Error_1

In Chrome the error claims

This site can’t provide a secure connection localhost sent an invalid response.

Try running Windows Network Diagnostics.

ERR_SSL_PROTOCOL_ERROR

Firefox is even more alarming and says

Secure Connection Failed

An error occurred during a connection to localhost:7443. Peer reports it experienced an internal error. Error code: SSL_ERROR_INTERNAL_ERROR_ALERT

The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.

Please contact the website owners to inform them of this problem.

Learn more…

Report errors like this to help Mozilla identify and block malicious sites

Edge complains

Can’t connect securely to this page

This might be because the site uses outdated or unsafe TLS security settings. If this keeps happening, try contacting the website’s owner.

Try this:

Go back to the last page

It looks like the usual SSL issue with the standard certificate, but it actually is not. It does not allow you to proceed and add an exception to connect to the web site.

You are basically stuck.

I had the same issue yesterday with 6.0.5 and 6.0.6 and I knew it was working very recently – when I moved to my new laptop. I searched the internet and was not really sure about a solution still. The answers pointed at version issues and certification mismatches. So I decided to switch the Eclipse environment to a current JDK and that solved the problem. I have tried it with RTC 6.0.5 and 6.0.6 but I am pretty sure it would work with other versions as well.

Please note that the Extensions workshop suggests to use the same JDK the Server uses. I think since Browsers have recently increased their security measures, this is no longer true.

Solution

Here how to fix the problem.

  • Download and install a recent Java JDK/JRE that is compatible with the version of the RTC SDK you are working with.
    • 6.0.6 is compatible with Java 1.8
    • I use a JDK because there are other development tools that require a JDK
  • Open the Eclipse server development workspace
  • Open Window>preferences
  • Type JRE and navigate to Java>Installed JREs
    • Add the new JDK/JRE
    • Select the JDK/JRE as active click Apply and CloseNewJDK
    • Navigate to Java>Installed JREs>Execution Environments
    • Select the Execution Environment with the matching version
    • Select the new JDK/JRE as active click Apply and CloseExecEnvironment
  • Launch the Jetty debug serverLaunch

Now try to login to https://localhost:7443/jazz/admin again. You still get an error, however the browser provides the Advanced link to continue connection to the site.

SSL Error_2

The different browsers display it different, but the concept is the same for all of them.

Summary

Recent increases of browser security checks and invalidation of certificates requires to update to recent versions of the Java JDK/JRE to be used with the Extensions workshop. As always I hope that this helps people out there.

Jazz Community Contributions

The Jazz Community starts sharing their tools here: http://jazz-community.org/. The code for their tools can be found here.

There is a very active Jazz user community of members of several companies in Europe that are heavily using the Jazz products such as Rational Team Concert, Rational Quality Manager and Doors Next Generation.

The community members try to meet to share their experience with using, administrating and running the Jazz tools in their environments. It became clear that the different companies and community members face similar challenges and that it would be beneficial if they could share tools they created to make running such an environment easier.

The community has now started sharing their tools in this community project and in this code repository.

JazzCommunity2017-05-12_11-34-52

Some of the tools have already been shared on other sites. I have linked the ones I am aware of to the Interesting links page in the ‘Extensions Provided by the Community’ section. These are the ones I am aware of (and the code for some of them is already available in the community repository):

I am looking forward to see more community created tools soon. Visit the community to find out what they have to offer. The code for their tools can be found here.

Last but not least, a special thanks to Dani for getting this awesome user groups started and for the members of said community for their spirit, engagement and willingness to contribute and help each other. You know whom I address here!

The RTC SDK is about to change in 6.0.3

Since some time now, the RTC and Jazz Development teams are in discussion how to cope with the version compatibility requirements driven by Eclipse clients and the server API. In RTC 6.0.3 the SDK is about to be split into separate SDK’s for the Eclipse client and the Server. This will impact how the development environment needs to be set up and how extensions are developed. I will try to share a summary of what to expect here. I have so far only been able to experiment with development builds, there has not been an official release of the SDK for 6.0.3 yet.

Download the new workshop

Update * the new Extensions Workshop is finished for a while now. it can be found at the original Rational Team Concert Extensions Workshop location.

Why splitting the SDK?

The RTC Clients have been based on Eclipse 3.6 for a considerable amount of time now. This has been the case for the Jazz Servers as well. However, there is pressure on the server infrastructure for the need to support Eclipse 4.4.x and higher. On the other hand there are client applications that RTC needs to integrate with, that are lagging behind in adoption of new Eclipse versions.

As described in What API’s are Available for RTC and What Can You Extend? the RTC SDK currently contains the RTC Server API, the RTC Common API and the RTC Client API in one delivery. The RTC Common API is part of the RTC Server API as well as the RTC Client API. This is a potential problem when shipping the SDK and trying to keep the Server compatible to Eclipse 4.4.2 and above and being compatible with Eclipse 3.6 clients. As it looks, the RTC SDK will be split into two parts.

  1. A RTC Server SDK bundling the RTC Server API and the RTC Common API compatible with Eclipse 4.4.x and higher
  2. A RTC Client SDK bundling the RTC Client API and the RTC Common API compatible with Eclipse 3.6.x and higher

Impact of splitting the SDK

The split has various impacts on how extension development will now work. Please find below a short summary of changes that I have found necessary to perform the workshop.

Changes to section: 1.1 Download and Unzip the Required Files from jazz.net

The Server SDK Target Platform now requires an Eclipse 4.4.2 or higher. You can download the base Eclipse 4.4.2 client here. For example download the version Eclipse IDE for Java EE Developers. Install the client similar to described in the workshop. Then download the RTC Eclipse Client p2 install package and install this into your Eclipse 4.4.2.

In the Feature Based Launches download the new launcher442.zip. Unzip the zip file, browse to the enclosed JAR file and copy that into the dropins folder in the Eclipse client.

You might want to consider to do the following changes to the eclipse.ini file.

  • Add a section -vm with an additional line for the java virtual machine to use. If you run Eclipse with a different JVM, e.g. from Oracle, consider to specify a JRE or JDK that is compatible with the one that ships with RTC. This vm would also be used in the workspace setup section.
  • Add -showLocation in the org.eclipse.platform section; this shows the Eclipse workspace path in the upper border of the Eclipse client as below

    eclipse-workspace_2016-11-07_11-23-52

    This makes it possible to actually work with multiple workspaces and knowing which an Eclipse instance is responsible for.

  • A vmargs argument -Duser.language=en to make sure you get a consistent language in the menus if you want.

The image below shows the changes in my caseeclipse-ini-2016-11-07_11-09-02

Changes to section: 1.2 Setup for Development

Once the SDK is split into two parts the Rational Team Concert Extensions Workshop can no longer be performed using just one Eclipse Workspace. An SDK is set up as a Target Platform in the Plug-in Development section. Since the SDK’s are now split, it is necessary to have two target platforms. Since it is not possible to have more than one Target platform active in one Eclipse workspace it is not possible to launch a server for debugging while running an Eclipse client from the same workspace.

The RTC Extensions workshop will have to be changed to set up two separate workspaces.

  • One workspace will have to be set up with the RTC Server SDK as active Target Platform, for example using the path: C:\RTC603Dev\Workspaces\Dev1\Server
  • The other  workspace will have to be set up with the RTC Client SDK as active Target Platform, for example using the path: C:\RTC603Dev\Workspaces\Dev1\Client

Both workspaces will require to be set up as described in the RTC Extensions workshop document in section 1.2 Setup for Development.However, you will set up different target platforms in this step. Using the Server SDK for the server development workshop and the Client SDK for the Client development workspace.

Please note, it is a good idea to configure Eclipse to use an external browser as well in this step.

Changes to section: 1.3 Setup the RTC Tomcat Server

I am modifying the WorkshopSetup tool and data to setup the RTC project named RTC Extension Workshop to support an easier setup for the two workspaces. Basically two separate RTC Repository workspaces will be available. One will provide the launch, the configurations and the components needed to develop the RTC Eclipse server extension part of the workshop. The other one will provide the launch and the components needed to develop the RTC Eclipse client extension part of the workshop.

As long as this is not yet available it is possible to start with the existing setup tool and the related repository workspace and to load that into the two Eclipse workspaces. One workspace has to be set up with the RTC Client SDK will be used for development of the client part. The other with the RTC Server SDK set up is used to develop the server parts. When performing the workshop it will be necessary to work with the two workspaces and use one for all the server related tasks and the other one with the client related tasks. When Accepting changes into these workspaces it is necessary to understand what is part of the client and what is part of the server or what is shared. The image below shows what belongs to what.

  • The parts colored in blue are only related to server development
  • The parts colored in yellow are only related to client development
  • The uncolored parts are related to client and server development

client-server-common

Make sure to keep in mind which parts of the code are relevant for what. As an example, the project net.jazz.rtcext.workitem.extensions.ide.ui will not compile in the server development workspace. Similarly the net.jazz.rtcext.workitem.extensions.service project will not compile in the client development workspace.

Changes to section: 1.4 Complete Setup of Your RTC Eclipse Client

After Loading the repository workspace you have the choice to split the information into a sever part and a client part. For example you can duplicate RTC Extension Workshop Configuration and create one that only contains the client launches. Or you keep everything as it is and basically close the project areas you don’t need and ignore launches not needed. This is the easiest approach until a new Extension Workshop is available.

The initial step of copying the files services.xml and scr.xml is only needed in the server workspace. So when copying and importing, copy the files services.xml and scr.xml from your server’s ccm application in the installs\JazzTeamServer\server\conf\ccm folder into the RTC Extension Workshop Configuration project into the folder conf/jazz in the server development workspace.

When importing the plugins and features import the following into the server workspace:

  • com.ibm.team.common.tests.utils
  • com.ibm.team.jazz.foundation.server.licenses.enterprise-ea (or com.ibm.team.licensing.product.clm)
  • com.ibm.team.licensing.product.rtc-standalone

When importing the plugins and features import the following into the client workspace:

  • com.ibm.team.rtc.client.feature

Other considerations

As already mentioned, make sure to keep track which workspace you are working in and keep in mind that the server development part will not work in the client development workspace and vice versa.

Just Starting With Extending RTC?

If you just get started with extending Rational Team Concert, or create API based automation, start with the post Learning To Fly: Getting Started with the RTC Java API’s and follow the linked resources.

You should be able to use the following code in this environment and get your own automation or extension working.

Summary

We have major changes coming up in the RTC Extension development area. The RTC Extension workshop needs to be adjusted and parts of the workshop lab needs to be reorganized and rewritten. This post explains what to consider for experienced users. Once there is an update to the Extension workshop lab material this post will be updated.

As always, I hope this helps users out there and saves them some time.

Alien Skies II – JavaScript Yet Another Computer Language?

I tried to look into Web UI Development, especially in the context of RTC Extensions like dashboards and the like several times in the past. I couldn’t get it to fly. It was a total disaster so far. I couldn’t understand how all that works and what the fuzz was all about. The more I looked into it, the more confused I was.

Surprisingly, looking into Doors Next Extensions seems to have revealed the missing link(s) I needed to better understand the whole topic around JavaScript. I have talked to other people who had similar issues with JavaScript, so I hope it is worth sharing my thoughts here.

I am well aware that the topic and my approach to it could be very controversial. If you like to disagree with any of my assessment, please feel free to do so in the comments. I rarely dismiss comments, but please be aware that the comments in this blog are moderated xD.

I have a book about JavaScript, but I rarely see JavaScript like it is described in that book. So, how does this JavaScript stuff work?

Its Not The Language

As far as I can tell, the language itself does not really matter that much. The language syntax of JavaScript is very similar to other languages e.g. such as Java. The Language has inheritance mechanisms, the language can support event based, asynchronously processed and functional aspects. All in all, a collection of properties that are shared across several language families.

There are some issues with the language (at least from my, still uneducated, point of view). It is not type safe to get started. While this is very flexible it is also asking for trouble in my experience. JavaScript is very dynamic and does not get compiled. A lot of issues will be seen at run time and not during coding.

Anyway, to have a common language with a common syntax and semantic is probably useful if you have to switch the domains often. At least you don’t have to learn a new language over and over.

Its Not The Development Environment

I am missing a good library handling and context sensitive editor support. There might be some cool development environment out there, but it has eluded me so far. Any suggestions are welcome, as always.

I have seen the first computers making their appearance in schools. With enormous keyboards with two lines of 40 characters to display. Nothing one would associate with a computer today. I was lucky enough to be able to write my first own programs on Commodore PET, VIC-20, 64 and on the first IBM PC’s. Ignore the first program on a Texas Instruments calculator with 32 programmable steps. I have developed without debugger, because there were no debuggers available or not affordable e.g. you could buy in circuit debuggers for  embedded systems but that was sometimes unaffordable even for companies back then.

I have seen bad development environments and debugging by printf. It is not that bad with JavaScript, in browser based applications. It basically depends on what the use case is, if there is a good debugging environment available or not.

Feeding The Confusion

I am usually pretty good in spotting patterns. Looking into the JavaScript domain, I had a hard time finding the basic ones. A lot of the confusion I felt is also, in hind sight, probably related to the fact that there is not just one JavaScript. There are multiple. And they are used in different ways and using different patterns. In addition to that the expression JavaScript is often used in contexts, where it would probably be better to talk about Web or JavaScript libraries or frameworks instead.

  1. There is the JavaScript that runs in browsers. This JavaScript usually works together with CSS and HTML. It is used to create dynamic content and presentations in the Web to be able to provide more or less usable UI’s in a browser, designed to show more or less static HTML documents. The JavaScript engine is built into and shipped with the browser and there are different engines and development environments for different browsers. This type of JavaScript basically requires to understand how JavaScript, HTML and CSS work in combination and how to control their interaction.
  2. Browser based JavaScript is integrated in several different frameworks and tool kits such as Ajax/Dojo and others.
  3. There is NodeJS, which claims to be Java Script and actually is sever side JavaScript processing and a ton of libraries and frameworks that are available for it. The only common part I can identify here is the language that is used. I did not look into the details however, so maybe there is more e.g. the run time used could be one of the common ones.

So when I started to look at the Doors Next Extensions which said JavaScript, I realized quickly, that there were constructs in the examples that I had never seen before in any JavaScript example. The normal pattern again, every time I looked at something with JavaScript inside and thought I had some basic knowledge already, I saw something different. This time however I was able to figure out what I was missing and this seems to help with all the other cases as well.

The Doors Next Extension Mechanism

The Doors Next Extension mechanism is based on JavaScript, HTML and CSS. The API supports TypeScript which provides with a more type safe and compile like way of developing JavaScript. The Doors Next API itself comes with a Typescript interface definition. Some of the examples that are provided use jQuery a JavaScript based library and framework. The latter means that the extensions look a bit uncommon.

Uncommon Common

To some extend this seems to be fairly typical for my experience with JavaScript. Almost every solution I have seen has some common part that can be found in other solutions. But looking across all the examples there is only a tiny common core, which often is basic JavaScript functions. If the JavaScript based solution deals with UI’s, there is usually the common CSS and HTML, if it does not focus on UI’s this is missing. Since there are so many frameworks and libraries, Java Script based solutions can look very different and it is important to understand the detailed frameworks and resources that are used in the specific example.

Summary

Granted, it took a while for me to come to terms with JavaScript,  but looking into the Doors Next Generation Extension Mechanism provided me with some important insight. This will help me to understand these technologies in other contexts as well. I am looking forward to playing with Doors Next Extensions and add the results to this blog.

Alien Skies – Peeking Into Doors Next Extensibility

I started a recon mission into the alien territory of Doors Next extensions recently and almost lost my way.

Doors Next is one of the products I am responsible for. I have some experience with requirement management and I understand Doors Next usually good enough to find my way, but I am way more familiar with RTC, especially since I did a lot with its Java API and extensions. So I thought if this is similar to the situation with RTC, capabilities to extend Doors Next will be in high demand soon and I better get my hands dirty and into it. Since the language available to extend Doors Next is JavaScript, this looked also like a good opportunity to have some more exposure to JavaScript.

Update, a nice summary of RDNG API’s can be found here.

First Steps – Look into the documentation

So I browsed through the documentation a bit to learn what there is to learn! Other than me, please start with reading the most recent version of the documentation. This will save you detours like looking at WAS Liberty Profile and publishing data. I followed the guidance and played around with the simple examples and the more complex ones provided. I became adventurous and wanted to modify the Hello World open social gadget example and create one that should be published in the widget catalog along with the other examples. The example worked as open social gadget already and I thought, I could use it as widget as well.

That did not work so good. Why? If I used the same xml code the open social gadget showed and the Dashboard widget showed a blank content.

The Widget

This is how the basic widget looks like. All the rest such as CSS, JavaScript and the like can be added to it based on this structure.

DashboardWidget

The Open Social Gadget

This is how the open social gadget looks like. I tried to spot what the difference was for a long time. Until I finally noticed that the difference between a working gadget and a working widget is the tag that the open social gadget has and which is missing in the widget..

OpenSocialGadget

It took me quite some time to figure out what the problem was. Embarrassing, but on the other hand it shows again, that tool extensions, even the smallest, are complex business. Maybe there are better ways to debug these kinds of issues. Compared to RTC Java Extensions I felt like back in time, trying to debug C and Assembler with printf…..

See this site for some additional introduction into open social gadgets.

Deployment Structure

Similar to my description in Publishing XML and Other Data With the WAS Liberty Profile  the extensions where deployed in the apps folder like this:

WidgetDeployment

Compared to RTC Extensions, this is a relatively simple and accessible structure. As mentioned by Guido, in an enterprise context, it would make sense to have your own web server to publish the extensions. This would allow to have a more fine grained control about who can change what.

It would be possible to use a RTC Build workspace to publish these extensions. I think there is a need to come up with some naming pattern for the extensions, in order to avoid chaos and confusion over time.

This has been challenging in the past and I assume it will stay challenging. The structure above has a css folder, as well as a JavaScript folder just in case. There is also a need for some more documentation and the like.

Summary

Doors next has means to extend its capabilities. Provided some fundamental knowledge, this should allow for a lot of the automation needed in enterprise environment. My first steps into this alien territory where a bit shaky, but all in all, I think I now have the basic understanding where and how to get started with more complex extensions.