Registering Custom Resource Intensive Scenarios to CLM Applications


There is no such thing as limitless computing power. This is an unfortunate truth that can cause problems running the CLM and other tools, as the usage grows. To understand what systems actually do when getting under heavy load, more and more monitoring was introduced over the last years. Resource intensive scenarios where identified and the CLM tools have capabilities to record information about their frequency and duration. Plan loading and SCM compare workspace are examples in the product.

Custom Resource Intensive Scenarios

In addition to resource intensive scenarios that are built in, it is also possible to introduce custom resource intensive scenarios. Some examples are:

  • Custom automation that execute long running operations on work items, SCM data, requirements, test artifacts. Typical scenarios are custom export/import, mass updates, custom analysis of source code, baselines, linked work items.
  • Follow up actions
  • Long running custom dashboards

This is by no means a comprehensive list. It is possible to bring your clients and servers to their knees with custom themes that do not scale, by work item attribute customization adding more and more custom attributes, JavaScript providers, value providers with thousands of values to choose from and other customization.

What is your server up to?

When users complain about performance problems, even if a server is getting overloaded, it is hard to find the root causes, because a typical server does so many things.

Monitoring that has been added over time has helped, but it is still hard. It is sometimes even hard to understand the situation. As an example for how complex this can become. Users complained about performance.

Our performance architect looked at the server load and the build load and a huge amount of calls that we were not able to account for. The server was unarguably under heavy load created by builds, but the build users and SCM users where not complaining. The developers we talked to had no real issues. Some users, at a different location, using work items and running work item queries, had.

Because we could not explain the inconsistent feedback, I finally went to the location where the users where complaining. I met the users followed their day to day work and found the work item performance unacceptable. The web browser was even locking up on them.

Knowing this, we were able to reproduce the use case, and look into what happened. We found that the work item load was slow, especially on slow laptops, because it had to load so many team areas and iterations. This was specific to how the project area was configured and used.

We also found that the browser flooded the server with requests that where definitely not part of what the product UI sent. This basically forced the Web Browser to process and cache thousands of calls, reserving more and more memory and exhausting the CPU capabilities of the relatively weak laptops used by the users that complained.

The final verdict was, that there was a custom extension to the theme that created all these calls. It took us weeks and was luck that we found this out. If we had known there was such an extension, we would have been able to find this a lot faster. The server was still under a heavy build load, but the performance issue reported was not related to that.

Needless to say that this extension was also deployed in other environments. If it had a detrimental impact, it was heavily depended on the timeline and iteration structure of a project area. The more and deeper the worse.

It would have helped if we could have seen the extension working, and see how long it worked would also have helped.

Registering Custom Resource Intensive Scenarios

The same mechanism that is used to register resource intensive scenarios in the product code can be used to register custom resource intensive scenarios. Unfortunately, we where lacking a good description and supporting code that we could provide customers to use it for their extensions.

This has now changed. Some colleagues and I, independently, started creating a customer usable description how to register resource intensive scenarios. A colleague wrote some cURL code to do this. I wrote Java code to do this and started creating a presentation. When we found out, we decided to combine the effort. Here the result.

The Deployment Wiki page Register Custom Scripts as a Resource Intensive Scenario, explains, using an example, how the API works in general. It also explains how to retrieve and monitor this information.

Then it provides example code to perform this using cURL, Eclipse Lyo OSLC4J based java code, and RTC Plain Java Client Libraries based Java Code.

The Java Code comes with main classes to run it. This is basically example code, but it can also be directly used in command line based automation.

Open Source Code

Disclaimer and Download

Any code downloadable or accessible in this post is provided as is, without support, and used at your own risk. Part of the code was developed in Java using Eclipse and is based on the Eclipse Lyo Client. This was published as open source, under
 Eclipse Public License – v 1.0, in the incredible (mostly German speaking) Jazz Community and can be found here: custom-expensive-scenario-notifier-oslc4j.

Another part of the code was developed in Java using Eclipse and is based on the Plain Java Client Libraries. This was published as open source, under MIT license, in the incredible (mostly German speaking) Jazz Community and can be found here: custom-expensive-scenario-notifier-plainjava.

See the other examples the Deployment Wiki page Register Custom Scripts As a Resource Intensive Scenario.

How does it work?

There is basically a REST API to register the start and the stop of a scenario. All there is to register the start of the scenario at the beginning and then register the stop, after you are done. See Register Custom Scripts as a Resource Intensive Scenario for more details on the code.

What should your automation do?

If you have written automation tools or extensions, you should use the methods described in Register Custom Scripts as a Resource Intensive Scenario, to register your extension as an resource intensive scenario. Add the code to register the start and stop in a way that allows for disabling it easily.

Monitor the various resource intensive scenarios over time. For a scenario that takes only a fraction of a second, you could temporarily disable the registration. Scenarios that take a second or longer should continue to be monitored.

Related

Feedback

If you have questions around the Custom Resource Intensive Scenario code, ask them in the Jazz.net forum instead of commenting on the article or this blog post. Tag the question as a clm question and add the tag: custom-resource-intensive-scenarios to mark it for the reader.

Summary

Please use the method above to enhance your automation and extensions to allow monitoring their duration, frequency and deviation.

As always I hope this helps users out there with the Jazz products.

Advertisements

Type System Manager Part 2


We finally published Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 3: Automation tool deep dive on Jazz.net.This was a major effort and took a long time to do. This article provides a closer look at the source code, what it does and how it does it. It also provides some insight in how OSLC4J works and can be used. The information in the article, especially for setup and deployment of the automation prototype is very reusable for other scenarios and I hope to be able to reuse it in later articles and blog posts.

Type System Manager

When this effort was planned and performed last year, we had no idea what would come out of this effort. When we finished the first iterations and I started to write Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 3: Automation tool deep dive, we called what we where working on an automation prototype in the articles. I decided to keep it that way.

Since then, I started to call the prototype Type System Manager and I will continue to use that name.

The article Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 3: Automation tool deep dive, basically explains the details of the code that was created until end of January 2019. This is Release 1 of Type System Manager.

New versions?

While working on the article, I continued to refine the Type System Manager. See the releases tab for the latest release. The latest releases support more automation It integrates finding editable configurations based on sub-strings in the description, create a source to target mapping and import/deliver type system changes based on this mapping. The new commands avoid the creation of a CSV file.

The code is also slightly refactored. Most of the content from Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 3: Automation tool deep dive, still applies, but don’t be surprised if there are slight changes in names and locations. There basically where inconsistencies that I found along the way and decided to fix.

The latest versions also register Custom Resource Intensive Scenarios.

TSM 1.1

In addition to all this, the code on this branch already supports Registering Custom Resource Intensive Scenarios to CLM Applications. Each command uses its own scenario name composed from the command name plus the postfix “Scenario” to register the scenario execution.

Disclaimer and Download

Any code downloadable or accessible in this post is provided as is, without support, and used at your own risk.

The code was developed in Java using Eclipse and is based on the Eclipse Lyo Client.

Thanks to IBM approving, the code was published as open source, under 
 Eclipse Public License – v 1.0, in the incredible (mostly German speaking) Jazz Community and can be found here.

Related

Feedback

If you have questions around the Type System Manager, ask them in the Jazz.net forum instead of commenting on the article or this blog post. Tag the question as a Rational DOORS Next Generation question and add the tag: dng-type-system-management to mark it for the reader.

Summary

As always I hope that the artifacts created for this blog and on Jazz.net will useful for the Jazz user community out there.

Type System Manager Part 1


It has taken some time since I published Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 1: Manual procedures which is about this Jazz.net article.

I was very excited about it, because I was contributing to the effort myself. We finally have made progress and just released the next part. You can now read the article about Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 2: Automation.

So, what is that all about?

Type System Manager

I did some prototyping, and we experimented with different possible approaches for this automation. What we came up with is explained in the article above.

In summary, I created a prototype, lets name it Type System Manager (TSM), that can be used to automate the type system management, supporting the best practices identified in Part 1.

Example Execution Output

The prototype uses available public Doors Next Generation OSLC/REST API’s to perform the necessary tasks.

So I have been lucky to be able to do some prototyping and learn a lot about OSLC and REST API’s. The result is a prototype, that has a useful application, and is also a demonstrator for how to create automation using OSLC and REST API’s.

Usage of OSLC and REST API’s has been missing on this blog and this prepares the foundation for hopefully more examples, here in this blog, in the future.

Disclaimer and Download

I should not have to write this, but as this is the internet 8), so here goes: Any code downloadable or accessible in this post is provided as is, without support, and used at your own risk.

The code was developed in Java using Eclipse and is based on the Eclipse Lyo Client.

Thanks to IBM approving, the code was published as open source, under
 Eclipse Public License – v 1.0, in the incredible (mostly German speaking) Jazz Community and can be found here.

What is next?

The second part explains how the TSM prototype can be used and what it does and how. The next part is currently under review and will explain the details of the code, how it works, how to re-use and add to it. It also explains how to download and work with the code.

Related

Feedback?

As mentioned in the article Comments, feedback, ideas, and experiences are greatly appreciated.

If you have questions, ask them in the Jazz.net forum instead of commenting on the article or this blog post. Tag the question as a Rational DOORS Next Generation question and add the tag: dng-type-system-management to mark it for the reader.

Summary

As always I hope that the artifacts created for this blog will useful for the Jazz user community out there. Feedback, also usage, is greatly appreciated.

Work Item Command Line 4.3


I worked with a colleague on the Work Item Command Line who was adding a feature for usage with a customer. That colleague had also done some changes related to availability and performance running queries. These changes where merged back into the code. WCL was missing some capabilities that I always wanted to have. I had a quick look to understand how much effort hat would be and took the opportunity to implement these capabilities on the way. While doing so, some small enhancements where added and some defects where found and fixed. All that needed to be published as a new version of WCL.

Print a work item

New in Version 4.3

The latest release 4.3 is available here and has the following changes:

  • Readme.md updated
  • Readme documents for deployment in source code updated
  • Refactored the capability to create a string representation for a work item attribute, link or pseudo attribute (e.g. attachments) into a helper class WorkItemExportHelper
  • Migrated -exportworkitems to use the new helper class WorkItemExportHelper
  • Several fixes in -exportworkitems to
    • Enable error handling and make suppressing errors work
    • Introduce a new flag that exports all supported columns and links
    • If exporting all columns, order the attributes and links (except ID, Type and Summary that are by default at the beginning)
    • Checked functionality with newest OpenCSV
    • Refactoring class to make maintenance easier
  • Added a new command -printworkitem that prints the attributes, attachments and supported links of a work item
    • Using the WorkItemExportHelper and its capabilities
    • Also provides switch to print all attributes and links ordered as above
    • Exports the attachments to a provided folder
  • Added a new command -printtypeattributes that allows to get the attribute Id’s and Display Names for a given work item type in a project area
  • Added a command -validateoslclinks that validates OSLC links and their back links in other applications fixing missing back links
  • External library dependencies are added without a version number to allow newer versions

Disclaimer

Please keep in mind that the information in this blog is “as is”, unsupported, and may be outdated or inaccurate.

The Work Item Command Line (WCL) is not an officially supported application.

For information on released products, consult the product documentation, support tech notes, and the Jazz.net library.

Open Items

Some of the API used in the Work Item Command Line becomes deprecated.

  • Some changes especially for -importworkitems needed for versions of RTC e.g. 6.0.5 and later
    • These will likely not work with earlier versions of RTC
    • if you need that either try an earlier version of WCL such as 4.0 or 4.1 or open an issue and I could look into a backport
  • The Linking API in RTC has evolved and deprecates some API used in WCL
  • OpenCSV has evolved and newer versions deprecates some API used in WCL
    • OpenCSV requires a new library commons-lang3-3.1.jar
  • Migration to Maven might be something to consider

This is all not problematic yet, but needs to be addressed sooner or later.

License

WCL is released under the MIT License. See the License.txt and the license headers in the individual files.

Compatibility

This code has been used with RTC 4.x, 5.x  and 6.x with no or minimal changes and it is pretty safe to assume, that the code will work with newer versions of RTC to come.

An exception is the -importworkitems command that requires a library to use a mapping file. The capability to use the mapping file was moved into the plain Java Client libraries in RTC 6.0.5 as an internal class. To support work item import with the mapping moved to the Plain Java Client Libraries as internal API, the imports in ImportWorkItemsCommand where changed and the file com.ibm.team.workitem.rcp.core*.jar was removed from the jar file dependencies in the build path.

See Restore Compatibility to 6.0.4 and before for how to switch this back.

The code requires the RTC Plain Java Client Libraries.

The Export and import commands require additional external libraries that need to be downloaded and installed separately. See the document below.

Restore Compatibility to 6.0.4 and older

To support 6.0.5 and beyond the imports in ImportWorkItemsCommand where changed from:

com.ibm.team.workitem.rcp.core.internal.bugzilla.mappers.BugzillaMapping*

to: com.ibm.team.workitem.common.internal.importer.bugzilla.mappers.BugzillaMapping*

and the file com.ibm.team.workitem.rcp.core*.jar was removed from the jar file dependencies in the build path.

Follow the description in the sections beginning with Downloads how to get and rebuild the code. The steps in this section below are required to port WCL back to RTC 4.0.4 and earlier:

The original imports have been put back in past 4.0.3, but commented out. To enable the mapping for earlier versions of RTC, replace the imports *BugzillaMapping* with the versions commented out. Then add the com.ibm.team.workitem.rcp.core_*.jar file back in to the build path and add the required JAR file as below

From Work Item Command Line 4.1

The import commands of WCL can only provide the capability to use a mapping file by using a JAR file that only ships with the RTC Eclipse client and the SDK. The classes used for the mapping file capability are located in the library com.ibm.team.workitem.rcp.core.  Open the Install location of the RTC Eclipse client and search for com.ibm.team.workitem.rcp.core*. You should find a file names similar to this one: com.ibm.team.workitem.rcp.core_3.1.900.v20141010_0124.jar. The version numbers at the end could be different. Copy the JAR file into the into the libfolder of your version of WCL.

Download

The latest code can be downloaded from this GIT repository. The latest releases can be downloaded from here

There are different ways to download.

  • The easiest is to download the latest code as a zip file using the button Clone or download.
    • Use the option Download ZIP to download the code as ZIP file. Once the download finishes, browse the folder structure of the ZIP file. The top level contains the README.md file. An Eclipse project is contained in the folder com.ibm.js.team.workitem.commandline. This project can be used to develop the code and to generate an executable.
    • It is also possible to clone the repository and import the project using GIT. Clone the GIT project to your local file system. This requires GIT or some GIT GUI to be installed. How to use GIT is out of the scope of this blog. See the help how to set up GIT and how to use it with Eclipse.
  • It is also possible to download releases from the releases page. The zip code made available is in the same format as the one above can be used as described above. Note that the releases might be older, and there might be no release available for the latest code version.

See below how to import the enclosed project into Eclipse.

Some releases might contain a prepackaged executable for convenience e.g. named wcl-V4.3-20190117.zip. See the section Deploying the packaged application below for how to use this form of packaging. Please note that the prepackaged executable might not be available for all versions. The following sections explain how to create this executable. See the section Packaging the application for shipping for how to perform the packaging.

Prerequisites

Install a current Eclipse or RTC Eclipse client.

Install a current Java 1.8 JRE or Java 1.8 JDK set it as active in Eclipse as Installed JRE and Execution Environment.

Getting the code, install and deploy

Once you have installed such an environment do the following

Import the Code

  • Open the Java Perspective in Eclipse.
  • Import the project into Eclipse from the file system.
    • To import the ZIP file use File>Import, select Existing Projects into Workspace. Click Select archive file, browse to and select the ZIP file you downloaded. Import the enclosed Eclipse project com.ibm.js.team.workitem.commandline.
    • To import the GIT repository use the GIT perspective to add the local cloned repository and then import the
      project com.ibm.js.team.workitem.commandline.

Create a User Library for the Plain Java Client Libraries

  • Use Window>Preferences>Java>BuildPath>User Libraries and add a user library named PlainJavaApi .
  • Add the JAR files of the Plain Java Client Libraries for the version of RTC you are using.

Download and Install openCSV

Download and Install Apache commons-lang

Check the project and recompile

  • Select Project>Clean and clean the project that was just imported.
  • There should be no compiler errors or issues with the classpath

The project should now have built and compiled and show no errors.

Building the WCL or extract from a Release

Follow the description in the file ReadMe – HowToRelease.txt in the root folder of the project

  • To create a wcl.jar file
  • To crate a WCL/lib folder
  • To provide batch and license files

Follow the description in the file /lib/ReadMe.txt in the lib folder of the project

  • To provide the required libraries in the WCL/lib folder

Select the folder, for example C:\Temp\wcl\ and compress the file

  • Rename the archive file to wcl-Vx-YYYYMMDD.zip,
    where YYYY is the year, MM is the month and DD is the day
  • The file is now ready for publishing

Use the file wcl-Vx-YYYYMMDD.zip for shipping similar to the next section.

Deploying the packaged application

Assume a file was created following the steps above and the file is called
wcl-V4.3-20190117.zip. This binary contains WCL as JAR file and other information such as licenses and Readme files. It also contains script files to set the environment and run the Jar File.

The folder lacks libraries needed to execute that need to be provided.

To deploy from this file, create a new folder as deploy location for the automation prototype. In the description here we use C:\temp. It is possible to use any other folder. In this case replace the folder in the this description by the folder you chose.

Extract the binary wcl-V4.3-20190117.zip file into the location C:\temp. There should now be a folder C:\temp\WCL. The folder among others contains the files wcl.jar, wcl.bat, wcl.sh.

Get the libraries as described in Download and Install openCSV and Download and Install Apache commons-lang above, and rename them as described in /lib/ReadMe.txt.

Check the script file and make sure to provide a valid path to a folder containing the unzipped Plain Java Client Libraries and a recent Java 1.8 is available over a Java_Home.

Call WCL using the script files or call it as Jar File as described in said script files.

Just starting with extending RTC?

If you just get started with extending Rational Team Concert, or create API based automation, start with the post Learning To Fly: Getting Started with the RTC Java API’sand follow the linked resources.

You should be able to use the following code in this environment and get your own automation or extension working.

Summary

This is only the first blog. I will try to provide a little bit more about the commands and the parameters soon.

As always, I hope this helps users out there. 

Please ask questions on in https://jazz.net/forum/questions/ and tag them rational-team-concert and wcl.

You can create issues in https://github.com/jazz-community/work-item-command-line/issues as well. 

Keep in mind this is off hours work and not officially supported.

Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 1: Manual procedures


I have been involved in efforts around the configuration management features in the IBM Collaborative Lifecycle Management and Continuous Engineering Solution. These Configuration Management capabilities are a completely new approach to versioning and variant management across the different disciplines such as Requirements Management, Quality Management, Modeling, and Software Change and Configuration Management. We realize that there is a whole new area for best practices and our subject matter experts have been working on creating necessary information for some time. There is already some content available on Jazz.net. For example from Kathryn and Tim in the Deployment Wiki.

As we learn we try to create more content, involving more SMEs. Today we published the article Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 1: Manual procedures.

I am currently working on Part 2. If you want to know what that could be, check the article.

I am pretty excited, I hope to publish soon!

Jazz Community Contributions


The Jazz Community starts sharing their tools here: http://jazz-community.org/. The code for their tools can be found here.

There is a very active Jazz user community of members of several companies in Europe that are heavily using the Jazz products such as Rational Team Concert, Rational Quality Manager and Doors Next Generation.

The community members try to meet to share their experience with using, administrating and running the Jazz tools in their environments. It became clear that the different companies and community members face similar challenges and that it would be beneficial if they could share tools they created to make running such an environment easier.

The community has now started sharing their tools in this community project and in this code repository.

JazzCommunity2017-05-12_11-34-52

Some of the tools have already been shared on other sites. I have linked the ones I am aware of to the Interesting links page in the ‘Extensions Provided by the Community’ section. These are the ones I am aware of (and the code for some of them is already available in the community repository):

I am looking forward to see more community created tools soon. Visit the community to find out what they have to offer. The code for their tools can be found here.

Last but not least, a special thanks to Dani for getting this awesome user groups started and for the members of said community for their spirit, engagement and willingness to contribute and help each other. You know whom I address here!

Upgrading your CLM system? – Definitive checklist.


Upgrading the IBM CLM solution is sometimes received as complex and users struggle with it. Yesterday I was notified that there was a blog from a colleague providing a check list that helps making this easier and more likely successful. I thought this is worth reblogging to make it easier to find.

Please find the information in Paul’s blog post Upgrading your CLM system? – Definitive must-read checklist just published. The link points to our Deployment Wiki which is always worth looking at for all things related to deployment, install, setup and configuration.

How to filter RTC email notifications by project or by user


Spamming the in-boxes of your project with notification mails? This is a reblog of Marks internal blog.

The RTC Mail Filtering Problem

A common requirement for Rational Team Concert administrators is the need for limiting email notifications for a single RTC project or user.  There are many scenarios which might drive this need.  Here are some examples:

  • Thousands of work items are imported and you want to not cause notifications for the new work items.
  • A new field is added and data from the old field is copied over, affecting many work items.  We don’t want users to be notified.
  • A subset of users now use a different RTC server, and they no longer want notifications from the old project, but they need to remain active in the old project.

Unfortunately, the current RTC product as of version 6.0.1 does not support this requirement.  Notifications are controlled at the Jazz Team Server level.  The JTS may control multiple RTC, Rational Quality Manager, and Rational Doors Next Generation repositories.  Turning off notifications in the JTS turns off mail for all RTC, RQM, and RDNG projects from all repositories controlled by that JTS server.  Mail  generated for all of those RTC, RQM, and RDNG projects while notifications are off in the JTS are lost.

This leaves administrators with a tough choice:  lose all mail for everything connected to the JTS, or live with excessive and unwanted notifications for a single project.

A Solution:  Milters

We have found and implemented a solution for this requirement, milters.  Milters are plugins for Sendmail that add additional functionality to Sendmail.  The “milter-regex” milter plugin permits filtering mail using regular expressions.

These are the overall steps for utilizing the regex milter:

  • Set up a machine with Sendmail and install the milter-regex plugin
  • Configure Sendmail to allow mail from the JTS server machine in /etc/mail/access
  • Develop a set of regular expressions which cause mail from a particular project or user to be found and filtered out and update the /etc/mail/milter-regex.conf configuration file
  • Restart the milter-regex service
  • Edit the email settings of the JTS to point to your Sendmail machine as the mail SMTP Server

You can easily add and remove projects and users from the filtering list by editing the /etc/mail/milter-regex.conf file.  You can turn off filtering completely by restoring the JTS SMTP Server value back to its original setting.

Regular Expressions

You’ll have to examine your email templates to determine the project and user information for the various email formats.  Here’s an example of the configuration statements for our email templates to filter out all mail for project “Project XYZ”:

reject “Mail filtered for project: Project XYZ”

body ,Team Area:.* Project XYZ,i

body ,Project Area:.* Project XYZ,i

Both “body” statements are required.  The “reject” statement defines a message which is logged into the /var/log/messages log file when a piece of mail is filtered out.

This is an example of filtering for a single user:

reject “Mail filtered for user: Mark E. Ingebretson”

body ,The user ‘Mark E. Ingebretson’ made a .* request,i*

body ,Mark E. Ingebretson mentioned you in,i

body ,Mark E. Ingebretson.*changed on,i

If you are a user of our IBM Systems servers and need email filtering, you can submit an ITHELP request.

Other Approaches

There is another approach from Sam provided on the Jazz.net forum here: Manage User E-mail Preferences for Mass Updates. It has been sitting on my Interesting Links page for a while. Time to show it here.

A new solution s available for RTC 6.0 starting with iFix03 if you are using the Java API. Use the Skip Mail Save Parameter.

Summary

I found the topic very interesting and related questions also come up on Jazz.net, so I decided to re-blog and promote this when Mark showed this to me. I hope it helps our users all over the world. I hope that this solution can help other RTC administrators address this important requirement.

Alien Skies II – JavaScript Yet Another Computer Language?


I tried to look into Web UI Development, especially in the context of RTC Extensions like dashboards and the like several times in the past. I couldn’t get it to fly. It was a total disaster so far. I couldn’t understand how all that works and what the fuzz was all about. The more I looked into it, the more confused I was.

Surprisingly, looking into Doors Next Extensions seems to have revealed the missing link(s) I needed to better understand the whole topic around JavaScript. I have talked to other people who had similar issues with JavaScript, so I hope it is worth sharing my thoughts here.

I am well aware that the topic and my approach to it could be very controversial. If you like to disagree with any of my assessment, please feel free to do so in the comments. I rarely dismiss comments, but please be aware that the comments in this blog are moderated xD.

I have a book about JavaScript, but I rarely see JavaScript like it is described in that book. So, how does this JavaScript stuff work?

Its Not The Language

As far as I can tell, the language itself does not really matter that much. The language syntax of JavaScript is very similar to other languages e.g. such as Java. The Language has inheritance mechanisms, the language can support event based, asynchronously processed and functional aspects. All in all, a collection of properties that are shared across several language families.

There are some issues with the language (at least from my, still uneducated, point of view). It is not type safe to get started. While this is very flexible it is also asking for trouble in my experience. JavaScript is very dynamic and does not get compiled. A lot of issues will be seen at run time and not during coding.

Anyway, to have a common language with a common syntax and semantic is probably useful if you have to switch the domains often. At least you don’t have to learn a new language over and over.

Its Not The Development Environment

I am missing a good library handling and context sensitive editor support. There might be some cool development environment out there, but it has eluded me so far. Any suggestions are welcome, as always.

I have seen the first computers making their appearance in schools. With enormous keyboards with two lines of 40 characters to display. Nothing one would associate with a computer today. I was lucky enough to be able to write my first own programs on Commodore PET, VIC-20, 64 and on the first IBM PC’s. Ignore the first program on a Texas Instruments calculator with 32 programmable steps. I have developed without debugger, because there were no debuggers available or not affordable e.g. you could buy in circuit debuggers for  embedded systems but that was sometimes unaffordable even for companies back then.

I have seen bad development environments and debugging by printf. It is not that bad with JavaScript, in browser based applications. It basically depends on what the use case is, if there is a good debugging environment available or not.

Feeding The Confusion

I am usually pretty good in spotting patterns. Looking into the JavaScript domain, I had a hard time finding the basic ones. A lot of the confusion I felt is also, in hind sight, probably related to the fact that there is not just one JavaScript. There are multiple. And they are used in different ways and using different patterns. In addition to that the expression JavaScript is often used in contexts, where it would probably be better to talk about Web or JavaScript libraries or frameworks instead.

  1. There is the JavaScript that runs in browsers. This JavaScript usually works together with CSS and HTML. It is used to create dynamic content and presentations in the Web to be able to provide more or less usable UI’s in a browser, designed to show more or less static HTML documents. The JavaScript engine is built into and shipped with the browser and there are different engines and development environments for different browsers. This type of JavaScript basically requires to understand how JavaScript, HTML and CSS work in combination and how to control their interaction.
  2. Browser based JavaScript is integrated in several different frameworks and tool kits such as Ajax/Dojo and others.
  3. There is NodeJS, which claims to be Java Script and actually is sever side JavaScript processing and a ton of libraries and frameworks that are available for it. The only common part I can identify here is the language that is used. I did not look into the details however, so maybe there is more e.g. the run time used could be one of the common ones.

So when I started to look at the Doors Next Extensions which said JavaScript, I realized quickly, that there were constructs in the examples that I had never seen before in any JavaScript example. The normal pattern again, every time I looked at something with JavaScript inside and thought I had some basic knowledge already, I saw something different. This time however I was able to figure out what I was missing and this seems to help with all the other cases as well.

The Doors Next Extension Mechanism

The Doors Next Extension mechanism is based on JavaScript, HTML and CSS. The API supports TypeScript which provides with a more type safe and compile like way of developing JavaScript. The Doors Next API itself comes with a Typescript interface definition. Some of the examples that are provided use jQuery a JavaScript based library and framework. The latter means that the extensions look a bit uncommon.

Uncommon Common

To some extend this seems to be fairly typical for my experience with JavaScript. Almost every solution I have seen has some common part that can be found in other solutions. But looking across all the examples there is only a tiny common core, which often is basic JavaScript functions. If the JavaScript based solution deals with UI’s, there is usually the common CSS and HTML, if it does not focus on UI’s this is missing. Since there are so many frameworks and libraries, Java Script based solutions can look very different and it is important to understand the detailed frameworks and resources that are used in the specific example.

Summary

Granted, it took a while for me to come to terms with JavaScript,  but looking into the Doors Next Generation Extension Mechanism provided me with some important insight. This will help me to understand these technologies in other contexts as well. I am looking forward to playing with Doors Next Extensions and add the results to this blog.

Publishing XML and Other Data With the WAS Liberty Profile


Since CLM now ships the WebSphere Application Server Liberty Profile as default, you might want to be able to publish XML and other data similar to the easy way that was available in Tomcat, by just using the application server that is available. This avoids having to set up an additional Web or application server.

Problem

Sometimes it is necessary to provide access to some files using the HTTP/HTTPS protocol.

This is interesting for the Process Enactment Workshop, or if you want to publish your own XML data to be used with HTTP Filtered Value Sets, host custom Doors Next Extensions, or if you want to publish Build results.

It is convenient if there is already an application server available to host these files there and make them accessible, instead of having to set up a web server. This especially holds if it is necessary to provide the data using HTTPS, and not HTTP, because browsers these days reject to process content that is mixed from HTTP and HTTPS sources. The latter is important for Doors Next Generation extensions. Not having to set up a new server also avoids having to create a trusted certificate for it. It reduces the additional failure point, backup and maintenance and provides URI stability as well, since you don’t want to change the URI for the CLM applications hosted anyway.

Tomcat

With Tomcat, it was relatively easy to do that, by just adding a folder under \tomcat\webapps and placing your files there. Assuming the folder name is myfiles, Tomcat then provides the files with the public URI root https://serverpath:port/myfiles/fileName.

Another way was to simply drop the file into the tomcat/webapps/ROOT/ or a sub folder, which has the same effect.

Is there a similar way to achieve this with the WAS Liberty Profile shipped with CLM?

Example Scenario

Lets assume, like in the scenario explained in Publish and Host XML Data Using Tomcat – The Easy Way, we want to publish a file makers.xml to be accessible with a context root PEWEnactmentData on the server with the public URI root https://clm.example.com:9443/.

The expected outcome is to be able to access the XML file content using the URL https://clm.example.com:9443/PEWEnactmentData/makers.xml.

Solutions

There are two relative simple solutions for WAS Liberty. The solutions below are based on information from Lars, one of our WebSphere experts and the IBM WebSphere Application Server Liberty Profile Guide for Developers.

For WAS Liberty Profile, all solutions require to deploy an application. There are several ways to create and deploy an application in WAS Liberty Profile.

  1. Deploy an application in some folder and declare the context root and location of the application in the server.xml or specific location XML file
  2. Deploy an application in the dropins folder; for this to work, it is necessary to make sure the application server has the dropins monitoring enabled for this to work

Deploying an application in this context means

  1. Create a folder with some contend in a location (or copy the folder into the location)
  2. Deploy a compressed folder with some content

Please note: Any changes to any of the server configuration XML files are automatically picked up be WAS Liberty Profile. There is no need to reboot the server for the steps below to work.

1. Deploying an Application – Standard

The trick in this solution, is to pretend to deploy an uncompressed Web Application such as a WAR file. It is obviously possible to create a real WAR file or the structure that is contained in it, but that is not really necessary. See the last section here if you are interested in deploying the application as compressed WAR file.

1.1 Deploying the Application

The only thing necessary to trick the WAS Liberty Profile into making the data available with a correct context root is to create a folder with the context root as name and an extension .war, and to make the application location known in an XML file.

In the default setup of WAS Liberty Profile with CLM, after the first launch, all installed CLM applications end up in the folder <Install Folder>\server\liberty\servers\clm\apps. The screen shot below shows this structure.

FolderStructure

The applications are deployed as compressed WAR file and have been extracted to a folder with the name of the application. As an example the CCM application (RTC) came shipped as ccm.war.zip and was extracted to a folder ccm.war. The context root of the CCM application is ccm, which is basically the name without the extension war. The same pattern holds for all the other applications.

It is possible to create a folder to host the files to be published. The folder should be named <context root>.war to result in an application with the desired context root.

Files within this folder will be available with their file name.

A file in that folder with the name <filename> will be accessible as <public URI root>/<context root>/<filename>.

If the files are within sub folders, the folder path within the root folder and the file name compose the file name part of the URL. For example, the root folder has a sub folder names myfiles and a file test.xml in it, the file is accessible as <public URI root>/<context root>/myfiles/test.xml.

In our example with the given public URI root being https://clm.example.com:9443/, the folder name <context root> being PEWEnactmentData and <filename> being makers.xml the URL would be https://clm.example.com:9443/PEWEnactmentData/makers.xml.

1.1 Declaring the Application Location

The WAS Liberty Profile needs to know which application has to be published and where to find its resources. WAS Liberty Profile defines an XML element to declare which application is located where. The element is <appliction…./>. It supports various attributes and can come in many different forms. The most important attributes are

  • id: Must be unique and is used internally by the serve
  • location: The path to the application resource
  • context-root: The context root of the application
  • name: The name of the application
  • type: Specifies the type of application (war or ear)

The location and ID is required and either the name equal to the context root, or the context-root can be provided. The type can be omitted. I did not provide the ID and it worked for me, but since the server requires it, it should be provided. Use the same value as the name or context root for the ID.

It is necessary to add the application declaration to the server configuration for example by adding it to the server.xml file. In a CLM deployment a file appliction.xml is included in the server.xml, declaring all the applications and it makes sense to use that file instead.

Save the xml files after any change to activate the new setting. The server will pick up the change.

2. Deploying an Application – dropins

The WAS Liberty Profile has the capability to monitor a special dropins folder on a regular basis. If new content is detected, it is automatically made available by the server. The automatic monitoring needs to be enabled for this to work. A standard CLM deployment disables the monitoring to reduce server load.

2.1 Deploying the Application – dropins

In a default CLM install the dropins folder is located here:

<Install Folder>\server\liberty\servers\clm\dropins

Create a folder named <context root>.war in the dropins folder and add the files to publish. After detecting the addition, the files will be accessible like explained in 1.1.

The folder <Install Folder>\server\liberty\servers\clm\dropins should now look like below.

Published application with file

2.1 Enable Automatic Monitoring for Dropins

The automatic monitoring of the dropins folder is disabled for a standard CLM setup with WAS Liberty Profile. This saves processor cycles and I/O for the server operation. It has to be enabled once in the server.xml file for this solution to work. Find the XML element <applictionMonitor> and change the attribute dropinsEnabled from “false” to “true” and save the file. The image below shows the changed configuration file.

Enable Dropins Monitoring

The attribute pollingRate determines how often the server checks the location for changes. Adjust the value to your needs.

Save the server.xml file after any change to activate the new setting. The server will pick up the change.

The XML file content should now be accessible.

Example 1 – Standard

In the example we want to publish a document with the URI https://clm.example.com:9443/<context root>/makers.xml, where <context root>=PEWEnactmentData. The public URI root is already given by the server set up.

1.1 Create the “Application”

As first step create the application. Create a new folder named PEWEnactmentData.war in the folder <Install Folder>\server\liberty\servers\clm\apps. The folder could be created anywhere, but it makes sense to put it where the other applications are. If it is necessary to use a different folder, for example to allow other users to modify the content, place the folder in a different location and note the path. The path to the location will become important in the next step.

Put the files to be published underneath this folder. In the given example, put the file makers.xml into the folder.

The folder structure should look like the image below:

New Application folder

1.2 Publish the “Application”

The WAS Liberty Profile needs to know which application has to be published and where to find its resources. WAS Liberty Profile defines an XML element to declare which application is located where. The element is <appliction…./>. It supports various attributes and can come in many different forms. The most important attributes are

  • id: Must be unique and is used internally by the serve
  • location: The path to the application resource
  • context-root: The context root of the application
  • name: The name of the application
  • type: Specifies the type of application (war or ear)

The location and ID is required and either the name equal to the context root, or the context-root can be provided. The type can be omitted. I did not provide the ID and it worked for me, but since the server requires it, it should be provided. Use the same value as the name or context root for the ID.

The element can be put into several places. In a CLM installation, by default, the applications are declared in the file

<Install Folder>\server\liberty\servers\clm\conf\application.xml

The file is included in the file <Install Folder>\server\liberty\servers\clm\server.xml. It would be possible to place the application declaration in the file server.xml, but it seems to make more sense to put it into the file application.xml instead.

Add lines

<!–  My Application –>

<application id=”PEWEnactmentData” name=”PEWEnactmentData” location=”${server.config.dir}/apps/PEWEnactmentData.war”/>

to the file application.xml and save the file.

The file should look like below:

Declare new Application

In this case the location is relative to the server configuration folder which n our case is <Install Folder>\server\liberty\servers\clm\. The location could really be anywhere. If it is necessary to include a folder that, as an example, can be accessed by regular users, you could change the location to somewhere with less read and modification restrictions.

After the save of the application.xml, the published XML file should be accessible using the URL https://clm.example.com:9443/PEWEnactmentData/makers.xml.

The image below shows the browser access.

Browser Access To XML File

2. Deploying an Application – dropins

In the example we want to publish a document with the URI https://clm.example.com:9443/<context root>/makers.xml, where <context root>=PEWEnactmentData. The public URI root is already given by the server set up.

2.1 Create the “Application”

As first step create the application. Create a new folder named PEWEnactmentData.war in the folder <Install Folder>\server\liberty\servers\clm\dropins.

Put the files to be published underneath this folder. In the given example, put the file makers.xml into the folder.

The folder structure should look like the image below.

Published application with file

Assuming the automatic application monitoring is enabled, the file should be accessible after the next polling cycle, (at least after waiting the full pollingRate). If automatic application monitoring is not enabled, enable it as described in 2.2 and save the configuration file.

The XML file content should now be accessible using the URL https://clm.example.com:9443/PEWEnactmentData/makers.xml.

Deploying Compressed Folder Structures

It is also possible to publish real WAR, EAR similar to the way described above. For example create a WAR file as described in Publish XML Data Using Tomcat – Hotfix for The Process Enactment Workshop and attached to that post. Deploy the compressed WAR file, for example PEWEnactmentData.war in the dropins folder or declare it as an application and put it onto another location like explained below. This automatically deploys the application and makes the contained files accessible.

Just zipping the folder structure we created above into a zip file, name it PEWEnactmentData.war and trying to deploy does not seem to be working however. The WAR file needs to provide required supporting structures in this case.

Related Posts

The RM Extensions Hosting Guide for CLM 6.0.1 and later versions explain the dropins example as well.

Summary

This post shows how easy it is to publish supporting files on the WAS Liberty Profile. This is important if you want to host custom Doors Next Extensions, or if you want to publish your own XML data to be used with HTTP Filtered Value Sets, for the Process Enactment Workshop and it can possibly also used with Build result publishing. In the latter case, you would likely publish a root folder that contains all the build results.

As always I hope this post has some value to users out there, or is at least interesting to read.