Made some progress on the Documentum-Cocoon stuff over the Christmas break. I’m working on a piece that will allow me to dynamically build charts and graphs based on data stored in the docbase. Here’s what I’ve done and what’s left to do:

Step 1. Figure out how to build a bar chart in SVG

First, I needed to figure out how to build a chart using SVG. I found some Java code that builds charts using SVG but what I was looking to do was build the SVG programmatically using XSLT to transform my source data which will be XML coming from my Documentum Transformer.

I needed something I could use to graphically draw the chart to get me started on the SVG. I downloaded a drawing tool called Sodipodi for this. It worked great. It stores everything as SVG. So all I had to do was draw my chart and save it to get a nice XML file to start with.

Step 2: Create the SVG with XSLT

Now that I had a static SVG file I needed to create XSLT that would take source data XML and transform it into the bar chart SVG. First, I copied my static SVG file into my Cocoon install to make sure it was happy rendering it into JPEG and PNG. It was. Next, I downloaded the Batik binaries. I wanted a tool I could use to quickly view my SVG file as I tweaked it. Batik comes with an SVG viewer called Squiggle. In WSAD, I created a test XML source data file, copied my static SVG file from Step 1 into a new XSLT file, tweaked, and then viewed the results in Squiggle.

The first thing I did was add comments so I could figure out which part of the SVG did what. In Sodipodi I had set every object’s “id” attribute, so correlating the XML to the object it was describing was pretty easy. Adding comments made it easy to rearrange blocks of XML to work how I needed it to in the stylesheet.

The next thing I did was clean up the SVG by rounding off all of the numbers Sodipodi had used for the coordinates of my shapes. Obviously with a design tool everything has got to be very precise, but for what I needed, whole numbers worked just fine and it made the XML easier to look at.

I then figured out the algorithms I’d need to use to figure out how to dynamically size and position the bars and bar legends in the chart.

Finally, I replaced the hardcoded values with XSLT variables. I then updated my source XML data with new values, transformed, and voila: Dynamically generated bar chart.

Step 3: Grab the data from Documentum on a schedule

The data I need to chart is in Documentum. It changes over time. Because it’s historical data I need to get a snapshot from Documentum on a schedule and store the result somewhere. Then, my chart will be built on-demand via a Cocoon pipeline using the data that’s been captured for a given time period.

I already have a pipeline that gets the snapshot from Documentum. All I need to do is execute that pipeline on a schedule and store the result somewhere. I’ve tested out the “cron” feature of Cocoon and it works great. It allows you to execute a pipeline on a schedule. A perfect fit.

Now what I need is to put my snapshot data somewhere. I’m planning on using Xindice. It’s an XML database embedded in Cocoon (you can use a separate Xindice install if you want). My pipeline will pull data from Documentum, transform it (I really only need to store a subset/summarized cut of the data), and store it in Xindice.

To complete my work for this step, I need to look at the XML:DB examples in Cocoon to learn how to stick data in Xindice from a pipeline.

Step 4: Build the chart based on the historical data

Now that I’ve got my historical data summarized and sitting in Xindice, I need to create a pipeline that will query Xindice for the data and transform the query result using the SVG I build in Step 2.

To complete my work for this step, I need to look at the XML:DB examples in Cocoon to learn how to query XML data from Xindice using XPath. The rest should be a piece of cake.

The result

Once this is in place, I’ll have an approach and some reusable code I can use to capture and chart source data from Documentum.

Here‘s a new minor release of the Documentum-Cocoon integration components. It includes the reader and generator components I referenced a while back and a minor tweak to how I’m handling namespaces. (I’m not convinced I’m handling them the way I should, BTW). I’ve mentioned needing to develop a Documentum-specific protocol but now that I have a reader and a generator, that doesn’t seem as important–readers and generators could be used as “internal” resources within a pipeline to retrieve things like XSL from the docbase to perform transformations.

I’m a little torn on what to work on next regarding this integration. I’d like a more elegant way to handle sessions. I can add single sign-on fairly easily so that when called from a WDK application the credentials would be passed to Cocoon and then used to create a session as the currently logged in Documentum user.

I’d also like to migrate the administrator dashboard prototype I’ve developed to use the Cocoon Portal UI framework.

Another idea is to use Cocoon scheduler to wake up periodically, execute a pipeline that reads data from Documentum, and persist that data either back to Documentum or into the Xindice XML database. This could be used to run historical reports, for example.

At Momentum I ran across a presentation by a fellow doing XML pipelining with Cocoon and Documentum. We exchanged information. If he posts his components for public consumption I’ll link to them.

Got a Reader and a Generator added to my Cocoon-Documentum integration package last week. I also tested out the sendmail transformer that comes with Cocoon. Now, I can grab any file from Documentum from Cocoon using the Reader and then stream it to the browser or email it. If I want to transform an object stored in Documentum, I can use the Generator to snag it and then use it as I normally would in any other pipeline. For both I’m using the same protocol Documentum uses to reference documents through their “virtual link” support which is /<docbase>:<full path to content>.

I haven’t made the Reader or the Generator available yet. I’ll add it to the Transformer package when I get a chance.

What’s left to do is create a protocol. That will allow me to use any Documentum object anywhere in a pipeline rather than just being restricted to Readers and Generators. That means I could store my XSL stylesheets as objects in the Documentum repository instead of on the file system if I wanted/needed to, although that would degrade pipeline performance. 

Finally got the Documentum Cocoon Transformer cleaned up in WSAD to make it easier for me to share with others. Here is version 0.1. I’ve released it under the Apache License.

It isn’t anything fancy. Very little code. There are probably other Cocoon-Documentum integration points that would be useful but I haven’t knocked any out yet.

I’ve had some interest in the Documentum Cocoon Transformer. I’ll probably make it available to whoever wants it.

Finally got my Cocoon Documentum Transformer fully functional this evening thanks to the good folks at the Cocoon Users Mailing List. My problem was that the Documentum xDQL query was returning a full XML document. When I tried to parse that, the extra startDocument and endDocument calls were causing the runtime exception (archived mailing list thread).

Now that I have my transformer working, I can configure any number of XML docs I want, each with any number of xDQL queries against any docbase my server can see. This is real handy because there are times when you want to run the same query against multiple docbases. I’ve currently got the pipeline configured to style the query results using a generic xDQL-to-HTML stylesheet, but now that I’ve got everything working, the sky is the limit. Here are some ideas…

Documentum admin portal

Go to one page and see very quickly what the status is of key metrics across all Documentum servers, docbases, and environments (Dev, QA, Prod). This is easily done using Cocoon’s aggregation of my styled query results. If I really wanted to get fancy I could use the Cocoon portal framework and create an honest-to-goodness portal with multiple users and profiles (different users might want to monitor different things on different docbases) as well as security.

But, at some point, this thing starts to approach the out-of-the-box Documentum Administrator client and I’m definitely not up for reinventing that wheel. So, this one is a lower priority.

Documentum RSS alerts

Create one or more RSS feeds that will syndicate alerts for things in the environment that need attention. To do this, I’ll add another transformation in my pipeline that will apply business logic to the query results and format “alerts” as XML. I will then add an alert-XML-to-RSS stylesheet. (I’m separating the two because I might want to format alerts in HTML). Once that is in place, I can use a news aggregator like Amphetadesk or NewsMonster to poll my Documentum RSS alerts periodically for things that need attention.

This one is nice because it does not force the admins to go to a web page–they just leave their aggregator running on their desktop.

Return Documentum Query Results in a Spreadsheet

Sometimes you’d like to be able to query multiple docbases and have the results in a spreadsheet so you can then do some analysis or whatever. Today you have to query each docbase and then cut-and-paste the results. With my transformer in place, I ought to be able to spit my query results into an Excel spreadsheet, PDF, SVG, whatever.

Two things are stealing time away from the time budget previously allocated to blogging:

1. Netflix. Having a constant supply of movies on-hand that have been on my to-be-watched list for some time is awesome, and I love that I’m getting caught up. But, the blog has suffered.

2. Integrating Documentum with Cocoon. I got my DQL Transformer working some time ago but I ran into a problem transforming the DQL query result with my stylesheet. What sucks is that I had to upgrade my DFC and all of my WDK apps because of my current project, and after that, I discovered the problem. So now I’ve got to figure out if it was the upgrade or something else.

Until now I hadn’t been able to get logging to work from my custom transformer but this problem forced me to work through that. It turned out to be a simple thing that just wasn’t documented clearly. The solution was to use the logger attribute on my transformer declaration in sitemap.xmap like this:

<map:transformer name=”dqlquery” src=”com.navigatorsystems.cocoon.dctm.transformation.XdqlTransformer” logger=”sitemap.transformer.dqlquery”>

The category declaration in logkit.xconf matches up with that like this:

<category log-level=”DEBUG” name=”sitemap.transformer.dqlquery”>
  <log-target id-ref=”dqlquery”/>
  <log-target id-ref=”error”/>
</category>

The DQL Transformer works like the SQL Transformer. It receives SAX events looking for a query element. When it finds it, it executes the query against Documentum as an XDQL query (ie, the DQL query results get returned as a well-formed XML document). The XML that gets returned gets sent to the next component in the pipeline (like another transformer or a serializer).

I am now back to running on Cocoon 2.1 instead of the version that came with the Cocoon book. And I’m running it on Tomcat 4.1.