Monday, December 13, 2010

Mapping COBOL level 88 to Java

Once in a while comes the issue of mapping COBOL data items with associated level 88 condition values to Java. Since this is a complex question, I thought it deserves an entry of its own.

This is an example of simple COBOL level 88 data item descriptions.

     01  MY-ITEM PIC S9(4) COMP.
         88 WRONG-DATA VALUE -1.
         88 NULL-DATA VALUE 0.

First notice that the 88 levels are associated with an actual data item (MY-ITEM). Level 88 items do not occupy any space in memory themselves.

The way they are used is that somewhere in the code, you can state:

         IF WRONG-DATA THEN ...

The condition will be true if, and only if, MY-ITEM contains -1.

Such code is more readable than the equivalent:

         IF MY-ITEM = -1 THEN ...

So this is what it's all about, an improvement to the readability of your COBOL code.

So how can such a thing map to Java?

First of all, it is easy to confuse COBOL Level 88 with java enums . After all, enums are also a readibility improvement feature and they can be used to replace a value with a meaningful label.

It is tempting to map MY-ITEM with:

    enum MyItem {
        WRONG_DATA(-1), NULL_DATA(0);
        private MyItem(int value) {
            _value = value;
        }

        private int _value;

        final int getValue() {
            return _value;
        }
    };

But there is a fundamental issue in doing so: Enums are actually restrictions. This means only the values that are enumerated (-1 and 0 in our case) are possible.

In the case of COBOL level 88, there are no restrictions on actual values. This means MY-ITEM can contain value 2, which is not described by any level 88.

Furthermore, level 88 are not necessarily atomic values. They can be ranges:

       01  ANOTHER-ITEM PIC S9(4) COMP.
           88 CUST-DATA VALUE 9 THRU 975.

Or even list of values:

       01  YET-ANOTHER-ITEM PIC S9(4) COMP.
           88 VALID-DATA VALUE 2, 7, 15, 245.

By now it should be clear why COBOL level 88 does not automatically map to a java variable of some sort with LegStar.

The legstar-cob2xsd module actually offers an option to map level 88 to XML Schema restrictions in the unlikely case where you are sure the COBOL variable never contains any value outside the enumerated values and enumerated values are atomic. But past that, there is no real support for level 88.

Monday, October 25, 2010

LegStar for Pentaho Data Integration (Kettle)

I have spent the last few weeks digging into open source ETL tools.

The 3 most quoted products are:

It turns out Clover is only partially open source. The GUI is proprietary. So I spent more time on PDI and Talend.

PDI is the oldest product and, perhaps as a consequence, has the largest community. You can get a sense of that by comparing the Ohloh page for PDI to the Ohloh page for Talend.

But if you compare new threads per day on the PDI forum to that of the Talend forum, you can see that Talend is doing good too.

I decided to try out PDI first and developed a proof of concept implementation of LegStar for PDI

You can see the result on Google code as usual.

I have to say that I am very impressed with PDI, a product originally called Kettle and developed by Matt Casters.

PDI comes with a framework for people to develop additional plugins. For those who are interested, there is an excellent blog entry by Slawomir Chodnicki to get started.

I was able to reuse part of the PDI internal test framework to automate testing of my plugin. I have automated unit tests and integration tests

It is also quite easy to deploy new plugins in PDI. It is a matter of packaging the plugin as a jar, and dropping it to a particular location.

As usual, it is extremely helpful that the product is open source. In particular, I could easily debug my plugin in Eclipse, stepping through PDI code as well as my code.

My only regrets with PDI is that there is little Maven support and that the code is often not commented. This being said, that did not prevent me from using Maven for all lifecycle phases of my plugin and was able to find my way into the PDI code which is usually readable enough.

PDI also has support for parallel processing and clustering that I did not explore yet. I am looking forward to playing with these features next.

Tuesday, October 12, 2010

VSAM to CSV using LegStar

I was wondering how hard it would be to translate a z/OS VSAM file content to a CSV file using LegStar.

So I started with a very trivial case:

CICS comes with a sample VSAM KSDS file called FILEA (see hlq.CICS.SDFHINST(DFHDEFDS)). FILEA is an 80 characters, fixed sized record file.

The records in FILEA are described by a COBOL copybook called DFH0CFIL (in hlq.CICS.SDFHSAMP). The content of DFH0CFIL looks like this:

My first step was to extract the VSAM file content to a sequential file using a JCL like this one on z/OS:

I then downloaded the sequential file to my workstation using regular FTP in binary mode:

On the java side now, I created a LegStar Transformer from the DFH0CFIL COBOL copybook. This results in a FilerecTransformers class.

The following code snippet, is what I needed to write to get my CSV file:

Of course this is a very contrived example, both because the VSAM file is fixed size and the record data is only made of characters.

In a more realistic case, the records will contain all sorts of numerics: compressed, zoned or edited and chances are that some redefines and arrays will complexify the setting. This is where LegStar should really become useful.

There are other interesting questions when you get to that point:

  • Assuming you need to regularly extract the content of a file, how do you automate a distributed process like this one?
  • How would you do that reliably; meaning you don't process the same data twice or miss part of it?

ETL (Extract Transfer Load) tools are typically focused on these issues. I became quite curious about them recently.

Sunday, August 22, 2010

Mule 3.0 is here

Mule ESB is undergoing its next mutation with the upcoming version 3.0. Mule ESB has always been one of the easiest ESB's to setup and work with. The intelligent combination of Spring and other open source technologies gives outstanding results.

An example of a unique feature, is the JUnit-based framework that comes with Mule. It is enormously convenient to be able to create a complete test case and run the entire Mule server from a single JUnit class! When I work with other ESB's I really miss this one.

Another example is the Spring-based configuration system. It is difficult to grasp all the power that you get from this. By contrast, ESB's that use proprietary DSL's for their configurations are limited to the features they build. With Spring, you can pretty much inject anything into you Mule configuration and reference that from anywhere else in the configuration.

A third example is the leveraging of Maven in various areas. Build management of course but also providing partners with Maven archetypes to help them bootstrap their transports and modules developments. That's so much better than documentation and samples.

These features alone could explain why Mule has become so popular.

Inevitably though, as larger companies and organizations started using it, demand for entreprise-class features has increased. Message flows that users needed to describe to Mule have become more complex. The limitations of the "endpoint/transformer/service" buiding blocs became apparent (Behaviors such as "Remote Sync" are still pretty obscure to me).

Mule 3.0 is introducing a new "flow" concept that should allow description of complete pipelines. I haven't played much with that yet but really looking forward to.

When you deploy large number of services in a single server instance, the class loading mechanism comes under stress. Typical problems are the need to share libraries and multiple versions of libraries having to coexist.

You don't solve problems like these without some major overhaul. Mule 3.0 is introducing a completely new deployment mechanism with features such as hot deployment.

Because there are major architectural changes, Mule 3.0 is not backward compatible. This might sound like an heresy to Mainframe people but remember how IBM CICS went from macro-level API to command-level APIs.

The reality is that you can't deal with core architectural issues without fundamental refactoring.

The good news is that the migration process is not that painful and is well documented. It took me only a few days of work to migrate the LegStar for Mule transport.

In the mainframe world, version 3 was usually the most complete and stable version. Looking at Mule 3.0 feature set, it could very well be the case for Mule too.

Friday, July 23, 2010

JSON to COBOL

It turns out it is fairly simple to transform JSON strings to COBOL using LegStar and Jackson.
To do this you would start from a COBOL fragment and use the usual LegStar COBOL to XSD translator and then generate transformers using LegStar COBOL Binding Generator.
Here is a sample JUnit code performing transformation from JSON to COBOL and vice versa:
package json.lsfileae.test;

import java.io.StringWriter;

import org.codehaus.jackson.map.AnnotationIntrospector;
import org.codehaus.jackson.map.ObjectMapper;
import org.codehaus.jackson.xc.JaxbAnnotationIntrospector;

import com.legstar.coxb.host.HostData;
import com.legstar.test.coxb.lsfileae.Dfhcommarea;
import com.legstar.test.coxb.lsfileae.bind.DfhcommareaTransformers;

import junit.framework.TestCase;

/**
* Unit test JSON/COBOL transformation.
*/
public class JacksonTest
extends TestCase {

 /** Hex EBCDIC data orignating from mainframe. */
 public static final String HOST_DATA =
     /*0 0 0 1 0 0 */
     "f0f0f0f1f0f0"
     /*  S .  D .   B O R M A N                  */
     + "e24b40c44b40c2d6d9d4c1d54040404040404040"
     /*  L A B A S   S T R E E T                 */
     + "e2e4d9d9c5e86b40c5d5c7d3c1d5c44040404040"
     /*  3 2 1 5 6 7 7 8 */
     + "f3f2f1f5f6f7f7f8"
     /*  2 6   1 1   8 1  */
     + "f2f640f1f140f8f1"
     /*  $ 0 1 0 0 . 1 1  */
     + "5bf0f1f0f04bf1f1"
     /*  * * * * * * * * *  */
     + "5c5c5c5c5c5c5c5c5c"
 ;

 /** JSON serialization of the host data. */
 public static final String JSON_DATA =
   "{\"comNumber\":100," +
   "\"comPersonal\":{\"comName\":\"S. D. BORMAN\"," +
   "\"comAddress\":\"SURREY, ENGLAND\"," +
   "\"comPhone\":\"32156778\"}," +
   "\"comDate\":\"26 11 81\"," +
   "\"comAmount\":\"$0100.11\"," +
   "\"comComment\":\"*********\"}";

 /** Jackson mapper. */
 private ObjectMapper _mapper;

 /** LegStar transformer. */
 private DfhcommareaTransformers _transformer;

 public void setUp() {
   _mapper = new ObjectMapper();
   AnnotationIntrospector introspector = new JaxbAnnotationIntrospector();
   _mapper.getDeserializationConfig().setAnnotationIntrospector(introspector);
   _mapper.getSerializationConfig().setAnnotationIntrospector(introspector);
   _transformer = new DfhcommareaTransformers();
 }

 /**
  * Transform host data to JSON and check
  * @throws Exception if transformation fails
  */
 public void testHostToJson() throws Exception {
   StringWriter sw = new StringWriter();
   _mapper.writeValue( sw, _transformer.toJava(HostData.toByteArray(HOST_DATA)));
   assertEquals(JSON_DATA,  sw.toString());
 }

 /**
  * Transform JSON to host data and check
  * @throws Exception if transformation fails
  */
 public void testJsonToHost() throws Exception {
   Dfhcommarea dfhcommarea = _mapper.readValue(JSON_DATA, Dfhcommarea.class);
   byte[] hostData = _transformer.toHost(dfhcommarea);
   assertEquals(HOST_DATA, HostData.toHexString(hostData));
 }
}

Tuesday, July 13, 2010

MuleSoftwebinar with MDI and LegStar

MuleSoft is putting together a Webinar on wednesday July 14th which features Mule ESB + MDI + LegStar. You can join the webinar here.
It tells a very compelling story where you want to web enable a CICS/COBOL program but would like a very different message structure than the original COBOL program input and output.
In this scenario LegStar does the COBOL parsing and mainframe transport while MDI takes over the transformation. The result is a Web Service with a nice interface deployed in Mule ESB. MDI really adds value here in that it hides the ugly COBOL structure and expose a set of friendly java classes instead.
COBOL structures, which typically describe CICS commareas, are often complex and convoluted due to the limitation of COBOL but also the need to reduce external calls and therefore cramming as much data as possible in a single commarea.
Exposing such complex structures as web services results in WSDL that is quite difficult to manipulate. MDI can be used to visually map these complex structures with simpler java classes and hence expose a nicer web service.

Friday, July 9, 2010

Just finished a series of videos showcasing how to create a mainframe adapter with LegStar for JBoss ESB.
I have become a fan of video demoes. I find them a lot easier to follow than reading paper documentation.
So I embarked in doing my own video demoes.
There are 2 things that I found amazing while doing this:
  1. How difficult it is to get something as obvious as a product demo video done
  2. How much data you can store on YouTube for free!
I had 3 choices when I started:
  1. filming myself doing the demo
  2. doing screen captures of the demo and recording my voice
  3. doing screen captures of the demo and adding subtitles
I ended up doing option 2. I ruled out option 1 for aesthetic reasons and 3 because I couldn't figure out how to do it .
I have been told that it would have been much easier on a Mac. Unfortunately, I am still a Windows user (Although I have one machine running Ubuntu, another running Fedora core and several other unixes running in VirtualBox).
I started by using Microsoft MovieMaker which is quite easy to learn, but the result was disappointing. I couldn't get the quality to be acceptable once the video was uploaded.
One of the issues with such product demoes is that you want the GUI to be readable. This is impossible with small frame sizes. Typically MovieMaker would not allow me to pick up 1024x768, which is what I ended up needing for the Eclipse GUI.
So I turned to these free products:
I also had to bring in the Xvid MPEG-4 Codec for compression. It turns out this is a codec that YouTube supports very well so there is very little quality loss when you upload to YouTube.
The result is not too bad although it still feels quite amateurish.
I have spent several days doing this, but feel I would need a lot lot more to make it look professional.
Hopefully this will be enough to help people learn the product faster.

Saturday, April 24, 2010

From TUI to WUI, not that simple

Mainframes 3270 interfaces are usually text based. Even though, graphics are possible on mainframes, they are rarely used. Users of such systems are familiar with the black background and green/blue text but more importantly, they are also familiar with the function key paradigm.
With 3270, nothing is transmitted to the host until a function key is pressed. Very little validation occur when the user fills in the input forms. This is not unlike HTML forms which are only posted when a command button is clicked.
For that reason, most solutions available today for scraping 3270 interfaces are based on HTML/JSP technologies.
One of the drawbacks though is the "flickering effect" that HTML/JSP suffers from when forms are posted. The entire web page is refreshed every time you interact with the host.
The 3270 protocol was smarter and able to refresh only those parts of the display that had actually changed. Many users living the transition from TUI to WUI complain about this. 3270 could optimize screen refresh because terminals were dumb. Terminals new very little about the semantics of the data they received (hence the limited local validation they could perform). Data was fundamentally just a stream of characters and associated positions/attributes on screen.
WUI have evolved past the "function key" and flickering effect by adopting the Ajax technologies so you might think that Ajax-based screen scraping solutions would be a perfect solution.
In reality though, Ajax is useless if you don't transfer logic from the host to the browser. The real Ajax added value is in allowing web applications to behave more like "rich clients".
What this all means is that if you have a mainframe 3270-based application that you are not willing to modify, there is little chance you will be able to build a rich, web-based, user interface to it.

Sunday, March 7, 2010

Itemscript a serious JSON library for java

JSON is an interesting alternative to XML. It has originally grown from javaScript but is now widely available for a large number of languages and many libraries are built with JSON support.
Of particular interest is the constant focus of the JSON community on simplicity. The moto seems to be: anything you can do with XML you can do with JSON, faster and cheaper.
itemscript is an example of such a library.
Itemscript is a rich JSON API for java. This is a sample taken from the product test cases that illustrates some of the nice aspects of the API:
JsonArray array = system().createArray()
.a("foo")
.a(1)
.a(1.5)
.a(true)
.a(123L);
assertEquals("foo", array.getString(0));
assertEquals((Integer) 1, array.getInt(1));
assertEquals((Double) 1.5, array.getDouble(2));
assertEquals((Float) 1.5f, array.getFloat(2));
assertEquals((Boolean) true, array.getBoolean(3));
assertEquals((Long) 123L, array.getLong(4));
Notice the elegant chaining of add operations and the automatic type inferencing mechanism.
Itemscipt also has a binding mechanism called the Foundry that ties POJOs to JSON. The best way to illustrate this is to look at some code.
On top of the JSON API, itemscript provides a way to store and query JSON objects in a RESTful manner creating an in-memory database of linked objects. This is an example inspired by the itemscript samples:
/* create a hierarchy */
system.put("mem:/abc/def", "123");
system.put("mem:/abc/ghi", "456");
/* query the in-memory data */
assertEquals(2, (int) system.getInt("mem:/abc?countItems"));
JsonArray keys = system.getArray("mem:/abc?keys");
assertEquals("[\"ghi\",\"def\"]", keys.toCompactJsonString());
JsonArray pagedItems = system.getArray("mem:/abc?pagedItems&numRows=2&startRow=0");
assertEquals("[[\"def\",\"123\"],[\"ghi\",\"456\"]]", pagedItems.toCompactJsonString());
Last but not least, Itemscript is open source and is hosted on Google code.
I am thinking of ways to leverage libraries such as itemscript to provide COBOL to JSON binding. As JSON becomes more and more popular there are good chances its trajectory will cross the legacy applications one. Would be nice to have LegStar at the junction.