Thursday, January 6, 2011

Batch integration with CICS, ETL integration with ESB

On mainframes, integration between batch and online processes is not simple. In an IBM CICS environment for instance, if files and databases need to be shared between batch and CICS (and they most always do), you have to deal with a number of issues such as:

  • Batch processes might update numerous records. It is usually inefficient to commit those changes after each record change. Large number of uncommitted changes means large numbers of locks which in turn affect online activity by slowing down response times or even leading to time out errors.
  • Batch processes often deal with files in addition to databases. The most widely used technique to restart a batch after a failure is to backup these files before the batch is started and restore them in case of failure so that the batch can be restarted. Any online activity that dealt with the same files between the batch start and failure worked on uncommitted data.

To avoid such issues, many mainframe shops initially segregated batch and online activities in different time frames. Typically CICS systems were brought down in the evening while batch processes were running and restarted in the morning.

Because of this strong separation between online and batch activity, very little integration between batch and CICS systems was developed. There was some degree of code reuse with COBOL copybooks but you would never see binary reuse (a batch program calling a CICS program for instance). Actually this was not even possible before IBM introduced the EXCI technology.

The mainframe nightly batch window rapidly came under pressure though. As mainframes grew larger and merged with one another, they started serving populations across many time zones. It is not uncommon that users are all over the world. This forced mainframe shops to rearchitect their batch processes and lead IBM to develop new technologies such as EXCI and VSAM Record Level Sharing.

At the same time, the need for business logic reuse between batch and CICS became more important because systems became more complex. Mainframe developers resorted to database triggers, stored procedures or WebSphere MQ triggers, probably beyond the original intent of such technologies, because there was no other way of sharing logic at the binary level.

I am seeing some similarity with the ETL versus ESB situation in the Java world. These systems do very little to integrate with one another today.

By nature, ETL is similar to batch in the sense that it deals with large amounts of records and multiple file systems and databases. ESB is closer to online as it deals with smaller transactional events.

Both ETL and ESB products claim to be transformation engines though and indeed the term "Transformation" is widely used in both type of products documentation. So you might wonder why it is almost impossible to reuse a transformation written for an ETL in an ESB.

If IBM was forced to introduce EXCI for Batch to CICS communication, I wouldn't be surprised if users forced ETL and ESB vendors to integrate with one another more closely.

I don't mean that ETL and ESB technologies need to be tightly integrated, after all they do different jobs, but yet it would be nice if some level of transformation reusability can be achieved.

Monday, December 13, 2010

Mapping COBOL level 88 to Java

Once in a while comes the issue of mapping COBOL data items with associated level 88 condition values to Java. Since this is a complex question, I thought it deserves an entry of its own.

This is an example of simple COBOL level 88 data item descriptions.

     01  MY-ITEM PIC S9(4) COMP.
         88 WRONG-DATA VALUE -1.
         88 NULL-DATA VALUE 0.

First notice that the 88 levels are associated with an actual data item (MY-ITEM). Level 88 items do not occupy any space in memory themselves.

The way they are used is that somewhere in the code, you can state:

         IF WRONG-DATA THEN ...

The condition will be true if, and only if, MY-ITEM contains -1.

Such code is more readable than the equivalent:

         IF MY-ITEM = -1 THEN ...

So this is what it's all about, an improvement to the readability of your COBOL code.

So how can such a thing map to Java?

First of all, it is easy to confuse COBOL Level 88 with java enums . After all, enums are also a readibility improvement feature and they can be used to replace a value with a meaningful label.

It is tempting to map MY-ITEM with:

    enum MyItem {
        WRONG_DATA(-1), NULL_DATA(0);
        private MyItem(int value) {
            _value = value;
        }

        private int _value;

        final int getValue() {
            return _value;
        }
    };

But there is a fundamental issue in doing so: Enums are actually restrictions. This means only the values that are enumerated (-1 and 0 in our case) are possible.

In the case of COBOL level 88, there are no restrictions on actual values. This means MY-ITEM can contain value 2, which is not described by any level 88.

Furthermore, level 88 are not necessarily atomic values. They can be ranges:

       01  ANOTHER-ITEM PIC S9(4) COMP.
           88 CUST-DATA VALUE 9 THRU 975.

Or even list of values:

       01  YET-ANOTHER-ITEM PIC S9(4) COMP.
           88 VALID-DATA VALUE 2, 7, 15, 245.

By now it should be clear why COBOL level 88 does not automatically map to a java variable of some sort with LegStar.

The legstar-cob2xsd module actually offers an option to map level 88 to XML Schema restrictions in the unlikely case where you are sure the COBOL variable never contains any value outside the enumerated values and enumerated values are atomic. But past that, there is no real support for level 88.

Monday, October 25, 2010

LegStar for Pentaho Data Integration (Kettle)

I have spent the last few weeks digging into open source ETL tools.

The 3 most quoted products are:

It turns out Clover is only partially open source. The GUI is proprietary. So I spent more time on PDI and Talend.

PDI is the oldest product and, perhaps as a consequence, has the largest community. You can get a sense of that by comparing the Ohloh page for PDI to the Ohloh page for Talend.

But if you compare new threads per day on the PDI forum to that of the Talend forum, you can see that Talend is doing good too.

I decided to try out PDI first and developed a proof of concept implementation of LegStar for PDI

You can see the result on Google code as usual.

I have to say that I am very impressed with PDI, a product originally called Kettle and developed by Matt Casters.

PDI comes with a framework for people to develop additional plugins. For those who are interested, there is an excellent blog entry by Slawomir Chodnicki to get started.

I was able to reuse part of the PDI internal test framework to automate testing of my plugin. I have automated unit tests and integration tests

It is also quite easy to deploy new plugins in PDI. It is a matter of packaging the plugin as a jar, and dropping it to a particular location.

As usual, it is extremely helpful that the product is open source. In particular, I could easily debug my plugin in Eclipse, stepping through PDI code as well as my code.

My only regrets with PDI is that there is little Maven support and that the code is often not commented. This being said, that did not prevent me from using Maven for all lifecycle phases of my plugin and was able to find my way into the PDI code which is usually readable enough.

PDI also has support for parallel processing and clustering that I did not explore yet. I am looking forward to playing with these features next.

Tuesday, October 12, 2010

VSAM to CSV using LegStar

I was wondering how hard it would be to translate a z/OS VSAM file content to a CSV file using LegStar.

So I started with a very trivial case:

CICS comes with a sample VSAM KSDS file called FILEA (see hlq.CICS.SDFHINST(DFHDEFDS)). FILEA is an 80 characters, fixed sized record file.

The records in FILEA are described by a COBOL copybook called DFH0CFIL (in hlq.CICS.SDFHSAMP). The content of DFH0CFIL looks like this:

My first step was to extract the VSAM file content to a sequential file using a JCL like this one on z/OS:

I then downloaded the sequential file to my workstation using regular FTP in binary mode:

On the java side now, I created a LegStar Transformer from the DFH0CFIL COBOL copybook. This results in a FilerecTransformers class.

The following code snippet, is what I needed to write to get my CSV file:

Of course this is a very contrived example, both because the VSAM file is fixed size and the record data is only made of characters.

In a more realistic case, the records will contain all sorts of numerics: compressed, zoned or edited and chances are that some redefines and arrays will complexify the setting. This is where LegStar should really become useful.

There are other interesting questions when you get to that point:

  • Assuming you need to regularly extract the content of a file, how do you automate a distributed process like this one?
  • How would you do that reliably; meaning you don't process the same data twice or miss part of it?

ETL (Extract Transfer Load) tools are typically focused on these issues. I became quite curious about them recently.

Sunday, August 22, 2010

Mule 3.0 is here

Mule ESB is undergoing its next mutation with the upcoming version 3.0. Mule ESB has always been one of the easiest ESB's to setup and work with. The intelligent combination of Spring and other open source technologies gives outstanding results.

An example of a unique feature, is the JUnit-based framework that comes with Mule. It is enormously convenient to be able to create a complete test case and run the entire Mule server from a single JUnit class! When I work with other ESB's I really miss this one.

Another example is the Spring-based configuration system. It is difficult to grasp all the power that you get from this. By contrast, ESB's that use proprietary DSL's for their configurations are limited to the features they build. With Spring, you can pretty much inject anything into you Mule configuration and reference that from anywhere else in the configuration.

A third example is the leveraging of Maven in various areas. Build management of course but also providing partners with Maven archetypes to help them bootstrap their transports and modules developments. That's so much better than documentation and samples.

These features alone could explain why Mule has become so popular.

Inevitably though, as larger companies and organizations started using it, demand for entreprise-class features has increased. Message flows that users needed to describe to Mule have become more complex. The limitations of the "endpoint/transformer/service" buiding blocs became apparent (Behaviors such as "Remote Sync" are still pretty obscure to me).

Mule 3.0 is introducing a new "flow" concept that should allow description of complete pipelines. I haven't played much with that yet but really looking forward to.

When you deploy large number of services in a single server instance, the class loading mechanism comes under stress. Typical problems are the need to share libraries and multiple versions of libraries having to coexist.

You don't solve problems like these without some major overhaul. Mule 3.0 is introducing a completely new deployment mechanism with features such as hot deployment.

Because there are major architectural changes, Mule 3.0 is not backward compatible. This might sound like an heresy to Mainframe people but remember how IBM CICS went from macro-level API to command-level APIs.

The reality is that you can't deal with core architectural issues without fundamental refactoring.

The good news is that the migration process is not that painful and is well documented. It took me only a few days of work to migrate the LegStar for Mule transport.

In the mainframe world, version 3 was usually the most complete and stable version. Looking at Mule 3.0 feature set, it could very well be the case for Mule too.

Friday, July 23, 2010

JSON to COBOL

It turns out it is fairly simple to transform JSON strings to COBOL using LegStar and Jackson.
To do this you would start from a COBOL fragment and use the usual LegStar COBOL to XSD translator and then generate transformers using LegStar COBOL Binding Generator.
Here is a sample JUnit code performing transformation from JSON to COBOL and vice versa:
package json.lsfileae.test;

import java.io.StringWriter;

import org.codehaus.jackson.map.AnnotationIntrospector;
import org.codehaus.jackson.map.ObjectMapper;
import org.codehaus.jackson.xc.JaxbAnnotationIntrospector;

import com.legstar.coxb.host.HostData;
import com.legstar.test.coxb.lsfileae.Dfhcommarea;
import com.legstar.test.coxb.lsfileae.bind.DfhcommareaTransformers;

import junit.framework.TestCase;

/**
* Unit test JSON/COBOL transformation.
*/
public class JacksonTest
extends TestCase {

 /** Hex EBCDIC data orignating from mainframe. */
 public static final String HOST_DATA =
     /*0 0 0 1 0 0 */
     "f0f0f0f1f0f0"
     /*  S .  D .   B O R M A N                  */
     + "e24b40c44b40c2d6d9d4c1d54040404040404040"
     /*  L A B A S   S T R E E T                 */
     + "e2e4d9d9c5e86b40c5d5c7d3c1d5c44040404040"
     /*  3 2 1 5 6 7 7 8 */
     + "f3f2f1f5f6f7f7f8"
     /*  2 6   1 1   8 1  */
     + "f2f640f1f140f8f1"
     /*  $ 0 1 0 0 . 1 1  */
     + "5bf0f1f0f04bf1f1"
     /*  * * * * * * * * *  */
     + "5c5c5c5c5c5c5c5c5c"
 ;

 /** JSON serialization of the host data. */
 public static final String JSON_DATA =
   "{\"comNumber\":100," +
   "\"comPersonal\":{\"comName\":\"S. D. BORMAN\"," +
   "\"comAddress\":\"SURREY, ENGLAND\"," +
   "\"comPhone\":\"32156778\"}," +
   "\"comDate\":\"26 11 81\"," +
   "\"comAmount\":\"$0100.11\"," +
   "\"comComment\":\"*********\"}";

 /** Jackson mapper. */
 private ObjectMapper _mapper;

 /** LegStar transformer. */
 private DfhcommareaTransformers _transformer;

 public void setUp() {
   _mapper = new ObjectMapper();
   AnnotationIntrospector introspector = new JaxbAnnotationIntrospector();
   _mapper.getDeserializationConfig().setAnnotationIntrospector(introspector);
   _mapper.getSerializationConfig().setAnnotationIntrospector(introspector);
   _transformer = new DfhcommareaTransformers();
 }

 /**
  * Transform host data to JSON and check
  * @throws Exception if transformation fails
  */
 public void testHostToJson() throws Exception {
   StringWriter sw = new StringWriter();
   _mapper.writeValue( sw, _transformer.toJava(HostData.toByteArray(HOST_DATA)));
   assertEquals(JSON_DATA,  sw.toString());
 }

 /**
  * Transform JSON to host data and check
  * @throws Exception if transformation fails
  */
 public void testJsonToHost() throws Exception {
   Dfhcommarea dfhcommarea = _mapper.readValue(JSON_DATA, Dfhcommarea.class);
   byte[] hostData = _transformer.toHost(dfhcommarea);
   assertEquals(HOST_DATA, HostData.toHexString(hostData));
 }
}

Tuesday, July 13, 2010

MuleSoftwebinar with MDI and LegStar

MuleSoft is putting together a Webinar on wednesday July 14th which features Mule ESB + MDI + LegStar. You can join the webinar here.
It tells a very compelling story where you want to web enable a CICS/COBOL program but would like a very different message structure than the original COBOL program input and output.
In this scenario LegStar does the COBOL parsing and mainframe transport while MDI takes over the transformation. The result is a Web Service with a nice interface deployed in Mule ESB. MDI really adds value here in that it hides the ugly COBOL structure and expose a set of friendly java classes instead.
COBOL structures, which typically describe CICS commareas, are often complex and convoluted due to the limitation of COBOL but also the need to reduce external calls and therefore cramming as much data as possible in a single commarea.
Exposing such complex structures as web services results in WSDL that is quite difficult to manipulate. MDI can be used to visually map these complex structures with simpler java classes and hence expose a nicer web service.