When it comes to Mainframes, nothing is simple

The IEEE Software review has published an article by Belgian researchers who made an attempt at re-engineering a mainframe application using automated tooling they knew worked in other environments.

Although the article mentions important lessons learned, it proved once more that it is extremely difficult to reconstruct mainframe applications knowledge automatically from static code analysis.

They typically ran into an issue, I for one, encountered several times, where an apparently autonomous set of COBOL programs actually calls some Assembler magic that turns control to other hidden programs only identified at runtime. There is little you can't do on a mainframe using Assembler. One thing you can do, is load executables dynamically.

During the 80's, the heydays of Mainframe programming, every IT shop had one or more Assembler guru's. With much looser budget controls than today, it was possible to spend considerable time developing in-house frameworks, optimizing performances and so forth. Not to say that this only had negative effects, these optimized assembler routines probably saved large amounts of money by reducing CPU consumption, a major parameter on IBM bills.

Most code analysis tools are COBOL centric. One reason for that is that COBOL is not that hard to parse. To my knowledge, there are no automatic code analysis tools for MVS Assembler (DataTek has an impressive tool for Assembler to COBOL but it usually requires some level of human assistance). That's probably because the level of complexity such a tool would require would be several levels of magnitude higher than COBOL.

For a typical Mainframe shop, the volume of MVS Assembler programs is much smaller than COBOL. That might explain why vendors would have a hard time cost justifying writing a very complex Assembler code analysis tool.

So the result of such a situation is that COBOL code analysis tools fail on multilanguage boundaries. Of course there are other multilanguage boundaries on Mainframes for instance: COBOL to BMS or MFS macros, COBOL to JCL, COBOL to SQL, COBOL to CICS, COBOL to IMS not to mention all third party tools one can find on Mainframes. One thing for sure, I have never seen a pure COBOL application.

The problem is that without reliable code analysis tools, you can't reconstruct knowledge in a bottom up approach. What you end up doing is taking the top down approach, by chasing the last application expert on site, hoping that he hasn't retired yet.

The complexity revealed by this IEEE article explains a good part of the Mainframe applications longevity. I know IBM prefers the reliability, availability, security explanation. But I know many IT shops who would have happily migrated off their mainframes if it was easy.

LegStar is also affected by this complexity of course. I often have a hard time explaining to Java developers why the Java side of LegStar works so easily while the Mainframe side, which has much less code in it, is often much harder to get to work properly...

COBOL in a flat world

We just released a version of LegStar for Talend Open Studio. Talend is a well known ETL tool that also expands to the MDM and ESB spaces.

This is the second ETL tool we interface with. The first one was Pentaho Data Integration a.k.a Kettle for which we released legstar-pdi back in November 2010.

With this experiences behind us it becomes clear that ETL tools are row centric. This means that data flowing from one step to another needs to be modeled as a flat, fixed, list of fields. This exactly maps to a classical database row. Not surprising as ultimately, an ETL, must feed database tables somewhere.

When the starting point is COBOL though, where data structures tend to be very hierarchical, fitting in a flat model is challenging. Here are some considerations to keep in mind.

Name conflicts:

Flattening a simple hierarchy such as:

01 ITEM-A. 05 ITEM-B PIC X. 05 ITEM-C. 10 ITEM-D PIC 9(4) COMP.

is relatively easy, it intuitively maps to: [ItemB:String, ItemD:Short].

Now what happens for this one (perfectly valid in COBOL):

01 ITEM-A. 05 ITEM-B PIC X. 05 ITEM-C. 10 ITEM-B PIC 9(4) COMP.

In a flat model, field names must be unique so [ItemB:String, ItemB:Short] does not work. You need to disambiguate names and produce something like [ItemB:String, ItemC_ItemB:Short].

Arrays:

Assuming a COBOL data item such as:

05 ITEM-A OCCURS 5 TIMES PIC X.

we have an new issue since arrays are usually not handled by database schemas.

Now the solution is to expand each item into a different field. Something like [ItemA_0:String, ItemA_1:String, ItemA_2:String, ItemA_3:String, ItemA_4:String].

This is quite wasteful but no real alternatives here.

In COBOL, the DEPENDING ON clause is often used to limit array sizes and processing time. Here we hit another limitation of the flat models, they are usually fixed in the sense that all fields declared must be present in each row.

Filling unused fields with null values is a common technique used to tell downstream steps that fields have no value.

Redefines:

The COBOL REDEFINES clause, a cousin of the C union, is another interesting challenge. Since the flat model is fixed it can't be dynamically changed depending on the REDEFINES alternatives.

The best solution here is to manage a different set of flat fields for each combination of alternatives. This can be demonstrated with a simple example:

01 RDEF01-RECORD. 05 COM-SELECT PIC 9(4) COMP. 88 DETAIL1 VALUE 0. 88 DETAIL2 VALUE 1. 05 COM-DETAIL1. 10 COM-NAME PIC X(10). 05 COM-DETAIL2 REDEFINES COM-DETAIL1. 10 COM-AMOUNT PIC 9(5)V99 COMP-3.

Here the COM-SELECT field value determines if COM-DETAIL1 or COM-DETAIL2 is present in the COBOL data.

This would result in 2 field sets (schemas in ETL parlance):

• set1: [ComSelect:Short, ComName:String]
• set2: [ComSelect:Short, ComAmount:Decimal]

The number of field sets you need to contemplate depends on the number of alternatives in each REDEFINE group (an ITEM followed by a set of items redefining its location). Furthermore, if a COBOL structure contains multiple such REDEFINE groups, than all combinations are possible. So lets say a COBOL structure has a first group of 3 alternatives and another of 2 alternatives, there are 6 (3 x 2) possible field sets.

Fortunately, the number of REDEFINE groups and the number of alternatives in each groups are usually small.

What this all means is that COBOL structures need somehow to be shoehorned to fit the ETL data model. This is an important difference with ESBs where the data model is usually a much more versatile Java object.

See you at Maven central

Maven central has long been restricted to a few very large players such as the Apache foundation.

This has changed recently thanks to a new free offering by Sonatype for OSS projects.

LegStar has been using Maven from the very beginning and more and more users rely on the availability of artifacts in the LegStar Maven repository.

This proprietary repository is not very secure and not always available. Now that it is possible to push artifacts to Maven central I have been busy figuring out how to take advantage of this.

The major issue is that Maven central's policy is to host artifacts only if their dependencies are also in central. And of course LegStar has dependencies on oss libraries which are not in central

One of the bad players is the Eclipse foundation. So far, no complete sets of Eclipse bundles are available as Maven artifacts. There are lengthy discussions going on but no results yet.

Besides Eclipse the only other annoying dependency we have in LegStar is Websphere MQ. Of course this one being proprietary, it will never make it to central.

In order to bootstrap the process of moving to Maven central, I have started to split the modules into separate release units. the first, very limited, release is now in Maven central. You can see the result at this location by entering the legstar keyword.

It feels good to be part of the big league

Batch integration with CICS, ETL integration with ESB

On mainframes, integration between batch and online processes is not simple. In an IBM CICS environment for instance, if files and databases need to be shared between batch and CICS (and they most always do), you have to deal with a number of issues such as:

• Batch processes might update numerous records. It is usually inefficient to commit those changes after each record change. Large number of uncommitted changes means large numbers of locks which in turn affect online activity by slowing down response times or even leading to time out errors.
• Batch processes often deal with files in addition to databases. The most widely used technique to restart a batch after a failure is to backup these files before the batch is started and restore them in case of failure so that the batch can be restarted. Any online activity that dealt with the same files between the batch start and failure worked on uncommitted data.

To avoid such issues, many mainframe shops initially segregated batch and online activities in different time frames. Typically CICS systems were brought down in the evening while batch processes were running and restarted in the morning.

Because of this strong separation between online and batch activity, very little integration between batch and CICS systems was developed. There was some degree of code reuse with COBOL copybooks but you would never see binary reuse (a batch program calling a CICS program for instance). Actually this was not even possible before IBM introduced the EXCI technology.

The mainframe nightly batch window rapidly came under pressure though. As mainframes grew larger and merged with one another, they started serving populations across many time zones. It is not uncommon that users are all over the world. This forced mainframe shops to rearchitect their batch processes and lead IBM to develop new technologies such as EXCI and VSAM Record Level Sharing.

At the same time, the need for business logic reuse between batch and CICS became more important because systems became more complex. Mainframe developers resorted to database triggers, stored procedures or WebSphere MQ triggers, probably beyond the original intent of such technologies, because there was no other way of sharing logic at the binary level.

I am seeing some similarity with the ETL versus ESB situation in the Java world. These systems do very little to integrate with one another today.

By nature, ETL is similar to batch in the sense that it deals with large amounts of records and multiple file systems and databases. ESB is closer to online as it deals with smaller transactional events.

Both ETL and ESB products claim to be transformation engines though and indeed the term "Transformation" is widely used in both type of products documentation. So you might wonder why it is almost impossible to reuse a transformation written for an ETL in an ESB.

If IBM was forced to introduce EXCI for Batch to CICS communication, I wouldn't be surprised if users forced ETL and ESB vendors to integrate with one another more closely.

I don't mean that ETL and ESB technologies need to be tightly integrated, after all they do different jobs, but yet it would be nice if some level of transformation reusability can be achieved.

Mapping COBOL level 88 to Java

Once in a while comes the issue of mapping COBOL data items with associated level 88 condition values to Java. Since this is a complex question, I thought it deserves an entry of its own.

This is an example of simple COBOL level 88 data item descriptions.

     01  MY-ITEM PIC S9(4) COMP.
         88 WRONG-DATA VALUE -1.
         88 NULL-DATA VALUE 0.

First notice that the 88 levels are associated with an actual data item (MY-ITEM). Level 88 items do not occupy any space in memory themselves.

The way they are used is that somewhere in the code, you can state:

         IF WRONG-DATA THEN ...

The condition will be true if, and only if, MY-ITEM contains -1.

Such code is more readable than the equivalent:

         IF MY-ITEM = -1 THEN ...

So this is what it's all about, an improvement to the readability of your COBOL code.

So how can such a thing map to Java?

First of all, it is easy to confuse COBOL Level 88 with java enums . After all, enums are also a readibility improvement feature and they can be used to replace a value with a meaningful label.

It is tempting to map MY-ITEM with:

    enum MyItem {
        WRONG_DATA(-1), NULL_DATA(0);
        private MyItem(int value) {
            _value = value;
        }

        private int _value;

        final int getValue() {
            return _value;
        }
    };

But there is a fundamental issue in doing so: Enums are actually restrictions. This means only the values that are enumerated (-1 and 0 in our case) are possible.

In the case of COBOL level 88, there are no restrictions on actual values. This means MY-ITEM can contain value 2, which is not described by any level 88.

Furthermore, level 88 are not necessarily atomic values. They can be ranges:

       01  ANOTHER-ITEM PIC S9(4) COMP.
           88 CUST-DATA VALUE 9 THRU 975.

Or even list of values:

       01  YET-ANOTHER-ITEM PIC S9(4) COMP.
           88 VALID-DATA VALUE 2, 7, 15, 245.

By now it should be clear why COBOL level 88 does not automatically map to a java variable of some sort with LegStar.

The legstar-cob2xsd module actually offers an option to map level 88 to XML Schema restrictions in the unlikely case where you are sure the COBOL variable never contains any value outside the enumerated values and enumerated values are atomic. But past that, there is no real support for level 88.

LegStar for Pentaho Data Integration (Kettle)

I have spent the last few weeks digging into open source ETL tools.

The 3 most quoted products are:

• Pentaho Data Integration (Kettle)
• Talend
• Clover ETL

It turns out Clover is only partially open source. The GUI is proprietary. So I spent more time on PDI and Talend.

PDI is the oldest product and, perhaps as a consequence, has the largest community. You can get a sense of that by comparing the Ohloh page for PDI to the Ohloh page for Talend.

But if you compare new threads per day on the PDI forum to that of the Talend forum, you can see that Talend is doing good too.

I decided to try out PDI first and developed a proof of concept implementation of LegStar for PDI

You can see the result on Google code as usual.

I have to say that I am very impressed with PDI, a product originally called Kettle and developed by Matt Casters.

PDI comes with a framework for people to develop additional plugins. For those who are interested, there is an excellent blog entry by Slawomir Chodnicki to get started.

I was able to reuse part of the PDI internal test framework to automate testing of my plugin. I have automated unit tests and integration tests

It is also quite easy to deploy new plugins in PDI. It is a matter of packaging the plugin as a jar, and dropping it to a particular location.

As usual, it is extremely helpful that the product is open source. In particular, I could easily debug my plugin in Eclipse, stepping through PDI code as well as my code.

My only regrets with PDI is that there is little Maven support and that the code is often not commented. This being said, that did not prevent me from using Maven for all lifecycle phases of my plugin and was able to find my way into the PDI code which is usually readable enough.

PDI also has support for parallel processing and clustering that I did not explore yet. I am looking forward to playing with these features next.

VSAM to CSV using LegStar

I was wondering how hard it would be to translate a z/OS VSAM file content to a CSV file using LegStar.

So I started with a very trivial case:

CICS comes with a sample VSAM KSDS file called FILEA (see hlq.CICS.SDFHINST(DFHDEFDS)). FILEA is an 80 characters, fixed sized record file.

The records in FILEA are described by a COBOL copybook called DFH0CFIL (in hlq.CICS.SDFHSAMP). The content of DFH0CFIL looks like this:

02 FILEREC. 03 STAT PIC X. 03 NUMB PIC X(6). 03 NAME PIC X(20). 03 ADDRX PIC X(20). 03 PHONE PIC X(8). 03 DATEX PIC X(8). 03 AMOUNT PIC X(8). 03 COMMENT PIC X(9).

My first step was to extract the VSAM file content to a sequential file using a JCL like this one on z/OS:

//REPRO EXEC PGM=IDCAMS //SYSPRINT DD SYSOUT=* //IN DD DISP=SHR,DSN=hlq.CICS.FILEA //OUT DD DISP=(MOD,CATLG,DELETE),DSN=hlq.CICS.FILEA.BKP, // SPACE=(TRK,(45,15)),UNIT=3390, // DCB=(BLKSIZE=0,LRECL=80,RECFM=FB) //SYSIN DD * REPRO IFILE(IN) OFILE(OUT) //*

I then downloaded the sequential file to my workstation using regular FTP in binary mode:

ftp> bin 200 Representation type is Image ftp> get 'hlq.CICS.FILEA.BKP' 200 Port request OK. 125 Sending data set hlq.CICS.FILEA.BKP FIXrecfm 80 250 Transfer completed successfully.

On the java side now, I created a LegStar Transformer from the DFH0CFIL COBOL copybook. This results in a FilerecTransformers class.

The following code snippet, is what I needed to write to get my CSV file:

FileInputStream fs = null; try { String LS = System.getProperty("line.separator"); fs = new FileInputStream(new File("hlq.CICS.FILEA.BKP")); byte[] hostRecord = new byte[FilerecBinding.BYTE_LENGTH]; FilerecTransformers tf = new FilerecTransformers(); FileWriter fw = new FileWriter(new File("filea.csv")); fw.write("STAT;NUMB;NAME;ADDRX;PHONE;DATEX;AMOUNT;COMMENT"); fw.write(LS); int count = fs.read(hostRecord); while(count > 0) { Filerec javaRecord = tf.toJava(hostRecord); fw.write(javaRecord.getStat()); fw.write(";"); fw.write(javaRecord.getNumb()); fw.write(";"); fw.write(javaRecord.getName()); fw.write(";"); fw.write(javaRecord.getAddrx()); fw.write(";"); fw.write(javaRecord.getPhone()); fw.write(";"); fw.write(javaRecord.getDatex()); fw.write(";"); fw.write(javaRecord.getAmount()); fw.write(";"); fw.write(javaRecord.getComment()); fw.write(LS); count = fs.read(hostRecord); } fw.close(); } catch (IOException e) { e.printStackTrace(); } catch (HostTransformException e) { e.printStackTrace(); } finally { if (fs != null) { try { fs.close(); } catch (IOException e) { e.printStackTrace(); } } }

Of course this is a very contrived example, both because the VSAM file is fixed size and the record data is only made of characters.

In a more realistic case, the records will contain all sorts of numerics: compressed, zoned or edited and chances are that some redefines and arrays will complexify the setting. This is where LegStar should really become useful.

There are other interesting questions when you get to that point:

• Assuming you need to regularly extract the content of a file, how do you automate a distributed process like this one?
• How would you do that reliably; meaning you don't process the same data twice or miss part of it?

ETL (Extract Transfer Load) tools are typically focused on these issues. I became quite curious about them recently.