LegStar

LegStar for PDI, a primer

2011-11-27T04:51:00.001-08:00

Introduction

Mainframes, such as IBM z/OS, are still operating in many large corporations and will probably continue to do so for the foreseeable future.

Mainframe file systems contain huge amounts of data that are still waiting to be unlocked and made available to modern applications.

Traditionally, mainframe data is processed by batch programs often written in COBOL. Usually several batch programs are organized as sequential steps in a flow. On z/OS, the flow description language is JCL, a rather complex and proprietary language.

Assuming you would like to exploit mainframe data but would rather not write COBOL or JCL, this article shows an approach harnessing the power of ETL tools such as Pentaho Data Integration. With this type of solution, mainframe data can be made available to a very large and growing set of technologies.

The primary destination of mainframe data is BI systems, data warehouses and so forth. Such systems impose constraints on data models in order to achieve the usability and performance levels that users expect when they run complex queries. Mainframe data on the other hand, being optimized for OLTP activity and storage optimization, is rarely organized in a BI friendly way. Hence the need to transform it.

In this article we will walk you through a rather common use case where mainframe data is optimized to reduce storage and needs to be normalized with the help of ETL technology.

While this type of transformation has been possible in the past, the novelty here is that we can now achieve identical results for a fraction of the cost, using Open Source technologies.

Use case

In our use case the source data is stored in a mainframe sequential file (QSAM in mainframe parlance).

Records in such files are not delimited by a special character, such as carriage return or line feed, as is common on distributed systems. Furthermore, a record content is a mix of characters and non-characters. Characters are usually encoded in EBCDIC, while non-characters represent various forms of numeric data. Numeric data is often encoded in mainframe specific formats such as compressed numerics (COMP-3).

Mainframe file records are often variable in size. This was important to save storage resources at a time when these were very expensive.

Although there are yet more difficulties involved in interpreting mainframe data, it should be clear by now that you can’t do so without some meta data that describes records. On mainframes, such metadata is often a COBOL copybook. A copybook is a fragment of COBOL code that describes a data structure (very similar to a C structure). This is a sample of such a copybook we will be using as a use case:

       01  CUSTOMER-DATA.
           05 CUSTOMER-ID                    PIC 9(6).
           05 PERSONAL-DATA.
              10 CUSTOMER-NAME               PIC X(20).
              10 CUSTOMER-ADDRESS            PIC X(20).
              10 CUSTOMER-PHONE              PIC X(8).
           05 TRANSACTIONS.
              10 TRANSACTION-NBR             PIC 9(9) COMP.
              10 TRANSACTION OCCURS 0 TO 5
                 DEPENDING ON TRANSACTION-NBR.
                 15 TRANSACTION-DATE         PIC X(8).
                 15 TRANSACTION-AMOUNT       PIC S9(13)V99 COMP-3.
                 15 TRANSACTION-COMMENT      PIC X(9).

Things to notice about this record description are:

This is 5 levels deep hierarchy
PIC X(n) denotes text fields containing EBCDIC encoded characters
CUSTOMER-ID, TRANSACTION-NBR and TRANSACTION-AMOUNT are 3 different forms of numerics
The array described with OCCURS and DEPENDING ON is a variable size array whose actual size is given by the TRANSACTION-NBR variable.

Now assuming we take a peek, with an hexadecimal editor, at a file record containing data described by this COBOL copybook, we would see something like this:

    00000000h: F0 F0 F0 F0 F0 F1 C2 C9 D3 D3 40 E2 D4 C9 E3 C8 ;
    00000010h: 40 40 40 40 40 40 40 40 40 40 C3 C1 D4 C2 D9 C9 ;
    00000020h: C4 C7 C5 40 40 40 40 40 40 40 40 40 40 40 F3 F8 ; 
    00000030h: F7 F9 F1 F2 F0 F6 00 00 00 00                   ;

This first record size is only 58 bytes long. This is because the TRANSACTION-NBR field contains a value of zero, hence there are no array items stored.

The second record though, which starts at offset 59, looks like this:

    0000003ah: F0 F0 F0 F0 F0 F2 C6 D9 C5 C4 40 C2 D9 D6 E6 D5 ; 
    0000004ah: 40 40 40 40 40 40 40 40 40 40 C3 C1 D4 C2 D9 C9 ;
    0000005ah: C4 C7 C5 40 40 40 40 40 40 40 40 40 40 40 F3 F8 ; 
    0000006ah: F7 F9 F1 F2 F0 F6 00 00 00 04 F3 F0 61 F1 F0 61 ; 
    0000007ah: F1 F0 00 00 00 00 00 03 68 2C 5C 5C 5C 5C 5C 5C ; 
    0000008ah: 5C 5C 5C F3 F0 61 F1 F0 61 F1 F0 00 00 00 00 00 ; 
    0000009ah: 17 59 3C 5C 5C 5C 5C 5C 5C 5C 5C 5C F3 F0 61 F1 ; 
    000000aah: F0 61 F1 F0 00 00 00 00 00 11 49 2C 5C 5C 5C 5C ; 
    000000bah: 5C 5C 5C 5C 5C F1 F0 61 F0 F4 61 F1 F1 00 00 00 ; 
    000000cah: 00 00 22 96 5C 5C 5C 5C 5C 5C 5C 5C 5C 5C       ;

This one is 158 bytes long because the TRANSACTION-NBR field contains a value of 4. There are 4 items in the variable size array.

As you can see, there is not a single byte of excess storage in that file!

Now let us assume this file was transferred in binary mode to our workstation and that we want to create an excel worksheet out of its content.

Building a simple PDI Transform

To transform the mainframe file into an Excel worksheet, we will be using Pentaho Data Integration (PDI) and the LegStar plugin for PDI. LegStar for PDI provides the COBOL processing capabilities that are needed to transform the mainframe data into PDI rows.

The PDI community edition product is open source and freely available from this download link. This article was written using version 4.1.0.

LegStar for PDI is also an open source product, freely available at this download link. For this article, we used release 0.4.

Once you download legstar-pdi, you need to unzip the archive to the PDI plugins/steps folder. This will add the z/OS File Input plugin to PDI standards plugins.

The PDI GUI designer is called Spoon, it can be started using the spoon.bat or spoon.sh scripts.

On the first Spoon screen we create a new Transformation (using menu option File→New→Transformation).

From the designer palette’s Input folder, we drag and drop the z/OS file input step:

This will be the first step of our transformation process. Let us double click on it to bring up the settings dialog:

On the File tab, we pick up the z/OS binary File which was transferred to our workstation. Since this is a variable length record file, we check the corresponding box on the dialog.

This file does not contain record descriptor words. RDWs are 4 bytes that z/OS adds to each variable record. When these are present, LegStar can more efficiently process the file.

The z/OS character set is the EBCDIC encoding used for text fields. For french EBCDIC for instance, with accented characters and Euro sign, you would pick up IBM01147.

We now select the COBOL tab and copy/paste the COBOL structure that describes our file records:

At this stage, we need to click the Get Fields button which will start the process of translating the COBOL structure into a PDI row.

A PDI row is a list of fields, similar to a database row. The row model is fundamental in ETL tools as it nicely maps to the RDBMS model.

The Fields tab shows the result:

As you can see, several things happened:

The COBOL hierarchy has been flattened (LegStar has a name conflict resolution mechanism)
Data items, according to their COBOL type, have been mapped to Strings, Integers or BigNumbers with the appropriate precision
The array items have been flattened using the familiar _n suffix where n is the item index in the array

We are now done with setting up the z/OS file input step but before we continue building our PDI Transformation, it is a good idea to use one of the great features in PDI, which is the Preview capability. The Preview button should now be enabled, if you click on it and select a number of rows you would like to preview, you should see this result:

Time to go back to the PDI Transformation, add an Excel output step and create a hop between the z/OS file input step and the Excel output step:

You can now run the Transformation, this will require that you save your work. In our case we named our Transformation rcus-simple.ktr. Ktr files are XML documents that completely describe the PDI Transformation.

There is a launch dialog on which we simply chose to click launch and then the result showed up as this:

As you can see, 10000 records were read off the z/OS file and an identical number of rows were written in the Excel worksheet (plus a header row).

It is time to take a look at the Excel worksheet we created:

Everything is in there but you might notice the variable size array results in a lot of oolumns and a lot of empty cells since we need to fill all columns. Indexed column names and sparsely filled cells result in a worksheet that is hard to play with.

This reveals the fundamental issue with mainframe data models, they were not intended for end users to see. So putting such raw data in an excel worksheet is unlikely to be satisfactory.

This is where ETL tools take all their meaning. To illustrate the point we will next enhance our transformation to get rid of the variable size array effect.

Enhancing the PDI Transformation

Our first enhancement it to reduce the number of columns. We will apply a normalization transformation that is best described with an example.

This is a partial view of our current result row:

CustomerId	CustomerName	…	TransactionAmount_0	TransactionAmount_1	TransactionAmount_2	TransactionAmount_3	…
2	FRED BROWN	…	36,82	175,93	114,92	229,65	…

The COBOL array flattening has had the effect of multiplying the number of columns. Here would be a more desirable, normalized, view of that same data:

CustomerId	CustomerName	…	TransactionIdx	TransactionAmount	…
2	FRED BROWN	…	0	36,82	…
2	FRED BROWN	…	1	175,93	…
2	FRED BROWN	…	2	114,92	…
2	FRED BROWN	…	3	229,65	…

What happened here is that Columns were traded for Rows. Instead of 5 TransactionAmount_n columns there is a single one and the new TransactionIdx column identifies each transaction. The result is a normalized table in the sense of the first normal form in RDBMS theory.

Normalizing has an effect on volumes of course but the result is much easier to manipulate with traditional RDBMS semantics.

Let us now modify our PDI Transformation and introduce a Normalizer step (Palette’s Transform category):

Setting up the Normalizer involves specifying the new TransactionIdx column and then mapping the indexed columns to a TransactionIdx value and the single column that will replace each repeatable group:

If we now run our transformation, this is how the result looks like:

This is already much nicer and easier to manipulate. From the original 20 columns, we are now down to 9.

The PDI execution statistics should show that 50000 rows were created in the Excel worksheet out from the 10000 that we read from the z/OS file. This is an negative effect of normalizing that we should now try to alleviate.

You might notice that the Excel worksheet still contains a large number of empty cells corresponding to empty transactions.

Our next step will be to get rid of these empty transactions. For that purpose, we will use the PDI Filter Rows step (under the palette’s Flow category).

The Filter step will be setup to send empty transaction rows to the trash can and forward rows with transactions to the Excel worksheet. The PDI equivalent of a trash can is the Dummy step, also found under the Flow category so we go ahead an add it to the canvas too:

Let us now double click on the Filter step to bring up the setting dialog:

Here we specify that the filter condition is true if the TransactionDate column is not empty. Back to the canvas, we can now create 2 hops, one for the true condition that will lead to the Excel worksheet and one for the false condition which brings to the Dummy step:

We are now ready to execute the PDI Transformation. The metrics should display something like this:

Now, only 25163 rows made it to the Excel worksheet while 24837 were trashed. The resulting Excel worksheet is finally much closer to what an end user might expect:

Conclusion

In this article we have seen how mainframe data, which is by construct obscure and hardly usable by non-programmers, can be transformed into something as easy to manipulate as an Excel worksheet.

Of course, the example given remains simplistic compared to true life COBOL structures and mainframe data organizations but we have seen a small part of the PDI capabilities, some of which are pretty powerful.

Historically the type of features that you have seen in this article were only available from very expensive and proprietary products. The fact that you can now do a lot of the same things, entirely with open source software, will hopefully trigger many more opportunities to exploit the massive untapped mainframe data.

We hope that readers with mainframe background, as well as readers with open systems background, will find this useful and come out with new ideas for Open Source mainframe integration solutions.

A LegStar Commercial License

2011-11-18T04:55:00.001-08:00

Entreprises still intensively using mainframes these days tend to be very large. Although most of these companies are more and more comfortable using open source software, they are not comfortable at all with running unsupported code in production.

That creates a specific problem to LegStar since it is both Open Source an primarily of use for such large companies.

After receiving several requests from customers at LegSem we started looking into building some kind of commercial offering around LegStar.

Of course, we are not the first open source company to bang our head against this thorny issue, infoworld has a honest and funny write-up about this.

So this is what we came up with:

Committed to Open Source:

First, LegStar is and will remain Open Source
The core features will stay under the permissive LGPL as they are today and the advanced features will be GPL
There won't be an entreprise product separate from the Open Source one. This means every single line of code will be available under an Open Source license

For Customers who need it:

Besides the Open Source licenses, there will be a Commmercial License available. Yes, this a dual-licensing scheme
The Commercial License describes the level of support that LegSem, or a business partner, provides
LegSem will also shield Commercial Licensees from backward compatibility issues

This last point requires a little bit of explanation:

As we add new features or fix bugs in LegStar we generally pay attention to backward compatibility but we don't necessarily test it. As a result, new releases are not guaranteed to be backward compatible with your own developments. With the commercial offering, we will do additional, less frequent, releases that we will explicitly check for backward compatibility. These extra releases will only be available to commercial licensees.

LegSem is a small company and will not cover the entire world with that commercial license. What we intend to do is to work with business partners in territories where we are not present. So if your company would be interested in entering this type of deal with us please send a mail at contact@legsem.com.

Finally a LegStar JCA Connector

2011-10-13T08:54:00.000-07:00

This article refers to JCA, the Java Connector Architecture, not the Java Cryptography Architecture.

When we considered J2EE several years ago as a potential target for mainframe integration, I was horrified and decided not to pursue that route. This has proven the right choice and the J2EE reputation was so bad that the name was later changed to Java EE.

The idea that overly complex technologies provide great opportunities to sell services and consulting is not new. Some very large companies made a fortune this way. Some of these same companies were very active during the J2EE specifications...

Of course J2EE completely failed to become the universal web engine it was meant to be, but it made it to most large Entreprise Systems. Or at least part of the technology was adopted by large IT departements and some of the complexity alleviated thanks to frameworks like Spring.

So the reality today is that most companies still using mainframes also use Java EE.

Another interesting development concerns the latest Java EE specifications (5 and 6). I don't know if the J2EE designers were humbled by their past failures but I have to say they did a much better job this time. Today, the technology is almost usable without Spring. For instance, you can inject a JNDI resource with a single annotation which would have taken about 10 lines of Java code previously. If you want to learn more about the latest Java EE, I recommend reading Antonio Goncalves's blog.

So with Java EE specs getting better and almost all mainframe shops using it, I thought it was time for LegStar to start supporting Java EE containers. The result is part of a series of extensions to LegStar that I am working on.

The first deliverable is a JCA Resource Adapter that uses CICS Sockets as its underlying connectivity.

It is conformant to the JCA 1.0 specifications but does not implement some of the CCI chapter (which is optional anyway). In practice we do not support CCI Records which are awkward indexed or mapped objects. In my view, Records are not suitable to represent COBOL structures. With the LegStar JCA connector, Instead of Records, you simply use regular LegStar Transformers.

I am convinced that separating Resource Adapter fonctionalities (Connection pooling, Transactions and Security) from Transformation is a much better approach than what most JCA connectors available on the market do. For one thing, you can reuse the Transformers outside Java EE which is good because, even if Java EE is largely used by mainframe shops, they also use many other java based technologies (ESBs, ETLs, ...).

I intend to support the other LegStar transports (HTTP, WebSphere MQ) in the future as well as some other transports I have in mind. There will be a different JCA Resource Adapter for each transport.

One thing you might notice is that the licence has changed for these extensions. While the core LegStar project remains LGPL for the time being, the extensions use the more restrictive GPL. LGPL is often used when OSS projects start and need to get the wider acceptance possible, while GPL is for more mature OSS projects. LegStar is 7 years old now and if you look at what Google has to say about open source mainframe integration:

You will agree with me that it has gained sufficient traction to become part of the higher class GPL projects out there.

This is all thanks to you of course since I haven't devised a scheme to get Google to better rank my sites and never paid Google a dime either :-).

Mapping Dates from COBOL to Java

2011-07-23T06:08:00.000-07:00

Date (and Time) types are difficult to map between COBOL and java. Here a few comments on why this is hard.

Dates in COBOL

For a very long time, the COBOL compiler on IBM mainframes didn't know anything about dates. Developers typically used structures and managed dates semantics by hand. The Y2K issue was largely a consequence of this situation since the compiler couldn't solve the problem all by itself.

Recently ("recently" on the mainframe time scale means in the last 15 years or so), IBM introduced the DATE FORMAT keyword. This is used to further qualify a regular numeric or alphabetic data item. It introduces some restrictions on what you can store in the data item but fundamentally, the data item remains a numeric (binary, compressed or zoned decimal) or an alphanumeric.

The DATE FORMAT keyword must be followed by a pattern using Y and X characters (no relationship to chromosomes) such as YYXXXX or YYYYXXXX.

YY designates a "windowed" date. This is relative to a century window given by the YEARWINDOW compiler option.
YYYY is an "expanded" date. This is a regular century + year date.

MOVE of a windowed date to an expanded date will expand it as expected. There are also intrinsic functions such as DATEVAL and UNDATE to convert dates to non date data items.

For more details, see the programming reference in IBM COBOL documentation.

Dates in Java

If dates are messy in COBOL, well, they are even messier in Java.

At the beginning there was an overly simple java.util.Date (with a weird java.sql.Date variation with 9 methods, 6 of which are deprecated!).

Then came java.sql.Timestamp which inherits from java.util.Date but about which the documentation states: "it is recommended that code not view Timestamp values generically as an instance of java.util.Date". It is as if to say: Timestamp inherits from Date, but please don't use that fact in your programs...

But worst, came the dreaded java.util.Calendar. In this article, Joshua Bloch, an ex Sun engineer, says:

As an extreme example of what not to do, consider the case of java.util.Calendar. Very few people understand its state-space -- I certainly don't -- and it's been a constant source of bugs for years.

java.util.Calendar was designed at a time (long gone now) when Java was to overtake the programming world, so this monster was born and several, often used, java.util.Date methods were deprecated as a consequence.

To further complexify things, JAXB has introduced the javax.xml.datatype.XMLGregorianCalendar in order to map the XML Schema date and time.

All in all, at least 5 different ways of representing a Date

Mapping COBOL dates to Java Dates

So far, Legstar has taken the, rather lame, approach of not attempting to map COBOL dates to Java dates automatically.

When starting from COBOL, this means you will never get a java Date/Timestamp/Calendar property, even if the corresponding COBOL data item is marked with a DATE FORMAT. Date semantics are lost in translation...

When starting from Java though, chances are that some property is a Date/Timestamp/Calendar. Actually Date/Calendar, because with Timestamp you would get "error: java.sql.Timestamp does not have a no-arg default constructor" from JAXB.

LegStar maps Java Date/Calendar properties to a COBOL PIC X(32).

At runtime, the PIC X(32) content is assumed to follow the form: "yyyy-mm-dd hh:mm:ss.fffffffff" known as the JDBC timestamp escape format.

This is rudimentary but following this conversation, the latest LegStar has introduced a new annotation, inspired from Patrick's comment: @CobolJavaTypeAdapter(value = SomeClass.class) where SomeClass is code that a developer implements. (Here is a sample).

An extension mechanism is probably the best way to handle COBOL to Java date binding.

Migrating off the CTG?

2011-04-10T10:41:00.000-07:00

CTG, the CICS Transaction Gateway, is an IBM product that has been around for a long time and is largely deployed in mainframe shops.

CTG is usually used from a Java client to make outbound, synchronous, calls to a CICS program. In this case, the API offered to Java clients is known as ECI (External Call Interface ). Because ECI is how most users access the CTG some of them refer to the CTG itself as ECI.

CTG is a separate product from CICS itself. Some people confuse both (probably because the names are so close: CICS Transaction Server and CICS Transaction Gateway).

CTG runs in a Java VM that could sit on z/OS (in the Unix System Services environment) but most frequently runs off a distributed server. It is also frequently used in conjunction with WebSphere and as such, can be thought of as middleware between WebSphere and CICS.

CTG’s major strength is transaction support and J2EE integration. It supports the JCA connector architecture and provides 2-phase commit and XA support.

Although CTG might sound like the definitive solution for CICS/java integration, a number of users are considering moving away from it.

I can see 3 reasons why this is happening:

The emergence of integration standards such as Web Services (and the slow erosion of older standards such as J2EE/JCA)

Loose coupling as a preferred architecture pattern over tight coupling

The Open Source revolution which drives more and more enterprises to consider Open Source software as a valid alternative to closed products

An uncomfortable position within IBM

Today, CTG is competing internally with other IBM integration features particularly when it comes to SOA. You can of course create Web Services on top of JCA but there are more efficient alternatives within IBM.

CICS itself has built-in support for SOAP Web Services. This feature, known as CICS Web Services, offers programmers a standard way to call CICS programs without the need for CTG.

WebSphere, when it is installed on z/OS, also now offers CICS integration directly (an option known as OLA for Optimized Local Adapters).

WebSphere MQ can also be used to support Web Services.

CTG seems to have less and less room within the IBM offering.

Not a serious contender for loose coupling

CTG has support for asynchronous calls but it is hard to justify the heavyweight 2-phase commit support in such a setting.

Asynchronous calls can’t support 2-phase commit and therefore could run with a much more lightweight infrastructure.

Furthermore, IBM has WebSphere MQ as its preferred asynchronous programming interface so it is hard to see how CTG could be justified to support a loosely coupled architecture.

Not Open Source

CTG is not Open Source and is a complex black box that users struggle to operate.

It is hard to tell how much this costs companies but I have met with very frustrated customers.

CTG is not free of charge but it is often bundled with WebSphere, CICS and other products and it is difficult to isolate its cost from the rest.

The true cost becomes apparent though for customers migrating off WebSphere because the bundle offer does not apply anymore.

Poor and confusing tooling

So far we have discussed the runtime aspects of CTG but how about the development time tooling? How do you map the target COBOL data structures to java beans?

Here there is some confusion. This mainly stems from IBM changing names and branding several times over the last few years.

The COBOL mapping tools are usually an option buried deep into a much larger developer product:

Enterprise Application Builder (EAB) within VisualAge for Java (No longer supported)

J2C within RAD (Rational Application Developer for WebSphere Software which used to be named WSAD for WebSphere Application Developer)

J2C(?) within RDz (Rational Developer for z/OS which seems to have been recently renamed Rational Developer for zEnterprise)

I am not sure about J2C being part of RDz. RDz seems to cover CICS Web Services with tooling that looks quite different from J2C. I know CICS had a “Web Service Assistant” a sort of command line utility to map COBOL to XML. RDz might have some GUI on top of that.

Although these tools are Eclipse based, they are not open source. There is usually a per seat license to be purchased.

There are several issues with J2C:

Users complain that the java beans mapping COBOL structures that J2C produces are awkward, the morphology does not always match the original COBOL structure (flattening), property names derived from COBOL data items are unfriendly (use of double underscores), arrays are turned to java arrays instead of more flexible java.util.List, no support for REDEFINES, …

J2C also produces java classes that intermingle java to COBOL transformation with remote execution of CICS programs. This tight coupling of language translation features with RPC mechanism was customary 10 years ago but has proven quite limited since then. This is because integration today is a lot more than calling CICS programs. Mainframe data can originate from files, messages, non IBM transaction monitors, etc… Also data might have to be processed asynchronously, as part of a flow (BPEL) or even in batch which was completely missed by the JCA specs.

If LegStar was architected on the same principles, it would be impossible to support ESB’s or ETL’s for instance.

LegStar might help

To summarize the CTG situation today, it has relatively limited tooling, is tightly related to JCA and has a weak position within the IBM offering.

For users considering Open Source and loosely coupled alternatives to CTG, LegStar could come handy.

At LegSem, with help from some of our pioneer users, we are building a service offering to help customers move away from RAD/CTG.

Our first step is to develop migration tools that can exploit meta data produced by tools such as RAD.

We are also working on an ECI transport for users who want to adopt LegStar COBOL Transformers (to replace the J2C beans) but would rather stick to CTG to get to their mainframe.

If you are interested, drop us a message at admin (at) legsem (dot) com or use this form.

When it comes to Mainframes, nothing is simple

2011-03-26T11:02:00.000-07:00

The IEEE Software review has published an article by Belgian researchers who made an attempt at re-engineering a mainframe application using automated tooling they knew worked in other environments.

Although the article mentions important lessons learned, it proved once more that it is extremely difficult to reconstruct mainframe applications knowledge automatically from static code analysis.

They typically ran into an issue, I for one, encountered several times, where an apparently autonomous set of COBOL programs actually calls some Assembler magic that turns control to other hidden programs only identified at runtime. There is little you can't do on a mainframe using Assembler. One thing you can do, is load executables dynamically.

During the 80's, the heydays of Mainframe programming, every IT shop had one or more Assembler guru's. With much looser budget controls than today, it was possible to spend considerable time developing in-house frameworks, optimizing performances and so forth. Not to say that this only had negative effects, these optimized assembler routines probably saved large amounts of money by reducing CPU consumption, a major parameter on IBM bills.

Most code analysis tools are COBOL centric. One reason for that is that COBOL is not that hard to parse. To my knowledge, there are no automatic code analysis tools for MVS Assembler (DataTek has an impressive tool for Assembler to COBOL but it usually requires some level of human assistance). That's probably because the level of complexity such a tool would require would be several levels of magnitude higher than COBOL.

For a typical Mainframe shop, the volume of MVS Assembler programs is much smaller than COBOL. That might explain why vendors would have a hard time cost justifying writing a very complex Assembler code analysis tool.

So the result of such a situation is that COBOL code analysis tools fail on multilanguage boundaries. Of course there are other multilanguage boundaries on Mainframes for instance: COBOL to BMS or MFS macros, COBOL to JCL, COBOL to SQL, COBOL to CICS, COBOL to IMS not to mention all third party tools one can find on Mainframes. One thing for sure, I have never seen a pure COBOL application.

The problem is that without reliable code analysis tools, you can't reconstruct knowledge in a bottom up approach. What you end up doing is taking the top down approach, by chasing the last application expert on site, hoping that he hasn't retired yet.

The complexity revealed by this IEEE article explains a good part of the Mainframe applications longevity. I know IBM prefers the reliability, availability, security explanation. But I know many IT shops who would have happily migrated off their mainframes if it was easy.

LegStar is also affected by this complexity of course. I often have a hard time explaining to Java developers why the Java side of LegStar works so easily while the Mainframe side, which has much less code in it, is often much harder to get to work properly...

COBOL in a flat world

2011-02-21T12:44:00.000-08:00

We just released a version of LegStar for Talend Open Studio. Talend is a well known ETL tool that also expands to the MDM and ESB spaces.

This is the second ETL tool we interface with. The first one was Pentaho Data Integration a.k.a Kettle for which we released legstar-pdi back in November 2010.

With this experiences behind us it becomes clear that ETL tools are row centric. This means that data flowing from one step to another needs to be modeled as a flat, fixed, list of fields. This exactly maps to a classical database row. Not surprising as ultimately, an ETL, must feed database tables somewhere.

When the starting point is COBOL though, where data structures tend to be very hierarchical, fitting in a flat model is challenging. Here are some considerations to keep in mind.

Name conflicts:

Flattening a simple hierarchy such as:

is relatively easy, it intuitively maps to: [ItemB:String, ItemD:Short].

Now what happens for this one (perfectly valid in COBOL):

In a flat model, field names must be unique so [ItemB:String, ItemB:Short] does not work. You need to disambiguate names and produce something like [ItemB:String, ItemC_ItemB:Short].

Arrays:

Assuming a COBOL data item such as:

we have an new issue since arrays are usually not handled by database schemas.

Now the solution is to expand each item into a different field. Something like [ItemA_0:String, ItemA_1:String, ItemA_2:String, ItemA_3:String, ItemA_4:String].

This is quite wasteful but no real alternatives here.

In COBOL, the DEPENDING ON clause is often used to limit array sizes and processing time. Here we hit another limitation of the flat models, they are usually fixed in the sense that all fields declared must be present in each row.

Filling unused fields with null values is a common technique used to tell downstream steps that fields have no value.

Redefines:

The COBOL REDEFINES clause, a cousin of the C union, is another interesting challenge. Since the flat model is fixed it can't be dynamically changed depending on the REDEFINES alternatives.

The best solution here is to manage a different set of flat fields for each combination of alternatives. This can be demonstrated with a simple example:

Here the COM-SELECT field value determines if COM-DETAIL1 or COM-DETAIL2 is present in the COBOL data.

This would result in 2 field sets (schemas in ETL parlance):

set1: [ComSelect:Short, ComName:String]
set2: [ComSelect:Short, ComAmount:Decimal]

The number of field sets you need to contemplate depends on the number of alternatives in each REDEFINE group (an ITEM followed by a set of items redefining its location). Furthermore, if a COBOL structure contains multiple such REDEFINE groups, than all combinations are possible. So lets say a COBOL structure has a first group of 3 alternatives and another of 2 alternatives, there are 6 (3 x 2) possible field sets.

Fortunately, the number of REDEFINE groups and the number of alternatives in each groups are usually small.

What this all means is that COBOL structures need somehow to be shoehorned to fit the ETL data model. This is an important difference with ESBs where the data model is usually a much more versatile Java object.

See you at Maven central

2011-02-08T09:51:00.000-08:00

Maven central has long been restricted to a few very large players such as the Apache foundation.

This has changed recently thanks to a new free offering by Sonatype for OSS projects.

LegStar has been using Maven from the very beginning and more and more users rely on the availability of artifacts in the LegStar Maven repository.

This proprietary repository is not very secure and not always available. Now that it is possible to push artifacts to Maven central I have been busy figuring out how to take advantage of this.

The major issue is that Maven central's policy is to host artifacts only if their dependencies are also in central. And of course LegStar has dependencies on oss libraries which are not in central

One of the bad players is the Eclipse foundation. So far, no complete sets of Eclipse bundles are available as Maven artifacts. There are lengthy discussions going on but no results yet.

Besides Eclipse the only other annoying dependency we have in LegStar is Websphere MQ. Of course this one being proprietary, it will never make it to central.

In order to bootstrap the process of moving to Maven central, I have started to split the modules into separate release units. the first, very limited, release is now in Maven central. You can see the result at this location by entering the legstar keyword.

It feels good to be part of the big league

Batch integration with CICS, ETL integration with ESB

2011-01-06T00:29:00.000-08:00

On mainframes, integration between batch and online processes is not simple. In an IBM CICS environment for instance, if files and databases need to be shared between batch and CICS (and they most always do), you have to deal with a number of issues such as:

Batch processes might update numerous records. It is usually inefficient to commit those changes after each record change. Large number of uncommitted changes means large numbers of locks which in turn affect online activity by slowing down response times or even leading to time out errors.
Batch processes often deal with files in addition to databases. The most widely used technique to restart a batch after a failure is to backup these files before the batch is started and restore them in case of failure so that the batch can be restarted. Any online activity that dealt with the same files between the batch start and failure worked on uncommitted data.

To avoid such issues, many mainframe shops initially segregated batch and online activities in different time frames. Typically CICS systems were brought down in the evening while batch processes were running and restarted in the morning.

Because of this strong separation between online and batch activity, very little integration between batch and CICS systems was developed. There was some degree of code reuse with COBOL copybooks but you would never see binary reuse (a batch program calling a CICS program for instance). Actually this was not even possible before IBM introduced the EXCI technology.

The mainframe nightly batch window rapidly came under pressure though. As mainframes grew larger and merged with one another, they started serving populations across many time zones. It is not uncommon that users are all over the world. This forced mainframe shops to rearchitect their batch processes and lead IBM to develop new technologies such as EXCI and VSAM Record Level Sharing.

At the same time, the need for business logic reuse between batch and CICS became more important because systems became more complex. Mainframe developers resorted to database triggers, stored procedures or WebSphere MQ triggers, probably beyond the original intent of such technologies, because there was no other way of sharing logic at the binary level.

I am seeing some similarity with the ETL versus ESB situation in the Java world. These systems do very little to integrate with one another today.

By nature, ETL is similar to batch in the sense that it deals with large amounts of records and multiple file systems and databases. ESB is closer to online as it deals with smaller transactional events.

Both ETL and ESB products claim to be transformation engines though and indeed the term "Transformation" is widely used in both type of products documentation. So you might wonder why it is almost impossible to reuse a transformation written for an ETL in an ESB.

If IBM was forced to introduce EXCI for Batch to CICS communication, I wouldn't be surprised if users forced ETL and ESB vendors to integrate with one another more closely.

I don't mean that ETL and ESB technologies need to be tightly integrated, after all they do different jobs, but yet it would be nice if some level of transformation reusability can be achieved.

Mapping COBOL level 88 to Java

2010-12-13T11:38:00.000-08:00

Once in a while comes the issue of mapping COBOL data items with associated level 88 condition values to Java. Since this is a complex question, I thought it deserves an entry of its own.

This is an example of simple COBOL level 88 data item descriptions.

     01  MY-ITEM PIC S9(4) COMP.
         88 WRONG-DATA VALUE -1.
         88 NULL-DATA VALUE 0.

First notice that the 88 levels are associated with an actual data item (MY-ITEM). Level 88 items do not occupy any space in memory themselves.

The way they are used is that somewhere in the code, you can state:

         IF WRONG-DATA THEN ...

The condition will be true if, and only if, MY-ITEM contains -1.

Such code is more readable than the equivalent:

         IF MY-ITEM = -1 THEN ...

So this is what it's all about, an improvement to the readability of your COBOL code.

So how can such a thing map to Java?

First of all, it is easy to confuse COBOL Level 88 with java enums . After all, enums are also a readibility improvement feature and they can be used to replace a value with a meaningful label.

It is tempting to map MY-ITEM with:

    enum MyItem {
        WRONG_DATA(-1), NULL_DATA(0);
        private MyItem(int value) {
            _value = value;
        }

        private int _value;

        final int getValue() {
            return _value;
        }
    };

But there is a fundamental issue in doing so: Enums are actually restrictions. This means only the values that are enumerated (-1 and 0 in our case) are possible.

In the case of COBOL level 88, there are no restrictions on actual values. This means MY-ITEM can contain value 2, which is not described by any level 88.

Furthermore, level 88 are not necessarily atomic values. They can be ranges:

       01  ANOTHER-ITEM PIC S9(4) COMP.
           88 CUST-DATA VALUE 9 THRU 975.

Or even list of values:

       01  YET-ANOTHER-ITEM PIC S9(4) COMP.
           88 VALID-DATA VALUE 2, 7, 15, 245.

By now it should be clear why COBOL level 88 does not automatically map to a java variable of some sort with LegStar.

The legstar-cob2xsd module actually offers an option to map level 88 to XML Schema restrictions in the unlikely case where you are sure the COBOL variable never contains any value outside the enumerated values and enumerated values are atomic. But past that, there is no real support for level 88.

LegStar for Pentaho Data Integration (Kettle)

2010-10-25T08:06:00.000-07:00

I have spent the last few weeks digging into open source ETL tools.

The 3 most quoted products are:

It turns out Clover is only partially open source. The GUI is proprietary. So I spent more time on PDI and Talend.

PDI is the oldest product and, perhaps as a consequence, has the largest community. You can get a sense of that by comparing the Ohloh page for PDI to the Ohloh page for Talend.

But if you compare new threads per day on the PDI forum to that of the Talend forum, you can see that Talend is doing good too.

I decided to try out PDI first and developed a proof of concept implementation of LegStar for PDI

You can see the result on Google code as usual.

I have to say that I am very impressed with PDI, a product originally called Kettle and developed by Matt Casters.

PDI comes with a framework for people to develop additional plugins. For those who are interested, there is an excellent blog entry by Slawomir Chodnicki to get started.

I was able to reuse part of the PDI internal test framework to automate testing of my plugin. I have automated unit tests and integration tests

It is also quite easy to deploy new plugins in PDI. It is a matter of packaging the plugin as a jar, and dropping it to a particular location.

As usual, it is extremely helpful that the product is open source. In particular, I could easily debug my plugin in Eclipse, stepping through PDI code as well as my code.

My only regrets with PDI is that there is little Maven support and that the code is often not commented. This being said, that did not prevent me from using Maven for all lifecycle phases of my plugin and was able to find my way into the PDI code which is usually readable enough.

PDI also has support for parallel processing and clustering that I did not explore yet. I am looking forward to playing with these features next.

VSAM to CSV using LegStar

2010-10-12T08:34:00.000-07:00

I was wondering how hard it would be to translate a z/OS VSAM file content to a CSV file using LegStar.

So I started with a very trivial case:

CICS comes with a sample VSAM KSDS file called FILEA (see hlq.CICS.SDFHINST(DFHDEFDS)). FILEA is an 80 characters, fixed sized record file.

The records in FILEA are described by a COBOL copybook called DFH0CFIL (in hlq.CICS.SDFHSAMP). The content of DFH0CFIL looks like this:

My first step was to extract the VSAM file content to a sequential file using a JCL like this one on z/OS:

I then downloaded the sequential file to my workstation using regular FTP in binary mode:

On the java side now, I created a LegStar Transformer from the DFH0CFIL COBOL copybook. This results in a FilerecTransformers class.

The following code snippet, is what I needed to write to get my CSV file:

FileInputStream fs = null;
      try {
        String LS = System.getProperty("line.separator");

fs = new FileInputStream(new File("hlq.CICS.FILEA.BKP"));
        byte[] hostRecord = new byte[FilerecBinding.BYTE_LENGTH];
        FilerecTransformers tf = new FilerecTransformers();

FileWriter fw = new FileWriter(new File("filea.csv"));
        fw.write("STAT;NUMB;NAME;ADDRX;PHONE;DATEX;AMOUNT;COMMENT");
        fw.write(LS);

int count = fs.read(hostRecord);
        while(count > 0) {
          Filerec javaRecord = tf.toJava(hostRecord);
          fw.write(javaRecord.getStat());
          fw.write(";");
          fw.write(javaRecord.getNumb());
          fw.write(";");
          fw.write(javaRecord.getName());
          fw.write(";");
          fw.write(javaRecord.getAddrx());
          fw.write(";");
          fw.write(javaRecord.getPhone());
          fw.write(";");
          fw.write(javaRecord.getDatex());
          fw.write(";");
          fw.write(javaRecord.getAmount());
          fw.write(";");
          fw.write(javaRecord.getComment());
          fw.write(LS);
          count = fs.read(hostRecord);
        }
        fw.close();

} catch (IOException e) {
        e.printStackTrace();
      } catch (HostTransformException e) {
        e.printStackTrace();
      } finally {
        if (fs != null) {
          try {
            fs.close();
          } catch (IOException e) {
            e.printStackTrace();
          }
        }
      }

Of course this is a very contrived example, both because the VSAM file is fixed size and the record data is only made of characters.

In a more realistic case, the records will contain all sorts of numerics: compressed, zoned or edited and chances are that some redefines and arrays will complexify the setting. This is where LegStar should really become useful.

There are other interesting questions when you get to that point:

Assuming you need to regularly extract the content of a file, how do you automate a distributed process like this one?
How would you do that reliably; meaning you don't process the same data twice or miss part of it?

ETL (Extract Transfer Load) tools are typically focused on these issues. I became quite curious about them recently.

Mule 3.0 is here

2010-08-22T00:43:00.000-07:00

Mule ESB is undergoing its next mutation with the upcoming version 3.0. Mule ESB has always been one of the easiest ESB's to setup and work with. The intelligent combination of Spring and other open source technologies gives outstanding results.

An example of a unique feature, is the JUnit-based framework that comes with Mule. It is enormously convenient to be able to create a complete test case and run the entire Mule server from a single JUnit class! When I work with other ESB's I really miss this one.

Another example is the Spring-based configuration system. It is difficult to grasp all the power that you get from this. By contrast, ESB's that use proprietary DSL's for their configurations are limited to the features they build. With Spring, you can pretty much inject anything into you Mule configuration and reference that from anywhere else in the configuration.

A third example is the leveraging of Maven in various areas. Build management of course but also providing partners with Maven archetypes to help them bootstrap their transports and modules developments. That's so much better than documentation and samples.

These features alone could explain why Mule has become so popular.

Inevitably though, as larger companies and organizations started using it, demand for entreprise-class features has increased. Message flows that users needed to describe to Mule have become more complex. The limitations of the "endpoint/transformer/service" buiding blocs became apparent (Behaviors such as "Remote Sync" are still pretty obscure to me).

Mule 3.0 is introducing a new "flow" concept that should allow description of complete pipelines. I haven't played much with that yet but really looking forward to.

When you deploy large number of services in a single server instance, the class loading mechanism comes under stress. Typical problems are the need to share libraries and multiple versions of libraries having to coexist.

You don't solve problems like these without some major overhaul. Mule 3.0 is introducing a completely new deployment mechanism with features such as hot deployment.

Because there are major architectural changes, Mule 3.0 is not backward compatible. This might sound like an heresy to Mainframe people but remember how IBM CICS went from macro-level API to command-level APIs.

The reality is that you can't deal with core architectural issues without fundamental refactoring.

The good news is that the migration process is not that painful and is well documented. It took me only a few days of work to migrate the LegStar for Mule transport.

In the mainframe world, version 3 was usually the most complete and stable version. Looking at Mule 3.0 feature set, it could very well be the case for Mule too.

JSON to COBOL

2010-07-23T09:42:00.000-07:00

It turns out it is fairly simple to transform JSON strings to COBOL using LegStar and Jackson.

To do this you would start from a COBOL fragment and use the usual LegStar COBOL to XSD translator and then generate transformers using LegStar COBOL Binding Generator.

Here is a sample JUnit code performing transformation from JSON to COBOL and vice versa:

package json.lsfileae.test;

import java.io.StringWriter;

import org.codehaus.jackson.map.AnnotationIntrospector;
import org.codehaus.jackson.map.ObjectMapper;
import org.codehaus.jackson.xc.JaxbAnnotationIntrospector;

import com.legstar.coxb.host.HostData;
import com.legstar.test.coxb.lsfileae.Dfhcommarea;
import com.legstar.test.coxb.lsfileae.bind.DfhcommareaTransformers;

import junit.framework.TestCase;

/**
* Unit test JSON/COBOL transformation.
*/
public class JacksonTest
extends TestCase {

 /** Hex EBCDIC data orignating from mainframe. */
 public static final String HOST_DATA =
     /*0 0 0 1 0 0 */
     "f0f0f0f1f0f0"
     /*  S .  D .   B O R M A N                  */
     + "e24b40c44b40c2d6d9d4c1d54040404040404040"
     /*  L A B A S   S T R E E T                 */
     + "e2e4d9d9c5e86b40c5d5c7d3c1d5c44040404040"
     /*  3 2 1 5 6 7 7 8 */
     + "f3f2f1f5f6f7f7f8"
     /*  2 6   1 1   8 1  */
     + "f2f640f1f140f8f1"
     /*  $ 0 1 0 0 . 1 1  */
     + "5bf0f1f0f04bf1f1"
     /*  * * * * * * * * *  */
     + "5c5c5c5c5c5c5c5c5c"
 ;

 /** JSON serialization of the host data. */
 public static final String JSON_DATA =
   "{\"comNumber\":100," +
   "\"comPersonal\":{\"comName\":\"S. D. BORMAN\"," +
   "\"comAddress\":\"SURREY, ENGLAND\"," +
   "\"comPhone\":\"32156778\"}," +
   "\"comDate\":\"26 11 81\"," +
   "\"comAmount\":\"$0100.11\"," +
   "\"comComment\":\"*********\"}";

 /** Jackson mapper. */
 private ObjectMapper _mapper;

 /** LegStar transformer. */
 private DfhcommareaTransformers _transformer;

 public void setUp() {
   _mapper = new ObjectMapper();
   AnnotationIntrospector introspector = new JaxbAnnotationIntrospector();
   _mapper.getDeserializationConfig().setAnnotationIntrospector(introspector);
   _mapper.getSerializationConfig().setAnnotationIntrospector(introspector);
   _transformer = new DfhcommareaTransformers();
 }

 /**
  * Transform host data to JSON and check
  * @throws Exception if transformation fails
  */
 public void testHostToJson() throws Exception {
   StringWriter sw = new StringWriter();
   _mapper.writeValue( sw, _transformer.toJava(HostData.toByteArray(HOST_DATA)));
   assertEquals(JSON_DATA,  sw.toString());
 }

 /**
  * Transform JSON to host data and check
  * @throws Exception if transformation fails
  */
 public void testJsonToHost() throws Exception {
   Dfhcommarea dfhcommarea = _mapper.readValue(JSON_DATA, Dfhcommarea.class);
   byte[] hostData = _transformer.toHost(dfhcommarea);
   assertEquals(HOST_DATA, HostData.toHexString(hostData));
 }
}

MuleSoftwebinar with MDI and LegStar

2010-07-13T11:10:00.000-07:00

MuleSoft is putting together a Webinar on wednesday July 14th which features Mule ESB + MDI + LegStar. You can join the webinar here.

It tells a very compelling story where you want to web enable a CICS/COBOL program but would like a very different message structure than the original COBOL program input and output.

In this scenario LegStar does the COBOL parsing and mainframe transport while MDI takes over the transformation. The result is a Web Service with a nice interface deployed in Mule ESB. MDI really adds value here in that it hides the ugly COBOL structure and expose a set of friendly java classes instead.

COBOL structures, which typically describe CICS commareas, are often complex and convoluted due to the limitation of COBOL but also the need to reduce external calls and therefore cramming as much data as possible in a single commarea.

Exposing such complex structures as web services results in WSDL that is quite difficult to manipulate. MDI can be used to visually map these complex structures with simpler java classes and hence expose a nicer web service.

2010-07-09T02:29:00.000-07:00

Just finished a series of videos showcasing how to create a mainframe adapter with LegStar for JBoss ESB.

I have become a fan of video demoes. I find them a lot easier to follow than reading paper documentation.

So I embarked in doing my own video demoes.

There are 2 things that I found amazing while doing this:

How difficult it is to get something as obvious as a product demo video done
How much data you can store on YouTube for free!

I had 3 choices when I started:

filming myself doing the demo
doing screen captures of the demo and recording my voice
doing screen captures of the demo and adding subtitles

I ended up doing option 2. I ruled out option 1 for aesthetic reasons and 3 because I couldn't figure out how to do it .

I have been told that it would have been much easier on a Mac. Unfortunately, I am still a Windows user (Although I have one machine running Ubuntu, another running Fedora core and several other unixes running in VirtualBox).

I started by using Microsoft MovieMaker which is quite easy to learn, but the result was disappointing. I couldn't get the quality to be acceptable once the video was uploaded.

One of the issues with such product demoes is that you want the GUI to be readable. This is impossible with small frame sizes. Typically MovieMaker would not allow me to pick up 1024x768, which is what I ended up needing for the Eclipse GUI.

So I turned to these free products:

CamStudio for video recording
Audacity for audio recording
Virtualdub for video editing

I also had to bring in the Xvid MPEG-4 Codec for compression. It turns out this is a codec that YouTube supports very well so there is very little quality loss when you upload to YouTube.

The result is not too bad although it still feels quite amateurish.

I have spent several days doing this, but feel I would need a lot lot more to make it look professional.

Hopefully this will be enough to help people learn the product faster.

From TUI to WUI, not that simple

2010-04-24T01:15:00.000-07:00

Mainframes 3270 interfaces are usually text based. Even though, graphics are possible on mainframes, they are rarely used. Users of such systems are familiar with the black background and green/blue text but more importantly, they are also familiar with the function key paradigm.

With 3270, nothing is transmitted to the host until a function key is pressed. Very little validation occur when the user fills in the input forms. This is not unlike HTML forms which are only posted when a command button is clicked.

For that reason, most solutions available today for scraping 3270 interfaces are based on HTML/JSP technologies.

One of the drawbacks though is the "flickering effect" that HTML/JSP suffers from when forms are posted. The entire web page is refreshed every time you interact with the host.

The 3270 protocol was smarter and able to refresh only those parts of the display that had actually changed. Many users living the transition from TUI to WUI complain about this. 3270 could optimize screen refresh because terminals were dumb. Terminals new very little about the semantics of the data they received (hence the limited local validation they could perform). Data was fundamentally just a stream of characters and associated positions/attributes on screen.

WUI have evolved past the "function key" and flickering effect by adopting the Ajax technologies so you might think that Ajax-based screen scraping solutions would be a perfect solution.

In reality though, Ajax is useless if you don't transfer logic from the host to the browser. The real Ajax added value is in allowing web applications to behave more like "rich clients".

What this all means is that if you have a mainframe 3270-based application that you are not willing to modify, there is little chance you will be able to build a rich, web-based, user interface to it.

Itemscript a serious JSON library for java

2010-03-07T07:41:00.000-08:00

JSON is an interesting alternative to XML. It has originally grown from javaScript but is now widely available for a large number of languages and many libraries are built with JSON support.

Of particular interest is the constant focus of the JSON community on simplicity. The moto seems to be: anything you can do with XML you can do with JSON, faster and cheaper.

itemscript is an example of such a library.

Itemscript is a rich JSON API for java. This is a sample taken from the product test cases that illustrates some of the nice aspects of the API:

JsonArray array = system().createArray()

.a("foo")

.a(1)

.a(1.5)

.a(true)

.a(123L);

assertEquals("foo", array.getString(0));

assertEquals((Integer) 1, array.getInt(1));

assertEquals((Double) 1.5, array.getDouble(2));

assertEquals((Float) 1.5f, array.getFloat(2));

assertEquals((Boolean) true, array.getBoolean(3));

assertEquals((Long) 123L, array.getLong(4));

Notice the elegant chaining of add operations and the automatic type inferencing mechanism.

Itemscipt also has a binding mechanism called the Foundry that ties POJOs to JSON. The best way to illustrate this is to look at some code.

On top of the JSON API, itemscript provides a way to store and query JSON objects in a RESTful manner creating an in-memory database of linked objects. This is an example inspired by the itemscript samples:

/* create a hierarchy */

system.put("mem:/abc/def", "123");

system.put("mem:/abc/ghi", "456");

/* query the in-memory data */

assertEquals(2, (int) system.getInt("mem:/abc?countItems"));

JsonArray keys = system.getArray("mem:/abc?keys");

assertEquals("[\"ghi\",\"def\"]", keys.toCompactJsonString());

JsonArray pagedItems = system.getArray("mem:/abc?pagedItems&numRows=2&startRow=0");

assertEquals("[[\"def\",\"123\"],[\"ghi\",\"456\"]]", pagedItems.toCompactJsonString());

Last but not least, Itemscript is open source and is hosted on Google code.

I am thinking of ways to leverage libraries such as itemscript to provide COBOL to JSON binding. As JSON becomes more and more popular there are good chances its trajectory will cross the legacy applications one. Would be nice to have LegStar at the junction.

A Maven plugin to upload sources on z/OS

2009-12-18T02:05:00.000-08:00

LegStar is a mix of Java code and z/OS native code written in C370 and COBOL.

The LegStar build system makes extensive use of Maven, a widely used build automation tool. Maven drives the LegStar release process by pulling source code from SCM, compiling, testing and generating documentation.

All the legStar z/OS native source code is managed locally using Subversion. What was missing so far was the capability to upload and compile these sources on z/OS as part of the Maven build lifecycle.

After several attempts at using the ant FTP task, I decided to write my own Maven plugin. Following the tradition of open sourcing everything in LegStar, you can find the zosupload project on google hosting.

It is quite rudimentary at this stage but it has the minimal capability to upload sources in various PDS libraries, submitting JCL for execution and checking condition codes returned as part of a Maven build.

I expect others will have the same mix of local Java and native z/OS code they would like to keep in sync so I am hoping they will contribute to that project.

COBOL-aware middleware

2009-12-07T10:14:00.000-08:00

Two phenomenons with opposing effects have affected COBOL structures over time.

The first one is a tendency to construct complex data structures with many fields. This would allow to store a maximum amount of information in a VSAM record or database segment for instance. This was an important factor in reducing physical I/O and getting the best performances possible. When data structures were passed from program to program, they also tended to be large structures because developers would typically try to reduce the number of programs and hence, would rather write big programs that handle many cases than many smaller programs.

When the mainframe opened up to program to program communication over networks though, these large structures became a problem because networks were slow. So the next tendency was to try to find ways to reduce the size of the data sent over the network even if the structures stayed complex and large.

The most widely used mechanism to reduce the size of the data is the variable size array (an array with the DEPENDING ON clause). This is a awkward type of array for people used to java and C. It is an array which actual size is given by a variable, appearing somewhere in the structure (before the array itself).

The second mechanism is not related to COBOL per-se but more with the middleware used. Because structures contain many fields, it often happens that these fields contain default values. For instance, alphanumeric fields might contain binary zeros, which is usually interpreted as "no content". If the middleware is smart enough, it can avoid sending these non initialized fields over the network thus saving a lot of bandwidth.

Variable size arrays also need to be handled correctly by middleware. For instance sending the maximum number of items would be wasteful. A more tricky situation arises when the middelware invokes a mainframe program that returns a variable size array. Often, the memory needed by the target program to operate must be allocated beforehand by the middleware. But how can the middleware anticipate the number of rows that the program will return?

As you can see there are good reasons why middleware used to communicate with mainframe programs need to be COBOL-aware.

First experience with cloud computing (sort of)

2009-11-22T07:32:00.000-08:00

Like many of you I am sure, I have been pounded with loads of articles on cloud computing.

I start to consider that there is something in a technology beyond pure marketing hype when IEEE Internet Computing dedicates a complete issue to the topic, which they did in October 2009.

So I decided to take a look and started using Google App Engine because it is free below certain quotas and also because I had some early experience of the Google Web Toolkit which is well supported by GAE.

GAE allows you to run java applications, actually classical J2EE Servlets, with a number of restrictions (for instance you cannot write to a file or open a port).

Since we recently restarted the COBOL Structure to XML Schema project, it seemed like a good idea to deploy it as a service available on the cloud. This way, developers who would like to get a sense of what the product does without investing time downloading and installing can do so.

The result is now available and I almost immediately started to get hits... and problems.

The first problem of course is that the translator is not a COBOL validating parser. I mean it is not a complete syntax checker. It is meant to process COBOL fragments that are supposed to compile OK in the first place. And yet, it is tempting to type COBOL statements in the input textarea of an HTML page, starting at column 1 instead of column 8. Today you often end up with an empty XML schema because the parser dropped everything that it did not recognize.

I guess we'll have to add more syntax checks after all.

The second problem has to do with GAE itself and Java. It has been discussed extensively on the Google App Engine group. When requests are received by GAE it picks up an instance of a VM somewhere on the cloud to service it. Chances are that this VM was last used for something totally different from running your own application. For this reason, Google actually cold starts the VM, which results in a large consumption of CPU... that counts against your quota!

Humm. That first experience has changed my view on cloud computing!

COBOL is weakly typed

2009-11-18T09:09:00.000-08:00

Yes, whatever proponents think, COBOL is far from an ideal programming language.

Try to push an integer into a Java String and you will be stopped at compilation time. Now try something like "MOVE 15 TO A" where A has been defined as PIC X(2) and not only the compiler will let you through but even at runtime, that will actually work. You end up with the EBCDIC representation of characters '1' and '5' as the content of A.

Actually, you can move pretty much anything into a COBOL PIC X. You commonly do things like "MOVE LOW-VALUES TO A" where you end up with A filled with binary zeros. This is particularly useful in CICS/BMS applications where a field filled with low values will result in no data sent to a screen while a field filled with space characters will increase the volume of data sent (and therefore use up the, once precious, network bandwidth).

This is always a source of problems with integration solutions because the most natural mapping to a COBOL PIC X is a Java String. So you actually map a COBOL type that is not limited to characters to a Java type that only accepts characters.

It is important that an alternative mapping be available. Typically it is sometimes necessary to map a PIC X to a Java byte[] because the content cannot predictably be converted to characters.

When mapping to a Java String is mandatory (it usually is), it is also important that low level conversion routines remove non character content coming from the COBOL program before the Java String is populated.

Conversely, it is important that non character content gets inserted in the COBOL data when needed. For instance, if a COBOL field needs to be filled with low values (as opposed to space characters) then the integration conversion routines should provide an option to do so.

Keep in mind that the mainframe program might react differently to a field filled with low values as opposed to one filled with characters!

A new COBOL structure to XML schema translator

2009-11-13T03:22:00.000-08:00

I have posted an initial release of a new project called legstar-cob2xsd that basically does COBOL structure to XML Schema translation.

There were several factors that led to this project:

The legstar-schemagen module in LegStar that does the job today was written in C and the parsing logic was hand coded. I think this might have driven away some users. legstar-cob2xsd is pure java and uses ANTLR grammars for COBOL structure parsing.
There is a clear need for an autonomous open source COBOL to XML Schema utility. So it makes sense to isolate that feature in its own project. People who need just that functionality won't have to figure out how to extract it from the other LegStar modules.
legstar-schemagen was systematically adding LegStar JAXB style annotations to the XML Schema produced. While this is still needed if you want to use LegStar for runtime transformations, this is not the default option anymore. This means people can use the resulting XML Schema totally outside LegStar if they want.
The clear separation of the COBOL parsing logic from the XML Schema generation means it is much easier now to create other targets. For instance JSON schemas.
Finally the fact that legstar-cob2xsd is in java allows JUnit tests to be much more comprehensive (and they are!).

What does CICS equates to in the Java world

2009-11-10T07:39:00.000-08:00

It is always tough to explain what CICS exactly is to someone with a Java background.

One misconception I often hear is that it is similar to JBoss TS (Arjuna), the transaction manager in JBoss.

I think the issue is with the "Transaction" moniker. In the Java world, it is generally understood as a "Database transaction", something that has to do with coordinating resource updates.

In CICS, the meaning is different. If you monitor an active CICS system, what you will see are a number of active transactions. CICS transactions are primarily scheduling units. A CICS transaction is associated with an initial executable program which can in turn call other programs. CICS transactions get alloted chunks of memory for their programs to use, get CPU cycles when CICS decides, are authorized or not to a given user, etc. That's why CICS is called a Transaction Monitor: what is does is monitor CICS transactions.

A CICS transaction might very well never perform any update to any resource. It would still be a CICS transaction. A CICS transaction can send a form to a screen or receive data entered from a keyboard. CICS transactions can save conversation contexts, pretty much like servlets do.

Now, if a CICS transaction (or to be more precise some program it is associated with) does perform an update to a resource (say a VSAM file), then the CICS transaction is actually the default boundary from a "database transaction" standpoint. This means that when the transaction ends fine, all updates are committed but if the transaction fails (abends in CICS jargon), then all updates since the start of the CICS transaction are undone.

The CICS API also has COMMIT/ROLLBACK to give programs more control over the database transaction. From this standpoint, CICS is a local transaction manager. Now this is true for resources such as VSAM but more complex systems, such as DB2 or IMS-DB, have their own transaction managers. So who does the distributed transaction coordination that is needed when a CICS transaction updates both a VSAM file and a DB2 table?

Well, that would be RRS (Resource Recovery Services), a separated address space (process) in z/OS. That would be the closest thing I can think of to JBoss TS in a z/OS world.

2009-11-03T08:38:00.000-08:00

When asked what the competition to LegStar is, I generally reply no one, because there are no other open source integration solutions for mainframes (to the best of my knowledge).

There are several commercial competitors though. They all seem to gather in the same places. DataDirect and HostBridge are the only ones missing from this list.

Most of these vendors (maybe Seagull to a lesser extent) are mainframe-centric. What I mean by that is that their products run natively on the mainframe, are priced/licensed accordingly and are sold to mainframe minded project leads. This is a closed circle. No place for open source there. Customers are not asking for it and vendors don't want to hear about it.

But out there, there is a growing population of java minded project leads and developers who are taking over higher and higher responsibilities in large IT departments. Many times, they have to deal with COBOL/PL/I/CICS/IMS applications as part of their projects.

This younger generation is a lot more responsive to open source. We are working for them.

ESBs: translators in an SOA world

2009-10-29T13:05:00.000-07:00

A strange article has popped up on zJournal about SOA. I am not sure I fully understand the content but it seems to be an advice to CIOs basically saying: SOA is difficult to get right and ESBs are a bad thing.

This is the first time I read something clearly hostile to the ESB architecture. The author seems to believe that they are too complex and that they mostly try to imitate old EAI systems, rather than embracing SOA.

The article seems to imply that the only way to pure SOA is rewriting applications is such a way that they are natively service enabled. It states, in several places, that "API"s are a bad thing.

Following this idea, I guess you would have to replace SQL, SMTP, JMS, Excel macros,... by Web Services.

It's a little bit as if you were saying: all these languages that people speak around the world make communication difficult, let's all speak Esperanto.

As long as we are not all speaking Esperanto, there will be a need for Translators. I guess that's why ESB's are so useful to an SOA.

Mainframes are not open source friendly

2009-10-18T10:11:00.000-07:00

IBM is a strange beast. On one hand, they gave away Eclipse and the Apache Software Foundation but on the other, they don't seem to embrace open initiatives when mainframes are involved.

You can read more about what happened to PSI and T3 Technologies on openmainframe.org. These examples are more on the hardware side but there is no sign of IBM opening up any of its z/OS software.

Most of the open source initiatives I have heard of such as http://www.opencobol.org/ or http://h3270.sourceforge.net/index.html are from third parties and do not attempt to run on z/OS itself.

I think that IBM's attitude toward open source on z/OS is harming the mainframe community and is counterproductive in the long term.

Mainframe-centric integration costs

2009-10-11T12:07:00.000-07:00

Gregg Wilhoit, DataDirect's chief architect, is one of the smartest professionals I have met. He continues to drive the Shadow line of products.

Shadow is rooted in the data integration space but has evolved over time to include programmatic integration as well.

From an architecture standpoint, Shadow is a mainframe-centric integration solution. It is probably one of the fastest and most stable products available that runs natively on the mainframe (Other such products are GT Software's Ivory, HostBridge, SOA Software SOLA and IBM's CICS Web Services support).

The major argument against mainframe centric integration solutions is usually one of cost. With the broad adoption of SOA for instance, all mainframe-centric solutions suffer from the cost of XML parsing/formatting. This is a CPU intensive activity that affects the overall mainframe cost of ownership (On mainframes, the more CPU you consume, the higher your software license fees are).

Gregg was one of the first (maybe the first as far as I know) who made it possible to reduce such CPU costs by exploiting IBM's specialty engines on system z. These co-processors can offload java and DB2 related payloads from the main processors therefore avoiding the license fees inflation. Since then, other companies have jumped in.

Although the ZIIP/zAAP offloading is good news for mainframe centric solutions, I doubt this will significantly reduce their total cost. One important aspect of integration is that it needs monitoring for instance. It is a whole lot trickier to tune mainframe centric solutions than distributed ones. It also requires very specific and costly skills and software.

When comparing distributed integration to mainframe-centric integration, it is important to compare development costs, administration costs and ultimately exit costs. I doubt that the total would be in favor of mainframe centricity.

In my opinion, the major argument for mainframe-centric solutions is one of performance and stability. The cost though will probably remain high.

Openworld Forum

2009-10-04T05:29:00.000-07:00

Went to the open world forum last week. This is the first time I attend and found the event quite interesting.

There were a lot more business suits than I expected and most of the topics revolved around the business aspects of open source.

Had a chance to listen to Nick Halsey from Jaspersoft, Josep Mitja from openbravo and Yves de Montcheuil from Talend. I find it much more interesting to listen to these relatively young, yet quite successful, open source companies, than to the older organizations such as Apache, Eclipse and Linux which seem out of reach.

It seems these smaller OSS companies make their money mostly by selling subscriptions. This model has been very successful for Redhat of course but I am still wondering if it is as efficient for server technologies. I thought services would be an important secondary source of revenue for them but this does not seem to be the case. The explanation could be that these companies heavily rely on SI's to get them into accounts and therefore they can't sell too much services for fear of openly competing with these same SI's. The bright side is that this forces them to focus on software engineering rather than diverting their resources on services.

The most impressive presentation I saw was from Mark Shuttleworth, the Ubuntu project founder. He had 3 slides with a single word on each of them. This is the first time I can remember 100% of a presenter slides content without taking a single note!

ANTLR and Terence Parr

2009-10-02T10:15:00.000-07:00

I am reading Terence Parr 's The definitive ANTLR reference. I have to say that it is not often that I regret not being 25 years younger to attend Terence courses at the University of San Francisco.

Theories behind compilers are still complex but Terence is doing a great job at making things clearer for people like us, with concrete problems on their hands.

Mainframe hardware & OS versus Mainframe Applications

2009-09-22T22:44:00.000-07:00

Another interesting study by Forrester has been published. Although it is a commissioned study by CA, there are some interesting figures in it, namely the 15000 mainframe sites remaining.

One company may own several mainframe sites and a mainframe site may host more than one physical machine. A mainframe site size is usually measured in MIPS, the accumulated power of all the mainframe hardware running on that site.

The total amount of MIPS is growing 20% annually according to Forrester.

BMC published similar figures some time ago.

The problem with figures coming from IBM, CA or BMC is that they all focus on MIPS, i.e. hardware.

But a mainframe today can run z/VM and several hundred z/Linux images. As such, it is essentially a virtualization environment for Linux. These Linux images probably run Java/J2EE applications of some form. No doubt that there is a need for such centralized architectures which explains part of the MIPS growth.

But how about the more traditional COBOL-CICS or PL/I-IMS applications? Do they have a share in the 20% annual growth rate?

It is probably not a good idea to look at the compound MIPS growth rate anymore since it now sustains workloads that are totally unrelated.

I would love to see a study focusing on legacy-MIPS.

First post on the LegStar blog

2009-09-22T05:31:00.000-07:00

I have been working on open source software for the last 3 years or so. One of the most frustrating aspects to me has been the lack of feedback from users.

Most people who download the software never leave a message of any kind. Only a tiny minority do.

LegStar is an open source integration solution for mainframe applications. It deals with COBOL and CICS applications and it does so by leveraging J2EE and ESB servers such as Mule and JBoss ESB.

I have decided to start this blog hoping it will help gather more feedback.