Runtime/Internals

This section discusses the runtime execution of a map, and various behind the scenes internal issues.

Runtime

The runtime support is used to execute maps in a production environment. We provide both a command line and Java API which can be used to execute a map.

A map can be executed for only one file (or whatever files it depends on) or for all of the files in a given directory.

The runtime environment depends only on having access to the map and structure definition files, which can be conveniently moved anywhere for deployment.

Map Execution

When a map is executed, the map metadata is translated into X/Query and compiled by the X/Query engine. Different X/Query engines can be plugged in as required. We currently use the Saxon X/Query engine, probably the most widely used open source engine and it has the best conformance to the latest editions of the standard.

All input for the execution of the map (flat files, database rows, EDI, etc) is translated into SAX events for processing by the X/Query engine. The actual map execution handles all types of data uniformly. Similarly, the output is translated from SAX events to the desired output format.

Because of this architecture it is easy to add additional adaptors for different formats of data.

Metadata

All metadata is stored in XML, some in local files for user projects and others as part of plugin projects (for the standards). The metadata format is fully versioned and extensible so the definition investment in maps and structures can be preserved permanently across all versions of the product.

As the amount of metadata can be quite large (consider the EDI X12 856 transaction with all of its code tables), there are many tricks and optimizations required to not store duplicate data.

These are accomplished both in the XML serialization process by detecting the removing redundant blocks and by the use of the inheritance mechanism to reuse definitions.

Code Generation

Currently X/Query is generated from the map metadata. However the code generation portion of the product is isolated and generating alternative code is a relatively small task. XSLT could be generated to support high-performance appliances that do transformation. Native Java (or code in another language could be generated).

Scripted Editing

A command line tool is provided to do structure and map editing, allowing these to be scripted. As an example, the GS1/XML structure and example documents are provided in a large set of Zip files. The process of importing the structure data and the sample documents, including various mappings to show structures both inside of a message and separately is entirely scripted and based on the command line tool.

Diagnostics and Support

Extensive logging (using Log4J) is provided and can be enabled in the field. An automatic bug-reporting mechanism is provided in the designer to allow the user to submit a bug report if the product catches any sort of unexpected exception. The user can also directly request support with a comment or question through the designer.

Product Editions

The data mapping technology can easily be subsetted and packaged into different products for different purposes and with different branding. The GS1 Viewer is an example of this.

Future Work

  1. Map set editor - An editor for specifying chains of maps needs to be added. Most of the infrastructure to allow maps to be connected is present. This would allow maps to invoke each other through events, some of which may be asynchronous. This will allows maps to be invoked in response to things like a file appearing in a directory.
  2. Runtime management - System management and better logging is required for the runtime to work in production environments.