!doctyp html> TemplaGen — Template-driven artifact generation from repository data


TemplaGen — Template-Driven Artifact Generation from Repository Data



Scenario 1 — you are a Master Data Manager (MDM) in charge of a "single source of truth" enterprise data repositoryMultiple artifacts (code, documentation, etc.) are individually generated from the data in that repository.  So every time there are changes in the data, all of those artifacts must be painfully recreated to reflect those changes — a laborious and error-prone process.

Scenario 2 — you need to do sophisticated data transformations from multiple data sources, including cleanup, formatting, and data selection, perhaps to prepare the data for Big Data or AI / Machine Learning, and/or to process the results of such analysis for use in the real world, including normalizing the data for input and denormalizing the results.

Scenario 3 — you are the Webmaster for a large and complex Web site, and you have two problems:

Scenario 4 — you are a software architect in a sizable development shop; your code is riddled with dependencies on data from your MDM's repository.  Whenever the repository changes, your developers must manually update all of the code tables affected by the changes.

TemplaGen to the rescue!

TemplaGen is a data utility implemented in XTRAN's rules language ("meta-code"), comprising 2,100 non-comment lines of rules, which automates the template-driven generation of artifacts from dataTemplaGen's original XTRAN rules took 1¾ hours to design, 4¾ hours to create, and 3¾ hours to debug.  (That's right, only 10¼ hours total!)  We have since added features, and TemplaGen's total development + debugging time is up to about 40 hours.

TemplaGen uses text data sources to instantiate a destination file from a text template, using data from one or more delimiter separated value (DSV) data files. (The most common form of DSV is comma separated values, or CSV.)  Such a data file might, for instance, be generated from a repository or exported from a spreadsheet.

TemplaGen's capabilities are powerful enough to automate the generation of fully readable, maintainable, well-styled artifactscomputer code, Web content, XML, documentation, scripts, processed DSV, text files, etc.

TemplaGen assumes that the first non-comment line in a data file is a DSV line of field labels, which are then used in templates to identify field positions in the DSV data lines that follow the field labels line.  This is a common convention used with DSV data, especially when exported from a data base or spreadsheet.

TemplaGen provides the following features:

These capabilities allow TemplaGen to automatically generate a variety of artifacts from your data repository.  A change in the data can then trigger automatic regeneration of those artifacts, ensuring that they all remain synchronized and up to date.

A typical DevOps scenario would create procedures so that, whenever there is a change in repository data, the relevant data are automatically extracted from the repository in DSV form and TemplaGen is run to regenerate all artifacts affected by the change in the repository data.

Note that, by specifying different conditionalization and the use of different data fields and field value editing in different TemplaGen runs, you can generate a wide variety of artifacts from a single data source.  For instance, the first four examples below use the same data file to generate dramatically different artifacts.

How can such powerful and generalized template-driven artifact generation from a repository be automated in only about 40 hours and 2,100 code lines of XTRAN rules?  Because there is so much capability already available as part of XTRAN's rules language (in which TemplaGen is implemented).  TemplaGen's rules take advantage of the following functionality:

In-House Use

We actually use TemplaGen ourselves, to generate XTRAN code, documentation, and text artifacts for the following XTRAN entities:

For each of those entities, we use TemplaGen templates to automatically generate one or more of the following artifacts:

As an example of user-specified string replacements to be applied to a data field's value, when we use TemplaGen to generate XTRAN documentation from repository data, we specify the following string replacements for each data field whose value will be instantiated in a template as HTML non-markup text:

The result is that each occurrence of XTRAN in such a data value will be rendered in the generated HTML artifact as XTRAN, and each occurrence of XTRAN, LLC will be rendered as XTRAN, LLC.

The advantage of using TemplaGen's template-driven artifact generation capabilities to generate XTRAN code and documentation artifacts is that, after any change to the XTRAN data repository, we run TemplaGen in scripts that automatically update all affected code and documentation, both external and internal.

Process Flowchart

Here is a flowchart for this process, in which the elements are color coded:

data flowchart


Data file

The first four examples use the same TemplaGen data file, which contains:

! data1234.dsv -- data file for TemplaGen examples
! Revised 2019-03-01.1630 by Stephen F. Heffner
! Field labels:  Function name, return type, flags, comment
! Data:
FNC1,INTEGER,21,Comment 1
F2,real,1fff,comment 2
fnc4,integer,,Comment 4 with "quotes"
! End of data file

Because this data file uses a comment character other than the default of ; (semicolon), in our template we specify ! as the data file's comment character in repetition commands that name it.  However, it uses the default DSV data delimiter of , (comma), so we don't need to specify that.

Each of the examples also assumes environment variables set to:

For each example, the instantiated output file is shown.

Example 1 — Generate an HTML table

In the template for this example, we specify the following editing for each field it uses.  For "Function name", it includes a group of two string replacements.

Field description Force case Language String
Function name Lower   fnc1
Return type Upper      
Comment   HTML    

Instantiated target file, as HTML:

Function return type table; automatically generated 2018-01-01
by TemplaGen from data1234.dsv



<th>Function name&nbsp; &nbsp;</th>
<th align="right">Return result type</th>
<th align="left">Comment</th>

<td align="right">INTEGER</td>
<td>Comment 1</td>

<td align="right">REAL</td>
<td>comment 2</td>

<td align="right">REAL</td>
<td>COMMENT 3</td>

<td align="right">INTEGER</td>
<td>Comment 4 with &quot;quotes&quot;</td>




Instantiated target file, as rendered by this browser:

Function name    Return result type Comment
fnc1 INTEGER Comment 1
f2 REAL comment 2
fnc4 INTEGER Comment 4 with "quotes"

Example 2 — Generate a C array of structures

In the template for this example, we specify the following editing for each field it uses:

Field description Force case Prefix Suffix Min field width Justification Fill character Language
Function name     ", 10 (defaulted to left) (defaulted to <SPACE>)  
Flags Upper     8 Right 0  
Comment Lower           C

Instantiated target file:

#include "stg.h"
 * Function flags table; automatically generated 2018-01-01
 * by TemplaGen from data1234.dsv
    struct stg p_stg_table[] =
        { "FNC1",   0x00000021, "comment 1" },
        { "F2",     0x00001FFF, "comment 2" },
        { "func3",  0x00FFF1AF, "comment 3" },
        { "fnc4",   0x00000000, "comment 4 with \"quotes\"" },
        { NULL } /*end of table*/

Note that, because data1.dsv's data line with fnc4 has no value for the "flags" field, TemplaGen zero-filled that field to the minimum field width of 8.

Example 3 — Generate a text document

In the template for this example, we specify the following editing for each field it uses:

Field description Force case Min field width Fill character
Function name      
Return type Upper    
Flags   1 0
Comment Lower    

Instantiated target file:

This document was automatically generated 2018-01-01 by TemplaGen
from data1234.dsv, which was extracted from our repository after
changes in repository data.

The function FNC1 has a return type of "INTEGER".  Its flags are
0x21.  Its comment is 'comment 1'.

The function F2 has a return type of "REAL".  Its flags are
0x1fff.  Its comment is 'comment 2'.

The function func3 has a return type of "REAL".  Its flags are
0xFFF1AF.  Its comment is 'comment 3'.

The function fnc4 has a return type of "INTEGER".  Its flags are
0x0.  Its comment is 'comment 4 with "quotes"'.

For more information about this document, please contact our repository

Note that, because data1.dsv's data line with fnc4 has no value for the "flags" field, TemplaGen zero-filled that field to the minimum field width of 1.

Example 4 — Data selection and transformation

This example is the same as example 1, except that we use TemplaGen's conditionalization capabilities to select data based on criteria, and we use its data transformation capabilities to to create new data forms from our DSV data.  Both use DSV expressions, which reference the values of DSV field labels and environment variables.

In our template, we use a DSV expression to specify that only DSV data lines whose rtntyp field has a value of REAL (case-insensitive) are to cause instantiation.  We also use a DSV expression with regular expression processing to remove "comment " from the value of the comment field, if it's there, and instantiate the result.

In the template for this example, we specify the following editing for each field it uses.

Field description Force case Language Transformation
Comment Lower HTML Remove "comment " if there

Instantiated target file, as HTML:

"REAL" function return type table; automatically generated 2019-01-01
by TemplaGen from data1234.dsv



<th>"REAL" function name&nbsp; &nbsp;</th>
<th align="left">Comment</th>






Instantiated target file, as rendered by this browser:

"REAL" function name    Comment
F2 2
func3 3

Example 5 — Use regular expressions to embed HTML markup in normal text

In this example, we use regular expressions (regexps) to embed HTML markup in normal text data, such that it survives when we're instantiating it as HTML non-markup text, but it's deleted when we're instantiating it as, e.g., a text literal in C code.  To do that, we specify the same regexp for both scenarios, but with different replacement strings:

(We use #...# for our embedded HTML markup instead of <...> because we want TemplaGen to "escape" characters in the rest of the text for HTML when that's what we're generating, and <...> would then be "escaped" to &lt;...&gt;.)

With that scheme in place, let's say we have a data field value in which we've embedded some HTML markup and some double quotes:

    The #b#frammis#/b# will be "pulverized".

When we're generating HTML, our regexp match and replace plus character "escaping" for HTML will generate:

    The <b>frammis</b> will be &quot;pulverized&quot;.

…which will then be rendered as:

        The frammis will be "pulverized".

But when we're generating C code and using our data field to generate a C text literal, surrounded in the template by double quotes, our regexp match and replace plus character "escaping" for C will generate:

    "The frammis will be \"pulverized\"."

Example 6 — Combine data from multiple sources

In this example, we use a template that draws on multiple data sources to create a C "include" file.

This example is an abbreviated form of what we actually do in real life.  XTRAN, our Expert System in whose rules language TemplaGen is implemented, manipulates over 40 computer languages (as well as data and text).  It also handles multiple dialects of some of those languages.  The XTRAN repository contains a table of languages and a table of their dialects.

The following C "include" file, generated with TemplaGen, defines both the internal numbers that represent languages and the flags that represent their dialects.

This example's TemplaGen template specifies the following DSV data files, extracted from the XTRAN repository and abbreviated for demonstration purposes:

Languages information

; demo-mult-langs.dsv -- Languages info for TemplaGen demo
; Extracted 2019-08-14.2001 from the XTRAN repository
; COPYRIGHT 2019 by XTRAN, LLC; reproduction or use prohibited without
; permission.
; Label row:
; Data:
6~ima~IBM mainframe assembler
8~jvs~JavaScript / ECMAScript~Y
9~mta~XTRAN meta-language
10~nat~Adabas Natural
15~vxm~HPE VAX MACRO assembler
; End of demo-mult-langs.dsv

Language dialects information

; demo-mult-dlects.dsv -- Language dialects info for TemplaGen demo
; Extracted 2019-09-24.1515 from the XTRAN repository
; COPYRIGHT 2019 by XTRAN, LLC; reproduction or use prohibited without
; permission.
; Label row:
; Data:
cbl~s38~0x0001~IBM S/38 dialect
cbl~vms~0x0002~VAX/VMS dialect
cbl~vs~0x0004~Wang VS COBOL
cee~vax~0x0001~VAX C
ftn~vms~0x0001~VAX/VMS dialect
pas~vms~0x0001~VAX/VMS dialect
pas~ms~0x0002~Microsoft dialect
pas~ibm~0x0004~IBM VS Pascal
pli~os~0x0001~IBM OS/DOS dialect
pli~vms~0x0002~VAX/VMS dialect
rpg~400~0x0001~RPG/400 dialect
rpg~ii~0x0002~RPG II dialect
rpg~iii~0x0004~RPG III dialect
xml~xhtml~0x0001~XHTML -- HTML as XML dialect
; End of demo-mult-dlects.dsv

Resulting C "include" file

Here's the resulting C "include" file created using TemplaGen:

/* demo-mult.h -- Demonstrate TemplaGen combining info from multiple sources
 * Created 2019-09-24.1520 by TemplaGen from XTRAN repository data
 * COPYRIGHT 2019 by XTRAN, LLC; reproduction or use prohibited without
 * permission.
 * NOTE -- this file is automatically generated by TemplaGen (powered by
 * XTRAN); do not edit it!

 * The following are the numbers XTRAN uses to identify languages it handles,
 * plus their dialect flags.  (Both sets are abbreviated for demo purposes.)
#define CBL_NUM   1             /*COBOL*/
#define     CBL_S38_F    0x0001 /*  IBM S/38 dialect*/
#define     CBL_VMS_F    0x0002 /*  VAX/VMS dialect*/
#define     CBL_VS_F     0x0004 /*  Wang VS COBOL*/
#define CEE_NUM   2             /*C*/
#define     CEE_VAX_F    0x0001 /*  VAX C*/
#define     CEE_KR_F     0x0002 /*  K&R*/
#define CPP_NUM   3             /*C++*/
#define HTM_NUM   4             /*HTML*/
#define FTN_NUM   5             /*Fortran*/
#define     FTN_VMS_F    0x0001 /*  VAX/VMS dialect*/
#define IMA_NUM   6             /*IBM mainframe assembler*/
#define JVA_NUM   7             /*Java*/
#define JVS_NUM   8             /*JavaScript / ECMAScript*/
#define MTA_NUM   9             /*XTRAN meta-language*/
#define NAT_NUM  10             /*Adabas Natural*/
#define PAS_NUM  11             /*Pascal*/
#define     PAS_VMS_F    0x0001 /*  VAX/VMS dialect*/
#define     PAS_MS_F     0x0002 /*  Microsoft dialect*/
#define     PAS_IBM_F    0x0004 /*  IBM VS Pascal*/
#define PLI_NUM  12             /*PL/I*/
#define     PLI_OS_F     0x0001 /*  IBM OS/DOS dialect*/
#define     PLI_VMS_F    0x0002 /*  VAX/VMS dialect*/
#define RPG_NUM  13             /*RPG*/
#define     RPG_400_F    0x0001 /*  RPG/400 dialect*/
#define     RPG_II_F     0x0002 /*  RPG II dialect*/
#define     RPG_III_F    0x0004 /*  RPG III dialect*/
#define SQL_NUM  14             /*SQL*/
#define VXM_NUM  15             /*HPE VAX MACRO assembler*/
#define XML_NUM  16             /*XML*/
#define     XML_XHTML_F  0x0001 /*  XHTML -- HTML as XML dialect*/

/* End of demo-mult.h*/