TemplaGen — Template-Driven Generation from Repository Data



Menu




Description

Scenario — you are a Master Data Manager (MDM) in charge of a "single source of truth" enterprise data repositoryMultiple artifacts (code, documentation, etc.) are individually generated from the data in that repository.  So every time there are changes in the data, all of those artifacts must be painfully recreated to reflect those changes — a laborious and error-prone process.

XTRAN to the rescue!

The following examples use TemplaGen, a utility implemented as an XTRAN rule set comprising 1,282 non-comment lines of XTRAN's rules language ("meta-code"), which automates the template-driven generation of artifacts from dataTemplaGen's original XTRAN rules took 1¾ hours to design, 4¾ hours to create, and 3¾ hours to debug.  (That's right, only 10¼ hours total!)  We have since added features, and TemplaGen's total development + debugging time is up to about 30 hours.

TemplaGen uses text data sources to instantiate a destination file from a text template, using data from one or more delimiter separated value (DSV) data files. (The most common form of DSV is comma separated values, or CSV.)  Such a data file might, for instance, be generated from a repository or exported from a spreadsheet.

TemplaGen assumes that the first non-comment line in a data file is a DSV line of field names, which are then used in templates to identify field positions in the DSV data that follows the field names.  This is a common convention used with DSV data, especially when exported from a data base or ispreadsheet.

TemplaGen provides the following features:

These capabilities allow TemplaGen to automatically generate a variety of artifacts from your data repository.  A change in the data can then trigger automatic regeneration of those artifacts, ensuring that they all remain synchronized and up to date.

A typical DevOps scenario would be to create procedures so that, whenever there is a change in repository data, the relevant data are automatically extracted from the repository in DSV form and TemplaGen is run to regenerate all artifacts affected by the change in the repository data.

Note that, by specifying the use of different data fields and field value editing in different TemplaGen runs, you can generate a wide variety of artifacts from a single data source.  For instance, the first three examples below use the same data file to generate dramatically different artifacts.

How can such powerful and generalized template-driven generation from a repository be automated in only about 30 hours and 1,282 code lines of XTRAN rules?  Because there is so much capability already available as part of XTRAN's rules languageTemplaGen's rules take advantage of the following functionality:




Process Flowchart

Here is a flowchart for this process, in which the elements are color coded:

data flowchart

In-House Use

We actually use TemplaGen ourselves, to generate XTRAN code and documentation artifacts for the following XTRAN entities:

For each of those entities, we automatically generate one or more of the following artifacts:

As an example of user-specified string replacements to be applied to a data field's value, when we use TemplaGen to generate XTRAN documentation from repository data, we specify the following string replacements for each data field whose value will be instantiated in a template as HTML non-markup text:

The result is that each occurrence of XTRAN in such a data value will be rendered in the generated HTML artifact as XTRAN, and each occurrence of XTRAN, LLC will be rendered as XTRAN, LLC.

The advantage of using TemplaGen's template-driven artifact generation capabilities to generate XTRAN code and documentation artifacts is that, after any change to the XTRAN data repository, we run TemplaGen in scripts that automatically update all affected code and documentation, both external and internal.



Examples




Data file

The first three examples use the same TemplaGen data file, data1.dsv, which contains:

! Data file for TemplaGen examples
! Revised 2019-03-01.1630 by Stephen F. Heffner
!
! Field labels:  Function name, return type, flags, comment
!
fncnam,rtntyp,flags,comment
!
! Data:
!
FNC1,INTEGER,21,Comment 1
F2,real,1fff,comment 2
func3,REAL,FFF1AF,COMMENT 3
fnc4,integer,,Comment 4 with "quotes"
!
! End of data file

Because this data file uses a comment character other than the default of ; (semicolon), in our template we specify ! as the data file's comment character in repetition commands that name it.  However, it uses the default DSV data delimiter of , (comma), so we don't need to specify that.

Each of the examples also assumes environment variables set to:

For each example, the instantiated output file is shown.




Example 1 — generate an HTML table

In the template for this example, we specify the following editing for each field it uses.  For "Function name", it includes a group of two string replacements.

Field description Force case Language String
originals
String
replacements
Function name Lower   fnc1
func3
<i>fnc1</i>
<b>func3</b>
Return type Upper      
Comment   HTML    

Instantiated target file, as HTML:

<html>
<!--
Function return type table; automatically generated 2018-01-01
with XTRAN from data1.dsv
-->

<div>

<table>

<tr>
<th>Function name&nbsp; &nbsp;</th>
<th align="right">Return result type</th>
<th>Comment</th>
</tr>

<tr>
<td><i>fnc1</i></td>
<td align="right">INTEGER</td>
<td>fnc1</td>
<td>Comment 1</td>
<tr>

<tr>
<td>f2</td>
<td align="right">REAL</td>
<td>comment 2</td>
<tr>

<tr>
<td><b>func3</b></td>
<td align="right">REAL</td>
<td>COMMENT 3</td>
<tr>

<tr>
<td>fnc4</td>
<td align="right">INTEGER</td>
<td>Comment 4 with &quot;quotes&quot;</td>
<tr>

</table>

</div>

</html>

Instantiated target file, as rendered by this browser:

Function name    Return result type Comment
fnc1 INTEGER Comment 1
f2 REAL comment 2
func3 INTEGER COMMENT 3
fnc4 REAL Comment 4 with "quotes"


Example 2 — generate a C array of structures


In the template for this example, we specify the following editing for each field it uses:

Field description Force case Prefix Suffix Min field width Justification Fill character Language
Function name     ", 10 (defaulted to left) (defaulted to <SPACE>)  
Flags Upper     8 Right 0  
Comment Lower           C

Instantiated target file:

#include "stg.h"
/*
 * Function flags table; automatically generated 2018-01-01
 * with XTRAN from data1.dsv
 */
    struct stg p_stg_table[] =
        {
        { "FNC1",   0x00000021, "comment 1" },
        { "F2",     0x00001FFF, "comment 2" },
        { "func3",  0x00FFF1AF, "comment 3" },
        { "fnc4",   0x00000000, "comment 4 with \"quotes\"" },
        { NULL } /*end of table*/
        };

Note that, because data1.dsv's data line with fnc4 has no value for the "flags" field, TemplaGen zero-filled that field to the minimum field width of 8.



Example 3 — generate a text document

In the template for this example, we specify the following editing for each field it uses:

Field description Force case Min field width Fill character
Function name      
Return type Upper    
Flags   1 0
Comment Lower    

Instantiated target file:

This document was automatically generated 2018-01-01 with XTRAN
from data1.dsv, which was extracted from our repository after
changes in repository data.

The function FNC1 has a return type of "INTEGER".  Its flags are
0x21.  Its comment is 'comment 1'.

The function F2 has a return type of "REAL".  Its flags are
0x1fff.  Its comment is 'comment 2'.

The function func3 has a return type of "REAL".  Its flags are
0xFFF1AF.  Its comment is 'comment 3'.

The function fnc4 has a return type of "INTEGER".  Its flags are
0x0.  Its comment is 'comment 4 with "quotes"'.

For more information about this document, please contact our repository
administrator.

Note that, because data1.dsv's data line with fnc4 has no value for the "flags" field, TemplaGen zero-filled that field to the minimum field width of 1.



Example 4 — use regular expressions to embed HTML markup in normal text

In this example, we use regular expressions (regexps) to embed HTML markup in normal text data, such that it survives when we're instantiating it as HTML non-markup text, but it's deleted when we're instantiating it as, e.g., a text literal in C code.  To do that, we specify the same regexp for both scenarios, but with different replacement strings:

(We use #...# for our embedded HTML markup instead of <...> because we want TemplaGen to "escape" characters in the rest of the text for HTML when that's what we're generating, and <...> would then be "escaped" to &lt;...&gt;.)

With that scheme in place, let's say we have a data field value in which we've embedded some HTML markup and some double quotes:

    The #b#frammis#/b# will be "pulverized".

When we're generating HTML, our regular expression match and replace plus character "escaping" for HTML will generate:

    The <b>frammis</b> will be &quot;pulverized&quot;.

…which will then be rendered as:

        The frammis will be "pulverized".

But when we're generating C code and using our data field to generate a C text literal, surrounded in the template by double quotes, our regular expression match and replace plus character "escaping" for C will generate:

    "The frammis will be \"pulverized\"."

###