XTRAN Example — Error Check HTML Links & Generate Site Map

Scenario — as a Webmaster, you want to provide a site map for your Web site visitors' convenience. But your site is constantly changing, and manually maintaining a site map for it is turning into a real pain.

You're also worn out from clicking on each link to make sure it's not an orphan, and you're afraid you might have missed one.

XTRAN to the rescue!

The following map of this Web site was generated using rules for XTRAN, our computer language expert system, to process the site's HTML.

The original site map generation rules took about 8 hours to create and debug. (That's right, only 8 hours!)

We recently upgraded the rules to track all on-site links, using an XTRAN data base, and report any orphaned links. The upgraded rules comprise 336 code lines, and the upgrade took 6½ hours to create and debug.

The rules provide the following features when creating a site map (in a single XTRAN run):

Crawl the entire Web site, following all on-site page links to do so.
Use each HTML page's <TITLE> for its entry in the map.
Strip off a specified prefix, if it exists, from that <TITLE>.
Make each page's entry a link to that page.
Once a page's cited pages are listed under it, don't do so again for that page, to avoid infinite recursion due to cycles in the site's link graph.
Ignore links that are inside any of a specified list of tags, e.g. <FOOTER>.
Ignore links that are on any of a specified list of pages, e.g. this page.
Report any orphaned links, either to other pages or to NAMEs within pages, whose targets were not seen.

XTRAN took 10 seconds on a laptop computer to parse the 19,000+ HTML code lines in 89 HTML pages that comprise this Web site, error check its 600+ on-site links (335+ to pages and 275+ to NAMEs), and create the site map below. After making changes to the Web site, we rerun this process to reflect (and error check) the changes, and we update this page with the results.

The HTML mining and generation rules for this example can easily be enhanced to produce DSV output that can be interactively queried using existing XTRAN rules.

How can such powerful and generalized HTML mining and generation be automated in only 16½ hours and 336 lines of rules? Because there is so much capability already available as part of XTRAN's rules language. These rules take advantage of the following functionality:

Text file input and output
Text manipulation
Text formatting
Delimited list manipulation
Environment variable manipulation
Content-addressable data bases
"Per statement" recursive iterator
Access to XTRAN's Internal Representation (XIR)
Navigation in XIR
Creating new meta-functions written in meta-code, which we call user meta-functions
Recursive user meta-functions
Delayed / forced rules evaluation
Meta-variable and meta-function pointers

NOTE that the rendered HTML shown as XTRAN's output below was done with default conditions; XTRAN provides many options for controlling the way it renders code for output.

Process Flowchart

Here is a flowchart for this process, in which the elements are color coded:

BLUE for XTRAN versions (runnable programs)
ORANGE for XTRAN rules (text files)
RED for code
PURPLE for text data files

Site Map

XTRAN, LLC — Automating Software Development
    XTRAN — a Computer Language Expert System
        XTRAN, LLC — XTRAN examples
            XTRAN — a Computer Language Expert System
            XTRAN Example — Analyze HP (Compaq, Digital) MACRO-11 Assembler
            XTRAN Example — Analyze Year 2000 (Y2K) Impact on MACRO-11
                XTRAN Example — Analyze Year 2000 (Y2K) Impact on C, C++, Java, or C#
                    XTRAN Example — Analyze Year 2000 (Y2K) Impact on MACRO-11
                    XTRAN Example — Analyze Year 2000 (Y2K) Impact on PL/M
                        XTRAN Example — Analyze Year 2000 (Y2K) Impact on MACRO-11
                        XTRAN Example — Analyze Year 2000 (Y2K) Impact on C, C++, Java, or C#
                        XTRAN, LLC — Contact Us
                            XTRAN, LLC — Automating Software Development
                            XTRAN — a Computer Language Expert System
                            XTRAN, LLC — Services
                                XTRAN — a Computer Language Expert System
                                XTRAN, LLC — Contact Us
                                XTRAN, LLC — Stephen F. Heffner's Résumé
                                    XTRAN — a Computer Language Expert System
                                    XTRAN Example — Cross-Link & Index HTML Documents
                                        XTRAN User's Manual Title Page (Variant 1)
                                            XTRAN User's Manual — Chapter 1 ("mini" for demo)
                                            XTRAN User's Manual — Chapter 2 ("mini" for demo)
                                        XTRAN User's Manual Title Page (Variant 2)
                                            XTRAN User's Manual — Chapter 1 ("mini" for demo)
                                            XTRAN User's Manual — Chapter 2 ("mini" for demo)
                                            XTRAN User's Manual — Chapter 3 ("mini" for demo)
                                        XTRAN User's Manual — Chapter 1 ("mini" for demo)
                                        XTRAN User's Manual — Chapter 2 ("mini" for demo)
                                        XTRAN User's Manual — Chapter 3 ("mini" for demo)
                                        XTRAN User's Manual Title Page (Variant 1)
                                            XTRAN User's Manual — Chapter 1 ("mini" for
                                            XTRAN User's Manual — Chapter 2 ("mini" for
                                        XTRAN User's Manual — Chapter 1 ("mini" for
                                        XTRAN User's Manual — Chapter 2 ("mini" for
                                    XTRAN Example — Translate Encore (SEL, Gould) Assembler to C
                                    XTRAN Example — Translate HP (Compaq, Digital) MACRO-11 Assembler to C
                                    XTRAN Example — Translate HP (Digital, Compaq) VAX MACRO to C
                                    XTRAN Example — Translate IBM Series/1 EDL to C
                                    XTRAN Example — Create File Renumbering Script
                                    XTRAN, LLC — Stephen F. Heffner's Curriculum Vitæ
                                        XTRAN — a Computer Language Expert System
                                        XTRAN, LLC — Stephen F. Heffner's Résumé
                                XTRAN, LLC — Stephen F. Heffner's Curriculum Vitæ
                    XTRAN, LLC — Contact Us
                XTRAN Example — Analyze Year 2000 (Y2K) Impact on PL/M
            XTRAN Example — Translate HP (Compaq, Digital) MACRO-11 Assembler to C
            XTRAN Example — Analyze HP (Digital, Compaq) VAX MACRO
            XTRAN Example — Analyze HP (Digital, Compaq) VAX MACRO Problem Operands
                XTRAN Example — Translate HP (Digital, Compaq) VAX MACRO Problem Operands
                    XTRAN Example — Analyze HP (Digital, Compaq) VAX MACRO Problem Operands
            XTRAN Example — Translate HP (Digital, Compaq) VAX MACRO to C
            XTRAN Example — Translate User Function Entry Macro in HP (Digital, Compaq) VAX MACRO to C
            XTRAN Example — Translate Labeled Structures in HP (Digital, Compaq) VAX MACRO to C
            XTRAN Example — Translate HP (Digital, Compaq) VAX MACRO Problem Operands
            XTRAN Example — Translate Encore (SEL, Gould) Assembler to C
            XTRAN Example — Translate In-Line Function Arguments in Encore (SEL, Gould) Assembler
            XTRAN Example — Analyze Structure/Union Member Offsets in C, C++, Java, or C#
            XTRAN Example — Analyze Cyclomatic Complexity of C, C++, Java, or C#
            XTRAN Example — C, C++, Java, or C# Module/Function Cross-Reference
            XTRAN Example — Analyze Year 2000 (Y2K) Impact on C, C++, Java, or C#
            XTRAN Example — Restructure C, C++, Java, or C#
            XTRAN Example — Add I/O Documentation to C
            XTRAN Example — Match & Replace C, C++, Java, or C# Expressions
            XTRAN Example — Combine C, C++, Java, or C# Expression Statements
            XTRAN Example — Eliminate Dead Code in C, C++, Java, or C#
            XTRAN Example — Protect C or C++ Included Files from Multiple Inclusion
            XTRAN Example — Markov Analysis
            XTRAN Example — Analyze Symbol Usage in COBOL
            XTRAN Example — Translate IBM Series/1 EDL to C
            XTRAN Example — Module/Function Cross-Reference of Fortran
            XTRAN Example — Translate Fortran COMMONs & EQUIVALENCEs to C
            XTRAN Example — Analyze HTML Tag & Attribute Usage
            XTRAN Example — Analyze HTML Embedded Substitutions & Their Occurrences
            XTRAN Example — Cross-Link & Index HTML Documents
            XTRAN Example — Generate HTML Table of Images
            XTRAN Example — Mine Web Pages for Keywords
                XTRAN Example — Count Words in Text
                    XTRAN Example — Sum Integer Field in DSV Data
                        XTRAN Example — Count Words in Text
                        XTRAN Example — Mine Web Pages for Keywords
                    XTRAN Example — Mine Web Pages for Keywords
                XTRAN Example — Sum Integer Field in DSV Data
            XTRAN Example — Colorize C, C++, Java, or C# Code by Nesting Depth
                XTRAN Example — Highlight File & Terminal I/O Code in PL/I, Display as HTML
                    XTRAN Example — Highlight Expressions in C, C++, Java, or C#
                        XTRAN Example — Highlight File & Terminal I/O Code in PL/I, Display as HTML
                        XTRAN Example — Colorize C, C++, Java, or C# Code by Nesting Depth
                    XTRAN Example — Colorize C, C++, Java, or C# Code by Nesting Depth
                XTRAN Example — Highlight Expressions in C, C++, Java, or C#
            XTRAN Example — Highlight Expressions in C, C++, Java, or C#
            XTRAN Example — Filter Natural Code on Statement Type, Display as HTML
            XTRAN Example — Module/Function Cross-Reference of Pascal
            XTRAN Example — Translate Pascal to C
            XTRAN Example — Translate Pascal Array Subscripts to C
            XTRAN Example — Automatically Unnest Pascal PROCEDUREs & FUNCTIONs
            XTRAN Example — Translate Pascal String Operations to C
            XTRAN Example — Analyze Constant/Expression PL/I PROCEDURE Arguments
                XTRAN Example — Translate PL/I PROCEDURE Parameters/Arguments as "By Value"
                    XTRAN Example — Analyze Constant/Expression PL/I PROCEDURE Arguments
            XTRAN Example — Analyze Knots in PL/I
                XTRAN Example — Restructure PL/I
            XTRAN Example — Highlight File & Terminal I/O Code in PL/I, Display as HTML
            XTRAN Example — Module/Procedure Cross-Reference of PL/I
            XTRAN Example — Restructure PL/I
            XTRAN Example — Eliminate Dead PL/I Code
            XTRAN Example — Translate PL/I to C
            XTRAN Example — Translate PL/I Calling Arguments to C
            XTRAN Example — Translate PL/I PROCEDURE Parameters/Arguments as "By Value"
            XTRAN Example — Impose Included Files on Existing PL/ix Code
            XTRAN Example — Module/Procedure Cross-Reference of PL/M
            XTRAN Example — Analyze Year 2000 (Y2K) Impact on PL/M
            XTRAN Example — Translate PL/M to C
            XTRAN Example — PL/M $INCLUDE Implies Typedef'd Data Type
            XTRAN Example — Translate PL/M CALL Statement to C
            XTRAN Example — Translate PL/M BASED() Attribute, Using Cast Pointers
            XTRAN Example — Translate PL/M $IF/$ELSEIF/$ELSE Statements to C
            XTRAN Example — Analyze XML Tags & Attributes
            XTRAN Example — Analyze XML Entity References
            XTRAN Example — Create File Renumbering Script
            XTRAN Example — Manipulate Dependency Trees
            XTRAN Example — XTRAN data bases
            XTRAN Example — Sum Integer Field in DSV Data
            XTRAN Example — Count Words in Text
            XTRAN Examples — Translate HP (Digital, Compaq) Runoff to HTML
            XTRAN Example — Translate HP (Digital, Compaq) Runoff to Rich Text Format (RTF)
            XTRAN Example — Forensic analysis of function signatures
        XTRAN in the Software Development Life Cycle (SDLC)
        XTRAN, LLC — Contact Us
        XTRAN, LLC — Stephen F. Heffner's Résumé
        XTRAN, LLC — Stephen F. Heffner's Curriculum Vitæ
        XTRAN Example — Eliminate Dead Code in C, C++, Java, or C#
        XTRAN Example — Eliminate Dead PL/I Code
        XTRAN Example — Analyze Structure/Union Member Offsets in C, C++, Java, or C#
        XTRAN Example — Markov Analysis
        XTRAN Example — Restructure C, C++, Java, or C#
        XTRAN Example — Restructure PL/I
        XTRAN Example — Combine C, C++, Java, or C# Expression Statements
        XTRAN Example — Add I/O Documentation to C
        XTRAN Example — Impose Included Files on Existing PL/ix Code
        XTRAN Example — Protect C or C++ Included Files from Multiple Inclusion
        XTRAN Example — Cross-Link & Index HTML Documents
        XTRAN Example — Sum Integer Field in DSV Data
        XTRAN Example — XTRAN data bases
        XTRAN Example — Colorize C, C++, Java, or C# Code by Nesting Depth
        XTRAN Example — Highlight File & Terminal I/O Code in PL/I, Display as HTML
        XTRAN Example — Filter Natural Code on Statement Type, Display as HTML
        XTRAN Example — Translate HP (Compaq, Digital) MACRO-11 Assembler to C
    XTRAN in the Software Development Life Cycle (SDLC)
    XTRAN, LLC — XTRAN examples
    XTRAN, LLC — Stephen F. Heffner's Curriculum Vitæ
    XTRAN, LLC — Contact Us
    XTRAN, LLC — OpenVMS Users
        XTRAN — a Computer Language Expert System
        XTRAN, LLC — Stephen F. Heffner's Curriculum Vitæ
        XTRAN, LLC — Stephen F. Heffner's Résumé