The LibTPT Book (Preliminary Documentation)

A Template Scripting language for C++

Isaac W. Foraker

This is preliminary documentation for the LibTPT C++ Class library, based on build 0.90-beta. For the most up to date documentation, go to http://tazthecat.net/~isaac/libtpt/.


Table of Contents
Introduction
What is LibTPT?
Why LibTPT?
1. Compilation and Installation
Supported Platforms/Compilers
Where to get LibTPT
Building on Windows
Building on Unix
Running Tests
2. The TPT template language
A quick sample
Reserved Characters
Variables
Comments
Expressions
Whitespace & Carriage Returns
Macros
3. The C++ Interface
A quick sample
4. Writing a TPT callback function
What is a callback?
Uses for Callbacks
The Callback Interface
TPT Callbacks and Security
A. TPT Reference
Preprocessor control
@# comment
@ignoreindent, @noignoreindent
@ignoreblankline, @noignoreblankline
@<, @>
Setting Variables
@set
@setif
@unset
Conditional Parsing
@if/@elsif/@else
Loops
@while
@foreach
String Functions
@compare, @comp, @strcmp
@concat
@substr
@length
@lc
@uc
Math Functions
@avg
@sum
Array Functions
@isarray
@size
Hash Functions
@ishash
General Functions
@eval
@rand
B. C++ Reference
Namespace TPT
Classes
TPT::Buffer
TPT::Symbols
TPT::Parser
C. Converting from PML to TPT
Strings
Carriage returns
Functions
Unsupported PML
D. Project Specifics
License
Credits
Reporting bugs
List of Examples
2-1. helloworld.tpt
2-2. Contents of a fruit basket
2-3. A fruit basket array
2-4. Random array index
4-1. A callback example: @fsum()
C-1. Strings in PML and TPT
C-2. Carriage Returns in PML and TPT

Introduction

What is LibTPT?

LibTPT is a thread-safe C++ Class library designed for express parsing of template files. The author intended to use LibTPT for CGI programs to replace slow and outdated Perl scripts that use the PML Perl module. PML is a template language module for Perl created by Peter Jones. TPT attempts to maintain compatibility with PML where possible. Differences are described in the TPT section of this document.


Why LibTPT?

I have been getting frustrated with with the amount of power I have to throw at my web server to make it perform. LibTPT is my attempt at making a new series of CGI programs to replace my slower CGI scripts.

Also, Peter is discontinuing support for PML. Given the option of maintaining the original Perl PML module or writing a new library in C++, I chose the latter.

My hope is that LibTPT will allow my web server to server more dynamicly generated pages per second than Perl, PHP, or Java.


Chapter 1. Compilation and Installation

Supported Platforms/Compilers

LibTPT has been compiled and tested under the following platforms:

FreeBSD 4.4 GCC 3.0.2
Linux GCC 3.0.2
Win32 MSVC 6.0 w/ STLPort 4.5


Building on Windows

The only supported compiler on Windows at this time is Microsoft Visual C++ 6.0 with the STLPort installed. I highly recommend that you install the latest Service Pack before developing software.

The STLPort is available from http://stlport.org. After installation, modify %INCLUDE%\stlport\stl_user_config.h and

#define _STLP_DO_IMPORT_CSTD_FUNCTIONS 1

to make the STL import the standard C library functions into the "std" namespace.

Do not forget to set %INCLUDE%\stlport as your first include path, or the wrong version of the header files will be included. Also, since MSVC 6 does not properly handle the standard library in the std namespace, avoid broad statements like

using namespace std;

Once your build environment is set up correctly, open w32/libtpt.dsw. The libtpt workspace includes all projects necessary to build libtpt and its tests. The default build configuration is intended for a generic x86 platform. Change these options if you so desire.

Both debug and release configurations are available for build. Each project should build with "debug" in its name when compiled as a debug project.

For the first build, batch build all the release versions, then proceed to the Section called Running Tests section of this document.


Building on Unix

At the time of the writing of this document, LibTPT has only been tested on FreeBSD 4.4 with GCC 3.0.2. Pre-GCC 3.0 compilers do not compile LibTPT.

There is a set of simple make files for building LibTPT. You may have to modify the make files to work on your platform.

cd src
make
cd ../test
make all
			

These steps will install the libtpt.a file in lib/.


Running Tests

There is a set of tests in the test/ directory. There is not yet an automated testing mechanism.

Under Unix, run

cd test/
make test
			

Under Windows, run

cd test/
test1 38
			

If any of these tests fail, please report which test, the platform, and compiler. See the Section called Reporting bugs in Appendix D for details.


Chapter 2. The TPT template language

This introduction to the TPT template language is intended for users who are already familiar with programming. If you are not familiar with the concepts of numbers, strings, arrays, functions, and expressions, you may want to read an introductory level programming book.


A quick sample

Example 2-1. helloworld.tpt

				
@set(hello_world, "Hello World!")\
@# Output Hello World or the value of ${hello_world} if it is set
@if (@empty(hello_world)) {
Hello World!
} @else {
${hello_world}
}

Reserved Characters

TPT reserves the following characters.

@ { } " ' \

To use one of these characters in text, use the backslash (\) escape character. For example, \\ in TPT will evaluate to a single \. Not escaping these reserved characters can lead to undesirable effects.


Variables

Variables are objects that hold values. In TPT, a variable may hold a value of type integer, string, array of values, or hash of values.

A variable in TPT is identified in plain-text as ${id} and in an expression as just id, where id is the name of the variable. The value of a variable is set using the @set TPT command.

@set(id, 123)

@set(id, "The red fox runs.")

A variable may also be cleared using the @unset TPT command.

@unset(id)

See Example 2-2 for an example of how variables may be used in a template.

Example 2-2. Contents of a fruit basket

Listing of fruitbasket.tpt

@set(fruit, "Apple")
@set(basket, 12)
The basket holds ${basket} ${fruit}s.

Output of fruitbasket.tpt


The basket holds 12 Apples.

Variables can also hold more complex values, such as arrays or hashes. Array variables look like,

${array[index]}

and hashes look like,

${array.member}

In arrays, the index can be either a number or an expression that evaluates to a number. Example 2-3 demonstrates how to set a variable to an array, then read the variable.

Example 2-3. A fruit basket array

Listing of fruity.tpt

@set(basket, "apple", "orange", "lime", "banana", "pear")\
The fruit basket contains \
@foreach fruit (basket) { ${fruit} }.

Output of fruity.tpt

The fruit basket contains apple orange lime banana pear .

Note the use of spaces in this example. Backslashes are used to join lines. For more details about controlling whitespace and carriage returns, see the Section called Whitespace & Carriage Returns.

Example 2-4 demonstrates how to use an array index in a creative fashion.

Example 2-4. Random array index

Listing of randfruit.tpt

@set(basket, "apple", "orange", "lime", "banana", "pear")\
The fruit of the day is ${basket[@rand(@size(basket))]}.

Possible output of randfruit.tpt

The fruit of the day is banana.

To determine the type of variable, use @isarray, @isscalar, and @ishash.


Comments

Comments are denoted by the @# (at-pound) symbols. TPT only supports to end-of-line comments. I.e. everything on the line following @# is a part of the comment.

Hello World!	@# This is the Hello World Comment.

In the above example, everything after Hello World! will be ignored by the TPT parser, including the whitespace. Comments affect whitespace depending on use. All whitespace immediately preceeding a comment is ignored. In the case of a full line comment,

@# This is a full line comment
	@# This is not

the entire line, including carriage returns, is ignored. In order to be classified as a full line comment, @# must be the very first characters on the line. Preceding the @# with whitespace will only cause the whitespace and comment to be ignored. This is useful if you need to insert a certain number of blank lines into a template.

@# Insert some blank lines
	@# blank line 1
	@# blank line 2
	@# blank line 3
	@# blank line 4
	@# blank line 5

Expressions

Expressions in TPT can only be executed when placed inside a TPT wrapper like a function, macro, array [], loop, or conditional expression. TPT employs a full Recursive Descent parser, so complex infix expressions are supported. Use @eval to evaluate an expression in text.


Whitespace & Carriage Returns

Since the primary purpose of TPT is to format templates into text, controls are provided to remove whitespace and carriage returns. Using a backslash (\) at the end of a line will join lines. Any whitespace before @< will be ignored, and any whitespace after @> will also be ignored.


Macros

Macros are a powerful feature of TPT. User defined Macros in TPT are very similar to user defined functions or subroutines in other languages. A macro must be defined before it is used. A macro definition looks like:

@macro(mymacro, param1, param2, ...) {
	macro body
}

To access this macro once its defined would look like:

@mymacro(param1, param2, ...)

Chapter 3. The C++ Interface

A quick sample

dumptpt.cxx

				
#include <libtpt/buffer.h>
#include <libtpt/parse.h>
#include <iostream>

// Dump the parsed template output straight to std::cout
void dumptemplate(const char* filename)
{
	TPT::Buffer tptbuf(filename);
	TPT::Parser p(tptbuf);

	if (p.run(std::cout))
	{
		// handle reporting of errors
	}
}

Chapter 4. Writing a TPT callback function

What is a callback?

In C++, it is sometimes handy to specify that a library function call a user defined function for customizable behavior. In the standard library, the generic binary search and quick sort functions do this.

void *bsearch( const void *key, const void *base, size_t num, size_t width, int
	(*compare ) ( const void *elem1, const void *elem2 ) );
void qsort( void *base, size_t num, size_t width, int (*compare )(const void
	*elem1, const void *elem2 ) );

Each of these functions allows the user to specify a custom compare function for a user defined data type. LibTPT provides a similar method to allow the user to specify a C++ function that will be called from a TPT template.


Uses for Callbacks

Callbacks give the user the power to extend TPT beyond its core functionality. For instance, a project may require TPT to do some floating point math, which is not available in the core language. On the more extreme, a user may decide to extend TPT with CGI or Database functionality. Whatever the reason, callbacks let the user implement fast TPT functions within the C++ code.

Remember, while it is tempting to write callback modules that allow TPT to do anything a full blown language can do, TPT was designed to take advantage of the speed of C++ for variable management. Therefore, functionality like database access or CGI processing should remain in your C++ code for optimal performance.


The Callback Interface

A TPT callback must be declared in the form:

	bool mycallback(std::ostream& os, TPT::Object& params);

Note that the first parameter is a reference to an output stream. All communication between the callback and the LibTPT parser are handled with streams. So, to have the callback print a message would look like:

	os << "My Message";

The second parameter is of type TPT::Object, which represents a generic object in LibTPT. The object may hold a type of scalar, array, or hash. The params variable passed to the callback is guaranteed to hold an array, but that array may hold scalars, arrays, hashes, or any combination of those types, depending on how the callback was called from within the TPT template. The enumerated types of object available to a callback function are TPT::Object::type_scalar, TPT::Object::type_array, and TPT::Object::type_hash, with the corresponding access methods TPT::Object::scalar(), TPT::Object::array(), and TPT::Object::hash(). If a call to the incorrect access method is made, an exception will be thrown. The parser does its best to handle these exceptions without interrupting the template translation.

When accessing a member of Object::array() or Object::hash(), be sure to use the get() method to get the member's real address. Member of Object arrays and hashes are stored in a smart pointer to ensure proper memory management. Refer to the example below for how to properly access a member of an Object array.

As mentioned earlier, TPT does not have any built-in functionality for dealing with floating point numbers. Suppose you needed a function to sum a list of floating point numbers populated into variables from a database or user form. See Example 4-1 for an example of what the code might look like for a floating point summation function.

Example 4-1. A callback example: @fsum()

			
#include <libtpt/tpt>
#include <ostream>

// Define a function for summing floating point numbers
bool fsum(std::ostream& os, TPT::Object& params)
{
	// Make a reference to the array for easier typing.
	TPT::Object::ArrayType& pl = params.array();
	// initialize the work variable
	double work=0;

	// Set up iterators and loop.
	TPT::Object::ArrayType::const_iterator it(pl.begin()), end(pl.end());
	for (; it != end; ++it)
	{
		// Make a reference to the object for easier typing.
		TPT::Object& obj = *(*it).get();
		// Only add the object if it is a scalar.
		if (obj.gettype() == TPT::Object::type_scalar)
			work+= atof(obj.scalar().c_str());
		else
			return true; // expected a scalar!
	}
	os << work;
	return false;
}

This callback is then registered with the TPT::Parser::addfunction() method.

	TPT::Parser p(buf);
	p.addfunction("fsum", &fsum);
		

The above example demonstrates how simple it is to loop through an array of scalar examples. As an exercise, modify fsum to be able to process sub-arrays of floats.


TPT Callbacks and Security

As with most programming interfaces, using TPT callbacks can open some serious security holes if you are not paying attention. It is recommended that you do not create callbacks that can execute arbitrary programs, and when processing user input, always bounds check the input. Bounds checking is already handled by the std::string class, so the best bet is to stick with std::string. Do not use sprintf for formatting strings. If you must format a string, and absolutely cannot use iostreams, use snprintf instead, and double check your format string for any errors.


Appendix A. TPT Reference

Preprocessor control

The following functions affect the flow of the Lexical Analyzer, and are never parsed.


@# comment

Ignore everything until the end of the line. If the comment is the first item on the line, the entire line will be ignored.


@ignoreindent, @noignoreindent

When @ignoreindent is called (with no parenthesis), all indentation on the following lines will be ignored. Use @noignoreindent to stop ignoring indentation.


@ignoreblankline, @noignoreblankline

When @ignoreblankline is called (with no parenthesis), any line with no content (including white-space) will be ignored.


@<, @>

The ignore space operators instruct the lexical analyzer to ignore any space preceding @<, or any space following @>.


Setting Variables

@set

Assign a value or values to a variable.

@set(identifier, expression)
@set(identifier, expression1, expression2, ...)

@setif

Assign a value or values to a variable only if the variable is empty.

@set(identifier, expression)
@set(identifier, expression1, expression2, ...)

@unset

Clear a variable.

@set(identifier)

Conditional Parsing

@if/@elsif/@else

Perform conditional processing.

@if (expression) {
	conditional block
} @elsif (expression) {
	conditional block
} @elsif (expression) {
	conditional block
} @else {
	conditional block
}

Loops

@while

Loop over a block of code while the loop expression is true.

@while (expression) {
	loop block
}

@foreach

Loop over an array, assigning the value of the current index to the specified identifier on each iteration.

@foreach identifier (array) {
	loop block
}

String Functions

@compare, @comp, @strcmp

Compare two strings, giving -1 if string1 is less than string2, 0 if string1 is equal to string2, or 1 if string1 is greater than string 2. @comp and @strcmp are aliases for @compare.


@concat

Concatenate a list of strings together.


@substr

Get a substring of the given string.


@length

Give the length of the given string.


@lc

Give the lowercase version of the given string.


@uc

Give the uppercase version of the given string.


Math Functions

@avg

Get the average of the specified list of integers. Numbers may be passed as part of an array, or individually as parameters. The result will be returned as a truncated integer (i.e. the result will always be rounded down).


@sum

Get the sum of the specified list of integers. Numbers may be passed as part of an array, or individually as parameters.


Array Functions

@isarray

Check a variable to see if it holds an array. Gives 1 if the variable is array, or 0 if the variable holds a different type or is not defined.


@size

Give the size of the specified array. Size will be one for any variable that is not an array.


Hash Functions

@ishash

Check a variable to see if it holds a hash. Gives 1 if the variable is a hash, or 0 if the variable holds a different type or is not defined.


General Functions

@eval

Evaluate the given expression and give the result of the evaluation.


@rand

Give a pseudorandom 32-bit integer. @rand can be given a modulus to reduce its range.

@rand()
	or
@rand(modulus)
			

Appendix B. C++ Reference

Namespace TPT

All LibTPT library classes, functions, and variables exist in the TPT C++ namespace.

All TPT header files can be included with:

#include <libtpt/tpt.h>

Classes

TPT::Buffer

#include <libtpt/buffer.h>

The TPT::Buffer class is the base IO class for reading TPT source.

explicit Buffer(const char* filename);
explicit Buffer(std::istream* is);
explicit Buffer(const char* buffer, unsigned long size);
explicit Buffer(const Buffer& buf, unsigned long start, unsigned long end);

TPT::Symbols

#include <libtpt/symbols.h>

The TPT::Symbols class may be used to predefine a symbols table for a TPT template before the template is passed to TPT::Parser. The TPT::Symbols object must be populated before it is passed to the TPT::Parser.


TPT::Parser

#include <libtpt/parse.h>


Appendix C. Converting from PML to TPT

Strings

In PML, strings may be passed bare (i.e. without quotes) to macros and functions. TPT requires strings to be in enclosed in either single or double quotes (' or ").

Example C-1. Strings in PML and TPT

String parameters in PML

@include(file.pmlh)

String parameters in TPT

@include("file.tph")
or
@include('file.tph')

Carriage returns

TPT does not automatically remove carriage returns on lines that only contain function calls. To make TPT ignore the carriage return, use a backslash.

Example C-2. Carriage Returns in PML and TPT

PML ignores carriage returns

@include(file.pmlh)

TPT must be told to ignore carriage returns

@include("file.tph")\

Functions

@concat is different in TPT. @concat concatenates a variable list of strings together, with no added spaces. The PML version concatenated values onto an existing symbol.

PML: @concat(id, value)

TPT: @set(id, @concat(id, "value"))

"value" may be either a string or an expression.

The new @concat also replaces prepend and append. Instead use:

PML: @prepend(id, value)

TPT: @set(id, @concat("value", " ", id))

PML: @append(id, value)

TPT: @set(id, @concat(id, " ", "value"))

Why is this different? Most functions in TPT are designed to be usable within the expression processor. Some functions of PML, like concat, prepend, and acted on a symbol, instead of returning a value that could be used by the expression parser.

TPT adds the following new functions for your pleasure.

@lc(expr) - lowercase string or expression
@uc(expr) - uppercase string or expression
@length(expr) - get number of characters in string or expression


Unsupported PML

@need keyword supported by PML for loading PML modules.
@perl keyword for eval'ing perl expressions.
@unless keyword. Just use if (!expr) instead.
@until keyword. Just use while (!expr) instead.
@rib keyword for default value. Just use @else { nbsp; } for now. Another workaround is to use if (@empty(var)) if you were using @rib to only test a variable.
${ARGV} array variable.
@append keyword for appending string to a symbol. Just use new @concat.
@prepend keyword for prepending string to a symbol. Just use new @concat.


Appendix D. Project Specifics

License

LibTPT Class Library Template Processor
Copyright (C) 2002 Isaac W. Foraker (isaac(at)tazthecat(dot)net)
All Rights Reserved

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

  2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

  3. Neither the name of the Author nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


Credits

LibTPT Documentation

  • Isaac W. Foraker (isaac(at)tazthecat(dot)net)

LibTPT Class Library

  • Isaac W. Foraker (isaac(at)tazthecat(dot)net)

Original PML Perl Module

  • Peter J. Jones (pjones(at)pmade(dot)org)

Additional Credits


Reporting bugs

If you encounter a bug in the LibTPT, please send an e-mail to isaac(at)tazthecat(dot)net.