Section
This tutorial focuses on ways to making XCLE handle more complex language constructs than the basic mecanism of scalars, lists and primitives. To fully understand the concepts we are contending with, you will need to be very familiar with the language structure, and XCL programmation. You will also need to have a good understanding of the reference counting mecanism, and at least a basic knowledge of the layout of the API.
Language extensions
The first way to extend XCLE is to provide new primitives. Actually, XCLE would not be very usefull without these "extensions". The default set of primitives, XCLstd, provides most of the basic ways data can be altered in XCLE, whether it is stored in a scalar or list type.
However, tailored use of the language often relies on ad-hoc handlers, references to external libraries, or optimized, computationally intensive algorithms. These can be integrated into XCLE through the definition of dedicated primitives, either defined through the C API, or loaded inside modules (for instance, XCLstd is one such module).
Modules and types
The second way provides new data types, by defining types as compound of existing types (making complex types usng lists is relatively easy), and utility primitives to handle them, that can be seen as constructor and methods.
We will see on an example that while this is relatively easy, these constructs are not always needed, and in any case rely heavily on the construction of new primitives. Thus you should be familiar with primitives coding before attempting to enact this part of the tutorial.
Writing primitives
Defining a primitive is essentially providing ways for your code to interact with the execution code of XCLE. That means how to pass arguments, how to return results, where are stored parameters, and identifiying the primitive.
The data needed by XCLE for this purpose is stored in the XCLE_CodeDef structure that you will learn to build step by step:
struct S_XCLE_CodeDef {
char name[CDEF_NAME_LEN+1] ; // Primitive name
unsigned char argc ; // Number of arguments
XCLE_Type argt[CDEF_ARGS_LEN] ; // Types of required arguments (OR-ed type magic numbers)
unsigned char retc ; // Number of results
XCLE_Type rett[CDEF_ARGS_LEN] ; // Types of results
XCLE_CodeOperator func ; // Machine code pointer
char desc[CDEF_DESC_LEN+1] ; // Short description string
} ;
On the contrary, all you need to obtain is provided through the prototype of a C function, called "machine code", with the XCLE_CodeOperator predefined prototype:
static const XCLE_Exception _MACHINECODE_myprimitive (XCLE_ExecCtx ctx, XCLE_Stack stk, XCLE_Hash hsh, XCLE_Object dat) ;
This function will be specified in the structure above as the func pointer, and will be called each time XCLE needs to execute this primitive.
Prototypes
The first thing XCLE checks when executing a primitive, before even using the func pointer, is the prototype, that is, witch arguments are needed for this primitive, and what it will return. These are the argc/argt fields for the arguments, and the retc/rett fields for the results. argc/retc hold the number of items taken / returned, while argt/rett hold a mask describing the allowed types. This mask is built by OR-ing the magic numbers (the constants XCLE_INTG, XCLE_FLTP, ...) of the types that can be taken / returned from the stack. The index 0 in these arrays means the first level of the stack.
Primitive data, names and parameters
The name field of this structure is the name under witch the primitive will be displayed (as my_primitive). The desc field is a static string providing a brief, human-readable, description of the primitive's usage and arguments.
Machine code
This is the most important part of the definition: it is where actual action takes places, and where all non-straightforward checks (that is, other that argument-checking) must be done. It is also where you must take care of object references.
This machine code function takes several arguments: an execution context, an arguments stack, a variables hash, and a parameter object. You will need to use these to obtain arguments, returning the results your code produced, and signal errors.
Getting args
Arguments are already on the stack when the program enters the primitive's machine code. Thus, getting primitive arguments is simply a matter of taking objects from the XCLE_Stack argument of the function, with XCLE_StackPop.
As the object is stored in a variable, and will probably not be used anymore after the function exists, it must be freed (by XCLE_ObjectFree) before returning. Remember taht it can do no harm to free an object you don't use anymore in a primitive's machine code, since this object is either on the stack, and in this case its reference count is non-null, or really not owned, and therefore can be freed.
Returning results
As for arguments, results are simply put on the stack. Once the result objects are created, use XCLE_StackPush to put on the first level each result object in order.
The stack methods will take care of reference count for you.
Raising exceptions
The normal way of exiting from a primitive handler is to return the predefined OK exception XCLE_EXCEPTION_OK. Any other exception returned is considered an error code, and will halt execution of the program that contains this primitive.
In case of an error, use XCLE_ExceptionNew to create a new exception, and simply return it to raise the exception. Remember to free unused object before.
Standard error codes
The first argument to XCLE_ExceptionNew is an error code. A few predefined error codes exist, of the form ERR_<name>, with the corresponding MSG_<name> predefined error message.
Error name |
Error number (#define ERR_<name>) |
Error message (#define MSG_<name>) |
OKSTAT | 0 | (no error) |
UNHDLD | 1 | "Unhandled error" |
RUNTIME | 2 | "Run time error" |
MEMORY | 3 | "Memory error" |
SYSTEM | 4 | "System error" |
IOSTREAM | 5 | "IO error" |
TOOFEWARG | 6 | "Too few arguments" |
INVARGTYP | 7 | "Invalid argument type" |
INVARGVAL | 8 | "Invalid argument value" |
UNIMPLEM | 9 | "Not implemented" |
NOSUCHVR | 10 | "No such variable" |
OUTRANGE | 11 | "Value out of range" |
PARSEERR | 12 | "Parse error" |
USERERR | 20 | "User-defined error" |
All values up to ERR_USERERR are reserved. User applications (i.e., your primitives) can define and use values bigger than ERR_USERERR for their own error system.
Modules
XCLE supports a dynamic primitive definition interface, through the use of precompiled modules. This let primitives sets to be distributed easily, and be used equally in a variety of interpreters.
Interface
XCL modules are shared objects (shared libraries, .so files on Linux, or .dll files on Windows) that define several symbols: XCL_Registry_Table, XCL_Registry_Size and XCL_Registry_Vers.
XCL_Registry_Table is a table of XCLE_CodeDef structures, that contains the list of primitives definitions. Each of these structures holds various information and data on a primitive, as we saw above.
XCL_Registry_Size is an integer holding the number of primitives we define (taht is, the size of XCL_Registry_Table
XCL_Registry_Vers contains a version identifier, that must match the current version of XCLE used. It should be built using the XCLE_MAKEVERSIONID macro, with three integer parameters: the major, minor and release numbers, as in XCLE_MAKEVERSIONID(1,2,0)
Each XCLE_CodeDef structure needs a function pointer, with type XCLE_CodeOperator. These functions must be defined in the same shared object as the symbols above, to complete the module's symbol table and ensure proper loading.
Dependancy control
The XCL_Registry_Vers ensures that a module built for a given version of XCLE will not be loaded by an incompatible version of the library. The module loader will try to match one or several of the numbers given to XCLE_MAKEVERSIONID with an internal version identifier, and will fail with a warning when trying to load incompatible modules.
Implementation guidelines
Memory
While the reference counting system ensures that most of the burdensome memory management is dealt with by XCLE itself, using the library with C programs does not let us wrap away completely the problem. However, we tried to ensure that microscopic memory management needs to be done only when it makes the most sense, and where the scope of data is the easisest to interpret
Thus, the programmer's part is reduced to freeing unused objects in primitives. Each argument that has been used and will be discarded, each temporary object not put on the stack, should be freeed (by a call to XCLE_FreeObject, or its type-specific counterpart) before exiting from the primitive handler.
Symbols
Symbol names (that is, function names) for the primitives machine code sections should be kept as distinctive of the module as possible (for instance, by prefixing them with the module name), to avoid symbol conflicts.
The use of global symbols, defined in the interpreter and not in the module, while unavoidable in some cases, should be kept to a minimum. The standard interpreters cxcl and gxcl define the XCL_parse_ctx and XCL_exec_ctx symbols (respectively, the parsing context and the execution context used by the interpreter, that are used by the standard modules in XCLstd.
Prototypes
More precise prototypes means more intensive use of the fast built-in argument checking system, and more precise results with the profiling tools. So you should to be as precise as possible when defining argument counts and types.
Example of module definition
Standard headers
These headers regroup the various XCLE components you will need, and define a few macros to ease primitive coding.
#include <stdlib.h>
#include <unistd.h>
#include <errno.h>
#include <DbgLog.h>
#include <XCLE/Containers.h>
#include <XCLE/Obj_generic.h>
#include <XCLE/Obj_void.h>
#include <XCLE/Obj_intg.h>
#include <XCLE/Obj_fltp.h>
#include <XCLE/Obj_strg.h>
#include <XCLE/Obj_list.h>
#include <XCLE/Obj_code.h>
#include <XCLE/Mod_string.h>
#include <XCLE/Mod_exec.h>
#include <XCLE/Mod_write.h>
#include <XCLE/Mod_parse.h>
#define XCLE_Primitive(name) \
static const XCLE_Exception name(XCLE_ExecCtx ctx, XCLE_Stack stk, XCLE_Hash hsh, XCLE_Object dat)
#define GETSTACK() stk
#define GETNAMES() hsh
#define GETDATA() dat
#define SETSTATUS(e) return e
#define SETSTATUSMSG(m) return XCLE_ExceptionNew(ERR_MAXSYSERR,m)
#define SETSTATUSNUM(e) return XCLE_ExceptionNew(ERR_##e,MSG_##e)
#define STATUSOK() return XCLE_EXCEPTION_OK
Machine code definition
We define here only two primitives: one that adds two numbers, and another that flips the sign of a number. We distinguish the case where the numbers are integers, and floating-point values.
XCLE_Primitive(_CodeCall_numadd) {
XCLE_Object obj = NULL ;
XCLE_Object obj1 = NULL ;
XCLE_Object obj2 = NULL ;
obj1 = XCLE_StackPop(GETSTACK()) ;
obj2 = XCLE_StackPop(GETSTACK()) ;
switch(XCLE_ObjectType(obj1)) {
case XCLE_OT_INTG:
switch(XCLE_ObjectType(obj2)) {
case XCLE_OT_INTG:
obj = XCLE_AnyToObject(XCLE_IntgNew( XCLE_IntgValue((XCLE_Intg)obj1) + XCLE_IntgValue((XCLE_Intg)obj2) )) ;
break ;
case XCLE_OT_FLTP:
obj = XCLE_AnyToObject(XCLE_FltpNew( XCLE_IntgValue((XCLE_Intg)obj1) + XCLE_FltpValue((XCLE_Fltp)obj2) )) ;
break ;
}
break ;
case XCLE_OT_FLTP:
switch(XCLE_ObjectType(obj2)) {
case XCLE_OT_INTG:
obj = XCLE_AnyToObject(XCLE_FltpNew( XCLE_FltpValue((XCLE_Fltp)obj1) + XCLE_IntgValue((XCLE_Intg)obj2) )) ;
break ;
case XCLE_OT_FLTP:
obj = XCLE_AnyToObject(XCLE_FltpNew( XCLE_FltpValue((XCLE_Fltp)obj1) + XCLE_FltpValue((XCLE_Fltp)obj2) )) ;
break ;
}
break ;
}
XCLE_ObjectFree(obj1) ;
XCLE_ObjectFree(obj2) ;
if(obj==NULL) { SETSTATUSNUM(RUNTIME) ; /* return ; */ }
if(XCLE_StackPush(GETSTACK(),obj)==-1) { SETSTATUSNUM(RUNTIME) ; /* return ; */ }
STATUSOK() ;
/* return ; */
}
XCLE_Primitive(_CodeCall_numneg) {
XCLE_Object obj = NULL ;
XCLE_Object obj1 = NULL ;
obj1 = XCLE_StackPop(GETSTACK()) ;
switch(XCLE_ObjectType(obj1)) {
case XCLE_OT_INTG:
obj = XCLE_AnyToObject(XCLE_IntgNew( - XCLE_IntgValue((XCLE_Intg)obj1) )) ;
break ;
case XCLE_OT_FLTP:
obj = XCLE_AnyToObject(XCLE_FltpNew( - XCLE_FltpValue((XCLE_Fltp)obj1) )) ;
break ;
}
XCLE_ObjectFree(obj1) ;
if(obj==NULL) { SETSTATUSNUM(RUNTIME) ; /* return ; */ }
if(XCLE_StackPush(GETSTACK(),obj)==-1) { SETSTATUSNUM(RUNTIME) ; /* return ; */ }
STATUSOK() ;
/* return ; */
}
Primitives definitions structures
We now need to implement the XCL module interface, taht is, to fill the required symbols with appropriate values. We begin by setting the version control identifier with the version number of the XCLE library we use for the build.
const unsigned long XCL_Registry_Vers = XCLE_MAKEVERSIONID(1,2,0) ;
We set XCL_Registry_Size with the number of defined primitives:
const unsigned long XCL_Registry_Size = 2 ;
And finally we fill the registry with the primitive definition data, specifying the primitives' arity, expected arguments, name...
XCLE_CodeDef XCL_Registry_Table[2] = {
{
"+", // Name of the primitive
2, { XCLE_OT_INTG|XCLE_OT_FLTP, XCLE_OT_INTG|XCLE_OT_FLTP }, // Number, and types, of needed arguments
1, { XCLE_OT_INTG|XCLE_OT_FLTP }, // Number, and types, of result(s)
(const XCLE_CodeOperator) _CodeCall_numadd, // Machine code pointer
"Numeric addition" // Short description
} ,
{
"-", // Name of the primitive
1, { XCLE_OT_INTG|XCLE_OT_FLTP }, // Number, and types, of needed arguments
1, { XCLE_OT_INTG|XCLE_OT_FLTP }, // Number, and types, of result(s)
(const XCLE_CodeOperator) _CodeCall_numneg, // Machine code pointer
"Numeric sign inversion" // Short description
} ,
} ;
Compiling
Once you have created, as above, your primitive definition file, you need to compile it to build a shared library. On Unices, assuming you have called that file my_primitives.c, this is easily done with:
$[shell]> cc -c my_primitives.c -o my_primitives.o
$[shell]> ld -shared -soname my_primitives.so my_primitives.o -o my_primitives.so
Custom data types
The <TYPE:data> construct
Defining the type handler
Commented examples
Getting arguments: <print>
Returning results and types: <inc>
Managing execution, exceptions: <loop>
The Matrix type
The <Matrix> built-in: type-name, and type checker
Handlers: overloading the <+>, <*>, </>, <^> operators