Root Zanli
Home
Console
Upload
information
Create File
Create Folder
About
Tools
:
/
proc
/
thread-self
/
root
/
usr
/
share
/
doc
/
re2c
/
Filename :
re2c.1
back
Copy
.\" Man page generated from reStructuredText. . .TH RE2C 1 "" "" "" .SH NAME re2c \- generate fast lexical analyzers for C/C++ and Go . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .SH SYNOPSIS .sp \fBre2c [OPTIONS] INPUT [\-o OUTPUT]\fP .sp \fBre2go [OPTIONS] INPUT [\-o OUTPUT]\fP .SH INTRODUCTION .sp Re2c works as a preprocessor. It reads the input file (which is usually a program in the target language, but can be anything) and looks for blocks of code enclosed in special\-form comments. The text outside of these blocks is copied verbatim into the output file. The contents of the blocks are processed by re2c. It translates them to code in the target language and outputs the generated code in place of the block. There are different kinds of blocks; see the \fI\%blocks and directives\fP section for a full list of them. The main block kinds are the following: .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B \fB/*!re2c[:<name>] ... */\fP A \fIglobal block\fP that contains a mixture of named definitions, configurations, directives and rules (regular expressions with associated semantic actions). Re2c compiles regular expressions to a deterministic finite automaton, encodes it in the form of conditional jumps and replaces the block with the generated code. Names and configurations defined in the block are added to the global scope and become visible to subsequent blocks (at the start of the program global scope is initialized with the \fI\%command\-line options\fP). The \fB:<name>\fP part is optional: if specified, the name can be used to refer to the block in another part of the program. .TP .B \fB/*!local:re2c[:<name>] ... */\fP A \fIlocal block\fP is like a global block, but the names and configurations defined in it have local scope (they do not affect other blocks). .TP .B \fB/*!rules:re2c[:<name>] ... */\fP A \fIrules block\fP is like a local block, but it does not generate any code and is meant to be reused in other blocks. This is a way of sharing code (more details in the \fI\%reusable blocks\fP section). .UNINDENT .UNINDENT .UNINDENT .sp A block may contain the following kinds of statements: .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B \fB<name> = <regular expression>;\fP A \fInamed definition\fP binds a name to a regular expression. Names may contain alphanumeric characters and underscore. The \fI\%regular expressions\fP section gives an overview of re2c syntax for regular expressions. Once defined, the name can be used in other regular expressions and in rules. Recursion in named definitions is not allowed, and each name should be defined before it is used. A block inherits named definitions from the global scope. Redefining a name that exists in the current scope is an error. .TP .B \fB<configuration> = <value>;\fP A \fIconfiguration\fP allows one to change re2c behavior and customize the generated code. For a full list of configurations supported by re2c see the \fI\%configurations\fP section. Depending on a particular configuration, the value can be a keyword, a nonnegative integer number or a one\-line string which should be enclosed in double or single quotes unless it consists of alphanumeric characters. A block inherits configurations from the global scope and may redefine them or add new ones. Configurations defined inside of a block affect the whole block, even if they appear at the end of it. .TP .B \fB<regular expression> { <code> }\fP A \fIrule\fP binds a regular expression to a semantic action (a block of code in the target language). If the regular expression matches, the associated semantic action is executed. If multiple rules match, the longest match takes precedence. If multiple rules match the same string, the earliest one takes precedence. There are two special rules: the default rule \fB*\fP and the end of input rule \fB$\fP\&. The default rule should always be defined, it has the lowest priority regardless of its place in the block, and it matches any code unit (not necessarily a valid character, see the \fI\%encoding support\fP section). The end of input rule should be defined if the corresponding method for \fI\%handling the end of input\fP is used. If \fI\%start conditions\fP are used, rules have more complex syntax. .TP .B \fB!<directive>;\fP A \fIdirective\fP is one of the special predefined statements. Each directive has a unique purpose. For example, the \fB!use\fP directive merges a rules block into the current one (see the \fI\%reusable blocks\fP section), and the \fB!include\fP directive allows one to include an outer file (see the \fI\%include files\fP section). .UNINDENT .UNINDENT .UNINDENT .sp The generated code interfaces with the outer program with the help of \fIprimitives\fP \-\- symbolic names that can be defined as variables, functions or macros in the target language (collectively referred to as the API). The primitives should be defined by the user. This approach gives the user freedom and flexibility in customizing the lexer, but it also requires some understanding of how the lexer works in order to implement the primitives correctly and efficiently. The manual provides examples for the most popular use cases. For the full list of primitives see the \fI\%interface primitives\fP section. .sp There are two \fIAPI flavours\fP that define the set of primitives used by re2c: .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B \fBPointer API\fP This API is also called \fIdefault API\fP, since it was historically the first, and for a long time the only one. This is a more restrictive API based on C pointer arithmetics. It consists of pointer\-like primitives \fBYYCURSOR\fP, \fBYYMARKER\fP, \fBYYCTXMARKER\fP, \fBYYLIMIT\fP (which are normally defined as pointers of type \fBYYCTYPE*\fP) and \fBYYFILL\fP\&. Pointer API is enabled by default for the C backend, and it cannot be used with backends that do not support pointer arithmetics. .TP .B \fBGeneric API\fP This is a more flexible API that does not assume pointer semantics. It consists of primitives \fBYYPEEK\fP, \fBYYSKIP\fP, \fBYYBACKUP\fP, \fBYYBACKUPCTX\fP, \fBYYSTAGP\fP, \fBYYSTAGN\fP, \fBYYMTAGP\fP, \fBYYMTAGN\fP, \fBYYRESTORE\fP, \fBYYRESTORECTX\fP, \fBYYRESTORETAG\fP, \fBYYSHIFT\fP, \fBYYSHIFTSTAG\fP, \fBYYSHIFTMTAG\fP, \fBYYLESSTHAN\fP and \fBYYFILL\fP\&. For the C backend generic API is enabled with \fB\-\-input custom\fP option or \fBre2c:flags:input = custom;\fP configuration; for the Go backend it is enabled by default. Generic API was added in version 0.14. It is intentionally designed to give the user as much freedom as possible in redefining the input model and the semantics of different actions performed by the generated code. .UNINDENT .UNINDENT .UNINDENT .sp There are two \fIAPI styles\fP determine the form in which the primitives should be defined: .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .TP .B \fBFree\-form\fP This style is enabled with \fBre2c:api:style = free\-form;\fP configuration. Interface primitives should be defined as free\-form pieces of code with interpolated variables of the form \fB@@{name}\fP or optionally just \fB@@\fP if there is only one variable. The \fB@@\fP marker can be customized with the \fBre2c:api:sigil\fP configuration. In C/C++ free\-form style generic API can be defined in terms of pointers \fBcursor\fP, \fBlimit\fP, \fBmarker\fP and \fBctxmarker\fP as follows: .INDENT 7.0 .INDENT 3.5 .sp .nf .ft C /*!re2c re2c:define:YYPEEK = "*cursor"; re2c:define:YYSKIP = "++cursor"; re2c:define:YYBACKUP = "marker = cursor"; re2c:define:YYRESTORE = "cursor = marker"; re2c:define:YYBACKUPCTX = "ctxmarker = cursor"; re2c:define:YYRESTORECTX = "cursor = ctxmarker"; re2c:define:YYRESTORETAG = "cursor = ${tag}"; re2c:define:YYLESSTHAN = "limit \- cursor < @@{len}"; re2c:define:YYSTAGP = "@@{tag} = cursor"; re2c:define:YYSTAGN = "@@{tag} = NULL"; re2c:define:YYSHIFT = "cursor += @@{shift}"; re2c:define:YYSHIFTSTAG = "@@{tag} += @@{shift}"; */ .ft P .fi .UNINDENT .UNINDENT .nf .fi .sp .TP .B \fBFunction\-like\fP This style is enabled with \fBre2c:api:style = functions;\fP configuration. Primitives should be defined as functions or macros with parentheses, accepting the necessary number of arguments. For historical reasons this API style is the default for C/C++ backend. Function style generic API can be defined in terms of pointers \fBcursor\fP, \fBlimit\fP, \fBmarker\fP and \fBctxmarker\fP as follows: .INDENT 7.0 .INDENT 3.5 .sp .nf .ft C #define YYPEEK() *cursor #define YYSKIP() ++cursor #define YYBACKUP() marker = cursor #define YYRESTORE() cursor = marker #define YYBACKUPCTX() ctxmarker = cursor #define YYRESTORECTX() cursor = ctxmarker #define YYRESTORETAG(tag) cursor = tag #define YYLESSTHAN(len) limit \- cursor < len #define YYSTAGP(tag) tag = cursor #define YYSTAGN(tag) tag = NULL #define YYSHIFT(shift) cursor += shift #define YYSHIFTSTAG(tag, shift) tag += shift .ft P .fi .UNINDENT .UNINDENT .UNINDENT .UNINDENT .UNINDENT .sp For \fBYYFILL\fP definition and instructions how to customize or disable end\-of\-input checks see the \fI\%handling the end of input\fP and \fI\%buffer refilling\fP sections. .SH EXAMPLE .sp Here is a simple re2c program: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C // re2c $INPUT \-o $OUTPUT \-\-case\-ranges \-i int lex(const char *YYCURSOR) { // C/C++ code /*!re2c re2c:define:YYCTYPE = char; // configurations re2c:yyfill:enable = 0; // // alpha = [a\-zA\-Z]; // named definitions ident = alpha (alpha | [0\-9])*; // // ident { return 0; } // rules * { return 1; } // */ } // // int main() { // C/C++ code return lex("qwerty42"); // } // .ft P .fi .UNINDENT .UNINDENT .sp The generated code looks like this: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C /* Generated by re2c */ // re2c $INPUT \-o $OUTPUT \-\-case\-ranges \-i int lex(const char *YYCURSOR) { // C/C++ code { char yych; yych = *YYCURSOR; switch (yych) { case \(aqA\(aq ... \(aqZ\(aq: case \(aqa\(aq ... \(aqz\(aq: goto yy4; default: goto yy2; } yy2: ++YYCURSOR; { return 1; } yy4: yych = *++YYCURSOR; switch (yych) { case \(aq0\(aq ... \(aq9\(aq: case \(aqA\(aq ... \(aqZ\(aq: case \(aqa\(aq ... \(aqz\(aq: goto yy4; default: goto yy6; } yy6: { return 0; } } } // // int main() { // C/C++ code return lex("qwerty42"); // } // .ft P .fi .UNINDENT .UNINDENT .SH COMMAND-LINE OPTIONS .sp Command\-line interface consists of options and warnings. Some of the options have corresponding \fI\%configurations\fP, others are global and cannot be changed after re2c starts reading the input file. Debug options generally require building re2c in debug configuration. Internal options are useful for experimenting with the algorithms used in re2c. Warnings can be invividually enabled, disabled and turned into an error. .INDENT 0.0 .TP .B \fB\-? \-h \-\-help\fP Show help message. .TP .B \fB\-1 \-\-single\-pass\fP Deprecated. Does nothing (single pass is the default now). .TP .B \fB\-8 \-\-utf\-8\fP Generate a lexer that reads input in UTF\-8 encoding. re2c assumes that character range is 0 \-\- 0x10FFFF and character size is 1 byte. .TP .B \fB\-b \-\-bit\-vectors\fP Optimize conditional jumps using bit masks. Implies \fB\-s\fP\&. .TP .B \fB\-c \-\-conditions \-\-start\-conditions\fP Enable support of Flex\-like "conditions": multiple interrelated lexers within one block. Option \fB\-\-start\-conditions\fP is a legacy alias; use \fB\-\-conditions\fP instead. .TP .B \fB\-\-case\-insensitive\fP Treat single\-quoted and double\-quoted strings as case\-insensitive. .TP .B \fB\-\-case\-inverted\fP Invert the meaning of single\-quoted and double\-quoted strings: treat single\-quoted strings as case\-sensitive and double\-quoted strings as case\-insensitive. .TP .B \fB\-\-case\-ranges\fP Collapse consecutive cases in a switch statements into a range of the form \fBcase low ... high:\fP\&. This syntax is an extension of the C/C++ language, supported by compilers like GCC, Clang and Tcc. The main advantage over using single cases is smaller generated C code and faster generation time, although for some compilers like Tcc it also results in smaller binary size. This option doesn\(aqt work for the Go backend. .TP .B \fB\-\-depfile FILE\fP Write dependency information to \fBFILE\fP in the form of a Makefile rule \fB<output\-file> : <input\-file> [include\-file ...]\fP\&. This allows to track build dependencies in the presence of \fB/*!include:re2c*/\fP directives, so that updating include files triggers regeneration of the output file. This option requires that \fB\-o \-\-output\fP option is specified. .TP .B \fB\-e \-\-ecb\fP Generate a lexer that reads input in EBCDIC encoding. re2c assumes that character range is 0 \-\- 0xFF an character size is 1 byte. .TP .B \fB\-\-empty\-class <match\-empty | match\-none | error>\fP Define the way re2c treats empty character classes. With \fBmatch\-empty\fP (the default) empty class matches empty input (which is illogical, but backwards\-compatible). With \fBmatch\-none\fP empty class always fails to match. With \fBerror\fP empty class raises a compilation error. .TP .B \fB\-\-encoding\-policy <fail | substitute | ignore>\fP Define the way re2c treats Unicode surrogates. With \fBfail\fP re2c aborts with an error when a surrogate is encountered. With \fBsubstitute\fP re2c silently replaces surrogates with the error code point 0xFFFD. With \fBignore\fP (the default) re2c treats surrogates as normal code points. The Unicode standard says that standalone surrogates are invalid, but real\-world libraries and programs behave in different ways. .TP .B \fB\-f \-\-storable\-state\fP Generate a lexer which can store its inner state. This is useful in push\-model lexers which are stopped by an outer program when there is not enough input, and then resumed when more input becomes available. In this mode users should additionally define \fBYYGETSTATE()\fP and \fBYYSETSTATE(state)\fP macros and variables \fByych\fP, \fByyaccept\fP and \fBstate\fP as part of the lexer state. .TP .B \fB\-F \-\-flex\-syntax\fP Partial support for Flex syntax: in this mode named definitions don\(aqt need the equal sign and the terminating semicolon, and when used they must be surrounded by curly braces. Names without curly braces are treated as double\-quoted strings. .TP .B \fB\-g \-\-computed\-gotos\fP Optimize conditional jumps using non\-standard "computed goto" extension (which must be supported by the compiler). re2c generates jump tables only in complex cases with a lot of conditional branches. Complexity threshold can be configured with \fBcgoto:threshold\fP configuration. This option implies \fB\-b\fP\&. This option doesn\(aqt work for the Go backend. .TP .B \fB\-I PATH\fP Add \fBPATH\fP to the list of locations which are used when searching for include files. This option is useful in combination with \fB/*!include:re2c ... */\fP directive. Re2c looks for \fBFILE\fP in the directory of including file and in the list of include paths specified by \fB\-I\fP option. .TP .B \fB\-i \-\-no\-debug\-info\fP Do not output \fB#line\fP information. This is useful when the generated code is tracked by some version control system or IDE. .TP .B \fB\-\-input <default | custom>\fP Specify the API used by the generated code to interface with used\-defined code. Option \fBdefault\fP is the C API based on pointer arithmetic (it is the default for the C backend). Option \fBcustom\fP is the generic API (it is the default for the Go backend). .TP .B \fB\-\-input\-encoding <ascii | utf8>\fP Specify the way re2c parses regular expressions. With \fBascii\fP (the default) re2c handles input as ASCII\-encoded: any sequence of code units is a sequence of standalone 1\-byte characters. With \fButf8\fP re2c handles input as UTF8\-encoded and recognizes multibyte characters. .TP .B \fB\-\-lang <c | go>\fP Specify the output language. Supported languages are C and Go (the default is C). .TP .B \fB\-\-location\-format <gnu | msvc>\fP Specify location format in messages. With \fBgnu\fP locations are printed as \(aqfilename:line:column: ...\(aq. With \fBmsvc\fP locations are printed as \(aqfilename(line,column) ...\(aq. Default is \fBgnu\fP\&. .TP .B \fB\-\-no\-generation\-date\fP Suppress date output in the generated file. .TP .B \fB\-\-no\-version\fP Suppress version output in the generated file. .TP .B \fB\-o OUTPUT \-\-output=OUTPUT\fP Specify the \fBOUTPUT\fP file. .TP .B \fB\-P \-\-posix\-captures\fP Enable submatch extraction with POSIX\-style capturing groups. .TP .B \fB\-r \-\-reusable\fP Deprecated since re2c version 2.2 (reusable blocks are allowed by default now). .TP .B \fB\-S \-\-skeleton\fP Ignore user\-defined interface code and generate a self\-contained "skeleton" program. Additionally, generate input files with strings derived from the regular grammar and compressed match results that are used to verify "skeleton" behavior on all inputs. This option is useful for finding bugs in optimizations and code generation. This option doesn\(aqt work for the Go backend. .TP .B \fB\-s \-\-nested\-ifs\fP Use nested \fBif\fP statements instead of \fBswitch\fP statements in conditional jumps. This usually results in more efficient code with non\-optimizing compilers. .TP .B \fB\-T \-\-tags\fP Enable submatch extraction with tags. .TP .B \fB\-t HEADER \-\-type\-header=HEADER\fP Generate a \fBHEADER\fP file that contains enum with condition names. Requires \fB\-c\fP option. .TP .B \fB\-u \-\-unicode\fP Generate a lexer that reads UTF32\-encoded input. Re2c assumes that character range is 0 \-\- 0x10FFFF and character size is 4 bytes. This option implies \fB\-s\fP\&. .TP .B \fB\-V \-\-vernum\fP Show version information in \fBMMmmpp\fP format (major, minor, patch). .TP .B \fB\-\-verbose\fP Output a short message in case of success. .TP .B \fB\-v \-\-version\fP Show version information. .TP .B \fB\-w \-\-wide\-chars\fP Generate a lexer that reads UCS2\-encoded input. Re2c assumes that character range is 0 \-\- 0xFFFF and character size is 2 bytes. This option implies \fB\-s\fP\&. .TP .B \fB\-x \-\-utf\-16\fP Generate a lexer that reads UTF16\-encoded input. Re2c assumes that character range is 0 \-\- 0x10FFFF and character size is 2 bytes. This option implies \fB\-s\fP\&. .UNINDENT .INDENT 0.0 .TP .B \fB\-D \-\-emit\-dot\fP Instead of normal output generate lexer graph in .dot format. The output can be converted to an image with the help of Graphviz (e.g. something like \fBdot \-Tpng \-odfa.png dfa.dot\fP). .TP .B \fB\-d \-\-debug\-output\fP Emit \fBYYDEBUG\fP in the generated code. \fBYYDEBUG\fP should be defined by the user in the form of a void function with two parameters: \fBstate\fP (lexer state or \-1) and \fBsymbol\fP (current input symbol of type \fBYYCTYPE\fP). .TP .B \fB\-\-dump\-adfa\fP Debug option: output DFA after tunneling (in .dot format). .TP .B \fB\-\-dump\-cfg\fP Debug option: output control flow graph of tag variables (in .dot format). .TP .B \fB\-\-dump\-closure\-stats\fP Debug option: output statistics on the number of states in closure. .TP .B \fB\-\-dump\-dfa\-det\fP Debug option: output DFA immediately after determinization (in .dot format). .TP .B \fB\-\-dump\-dfa\-min\fP Debug option: output DFA after minimization (in .dot format). .TP .B \fB\-\-dump\-dfa\-tagopt\fP Debug option: output DFA after tag optimizations (in .dot format). .TP .B \fB\-\-dump\-dfa\-tree\fP Debug option: output DFA under construction with states represented as tag history trees (in .dot format). .TP .B \fB\-\-dump\-dfa\-raw\fP Debug option: output DFA under construction with expanded state\-sets (in .dot format). .TP .B \fB\-\-dump\-interf\fP Debug option: output interference table produced by liveness analysis of tag variables. .TP .B \fB\-\-dump\-nfa\fP Debug option: output NFA (in .dot format). .UNINDENT .INDENT 0.0 .TP .B \fB\-\-dfa\-minimization <moore | table>\fP Internal option: DFA minimization algorithm used by re2c. The \fBmoore\fP option is the Moore algorithm (it is the default). The \fBtable\fP option is the "table filling" algorithm. Both algorithms should produce the same DFA up to states relabeling; table filling is simpler and much slower and serves as a reference implementation. .TP .B \fB\-\-eager\-skip\fP Internal option: make the generated lexer advance the input position eagerly \-\- immediately after reading the input symbol. This changes the default behavior when the input position is advanced lazily \-\- after transition to the next state. This option is implied by \fB\-\-no\-lookahead\fP\&. .TP .B \fB\-\-no\-lookahead\fP Internal option: use TDFA(0) instead of TDFA(1). This option has effect only with \fB\-\-tags\fP or \fB\-\-posix\-captures\fP options. .TP .B \fB\-\-no\-optimize\-tags\fP Internal optionL: suppress optimization of tag variables (useful for debugging). .TP .B \fB\-\-posix\-closure <gor1 | gtop>\fP Internal option: specify shortest\-path algorithm used for the construction of epsilon\-closure with POSIX disambiguation semantics: \fBgor1\fP (the default) stands for Goldberg\-Radzik algorithm, and \fBgtop\fP stands for "global topological order" algorithm. .TP .B \fB\-\-posix\-prectable <complex | naive>\fP Internal option: specify the algorithm used to compute POSIX precedence table. The \fBcomplex\fP algorithm computes precedence table in one traversal of tag history tree and has quadratic complexity in the number of TNFA states; it is the default. The \fBnaive\fP algorithm has worst\-case cubic complexity in the number of TNFA states, but it is much simpler than \fBcomplex\fP and may be slightly faster in non\-pathological cases. .TP .B \fB\-\-stadfa\fP Internal option: use staDFA algorithm for submatch extraction. The main difference with TDFA is that tag operations in staDFA are placed in states, not on transitions. .TP .B \fB\-\-fixed\-tags <none | toplevel | all>\fP Internal option: specify whether the fixed\-tag optimization should be applied to all tags (\fBall\fP), none of them (\fBnone\fP), or only those in toplevel concatenation (\fBtoplevel\fP). The default is \fBall\fP\&. "Fixed" tags are those that are located within a fixed distance to some other tag (called "base"). In such cases only tha base tag needs to be tracked, and the value of the fixed tag can be computed as the value of the base tag plus a static offset. For tags that are under alternative or repetition it is also necessary to check if the base tag has a no\-match value (in that case fixed tag should also be set to no\-match, disregarding the offset). For tags in top\-level concatenation the check is not needed, because they always match. .UNINDENT .INDENT 0.0 .TP .B \fB\-W\fP Turn on all warnings. .TP .B \fB\-Werror\fP Turn warnings into errors. Note that this option alone doesn\(aqt turn on any warnings; it only affects those warnings that have been turned on so far or will be turned on later. .TP .B \fB\-W<warning>\fP Turn on \fBwarning\fP\&. .TP .B \fB\-Wno\-<warning>\fP Turn off \fBwarning\fP\&. .TP .B \fB\-Werror\-<warning>\fP Turn on \fBwarning\fP and treat it as an error (this implies \fB\-W<warning>\fP). .TP .B \fB\-Wno\-error\-<warning>\fP Don\(aqt treat this particular \fBwarning\fP as an error. This doesn\(aqt turn off the warning itself. .UNINDENT .INDENT 0.0 .TP .B \fB\-Wcondition\-order\fP Warn if the generated program makes implicit assumptions about condition numbering. One should use either the \fB\-t, \-\-type\-header\fP option or the \fB/*!types:re2c*/\fP directive to generate a mapping of condition names to numbers and then use the autogenerated condition names. .TP .B \fB\-Wempty\-character\-class\fP Warn if a regular expression contains an empty character class. Trying to match an empty character class makes no sense: it should always fail. However, for backwards compatibility reasons \fBre2c\fP allows empty character classes and treats them as empty strings. Use the \fB\-\-empty\-class\fP option to change the default behavior. .TP .B \fB\-Wmatch\-empty\-string\fP Warn if a rule is nullable (matches an empty string). If the lexer runs in a loop and the empty match is unintentional, the lexer may unexpectedly hang in an infinite loop. .TP .B \fB\-Wswapped\-range\fP Warn if the lower bound of a range is greater than its upper bound. The default behavior is to silently swap the range bounds. .TP .B \fB\-Wundefined\-control\-flow\fP Warn if some input strings cause undefined control flow in the lexer (the faulty patterns are reported). This is the most dangerous and most common mistake. It can be easily fixed by adding the default rule \fB*\fP which has the lowest priority, matches any code unit, and consumes exactly one code unit. .TP .B \fB\-Wunreachable\-rules\fP Warn about rules that are shadowed by other rules and will never match. .TP .B \fB\-Wuseless\-escape\fP Warn if a symbol is escaped when it shouldn\(aqt be. By default, re2c silently ignores such escapes, but this may as well indicate a typo or an error in the escape sequence. .TP .B \fB\-Wnondeterministic\-tags\fP Warn if a tag has \fBn\fP\-th degree of nondeterminism, where \fBn\fP is greater than 1. .TP .B \fB\-Wsentinel\-in\-midrule\fP Warn if the sentinel symbol occurs in the middle of a rule \-\-\- this may cause reads past the end of buffer, crashes or memory corruption in the generated lexer. This warning is only applicable if the sentinel method of checking for the end of input is used. It is set to an error if \fBre2c:sentinel\fP configuration is used. .UNINDENT .SH BLOCKS AND DIRECTIVES .sp Below is the list of re2c directives (syntactic constructs that mark the beginning and end of the code that should be processed by re2c). Named blocks were added in re2c version 2.2. They are exactly the same as unnamed blocks, except that the name can be used to reference a block in other parts of the program. More information on each directive can be found in the related sections. .INDENT 0.0 .TP .B \fB/*!re2c[:<name>] ... */\fP A global re2c block with an optional name. The block may contain named definitions, configurations and rules in any order. Named definitions and configurations are defined in the global scope, so they are inherited by subsequent blocks. The code for a global block is generated at the point where the block is specified. .TP .B \fB/*!local:re2c[:<name>] ... */\fP A local re2c block with an optional name. Unlike global blocks, definitions and configurations inside of a local block are not added into the global scope. In all other respects local blocks are the same as global blocks. .TP .B \fB/*!rules:re2c[:<name>] ... */\fP A reusable block with an optional name. Rules blocks have the same structure as local or global blocks, but they do not produce any code and they can be reused multiple times in other blocks with the help of a \fB!use:<name>;\fP directive or a \fB/*!use:re2c[:<name>] ... */\fP block. A rules block on its own does not add any definitions into the global scope. The code for it is generated at the point of use. Prior to re2c version 2.2 rules blocks required \fB\-r \-\-reusable\fP option. .TP .B \fB/*!use:re2c[:<name>] ... */\fP A use block that references a previously defined rules block. If the name is specified, re2c looks for a rules blocks with this name. Otherwise the most recent rules block is used (either a named or an unnamed one). A use block can add definitions, configurations and rules of its own, which are added to those of the referenced rules block. Prior to re2c version 2.2 use blocks required \fB\-r \-\-reusable\fP option. .TP .B \fB!use:<name>;\fP An in\-block use directive that merges a previously defined rules block with the specified name into the current block. Named definitions, configurations and rules of the referenced block are added to the current ones. Conflicts between overlapping rules and configurations are resolved in the usual way: the first rule takes priority, and the latest configuration overrides the preceding ones. One exception is the special rules \fB*\fP, \fB$\fP and \fB<!>\fP for which a block\-local definition always takes priority. A use directive can be placed anywhere inside of a block, and multiple use directives are allowed. .TP .B \fB/*!max:re2c[:<name1>[:<name2>...]] ... */\fP A directive that generates \fBYYMAXFILL\fP definition. An optional list of block names specifies which blocks should be included when computing \fBYYMAXFILL\fP value (if the list is empty, all blocks are included). By default the generated code is a macro\-definition for C (\fB#define YYMAXFILL <n>\fP), or a global variable for Go (\fBvar YYMAXFILL int = <n>\fP). It can be customized with an optional configuration \fBformat\fP that specifies a template string where \fB@@{max}\fP (or \fB@@\fP for short) is replaced with the numeric value of \fBYYMAXFILL\fP\&. .TP .B \fB/*!maxnmatch:re2c[:<name1>[:<name2>...]] ... */\fP A directive that generates \fBYYMAXNMATCH\fP definition (it requires \fB\-P \-\-posix\-captures\fP option). An optional list of block names specifies which blocks should be included when computing \fBYYMAXNMATCH\fP value (if the list is empty, all blocks are included). By default the generated code is a macro\-definition for C (\fB#define YYMAXNMATCH <n>\fP), or a global variable for Go (\fBvar YYMAXNMATCH int = <n>\fP). It can be customized with an optional configuration \fBformat\fP that specifies a template string where \fB@@{max}\fP (or \fB@@\fP for short) is replaced with the numeric value of \fBYYMAXNMATCH\fP\&. .TP .B \fB/*!stags:re2c[:<name1>[:<name2>...]] ... */\fP, \fB/*!mtags:re2c[:<name1>[:<name2>...]] ... */\fP Directives that specify a template piece of code that is expanded for each s\-tag/m\-tag variable generated by re2c. An optional list of block names specifies which blocks should be included when computing the set of tag variables (if the list is empty, all blocks are included). There are two optional configurations: \fBformat\fP and \fBseparator\fP\&. Configuration \fBformat\fP specifies a template string where \fB@@(tag}\fP (or \fB@@\fP for short) is replaced with the name of each tag variable. Configuration \fBseparator\fP specifies a piece of code used to join the generated \fBformat\fP pieces for different tag variables. .TP .B \fB/*!getstate:re2c[:<name1>[:<name2>...]] ... */\fP A directive that generates conditional dispatch on the lexer state (it requires \fB\-f \-\-storable\-state\fP option). An optional list of block names specifies which blocks should be included in the state dispatch. The default transition goes to the start label of the first block on the list. If the list is empty, all blocks are included, and the default transition goes to the first block in the file that has a start label. .TP .B \fB/*!types:re2c[:<name1>[:<name2>...]] ... */\fP A directive that generates condition enumeration (it requires \fB\-c \-\-conditions\fP option). An optional list of block names specifies which blocks should be included when computing the set of conditions (if the list is empty, all blocks are included). By default the generated code is an enumeration \fBYYCONDTYPE\fP\&. It can be customized with optional configurations \fBformat\fP and \fBseparator\fP\&. Configuration \fBformat\fP specifies a template string where \fB@@(cond}\fP (or \fB@@\fP for short) is replaced with the name of each condition, and \fB@@{num}\fP is replaced with a numeric index of that condition. Configuration \fBseparator\fP specifies a piece of code used to join the generated \fBformat\fP pieces for different conditions. .TP .B \fB/*!include:re2c <file> */\fP This directive allows one to include \fB<file>\fP, which must be a double\-quoted file path. The contents of the file are literally substituted in place of the directive, in the same way as \fB#include\fP works in C/C++. This directive can be used together with the \fB\-\-depfile\fP option to generate build system dependencies on the included files. .TP .B \fB!include <file>;\fP This directive is the same as \fB/*!include:re2c <file> */\fP, except that it should be used inside of a re2c block. .TP .B \fB/*!header:re2c:on*/\fP This directive marks the start of header file. Everything after it and up to the following \fB/*!header:re2c:off*/\fP directive is processed by re2c and written to the header file specified with \fB\-t \-\-type\-header\fP option. .TP .B \fB/*!header:re2c:off*/\fP This directive marks the end of header file started with \fB/*!header:re2c:on*/\fP\&. .TP .B \fB/*!ignore:re2c ... */\fP A block which contents are ignored and removed from the output file. .TP .B \fB%{ ... %}\fP A global re2c block in \fB\-F \-\-flex\-support\fP mode. This is deprecated and exists for backward compatibility. .UNINDENT .SH INTERFACE PRIMITIVES .sp Here is a list of API primitives that may be used by the generated code in order to interface with the outer program. Which primitives are needed depends on multiple factors, including the complexity of regular expressions, input representation, buffering, the use of various features and so on. All the necessary primitives should be defined by the user in the form of macros, functions, variables, free\-form pieces of code or any other suitable form. Re2c does not (and cannot) check the definitions, so if anything is missing or defined incorrectly the generated code will not compile. .INDENT 0.0 .TP .B \fBYYCTYPE\fP The type of the input characters (code units). For ASCII, EBCDIC and UTF\-8 encodings it should be 1\-byte unsigned integer. For UTF\-16 or UCS\-2 it should be 2\-byte unsigned integer. For UTF\-32 it should be 4\-byte unsigned integer. .TP .B \fBYYCURSOR\fP A pointer\-like l\-value that stores the current input position (usually a pointer of type \fBYYCTYPE*\fP). Initially \fBYYCURSOR\fP should point to the first input character. It is advanced by the generated code. When a rule matches, \fBYYCURSOR\fP points to the one after the last matched character. It is used only in the default C API. .TP .B \fBYYLIMIT\fP A pointer\-like r\-value that stores the end of input position (usually a pointer of type \fBYYCTYPE*\fP). Initially \fBYYLIMIT\fP should point to the one after the last available input character. It is not changed by the generated code. Lexer compares \fBYYCURSOR\fP to \fBYYLIMIT\fP in order to determine if there is enough input characters left. \fBYYLIMIT\fP is used only in the default C API. .TP .B \fBYYMARKER\fP A pointer\-like l\-value (usually a pointer of type \fBYYCTYPE*\fP) that stores the position of the latest matched rule. It is used to restores \fBYYCURSOR\fP position if the longer match fails and lexer needs to rollback. Initialization is not needed. \fBYYMARKER\fP is used only in the default C API. .TP .B \fBYYCTXMARKER\fP A pointer\-like l\-value that stores the position of the trailing context (usually a pointer of type \fBYYCTYPE*\fP). No initialization is needed. It is used only in the default C API, and only with the lookahead operator \fB/\fP\&. .TP .B \fBYYFILL\fP API primitive with one argument \fBlen\fP\&. The meaning of \fBYYFILL\fP is to provide at least \fBlen\fP more input characters or fail. If EOF rule is used, \fBYYFILL\fP should always return to the calling function; the return value should be zero on success and non\-zero on failure. If EOF rule is not used, \fBYYFILL\fP return value is ignored and it should not return on failure. Maximal value of \fBlen\fP is \fBYYMAXFILL\fP, which can be generated with \fB/*!max:re2c*/\fP directive. The definition of \fBYYFILL\fP can be either function\-like or free\-form depending on the API style (see \fBre2c:api:style\fP and \fBre2c:define:YYFILL:naked\fP). .TP .B \fBYYMAXFILL\fP An integral constant equal to the maximal value of \fBYYFILL\fP argument. It can be generated with \fB/*!max:re2c*/\fP directive. .TP .B \fBYYLESSTHAN\fP A generic API primitive with one argument \fBlen\fP\&. It should be defined as an r\-value of boolean type that equals \fBtrue\fP if and only if there is less than \fBlen\fP input characters left. The definition can be either function\-like or free\-form depending on the API style (see \fBre2c:api:style\fP). .TP .B \fBYYPEEK\fP A generic API primitive with no arguments. It should be defined as an r\-value of type \fBYYCTYPE\fP that is equal to the character at the current input position. The definition can be either function\-like or free\-form depending on the API style (see \fBre2c:api:style\fP). .TP .B \fBYYSKIP\fP A generic API primitive with no arguments. The meaning of \fBYYSKIP\fP is to advance the current input position by one character. The definition can be either function\-like or free\-form depending on the API style (see \fBre2c:api:style\fP). .TP .B \fBYYBACKUP\fP A generic API primitive with no arguments. The meaning of \fBYYBACKUP\fP is to save the current input position, which is later restored with \fBYYRESTORE\fP\&. The definition should be either function\-like or free\-form depending on the API style (see \fBre2c:api:style\fP). .TP .B \fBYYRESTORE\fP A generic API primitive with no arguments. The meaning of \fBYYRESTORE\fP is to restore the current input position to the value saved by \fBYYBACKUP\fP\&. The definition should be either function\-like or free\-form depending on the API style (see \fBre2c:api:style\fP). .TP .B \fBYYBACKUPCTX\fP A generic API primitive with zero arguments. The meaning of \fBYYBACKUPCTX\fP is to save the current input position as the position of the trailing context, which is later restored by \fBYYRESTORECTX\fP\&. The definition should be either function\-like or free\-form depending on the API style (see \fBre2c:api:style\fP). .TP .B \fBYYRESTORECTX\fP A generic API primitive with no arguments. The meaning of \fBYYRESTORECTX\fP is to restore the trailing context position saved with \fBYYBACKUPCTX\fP\&. The definition should be either function\-like or free\-form depending on the API style (see \fBre2c:api:style\fP). .TP .B \fBYYRESTORETAG\fP A generic API primitive with one argument \fBtag\fP\&. The meaning of \fBYYRESTORETAG\fP is to restore the trailing context position to the value of \fBtag\fP\&. The definition should be either function\-like or free\-form depending on the API style (see \fBre2c:api:style\fP). .TP .B \fBYYSTAGP\fP A generic API primitive with one argument \fBtag\fP\&. The meaning of \fBYYSTAGP\fP is to set \fBtag\fP value to the current input position. The definition should be either function\-like or free\-form depending on the API style (see \fBre2c:api:style\fP). .TP .B \fBYYSTAGN\fP A generic API primitive with one argument \fBtag\fP\&. The meaning of \fBYYSTAGN\fP is to set \fBtag\fP value to null (or some default value). The definition should be either function\-like or free\-form depending on the API style (see \fBre2c:api:style\fP). .TP .B \fBYYMTAGP\fP A generic API primitive with one argument \fBtag\fP\&. The meaning of \fBYYMTAGP\fP is to append the current position to the history of \fBtag\fP\&. The definition should be either function\-like or free\-form depending on the API style (see \fBre2c:api:style\fP). .TP .B \fBYYMTAGN\fP A generic API primitive with one argument \fBtag\fP\&. The meaning of \fBYYMTAGN\fP is to append null (or some other default) value to the history of \fBtag\fP\&. The definition can be either function\-like or free\-form depending on the API style (see \fBre2c:api:style\fP). .TP .B \fBYYSHIFT\fP A generic API primitive with one argument \fBshift\fP\&. The meaning of \fBYYSHIFT\fP is to shift the current input position by \fBshift\fP characters (the shift value may be negative). The definition can be either function\-like or free\-form depending on the API style (see \fBre2c:api:style\fP). .TP .B \fBYYSHIFTSTAG\fP A generic API primitive with two arguments, \fBtag\fP and \fBshift\fP\&. The meaning of \fBYYSHIFTSTAG\fP is to shift \fBtag\fP by \fBshift\fP characters (the shift value may be negative). The definition can be either function\-like or free\-form depending on the API style (see \fBre2c:api:style\fP). .TP .B \fBYYSHIFTMTAG\fP A generic API primitive with two arguments, \fBtag\fP and \fBshift\fP\&. The meaning of \fBYYSHIFTMTAG\fP is to shift the latest value in the history of \fBtag\fP by \fBshift\fP characters (the shift value may be negative). The definition should be either function\-like or free\-form depending on the API style (see \fBre2c:api:style\fP). .TP .B \fBYYMAXNMATCH\fP An integral constant equal to the maximal number of POSIX capturing groups in a rule. It is generated with \fB/*!maxnmatch:re2c*/\fP directive. .TP .B \fBYYCONDTYPE\fP The type of the condition enum. It should be generated either with \fB/*!types:re2c*/\fP directive or \fB\-t\fP \fB\-\-type\-header\fP option. .TP .B \fBYYGETCONDITION\fP An API primitive with zero arguments. It should be defined as an r\-value of type \fBYYCONDTYPE\fP that is equal to the current condition identifier. The definition can be either function\-like or free\-form depending on the API style (see \fBre2c:api:style\fP and \fBre2c:define:YYGETCONDITION:naked\fP). .TP .B \fBYYSETCONDITION\fP An API primitive with one argument \fBcond\fP\&. The meaning of \fBYYSETCONDITION\fP is to set the current condition identifier to \fBcond\fP\&. The definition should be either function\-like or free\-form depending on the API style (see \fBre2c:api:style\fP and \fBre2c:define:YYSETCONDITION@cond\fP). .TP .B \fBYYGETSTATE\fP An API primitive with zero arguments. It should be defined as an r\-value of integer type that is equal to the current lexer state. Should be initialized to \fB\-1\fP\&. The definition can be either function\-like or free\-form depending on the API style (see \fBre2c:api:style\fP and \fBre2c:define:YYGETSTATE:naked\fP). .TP .B \fBYYSETSTATE\fP An API primitive with one argument \fBstate\fP\&. The meaning of \fBYYSETSTATE\fP is to set the current lexer state to \fBstate\fP\&. The definition should be either function\-like or free\-form depending on the API style (see \fBre2c:api:style\fP and \fBre2c:define:YYSETSTATE@state\fP). .TP .B \fBYYDEBUG\fP A debug API primitive with two arguments. It can be used to debug the generated code (with \fB\-d\fP \fB\-\-debug\-output\fP option). \fBYYDEBUG\fP should return no value and accept two arguments: \fBstate\fP (either a DFA state index or \fB\-1\fP) and \fBsymbol\fP (the current input symbol). .TP .B \fByych\fP An l\-value of type \fBYYCTYPE\fP that stores the current input character. User definition is necessary only with \fB\-f\fP \fB\-\-storable\-state\fP option. .TP .B \fByyaccept\fP An l\-value of unsigned integral type that stores the number of the latest matched rule. User definition is necessary only with \fB\-f\fP \fB\-\-storable\-state\fP option. .TP .B \fByynmatch\fP An l\-value of unsigned integral type that stores the number of POSIX capturing groups in the matched rule. Used only with \fB\-P\fP \fB\-\-posix\-captures\fP option. .TP .B \fByypmatch\fP An array of l\-values that are used to hold the tag values corresponding to the capturing parentheses in the matching rule. Array length must be at least \fByynmatch * 2\fP (usually \fBYYMAXNMATCH * 2\fP is a good choice). Used only with \fB\-P\fP \fB\-\-posix\-captures\fP option. .UNINDENT .SH CONFIGURATIONS .INDENT 0.0 .TP .B \fBre2c:flags:t\fP, \fBre2c:flags:type\-header\fP Specify the name of the generated header file relative to the directory of the output file. (Same as \fB\-t\fP, \fB\-\-type\-header\fP command\-line option except that the filepath is relative.) .TP .B \fBre2c:flags:input\fP Same as \fB\-\-input\fP command\-line option. .TP .B \fBre2c:api:style\fP Allows one to specify the style of generic API. Possible values are \fBfunctions\fP and \fBfree\-form\fP\&. With \fBfunctions\fP style (the default for the C backend) API primitives behave like functions, and re2c generates parentheses with an argument list after the name of each primitive. With \fBfree\-form\fP style (the default for the Go backend) re2c treats API definitions as interpolated strings and substitutes argument placeholders with the actual argument values. This option can be overridden by options for individual API primitives, e.g. \fBre2c:define:YYFILL:naked\fP for \fBYYFILL\fP\&. .TP .B \fBre2c:api:sigil\fP Allows one to specify the "sigil" symbol (or string) that is used to recognize argument placeholders in the definitions of generic API primitives. The default value is \fB@@\fP\&. Placeholders start with sigil, followed by the argument name in curly braces. For example, if sigil is set to \fB$\fP, then placeholders will have the form \fB${name}\fP\&. Single\-argument APIs may use shorthand notation without the name in braces. This option can be overridden by options for individual API primitives, e.g. \fBre2c:define:YYFILL@len\fP for \fBYYFILL\fP\&. .TP .B \fBre2c:define:YYCTYPE\fP Defines \fBYYCTYPE\fP (see the user interface section). .TP .B \fBre2c:define:YYCURSOR\fP Defines C API primitive \fBYYCURSOR\fP (see the user interface section). .TP .B \fBre2c:define:YYLIMIT\fP Defines C API primitive \fBYYLIMIT\fP (see the user interface section). .TP .B \fBre2c:define:YYMARKER\fP Defines C API primitive \fBYYMARKER\fP (see the user interface section). .TP .B \fBre2c:define:YYCTXMARKER\fP Defines C API primitive \fBYYCTXMARKER\fP (see the user interface section). .TP .B \fBre2c:define:YYFILL\fP Defines API primitive \fBYYFILL\fP (see the user interface section). .TP .B \fBre2c:define:YYFILL@len\fP Specifies the sigil used for argument substitution in \fBYYFILL\fP definition. Defaults to \fB@@\fP\&. Overrides the more generic \fBre2c:api:sigil\fP configuration. .TP .B \fBre2c:define:YYFILL:naked\fP Allows one to override \fBre2c:api:style\fP for \fBYYFILL\fP\&. Value \fB0\fP corresponds to free\-form API style. .TP .B \fBre2c:yyfill:enable\fP Defaults to \fB1\fP (\fBYYFILL\fP is enabled). Set this to zero to suppress the generation of \fBYYFILL\fP\&. Use warnings (\fB\-W\fP option) and \fBre2c:sentinel\fP configuration to verify that the generated lexer cannot read past the end of input, as this might introduce severe security issues to your programs. .TP .B \fBre2c:yyfill:parameter\fP Controls the argument in the parentheses that follow \fBYYFILL\fP\&. Defaults to \fB1\fP, which means that the argument is generated. If zero, the argument is omitted. Can be overridden with \fBre2c:define:YYFILL:naked\fP or \fBre2c:api:style\fP\&. .TP .B \fBre2c:eof\fP Specifies the sentinel symbol used with EOF rule \fB$\fP to check for the end of input in the generated lexer. The default value is \fB\-1\fP (EOF rule is not used). Other possible values include all valid code units. Only decimal numbers are recognized. .TP .B \fBre2c:sentinel\fP Specifies the sentinel symbol used with the sentinel method of checking for the end of input in the generated lexer (the case when bounds checking is disabled with \fBre2c:yyfill:enable = 0;\fP and EOF rule \fB$\fP is not used). This configuration does not affect code generation. It is used by re2c to verify that the sentinel symbol is not allowed in the middle of the rule, and prevent possible reads past the end of buffer in the generated lexer. The default value is \fB\-1\fP (re2c assumes that the sentinel symbol is \fB0\fP, which is the most common case). Other possible values include all valid code units. Only decimal numbers are recognized. .TP .B \fBre2c:define:YYLESSTHAN\fP Defines generic API primitive \fBYYLESSTHAN\fP (see the user interface section). .TP .B \fBre2c:yyfill:check\fP Setting this to zero allows to suppress the generation of \fBYYFILL\fP check (\fBYYLESSTHAN\fP in generic API of \fBYYLIMIT\fP\-based comparison in default C API). This configuration is useful when the necessary input is always available. it defaults to \fB1\fP (the check is generated). .TP .B \fBre2c:label:yyFillLabel\fP Allows one to change the prefix of \fBYYFILL\fP labels (used with EOF rule or with storable states). .TP .B \fBre2c:define:YYPEEK\fP Defines generic API primitive \fBYYPEEK\fP (see the user interface section). .TP .B \fBre2c:define:YYSKIP\fP Defines generic API primitive \fBYYSKIP\fP (see the user interface section). .TP .B \fBre2c:define:YYBACKUP\fP Defines generic API primitive \fBYYBACKUP\fP (see the user interface section). .TP .B \fBre2c:define:YYBACKUPCTX\fP Defines generic API primitive \fBYYBACKUPCTX\fP (see the user interface section). .TP .B \fBre2c:define:YYRESTORE\fP Defines generic API primitive \fBYYRESTORE\fP (see the user interface section). .TP .B \fBre2c:define:YYRESTORECTX\fP Defines generic API primitive \fBYYRESTORECTX\fP (see the user interface section). .TP .B \fBre2c:define:YYRESTORETAG\fP Defines generic API primitive \fBYYRESTORETAG\fP (see the user interface section). .TP .B \fBre2c:define:YYSHIFT\fP Defines generic API primitive \fBYYSHIFT\fP (see the user interface section). .TP .B \fBre2c:define:YYSHIFTMTAG\fP Defines generic API primitive \fBYYSHIFTMTAG\fP (see the user interface section). .TP .B \fBre2c:define:YYSHIFTSTAG\fP Defines generic API primitive \fBYYSHIFTSTAG\fP (see the user interface section). .TP .B \fBre2c:define:YYSTAGN\fP Defines generic API primitive \fBYYSTAGN\fP (see the user interface section). .TP .B \fBre2c:define:YYSTAGP\fP Defines generic API primitive \fBYYSTAGP\fP (see the user interface section). .TP .B \fBre2c:define:YYMTAGN\fP Defines generic API primitive \fBYYMTAGN\fP (see the user interface section). .TP .B \fBre2c:define:YYMTAGP\fP Defines generic API primitive \fBYYMTAGP\fP (see the user interface section). .TP .B \fBre2c:flags:T\fP, \fBre2c:flags:tags\fP Same as \fB\-T \-\-tags\fP command\-line option. .TP .B \fBre2c:flags:P\fP, \fBre2c:flags:posix\-captures\fP Same as \fB\-P \-\-posix\-captures\fP command\-line option. .TP .B \fBre2c:tags:expression\fP Allows one to customize the way re2c addresses tag variables. By default re2c generates expressions of the form \fByyt<N>\fP\&. This might be inconvenient, for example if tag variables are defined as fields in a struct. Re2c recognizes placeholder of the form \fB@@{tag}\fP or \fB@@\fP and replaces it with the actual tag name. Sigil \fB@@\fP can be redefined with \fBre2c:api:sigil\fP configuration. For example, setting \fBre2c:tags:expression = "p\->@@";\fP results in expressions of the form \fBp\->yyt<N>\fP in the generated code. .TP .B \fBre2c:tags:prefix\fP Allows one to override the prefix of tag variables (defaults to \fByyt\fP). .TP .B \fBre2c:flags:lookahead\fP Same as inverted \fB\-\-no\-lookahead\fP command\-line option. .TP .B \fBre2c:flags:optimize\-tags\fP Same as inverted \fB\-\-no\-optimize\-tags\fP command\-line option. .TP .B \fBre2c:define:YYCONDTYPE\fP Defines \fBYYCONDTYPE\fP (see the user interface section). .TP .B \fBre2c:define:YYGETCONDITION\fP Defines API primitive \fBYYGETCONDITION\fP (see the user interface section). .TP .B \fBre2c:define:YYGETCONDITION:naked\fP Allows one to override \fBre2c:api:style\fP for \fBYYGETCONDITION\fP\&. Value \fB0\fP corresponds to free\-form API style. .TP .B \fBre2c:define:YYSETCONDITION\fP Defines API primitive \fBYYSETCONDITION\fP (see the user interface section). .TP .B \fBre2c:define:YYSETCONDITION@cond\fP Specifies the sigil used for argument substitution in \fBYYSETCONDITION\fP definition. The default value is \fB@@\fP\&. Overrides the more generic \fBre2c:api:sigil\fP configuration. .TP .B \fBre2c:define:YYSETCONDITION:naked\fP Allows one to override \fBre2c:api:style\fP for \fBYYSETCONDITION\fP\&. Value \fB0\fP corresponds to free\-form API style. .TP .B \fBre2c:cond:goto\fP Allows one to customize the goto statements used with the shortcut \fB:=>\fP rules in conditions. The default value is \fBgoto @@;\fP\&. Placeholders are substituted with condition name (see \fBre2c:api;sigil\fP and \fBre2c:cond:goto@cond\fP). .TP .B \fBre2c:cond:goto@cond\fP Specifies the sigil used for argument substitution in \fBre2c:cond:goto\fP definition. The default value is \fB@@\fP\&. Overrides the more generic \fBre2c:api:sigil\fP configuration. .TP .B \fBre2c:cond:divider\fP Defines the divider for condition blocks. The default value is \fB/* *********************************** */\fP\&. Placeholders are substituted with condition name (see \fBre2c:api;sigil\fP and \fBre2c:cond:divider@cond\fP). .TP .B \fBre2c:cond:divider@cond\fP Specifies the sigil used for argument substitution in \fBre2c:cond:divider\fP definition. The default value is \fB@@\fP\&. Overrides the more generic \fBre2c:api:sigil\fP configuration. .TP .B \fBre2c:condprefix\fP Specifies the prefix used for condition labels. The default value is \fByyc_\fP\&. .TP .B \fBre2c:condenumprefix\fP Specifies the prefix used for condition identifiers. The default value is \fByyc\fP\&. .TP .B \fBre2c:define:YYGETSTATE\fP Defines API primitive \fBYYGETSTATE\fP (see the user interface section). .TP .B \fBre2c:define:YYGETSTATE:naked\fP Allows one to override \fBre2c:api:style\fP for \fBYYGETSTATE\fP\&. Value \fB0\fP corresponds to free\-form API style. .TP .B \fBre2c:define:YYSETSTATE\fP Defines API primitive \fBYYSETSTATE\fP (see the user interface section). .TP .B \fBre2c:define:YYSETSTATE@state\fP Specifies the sigil used for argument substitution in \fBYYSETSTATE\fP definition. The default value is \fB@@\fP\&. Overrides the more generic \fBre2c:api:sigil\fP configuration. .TP .B \fBre2c:define:YYSETSTATE:naked\fP Allows one to override \fBre2c:api:style\fP for \fBYYSETSTATE\fP\&. Value \fB0\fP corresponds to free\-form API style. .TP .B \fBre2c:state:abort\fP If set to a positive integer value, changes the form of the \fBYYGETSTATE\fP switch: instead of using default case to jump to the beginning of the lexer block, a \fB\-1\fP case is used, and the default case aborts the program. .TP .B \fBre2c:state:nextlabel\fP With storable states, allows to control if the \fBYYGETSTATE\fP block is followed by a \fByyNext\fP label (the default value is zero, which corresponds to no label). Instead of using \fByyNext\fP it is possible to use \fBre2c:startlabel\fP to force the generation of a specific start label. Instead of using labels it is often more convenient to generate \fBYYGETSTATE\fP code using \fB/*!getstate:re2c*/\fP\&. .TP .B \fBre2c:label:yyNext\fP Allows one to change the name of the \fByyNext\fP label. .TP .B \fBre2c:startlabel\fP Controls the generation of start label for the next lexer block. The default value is zero, which means that the start label is generated only if it is used. An integer value greater than zero forces the generation of start label even if it is unused by the lexer. A string value also forces start label generation and sets the label name to the specified string. This configuration applies only to the current block (it is reset to default for the next block). .TP .B \fBre2c:flags:s\fP, \fBre2c:flags:nested\-ifs\fP Same as \fB\-s \-\-nested\-ifs\fP command\-line option. .TP .B \fBre2c:flags:b\fP, \fBre2c:flags:bit\-vectors\fP Same as \fB\-b \-\-bit\-vectors\fP command\-line option. .TP .B \fBre2c:variable:yybm\fP Overrides the name of the \fByybm\fP variable. .TP .B \fBre2c:yybm:hex\fP Defaults to zero (a decimal bitmap table is generated). If set to nonzero, a hexadecimal table is generated. .TP .B \fBre2c:flags:g\fP, \fBre2c:flags:computed\-gotos\fP Same as \fB\-g \-\-computed\-gotos\fP command\-line option. .TP .B \fBre2c:cgoto:threshold\fP With \fB\-g\fP \fB\-\-computed\-gotos\fP option this value specifies the complexity threshold that triggers the generation of jump tables instead of nested \fBif\fP statements and bitmaps. The default value is \fB9\fP\&. .TP .B \fBre2c:flags:case\-ranges\fP Same as \fB\-\-case\-ranges\fP command\-line option. .TP .B \fBre2c:flags:e\fP, \fBre2c:flags:ecb\fP Same as \fB\-e \-\-ecb\fP command\-line option. .TP .B \fBre2c:flags:8\fP, \fBre2c:flags:utf\-8\fP Same as \fB\-8 \-\-utf\-8\fP command\-line option. .TP .B \fBre2c:flags:w\fP, \fBre2c:flags:wide\-chars\fP Same as \fB\-w \-\-wide\-chars\fP command\-line option. .TP .B \fBre2c:flags:x\fP, \fBre2c:flags:utf\-16\fP Same as \fB\-x \-\-utf\-16\fP command\-line option. .TP .B \fBre2c:flags:u\fP, \fBre2c:flags:unicode\fP Same as \fB\-u \-\-unicode\fP command\-line option. .TP .B \fBre2c:flags:encoding\-policy\fP Same as \fB\-\-encoding\-policy\fP command\-line option. .TP .B \fBre2c:flags:empty\-class\fP Same as \fB\-\-empty\-class\fP command\-line option. .TP .B \fBre2c:flags:case\-insensitive\fP Same as \fB\-\-case\-insensitive\fP command\-line option. .TP .B \fBre2c:flags:case\-inverted\fP Same as \fB\-\-case\-inverted\fP command\-line option. .TP .B \fBre2c:indent:string\fP Specifies the string to use for indentation. The default value is \fB"\et"\fP\&. Indent string should contain only whitespace characters. To disable indentation entirely, set this configuration to empty string \fB""\fP\&. .TP .B \fBre2c:indent:top\fP Specifies the minimum amount of indentation to use. The default value is zero. The value should be a non\-negative integer number. .TP .B \fBre2c:labelprefix\fP Allows one to change the prefix of DFA state labels. The default value is \fByy\fP\&. .TP .B \fBre2c:yych:emit\fP Set this to zero to suppress the generation of \fByych\fP definition. Defaults to \fB1\fP (the definition is generated). .TP .B \fBre2c:variable:yych\fP Overrides the name of the \fByych\fP variable. .TP .B \fBre2c:yych:conversion\fP If set to nonzero, re2c automatically generates a cast to \fBYYCTYPE\fP every time \fByych\fP is read. Defaults to zero (no cast). .TP .B \fBre2c:variable:yyaccept\fP Overrides the name of the \fByyaccept\fP variable. .TP .B \fBre2c:variable:yytarget\fP Overrides the name of the \fByytarget\fP variable. .TP .B \fBre2c:variable:yystable\fP Deprecated. .TP .B \fBre2c:variable:yyctable\fP When both \fB\-c\fP \fB\-\-conditions\fP and \fB\-g\fP \fB\-\-computed\-gotos\fP are active, re2c will use this variable to generate a static jump table for \fBYYGETCONDITION\fP\&. .TP .B \fBre2c:define:YYDEBUG\fP Defines \fBYYDEBUG\fP (see the user interface section). .TP .B \fBre2c:flags:d\fP, \fBre2c:flags:debug\-output\fP Same as \fB\-d \-\-debug\-output\fP command\-line option. .TP .B \fBre2c:flags:dfa\-minimization\fP Same as \fB\-\-dfa\-minimization\fP command\-line option. .TP .B \fBre2c:flags:eager\-skip\fP Same as \fB\-\-eager\-skip\fP command\-line option. .UNINDENT .SH REGULAR EXPRESSIONS .sp re2c uses the following syntax for regular expressions: .INDENT 0.0 .IP \(bu 2 \fB"foo"\fP case\-sensitive string literal .IP \(bu 2 \fB\(aqfoo\(aq\fP case\-insensitive string literal .IP \(bu 2 \fB[a\-xyz]\fP, \fB[^a\-xyz]\fP character class (possibly negated) .IP \(bu 2 \fB\&.\fP any character except newline .IP \(bu 2 \fBR \e S\fP difference of character classes \fBR\fP and \fBS\fP .IP \(bu 2 \fBR*\fP zero or more occurrences of \fBR\fP .IP \(bu 2 \fBR+\fP one or more occurrences of \fBR\fP .IP \(bu 2 \fBR?\fP optional \fBR\fP .IP \(bu 2 \fBR{n}\fP repetition of \fBR\fP exactly \fBn\fP times .IP \(bu 2 \fBR{n,}\fP repetition of \fBR\fP at least \fBn\fP times .IP \(bu 2 \fBR{n,m}\fP repetition of \fBR\fP from \fBn\fP to \fBm\fP times .IP \(bu 2 \fB(R)\fP just \fBR\fP; parentheses are used to override precedence or for POSIX\-style submatch .IP \(bu 2 \fBR S\fP concatenation: \fBR\fP followed by \fBS\fP .IP \(bu 2 \fBR | S\fP alternative: \fBR or S\fP .IP \(bu 2 \fBR / S\fP lookahead: \fBR\fP followed by \fBS\fP, but \fBS\fP is not consumed .IP \(bu 2 \fBname\fP the regular expression defined as \fBname\fP (or literal string \fB"name"\fP in Flex compatibility mode) .IP \(bu 2 \fB{name}\fP the regular expression defined as \fBname\fP in Flex compatibility mode .IP \(bu 2 \fB@stag\fP an \fIs\-tag\fP: saves the last input position at which \fB@stag\fP matches in a variable named \fBstag\fP .IP \(bu 2 \fB#mtag\fP an \fIm\-tag\fP: saves all input positions at which \fB#mtag\fP matches in a variable named \fBmtag\fP .UNINDENT .sp Character classes and string literals may contain the following escape sequences: \fB\ea\fP, \fB\eb\fP, \fB\ef\fP, \fB\en\fP, \fB\er\fP, \fB\et\fP, \fB\ev\fP, \fB\e\e\fP, octal escapes \fB\eooo\fP and hexadecimal escapes \fB\exhh\fP, \fB\euhhhh\fP and \fB\eUhhhhhhhh\fP\&. .SH HANDLING THE END OF INPUT .sp One of the main problems for the lexer is to know when to stop. There are a few terminating conditions: .INDENT 0.0 .IP \(bu 2 the lexer may match some rule (including default rule \fB*\fP) and come to a final state .IP \(bu 2 the lexer may fail to match any rule and come to a default state .IP \(bu 2 the lexer may reach the end of input .UNINDENT .sp The first two conditions terminate the lexer in a "natural" way: it comes to a state with no outgoing transitions, and the matching automatically stops. The third condition, end of input, is different: it may happen in any state, and the lexer should be able to handle it. Checking for the end of input interrupts the normal lexer workflow and adds conditional branches to the generated program, therefore it is necessary to minimize the number of such checks. re2c supports a few different methods for end of input handling. Which one to use depends on the complexity of regular expressions, the need for buffering, performance considerations and other factors. Here is a list of all methods: .INDENT 0.0 .IP \(bu 2 \fBSentinel character.\fP This method eliminates the need for the end of input checks altogether. It is simple and efficient, but limited to the case when there is a natural "sentinel" character that can never occur in valid input. This character may still occur in invalid input, but it is not allowed by the regular expressions, except perhaps as the last character of a rule. The sentinel character is appended at the end of input and serves as a stop signal: when the lexer reads it, it must be either the end of input, or a syntax error. In both cases the lexer stops. This method is used if \fBYYFILL\fP is disabled with \fBre2c:yyfill:enable = 0;\fP and \fBre2c:eof\fP has the default value \-1. .nf .fi .sp .IP \(bu 2 \fBSentinel character with bounds checks.\fP This method is generic: it allows to handle any input without restrictions on the regular expressions. The idea is to reduce the number of end of input checks by performing them only on certain characters. Similar to the "sentinel character" method, one of the characters is chosen as a "sentinel" and appended at the end of input. However, there is no restriction on where the sentinel character may occur (in fact, any character can be chosen for a sentinel). When the lexer reads this character, it additionally performs a bounds check. If the current position is within bounds, the lexer will resume matching and handle the sentinel character as a regular one. Otherwise it will try to get more input with \fBYYFILL\fP (unless \fBYYFILL\fP is disabled). If more input is available, the lexer will rematch the last character and continue as if the sentinel never occurred. Otherwise it is the real end of input, and the lexer will stop. This method is used if \fBre2c:eof\fP has non\-negative value (it should be set to the ordinal of the sentinel character). \fBYYFILL\fP must be either defined or disabled with \fBre2c:yyfill:enable = 0;\fP\&. .nf .fi .sp .IP \(bu 2 \fBBounds checks with padding.\fP This method is the default one. It is generic, and it is usually faster than the "sentinel character with bounds checks" method, but also more complex to use. The idea is to partition the underlying finite\-state automaton into strongly connected components (SCCs), and generate only one bounds check per SCC, but make it check for multiple characters at once (enough to cover the longest non\-looping path in the SCC). This way the checks are less frequent, which makes the lexer run much faster. If a check shows that there is not enough input, the lexer will invoke \fBYYFILL\fP, which may either supply enough input or else it should not return (in the latter case the lexer will stop). This approach has a problem with matching short lexemes at the end of input, because the multi\-character check requires enough characters to cover the longest possible lexeme. To fix this problem, it is necessary to append a few fake characters at the end of input. The padding should not form a valid lexeme suffix to avoid fooling the lexer into matching it as part of the input. The minimum sufficient length of padding is \fBYYMAXFILL\fP and it is autogenerated by re2c with \fB/*!max:re2c*/\fP\&. This method is used if \fBre2c:yyfill:enable\fP has the default nonzero value, and \fBre2c:eof\fP has the default value \-1. \fBYYFILL\fP must be defined. .nf .fi .sp .IP \(bu 2 \fBCustom methods with generic API.\fP Generic API allows to override basic operations like reading a character, which makes it possible to include the end of input checks as part of them. Such methods are error\-prone and should be used with caution, only if other methods cannot be used. These methods are used if generic API is enabled with \fB\-\-input custom\fP or \fBre2c:flags:input = custom;\fP and default bounds checks are disabled with \fBre2c:yyfill:enable = 0;\fP\&. Note that the use of generic API does not imply the use of custom methods, it merely allows it. .UNINDENT .sp The following subsections contain an example of each method. .SS Sentinel character .sp In this example the lexer uses a sentinel character to handle the end of input. The program counts space\-separated words in a null\-terminated string. Configuration \fBre2c:yyfill:enable = 0;\fP suppresses the generation of bounds checks and \fBYYFILL\fP invocations. The sentinel character is null. It is the last character of each input string, and it is not allowed in the middle of a lexeme by any of the rules (in particular, it is not included in the character ranges, where it is easy to overlook). If a null occurs in the middle of a string, it is a syntax error and the lexer will match default rule \fB*\fP, but it won\(aqt read past the end of input or crash. \fI\%\-Wsentinel\-in\-midrule\fP warning verifies that the rules do not allow sentinel in the middle (it is possible to tell re2c which character is used as a sentinel with \fBre2c:sentinel\fP configuration \-\-\- the default assumption is null, since this is the most common case). .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C // re2c $INPUT \-o $OUTPUT #include <assert.h> // expect a null\-terminated string static int lex(const char *YYCURSOR) { int count = 0; loop: /*!re2c re2c:define:YYCTYPE = char; re2c:yyfill:enable = 0; * { return \-1; } [\ex00] { return count; } [a\-z]+ { ++count; goto loop; } [ ]+ { goto loop; } */ } int main() { assert(lex("") == 0); assert(lex("one two three") == 3); assert(lex("f0ur") == \-1); return 0; } .ft P .fi .UNINDENT .UNINDENT .SS Sentinel character with bounds checks .sp In this example the lexer uses sentinel character with bounds checks to handle the end of input (this method was added in version 1.2). The program counts single\-quoted strings separated with spaces. The sentinel character is null, which is specified with \fBre2c:eof = 0;\fP configuration. Null is the last character of each input string \-\-\- this is essential to detect the end of input. Null, as well as any other character, is allowed in the middle of a rule (for example, \fB\(aqaaa\e0aa\(aq\e0\fP is valid input, but \fB\(aqaaa\e0\fP is a syntax error). Bounds checks are generated in each state that has a switch on an input character, in the conditional branch that corresponds to null (that branch may also cover other characters \-\-\- re2c does not split out a separate branch for sentinel, because increasing the number of branches degrades performance more than bounds checks do). Bounds checks are of the form \fBYYLIMIT <= YYCURSOR\fP or \fBYYLESSTHAN(1)\fP with generic API. If a bounds check succeeds, the lexer will continue matching. If a bounds check fails, the lexer has reached the end of input, and it should stop. In this example \fBYYFILL\fP is disabled with \fBre2c:yyfill:enable = 0;\fP and the lexer does not attempt to get more input (see another example that uses \fBYYFILL\fP in the \fI\%YYFILL with sentinel character\fP section). When the end of input has been reached, there are three possibilities: if the lexer is in the initial state, it will match the end of input rule \fB$\fP, otherwise it will either fallback to a previously matched rule (including default rule \fB*\fP) or go to a default state, causing \fI\%\-Wundefined\-control\-flow\fP\&. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C // re2c $INPUT \-o $OUTPUT #include <assert.h> // expect a null\-terminated string static int lex(const char *str, unsigned int len) { const char *YYCURSOR = str, *YYLIMIT = str + len, *YYMARKER; int count = 0; loop: /*!re2c re2c:define:YYCTYPE = char; re2c:yyfill:enable = 0; re2c:eof = 0; * { return \-1; } $ { return count; } [\(aq] ([^\(aq\e\e] | [\e\e][^])* [\(aq] { ++count; goto loop; } [ ]+ { goto loop; } */ } #define TEST(s, r) assert(lex(s, sizeof(s) \- 1) == r) int main() { TEST("", 0); TEST("\(aqqu\e0tes\(aq \(aqare\(aq \(aqfine: \e\e\(aq\(aq ", 3); TEST("\(aqunterminated\e\e\(aq", \-1); return 0; } .ft P .fi .UNINDENT .UNINDENT .SS Bounds checks with padding .sp In this example the lexer uses bounds checking with padding to handle the end of input (it is the default method). The program counts single\-quoted strings separated with spaces. There is a padding of \fBYYMAXFILL\fP null characters appended at the end of input, where \fBYYMAXFILL\fP value is autogenerated with \fB/*!max:re2c*/\fP directive. It is not necessary to use null for padding \-\-\- any characters can be used, as long as they do not form a valid lexeme suffix (in this example padding should not contain single quotes, as they may be mistaken for a suffix of a single\-quoted string). There is a "stop" rule that matches the first padding character (null) and terminates the lexer (it returns success only if it has matched at the beginning of padding, otherwise a stray null is syntax error). Bounds checks are generated only in some states that depend on the strongly connected components of the underlying automaton. They are of the form \fB(YYLIMIT \- YYCURSOR) < n\fP or \fBYYLESSTHAN(n)\fP with generic API, where \fBn\fP is the minimum number of characters that are needed for the lexer to proceed (it also means that the next bounds check will occur in at most \fBn\fP characters). If a bounds check succeeds, the lexer will continue matching. If a bounds check fails, the lexer has reached the end of input and will invoke \fBYYFILL(n)\fP, which should either supply at least \fBn\fP input characters, or it should not return. In this example \fBYYFILL\fP always fails and terminates the lexer with an error. This is fine, because in this example \fBYYFILL\fP can only be called when the lexer has advanced into the padding, which means that is has encountered an unterminated string and should return a syntax error. See the \fI\%YYFILL with padding\fP section for an example that refills the input buffer with \fBYYFILL\fP\&. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C // re2c $INPUT \-o $OUTPUT #include <assert.h> #include <stdlib.h> #include <string.h> /*!max:re2c*/ // expect YYMAXFILL\-padded string static int lex(const char *str, unsigned int len) { const char *YYCURSOR = str, *YYLIMIT = str + len + YYMAXFILL; int count = 0; loop: /*!re2c re2c:api:style = free\-form; re2c:define:YYCTYPE = char; re2c:define:YYFILL = "return \-1;"; * { return \-1; } [\ex00] { return YYCURSOR + YYMAXFILL \- 1 == YYLIMIT ? count : \-1; } [\(aq] ([^\(aq\e\e] | [\e\e][^])* [\(aq] { ++count; goto loop; } [ ]+ { goto loop; } */ } // make a copy of the string with YYMAXFILL zeroes at the end static void test(const char *str, unsigned int len, int res) { char *s = (char*) malloc(len + YYMAXFILL); memcpy(s, str, len); memset(s + len, 0, YYMAXFILL); int r = lex(s, len); free(s); assert(r == res); } #define TEST(s, r) test(s, sizeof(s) \- 1, r) int main() { TEST("", 0); TEST("\(aqqu\e0tes\(aq \(aqare\(aq \(aqfine: \e\e\(aq\(aq ", 3); TEST("\(aqunterminated\e\e\(aq", \-1); return 0; } .ft P .fi .UNINDENT .UNINDENT .SS Custom methods with generic API .sp In this example the lexer uses a custom end of input handling method based on generic API. The program counts single\-quoted strings separated with spaces. It is the same as the \fI\%sentinel character with bounds checks\fP example, except that the input is not null\-terminated (so this method can be used if it\(aqs not possible to have any padding at all, not even a single sentinel character). To cover up for the absence of sentinel character at the end of input, \fBYYPEEK\fP is redefined to perform a bounds check before it reads the next input character. This is inefficient, because checks are done very often. If the check succeeds, \fBYYPEEK\fP returns the real character, otherwise it returns a fake sentinel character. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C // re2c $INPUT \-o $OUTPUT #include <assert.h> #include <stdlib.h> #include <string.h> // expect a string without terminating null static int lex(const char *str, unsigned int len) { const char *cur = str, *lim = str + len, *mar; int count = 0; loop: /*!re2c re2c:yyfill:enable = 0; re2c:eof = 0; re2c:flags:input = custom; re2c:api:style = free\-form; re2c:define:YYCTYPE = char; re2c:define:YYLESSTHAN = "cur >= lim"; re2c:define:YYPEEK = "cur < lim ? *cur : 0"; // fake null re2c:define:YYSKIP = "++cur;"; re2c:define:YYBACKUP = "mar = cur;"; re2c:define:YYRESTORE = "cur = mar;"; * { return \-1; } $ { return count; } [\(aq] ([^\(aq\e\e] | [\e\e][^])* [\(aq] { ++count; goto loop; } [ ]+ { goto loop; } */ } // make a copy of the string without terminating null static void test(const char *str, unsigned int len, int res) { char *s = (char*) malloc(len); memcpy(s, str, len); int r = lex(s, len); free(s); assert(r == res); } #define TEST(s, r) test(s, sizeof(s) \- 1, r) int main() { TEST("", 0); TEST("\(aqqu\e0tes\(aq \(aqare\(aq \(aqfine: \e\e\(aq\(aq ", 3); TEST("\(aqunterminated\e\e\(aq", \-1); return 0; } .ft P .fi .UNINDENT .UNINDENT .SH BUFFER REFILLING .sp The need for buffering arises when the input cannot be mapped in memory all at once: either it is too large, or it comes in a streaming fashion (like reading from a socket). The usual technique in such cases is to allocate a fixed\-sized memory buffer and process input in chunks that fit into the buffer. When the current chunk is processed, it is moved out and new data is moved in. In practice it is somewhat more complex, because lexer state consists not of a single input position, but a set of interrelated posiitons: .INDENT 0.0 .IP \(bu 2 cursor: the next input character to be read (\fBYYCURSOR\fP in default API or \fBYYSKIP\fP/\fBYYPEEK\fP in generic API) .IP \(bu 2 limit: the position after the last available input character (\fBYYLIMIT\fP in default API, implicitly handled by \fBYYLESSTHAN\fP in generic API) .IP \(bu 2 marker: the position of the most recent match, if any (\fBYYMARKER\fP in default API or \fBYYBACKUP\fP/\fBYYRESTORE\fP in generic API) .IP \(bu 2 token: the start of the current lexeme (implicit in re2c API, as it is not needed for the normal lexer operation and can be defined and updated by the user) .IP \(bu 2 context marker: the position of the trailing context (\fBYYCTXMARKER\fP in default API or \fBYYBACKUPCTX\fP/\fBYYRESTORECTX\fP in generic API) .IP \(bu 2 tag variables: submatch positions (defined with \fB/*!stags:re2c*/\fP and \fB/*!mtags:re2c*/\fP directives and \fBYYSTAGP\fP/\fBYYSTAGN\fP/\fBYYMTAGP\fP/\fBYYMTAGN\fP in generic API) .UNINDENT .sp Not all these are used in every case, but if used, they must be updated by \fBYYFILL\fP\&. All active positions are contained in the segment between token and cursor, therefore everything between buffer start and token can be discarded, the segment from token and up to limit should be moved to the beginning of buffer, and the free space at the end of buffer should be filled with new data. In order to avoid frequent \fBYYFILL\fP calls it is best to fill in as many input characters as possible (even though fewer characters might suffice to resume the lexer). The details of \fBYYFILL\fP implementation are slightly different depending on which EOF handling method is used: the case of EOF rule is somewhat simpler than the case of bounds\-checking with padding. Also note that if \fB\-f \-\-storable\-state\fP option is used, \fBYYFILL\fP has slightly different semantics (desrbed in the section about storable state). .SS YYFILL with sentinel character .sp If EOF rule is used, \fBYYFILL\fP is a function\-like primitive that accepts no arguments and returns a value which is checked against zero. \fBYYFILL\fP invocation is triggered by condition \fBYYLIMIT <= YYCURSOR\fP in default API and \fBYYLESSTHAN()\fP in generic API. A non\-zero return value means that \fBYYFILL\fP has failed. A successful \fBYYFILL\fP call must supply at least one character and adjust input positions accordingly. Limit must always be set to one after the last input position in buffer, and the character at the limit position must be the sentinel symbol specified by \fBre2c:eof\fP configuration. The pictures below show the relative locations of input positions in buffer before and after \fBYYFILL\fP call (sentinel symbol is marked with \fB#\fP, and the second picture shows the case when there is not enough input to fill the whole buffer). .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C <\-\- shift \-\-> >\-A\-\-\-\-\-\-\-\-\-\-\-\-B\-\-\-\-\-\-\-\-\-C\-\-\-\-\-\-\-\-\-\-\-\-\-D#\-\-\-\-\-\-\-\-\-\-\-E\-> buffer token marker limit, cursor >\-A\-\-\-\-\-\-\-\-\-\-\-\-B\-\-\-\-\-\-\-\-\-C\-\-\-\-\-\-\-\-\-\-\-\-\-D\-\-\-\-\-\-\-\-\-\-\-\-E#\-> buffer, marker cursor limit token <\-\- shift \-\-> >\-A\-\-\-\-\-\-\-\-\-\-\-\-B\-\-\-\-\-\-\-\-\-C\-\-\-\-\-\-\-\-\-\-\-\-\-D#\-\-E (EOF) buffer token marker limit, cursor >\-A\-\-\-\-\-\-\-\-\-\-\-\-B\-\-\-\-\-\-\-\-\-C\-\-\-\-\-\-\-\-\-\-\-\-\-D\-\-\-E#........ buffer, marker cursor limit token .ft P .fi .UNINDENT .UNINDENT .sp Here is an example of a program that reads input file \fBinput.txt\fP in chunks of 4096 bytes and uses EOF rule. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C // re2c $INPUT \-o $OUTPUT #include <assert.h> #include <stdio.h> #include <string.h> #define SIZE 4096 typedef struct { FILE *file; char buf[SIZE + 1], *lim, *cur, *mar, *tok; int eof; } Input; static int fill(Input *in) { if (in\->eof) { return 1; } const size_t free = in\->tok \- in\->buf; if (free < 1) { return 2; } memmove(in\->buf, in\->tok, in\->lim \- in\->tok); in\->lim \-= free; in\->cur \-= free; in\->mar \-= free; in\->tok \-= free; in\->lim += fread(in\->lim, 1, free, in\->file); in\->lim[0] = 0; in\->eof |= in\->lim < in\->buf + SIZE; return 0; } static void init(Input *in, FILE *file) { in\->file = file; in\->cur = in\->mar = in\->tok = in\->lim = in\->buf + SIZE; in\->eof = 0; fill(in); } static int lex(Input *in) { int count = 0; loop: in\->tok = in\->cur; /*!re2c re2c:eof = 0; re2c:api:style = free\-form; re2c:define:YYCTYPE = char; re2c:define:YYCURSOR = in\->cur; re2c:define:YYMARKER = in\->mar; re2c:define:YYLIMIT = in\->lim; re2c:define:YYFILL = "fill(in) == 0"; * { return \-1; } $ { return count; } [\(aq] ([^\(aq\e\e] | [\e\e][^])* [\(aq] { ++count; goto loop; } [ ]+ { goto loop; } */ } int main() { const char *fname = "input"; const char str[] = "\(aqqu\e0tes\(aq \(aqare\(aq \(aqfine: \e\e\(aq\(aq "; FILE *f; Input in; // prepare input file: a few times the size of the buffer, // containing strings with zeroes and escaped quotes f = fopen(fname, "w"); for (int i = 0; i < SIZE; ++i) { fwrite(str, 1, sizeof(str) \- 1, f); } fclose(f); f = fopen(fname, "r"); init(&in, f); assert(lex(&in) == SIZE * 3); fclose(f); remove(fname); return 0; } .ft P .fi .UNINDENT .UNINDENT .SS YYFILL with padding .sp In the default case (when EOF rule is not used) \fBYYFILL\fP is a function\-like primitive that accepts a single argument and does not return any value. \fBYYFILL\fP invocation is triggered by condition \fB(YYLIMIT \- YYCURSOR) < n\fP in default API and \fBYYLESSTHAN(n)\fP in generic API. The argument passed to \fBYYFILL\fP is the minimal number of characters that must be supplied. If it fails to do so, \fBYYFILL\fP must not return to the lexer (for that reason it is best implemented as a macro that returns from the calling function on failure). In case of a successful \fBYYFILL\fP invocation the limit position must be set either to one after the last input position in buffer, or to the end of \fBYYMAXFILL\fP padding (in case \fBYYFILL\fP has successfully read at least \fBn\fP characters, but not enough to fill the entire buffer). The pictures below show the relative locations of input positions in buffer before and after \fBYYFILL\fP invocation (\fBYYMAXFILL\fP padding on the second picture is marked with \fB#\fP symbols). .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C <\-\- shift \-\-> <\-\- need \-\-> >\-A\-\-\-\-\-\-\-\-\-\-\-\-B\-\-\-\-\-\-\-\-\-C\-\-\-\-\-D\-\-\-\-\-\-\-E\-\-\-F\-\-\-\-\-\-\-\-G\-> buffer token marker cursor limit >\-A\-\-\-\-\-\-\-\-\-\-\-\-B\-\-\-\-\-\-\-\-\-C\-\-\-\-\-D\-\-\-\-\-\-\-E\-\-\-F\-\-\-\-\-\-\-\-G\-> buffer, marker cursor limit token <\-\- shift \-\-> <\-\- need \-\-> >\-A\-\-\-\-\-\-\-\-\-\-\-\-B\-\-\-\-\-\-\-\-\-C\-\-\-\-\-D\-\-\-\-\-\-\-E\-F (EOF) buffer token marker cursor limit >\-A\-\-\-\-\-\-\-\-\-\-\-\-B\-\-\-\-\-\-\-\-\-C\-\-\-\-\-D\-\-\-\-\-\-\-E\-F############### buffer, marker cursor limit token <\- YYMAXFILL \-> .ft P .fi .UNINDENT .UNINDENT .sp Here is an example of a program that reads input file \fBinput.txt\fP in chunks of 4096 bytes and uses bounds\-checking with padding. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C // re2c $INPUT \-o $OUTPUT #include <assert.h> #include <stdio.h> #include <string.h> /*!max:re2c*/ #define SIZE 4096 typedef struct { FILE *file; char buf[SIZE + YYMAXFILL], *lim, *cur, *mar, *tok; int eof; } Input; static int fill(Input *in, size_t need) { if (in\->eof) { return 1; } const size_t free = in\->tok \- in\->buf; if (free < need) { return 2; } memmove(in\->buf, in\->tok, in\->lim \- in\->tok); in\->lim \-= free; in\->cur \-= free; in\->mar \-= free; in\->tok \-= free; in\->lim += fread(in\->lim, 1, free, in\->file); if (in\->lim < in\->buf + SIZE) { in\->eof = 1; memset(in\->lim, 0, YYMAXFILL); in\->lim += YYMAXFILL; } return 0; } static void init(Input *in, FILE *file) { in\->file = file; in\->cur = in\->mar = in\->tok = in\->lim = in\->buf + SIZE; in\->eof = 0; fill(in, 1); } static int lex(Input *in) { int count = 0; loop: in\->tok = in\->cur; /*!re2c re2c:api:style = free\-form; re2c:define:YYCTYPE = char; re2c:define:YYCURSOR = in\->cur; re2c:define:YYMARKER = in\->mar; re2c:define:YYLIMIT = in\->lim; re2c:define:YYFILL = "if (fill(in, @@) != 0) return \-1;"; * { return \-1; } [\ex00] { return (in\->lim \- in\->cur == YYMAXFILL \- 1) ? count : \-1; } [\(aq] ([^\(aq\e\e] | [\e\e][^])* [\(aq] { ++count; goto loop; } [ ]+ { goto loop; } */ } int main() { const char *fname = "input"; const char str[] = "\(aqqu\e0tes\(aq \(aqare\(aq \(aqfine: \e\e\(aq\(aq "; FILE *f; Input in; // prepare input file: a few times the size of the buffer, // containing strings with zeroes and escaped quotes f = fopen(fname, "w"); for (int i = 0; i < SIZE; ++i) { fwrite(str, 1, sizeof(str) \- 1, f); } fclose(f); f = fopen(fname, "r"); init(&in, f); assert(lex(&in) == SIZE * 3); fclose(f); remove(fname); return 0; } .ft P .fi .UNINDENT .UNINDENT .SH INCLUDE FILES .sp re2c allows one to include other files using directive \fB/*!include:re2c FILE */\fP or \fB!include FILE ;\fP, where \fBFILE\fP is a path to the file to be included. The first form should be used outside of re2c blocks, and the second form allows one to include a file in the middle of a re2c block. re2c looks for included files in the directory of the including file and in include locations, which can be specified with \fB\-I\fP option. Include directives in re2c work in the same way as C/C++ \fB#include\fP: the contents of \fBFILE\fP are copy\-pasted verbatim in place of the directive. Include files may have further includes of their own. Use \fB\-\-depfile\fP option to track build dependencies of the output file on include files. re2c provides some predefined include files that can be found in the \fBinclude/\fP subdirectory of the project. These files contain definitions that can be useful to other projects (such as Unicode categories) and form something like a standard library for re2c. Below is an example of using include directive. .SS Include file 1 (definitions.h) .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C typedef enum { OK, FAIL } Result; /*!re2c number = [1\-9][0\-9]*; */ .ft P .fi .UNINDENT .UNINDENT .SS Include file 2 (extra_rules.re.inc) .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C // floating\-point numbers frac = [0\-9]* "." [0\-9]+ | [0\-9]+ "."; exp = \(aqe\(aq [+\-]? [0\-9]+; float = frac exp? | [0\-9]+ exp; float { return OK; } .ft P .fi .UNINDENT .UNINDENT .SS Input file .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C // re2c $INPUT \-o $OUTPUT \-i #include <assert.h> /*!include:re2c "definitions.h" */ Result lex(const char *YYCURSOR) { const char *YYMARKER; /*!re2c re2c:define:YYCTYPE = char; re2c:yyfill:enable = 0; number { return OK; } !include "extra_rules.re.inc"; * { return FAIL; } */ } int main() { assert(lex("123") == OK); assert(lex("123.4567") == OK); return 0; } .ft P .fi .UNINDENT .UNINDENT .SH HEADER FILES .sp Re2c allows one to generate header file from the input \fB\&.re\fP file using option \fB\-t\fP, \fB\-\-type\-header\fP or configuration \fBre2c:flags:type\-header\fP and directives \fB/*!header:re2c:on*/\fP and \fB/*!header:re2c:off*/\fP\&. The first directive marks the beginning of header file, and the second directive marks the end of it. Everything between these directives is processed by re2c, and the generated code is written to the file specified by the \fB\-t \-\-type\-header\fP option (or \fBstdout\fP if this option was not used). Autogenerated header file may be needed in cases when re2c is used to generate definitions of constants, variables and structs that must be visible from other translation units. .sp Here is an example of generating a header file that contains definition of the lexer state with tag variables (the number variables depends on the regular grammar and is unknown to the programmer). .SS Input file .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C // re2c $INPUT \-o $OUTPUT \-i \-\-type\-header src/lexer/lexer.h #include <assert.h> #include "src/lexer/lexer.h" // generated by re2c /*!header:re2c:on*/ typedef struct { const char *str, *cur, *mar; /*!stags:re2c format = "const char *@@{tag}; "; */ } LexerState; /*!header:re2c:off*/ int lex(LexerState *st) { /*!re2c re2c:flags:type\-header = "src/lexer/lexer.h"; re2c:yyfill:enable = 0; re2c:flags:tags = 1; re2c:define:YYCTYPE = char; re2c:define:YYCURSOR = "st\->cur"; re2c:define:YYMARKER = "st\->mar"; re2c:tags:expression = "st\->@@{tag}"; [x]{1,4} / [x]{3,5} { return 0; } // ambiguous trailing context * { return 1; } */ } int main() { LexerState st; st.str = st.cur = "xxxxxxxx"; assert(lex(&st) == 0 && st.cur \- st.str == 4); return 0; } .ft P .fi .UNINDENT .UNINDENT .SS Header file .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C /* Generated by re2c */ typedef struct { const char *str, *cur, *mar; const char *yyt1; const char *yyt2; const char *yyt3; } LexerState; .ft P .fi .UNINDENT .UNINDENT .SH SUBMATCH EXTRACTION .sp Re2c has two options for submatch extraction. .sp The first option is \fB\-T \-\-tags\fP\&. With this option one can use standalone tags of the form \fB@stag\fP and \fB#mtag\fP, where \fBstag\fP and \fBmtag\fP are arbitrary used\-defined names. Tags can be used anywhere inside of a regular expression; semantically they are just position markers. Tags of the form \fB@stag\fP are called s\-tags: they denote a single submatch value (the last input position where this tag matched). Tags of the form \fB#mtag\fP are called m\-tags: they denote multiple submatch values (the whole history of repetitions of this tag). All tags should be defined by the user as variables with the corresponding names. With standalone tags re2c uses leftmost greedy disambiguation: submatch positions correspond to the leftmost matching path through the regular expression. .sp The second option is \fB\-P \-\-posix\-captures\fP: it enables POSIX\-compliant capturing groups. In this mode parentheses in regular expressions denote the beginning and the end of capturing groups; the whole regular expression is group number zero. The number of groups for the matching rule is stored in a variable \fByynmatch\fP, and submatch results are stored in \fByypmatch\fP array. Both \fByynmatch\fP and \fByypmatch\fP should be defined by the user, and \fByypmatch\fP size must be at least \fB[yynmatch * 2]\fP\&. Re2c provides a directive \fB/*!maxnmatch:re2c*/\fP that defines \fBYYMAXNMATCH\fP: a constant equal to the maximal value of \fByynmatch\fP among all rules. Note that re2c implements POSIX\-compliant disambiguation: each subexpression matches as long as possible, and subexpressions that start earlier in regular expression have priority over those starting later. Capturing groups are translated into s\-tags under the hood, therefore we use the word "tag" to describe them as well. .sp With both \fB\-P \-\-posix\-captures\fP and \fBT \-\-tags\fP options re2c uses efficient submatch extraction algorithm described in the \fI\%Tagged Deterministic Finite Automata with Lookahead\fP paper. The overhead on submatch extraction in the generated lexer grows with the number of tags \-\-\- if this number is moderate, the overhead is barely noticeable. In the lexer tags are implemented using a number of tag variables generated by re2c. There is no one\-to\-one correspondence between tag variables and tags: a single variable may be reused for different tags, and one tag may require multiple variables to hold all its ambiguous values. Eventually ambiguity is resolved, and only one final variable per tag survives. When a rule matches, all its tags are set to the values of the corresponding tag variables. The exact number of tag variables is unknown to the user; this number is determined by re2c. However, tag variables should be defined by the user as a part of the lexer state and updated by \fBYYFILL\fP, therefore re2c provides directives \fB/*!stags:re2c*/\fP and \fB/*!mtags:re2c*/\fP that can be used to declare, initialize and manipulate tag variables. These directives have two optional configurations: \fBformat = "@@";\fP (specifies the template where \fB@@\fP is substituted with the name of each tag variable), and \fBseparator = "";\fP (specifies the piece of code used to join the generated pieces for different tag variables). .sp S\-tags support the following operations: .INDENT 0.0 .IP \(bu 2 save input position to an s\-tag: \fBt = YYCURSOR\fP with default API or a user\-defined operation \fBYYSTAGP(t)\fP with generic API .IP \(bu 2 save default value to an s\-tag: \fBt = NULL\fP with default API or a user\-defined operation \fBYYSTAGN(t)\fP with generic API .IP \(bu 2 copy one s\-tag to another: \fBt1 = t2\fP .UNINDENT .sp M\-tags support the following operations: .INDENT 0.0 .IP \(bu 2 append input position to an m\-tag: a user\-defined operation \fBYYMTAGP(t)\fP with both default and generic API .IP \(bu 2 append default value to an m\-tag: a user\-defined operation \fBYYMTAGN(t)\fP with both default and generic API .IP \(bu 2 copy one m\-tag to another: \fBt1 = t2\fP .UNINDENT .sp S\-tags can be implemented as scalar values (pointers or offsets). M\-tags need a more complex representation, as they need to store a sequence of tag values. The most naive and inefficient representation of an m\-tag is a list (array, vector) of tag values; a more efficient representation is to store all m\-tags in a prefix\-tree represented as array of nodes \fB(v, p)\fP, where \fBv\fP is tag value and \fBp\fP is a pointer to parent node. .sp Here is a simple example of using s\-tags to parse an IPv4 address (see below for a more complex example that uses \fBYYFILL\fP). .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C // re2c $INPUT \-o $OUTPUT #include <assert.h> #include <stdint.h> static uint32_t num(const char *s, const char *e) { uint32_t n = 0; for (; s < e; ++s) n = n * 10 + (*s \- \(aq0\(aq); return n; } static const uint64_t ERROR = ~0lu; static uint64_t lex(const char *YYCURSOR) { const char *YYMARKER, *o1, *o2, *o3, *o4; /*!stags:re2c format = \(aqconst char *@@;\en\(aq; */ /*!re2c re2c:yyfill:enable = 0; re2c:flags:tags = 1; re2c:define:YYCTYPE = char; octet = [0\-9] | [1\-9][0\-9] | [1][0\-9][0\-9] | [2][0\-4][0\-9] | [2][5][0\-5]; dot = [.]; end = [\ex00]; @o1 octet dot @o2 octet dot @o3 octet dot @o4 octet end { return num(o4, YYCURSOR \- 1) + (num(o3, o4 \- 1) << 8) + (num(o2, o3 \- 1) << 16) + (num(o1, o2 \- 1) << 24); } * { return ERROR; } */ } int main() { assert(lex("1.2.3.4") == 0x01020304); assert(lex("127.0.0.1") == 0x7f000001); assert(lex("255.255.255.255") == 0xffffffff); assert(lex("1.2.3.") == ERROR); assert(lex("1.2.3.256") == ERROR); return 0; } .ft P .fi .UNINDENT .UNINDENT .sp Here is a more complex example of using s\-tags with \fBYYFILL\fP to parse a file with IPv4 addresses. Tag variables are part of the lexer state, and they are adjusted in \fBYYFILL\fP like other input positions. Note that it is necessary for s\-tags because their values are invalidated after shifting buffer contents. It may not be necessary in a custom implementation where tag variables store offsets relative to the start of the input string rather than buffer, which may be the case with m\-tags. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C // re2c $INPUT \-o $OUTPUT \-\-tags #include <assert.h> #include <stdint.h> #include <stdio.h> #include <string.h> #include <vector> #define SIZE 4096 typedef struct { FILE *file; char buf[SIZE + 1], *lim, *cur, *mar, *tok; // Tag variables must be part of the lexer state passed to YYFILL. // They don\(aqt correspond to tags and should be autogenerated by re2c. /*!stags:re2c format = \(aqconst char *@@;\(aq; */ int eof; } Input; static int fill(Input *in) { if (in\->eof) return 1; const size_t free = in\->tok \- in\->buf; if (free < 1) return 2; memmove(in\->buf, in\->tok, in\->lim \- in\->tok); in\->lim \-= free; in\->cur \-= free; in\->mar \-= free; in\->tok \-= free; // Tag variables need to be shifted like other input positions. The check // for non\-NULL is only needed if some tags are nested inside of alternative // or repetition, so that they can have NULL value. /*!stags:re2c format = "if (in\->@@) in\->@@ \-= free;\en"; */ in\->lim += fread(in\->lim, 1, free, in\->file); in\->lim[0] = 0; in\->eof |= in\->lim < in\->buf + SIZE; return 0; } static void init(Input *in, FILE *file) { in\->file = file; in\->cur = in\->mar = in\->tok = in\->lim = in\->buf + SIZE; // Initialization is only needed to avoid "use of uninitialized" warnings // when shifting tags in YYFILL. In the lexer tags are guaranteed to be // set before they are used (either to a valid input position, or NULL). /*!stags:re2c format = "in\->@@ = in\->lim;\en"; */ in\->eof = 0; fill(in); } static uint32_t num(const char *s, const char *e) { uint32_t n = 0; for (; s < e; ++s) n = n * 10 + (*s \- \(aq0\(aq); return n; } static bool lex(Input *in, std::vector<uint32_t> &ips) { // User\-defined local variables that store final tag values. // They are different from tag variables autogenerated with \(gastags:re2c\(ga, // as they are set at the end of match and used only in semantic actions. const char *o1, *o2, *o3, *o4; loop: in\->tok = in\->cur; /*!re2c re2c:eof = 0; re2c:api:style = free\-form; re2c:define:YYCTYPE = char; re2c:define:YYCURSOR = in\->cur; re2c:define:YYMARKER = in\->mar; re2c:define:YYLIMIT = in\->lim; re2c:define:YYFILL = "fill(in) == 0"; // The way tag variables are accessed from the lexer (not needed if tag // variables are defined as local variables). re2c:tags:expression = "in\->@@"; octet = [0\-9] | [1\-9][0\-9] | [1][0\-9][0\-9] | [2][0\-4][0\-9] | [2][5][0\-5]; dot = [.]; eol = [\en]; @o1 octet dot @o2 octet dot @o3 octet dot @o4 octet eol { ips.push_back(num(o4, in\->cur \- 1) + (num(o3, o4 \- 1) << 8) + (num(o2, o3 \- 1) << 16) + (num(o1, o2 \- 1) << 24)); goto loop; } $ { return true; } * { return false; } */ } int main() { const char *fname = "input"; FILE *f; Input in; std::vector<uint32_t> have, want; // Write a few IPv4 addresses to the input file and save them to compare // against parse results. f = fopen(fname, "w"); for (int i = 0; i < 256; ++i) { fprintf(f, "%d.%d.%d.%d\en", i, i, i, i); want.push_back(i + (i << 8) + (i << 16) + (i << 24)); } fclose(f); f = fopen(fname, "r"); init(&in, f); assert(lex(&in, have) && have == want); fclose(f); remove(fname); return 0; } .ft P .fi .UNINDENT .UNINDENT .sp Here is an example of using POSIX capturing groups to parse an IPv4 address. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C // re2c $INPUT \-o $OUTPUT #include <assert.h> #include <stdint.h> static uint32_t num(const char *s, const char *e) { uint32_t n = 0; for (; s < e; ++s) n = n * 10 + (*s \- \(aq0\(aq); return n; } /*!maxnmatch:re2c*/ static const uint64_t ERROR = ~0lu; static uint64_t lex(const char *YYCURSOR) { const char *YYMARKER; const char *yypmatch[YYMAXNMATCH * 2]; uint32_t yynmatch; /*!stags:re2c format = \(aqconst char *@@;\en\(aq; */ /*!re2c re2c:yyfill:enable = 0; re2c:flags:posix\-captures = 1; re2c:define:YYCTYPE = char; octet = [0\-9] | [1\-9][0\-9] | [1][0\-9][0\-9] | [2][0\-4][0\-9] | [2][5][0\-5]; dot = [.]; end = [\ex00]; (octet) dot (octet) dot (octet) dot (octet) end { assert(yynmatch == 5); return num(yypmatch[8], yypmatch[9]) + (num(yypmatch[6], yypmatch[7]) << 8) + (num(yypmatch[4], yypmatch[5]) << 16) + (num(yypmatch[2], yypmatch[3]) << 24); } * { return ERROR; } */ } int main() { assert(lex("1.2.3.4") == 0x01020304); assert(lex("127.0.0.1") == 0x7f000001); assert(lex("255.255.255.255") == 0xffffffff); assert(lex("1.2.3.") == ERROR); assert(lex("1.2.3.256") == ERROR); return 0; } .ft P .fi .UNINDENT .UNINDENT .sp Here is an example of using m\-tags to parse a semicolon\-separated sequence of words (C++). Tag variables are stored in a tree that is packed in a vector. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C // re2c $INPUT \-o $OUTPUT #include <assert.h> #include <vector> #include <string> static const int ROOT = \-1; struct Mtag { int pred; const char *tag; }; typedef std::vector<Mtag> MtagTree; typedef std::vector<std::string> Words; static void mtag(int *pt, const char *t, MtagTree *tree) { Mtag m = {*pt, t}; *pt = (int)tree\->size(); tree\->push_back(m); } static void unfold(const MtagTree &tree, int x, int y, Words &words) { if (x == ROOT) return; unfold(tree, tree[x].pred, tree[y].pred, words); const char *px = tree[x].tag, *py = tree[y].tag; words.push_back(std::string(px, py \- px)); } #define YYMTAGP(t) mtag(&t, YYCURSOR, &tree) #define YYMTAGN(t) mtag(&t, NULL, &tree) static bool lex(const char *YYCURSOR, Words &words) { const char *YYMARKER; /*!mtags:re2c format = "int @@ = ROOT;\en"; */ MtagTree tree; int x, y; /*!re2c re2c:yyfill:enable = 0; re2c:flags:tags = 1; re2c:define:YYCTYPE = char; (#x [a\-z]+ #y [;])+ { words.clear(); unfold(tree, x, y, words); return true; } * { return false; } */ } int main() { Words w; assert(lex("one;two;three;", w) && w == Words({"one", "two", "three"})); return 0; } .ft P .fi .UNINDENT .UNINDENT .SH STORABLE STATE .sp With \fB\-f\fP \fB\-\-storable\-state\fP option re2c generates a lexer that can store its current state, return to the caller, and later resume operations exactly where it left off. The default mode of operation in re2c is a "pull" model, in which the lexer "pulls" more input whenever it needs it. This may be unacceptable in cases when the input becomes available piece by piece (for example, if the lexer is invoked by the parser, or if the lexer program communicates via a socket protocol with some other program that must wait for a reply from the lexer before it transmits the next message). Storable state feature is intended exactly for such cases: it allows one to generate lexers that work in a "push" model. When the lexer needs more input, it stores its state and returns to the caller. Later, when more input becomes available, the caller resumes the lexer exactly where it stopped. There are a few changes necessary compared to the "pull" model: .INDENT 0.0 .IP \(bu 2 Define \fBYYSETSTATE()\fP and \fBYYGETSTATE(state)\fP promitives. .IP \(bu 2 Define \fByych\fP, \fByyaccept\fP and \fBstate\fP variables as a part of persistent lexer state. The \fBstate\fP variable should be initialized to \fB\-1\fP\&. .IP \(bu 2 \fBYYFILL\fP should return to the outer program instead of trying to supply more input. Return code should indicate that lexer needs more input. .IP \(bu 2 The outer program should recognize situations when lexer needs more input and respond appropriately. .IP \(bu 2 Use \fB/*!getstate:re2c*/\fP directive if it is necessary to execute any code before entering the lexer. .IP \(bu 2 Use configurations \fBstate:abort\fP and \fBstate:nextlabel\fP to further tweak the generated code. .UNINDENT .sp Here is an example of a "push"\-model lexer that reads input from \fBstdin\fP and expects a sequence of words separated by spaces and newlines. The lexer loops forever, waiting for more input. It can be terminated by sending a special EOF token \-\-\- a word "stop", in which case the lexer terminates successfully and prints the number of words it has seen. Abnormal termination happens in case of a syntax error, premature end of input (without the "stop" word) or in case the buffer is too small to hold a lexeme (for example, if one of the words exceeds buffer size). Premature end of input happens in case the lexer fails to read any input while being in the initial state \-\-\- this is the only case when EOF rule matches. Note that the lexer may call \fBYYFILL\fP twice before terminating (and thus require hitting \fBCtrl+D\fP a few times). First time \fBYYFILL\fP is called when the lexer expects continuation of the current greedy lexeme (either a word or a whitespace sequence). If \fBYYFILL\fP fails, the lexer knows that it has reached the end of the current lexeme and executes the corresponding semantic action. The action jumps to the beginning of the loop, the lexer enters the initial state and calls \fBYYFILL\fP once more. If it fails, the lexer matches EOF rule. (Alternatively EOF rule can be used for termination instead of a special EOF lexeme.) .SS Example .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C // re2c $INPUT \-o $OUTPUT \-f #include <assert.h> #include <stdio.h> #include <string.h> #define DEBUG 0 #define LOG(...) if (DEBUG) fprintf(stderr, __VA_ARGS__); #define BUFSIZE 10 typedef struct { FILE *file; char buf[BUFSIZE + 1], *lim, *cur, *mar, *tok; unsigned yyaccept; int state; } Input; static void init(Input *in, FILE *f) { in\->file = f; in\->cur = in\->mar = in\->tok = in\->lim = in\->buf + BUFSIZE; in\->lim[0] = 0; // append sentinel symbol in\->yyaccept = 0; in\->state = \-1; } typedef enum {END, READY, WAITING, BAD_PACKET, BIG_PACKET} Status; static Status fill(Input *in) { const size_t shift = in\->tok \- in\->buf; const size_t free = BUFSIZE \- (in\->lim \- in\->tok); if (free < 1) return BIG_PACKET; memmove(in\->buf, in\->tok, BUFSIZE \- shift); in\->lim \-= shift; in\->cur \-= shift; in\->mar \-= shift; in\->tok \-= shift; const size_t read = fread(in\->lim, 1, free, in\->file); in\->lim += read; in\->lim[0] = 0; // append sentinel symbol return READY; } static Status lex(Input *in, unsigned int *recv) { char yych; /*!getstate:re2c*/ loop: in\->tok = in\->cur; /*!re2c re2c:eof = 0; re2c:api:style = free\-form; re2c:define:YYCTYPE = "char"; re2c:define:YYCURSOR = "in\->cur"; re2c:define:YYMARKER = "in\->mar"; re2c:define:YYLIMIT = "in\->lim"; re2c:define:YYGETSTATE = "in\->state"; re2c:define:YYSETSTATE = "in\->state = @@;"; re2c:define:YYFILL = "return WAITING;"; packet = [a\-z]+[;]; * { return BAD_PACKET; } $ { return END; } packet { *recv = *recv + 1; goto loop; } */ } void test(const char **packets, Status status) { const char *fname = "pipe"; FILE *fw = fopen(fname, "w"); FILE *fr = fopen(fname, "r"); setvbuf(fw, NULL, _IONBF, 0); setvbuf(fr, NULL, _IONBF, 0); Input in; init(&in, fr); Status st; unsigned int send = 0, recv = 0; for (;;) { st = lex(&in, &recv); if (st == END) { LOG("done: got %u packets\en", recv); break; } else if (st == WAITING) { LOG("waiting...\en"); if (*packets) { LOG("sent packet %u\en", send); fprintf(fw, "%s", *packets++); ++send; } st = fill(&in); LOG("queue: \(aq%s\(aq\en", in.buf); if (st == BIG_PACKET) { LOG("error: packet too big\en"); break; } assert(st == READY); } else { assert(st == BAD_PACKET); LOG("error: ill\-formed packet\en"); break; } } LOG("\en"); assert(st == status); if (st == END) assert(recv == send); fclose(fw); fclose(fr); remove(fname); } int main() { const char *packets1[] = {0}; const char *packets2[] = {"zero;", "one;", "two;", "three;", "four;", 0}; const char *packets3[] = {"zer0;", 0}; const char *packets4[] = {"goooooooooogle;", 0}; test(packets1, END); test(packets2, END); test(packets3, BAD_PACKET); test(packets4, BIG_PACKET); return 0; } .ft P .fi .UNINDENT .UNINDENT .SH REUSABLE BLOCKS .sp Reusable blocks are re2c blocks that can be reused any number of times and combined with other re2c blocks. They are defined with \fB/*!rules:re2c[:<name>] ... */\fP (the \fB<name>\fP is optional). A rules block can be used in two contexts: either in a use block, or in a use directive inside of another block. The code for a rules block is generated at every point of use. .sp Use blocks are defined with \fB/*!use:re2c[:<name>] ... */\fP\&. The \fB<name>\fP is optional; if not specified, the associated rules block is the most recent one (whether named or unnamed). A use block can add named definitions, configurations and rules of its own. An important use case for use blocks is a lexer that supports multiple input encodings: the same rules block is reused multiple times with encoding\-specific configurations (see the example below). .sp In\-block use directive \fB!use:<name>;\fP can be used from inside of a re2c block. It merges the referenced block \fB<name>\fP into the current one. If some of the merged rules and configurations overlap with the previously defined ones, conflicts are resolved in the usual way: the earliest rule takes priority, and latest configuration overrides preceding ones. One exception are the special rules \fB*\fP, \fB$\fP and (in condition mode) \fB<!>\fP, for which a block\-local definition overrides any inherited ones. Use directive allows one to combine different re2c blocks together in one block (see the example below). .sp Named blocks and in\-block use directive were added in re2c version 2.2. Since that version reusable blocks are allowed by default (no special option is needed). Before version 2.2 reuse mode was enabled with \fB\-r \-\-reusable\fP option. Before version 1.2 reusable blocks could not be mixed with normal blocks. .SS Example of a \fB!use\fP directive .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C // re2c $INPUT \-o $OUTPUT // This example shows how to combine reusable re2c blocks: two blocks // (\(aqcolors\(aq and \(aqfish\(aq) are merged into one. The \(aqsalmon\(aq rule occurs // in both blocks; the \(aqfish\(aq block takes priority because it is used // earlier. Default rule * occurs in all three blocks; the local (not // inherited) definition takes priority. #include <assert.h> enum What { COLOR, FISH, DUNNO }; /*!rules:re2c:colors * { assert(false); } "red" | "salmon" | "magenta" { return COLOR; } */ /*!rules:re2c:fish * { assert(false); } "haddock" | "salmon" | "eel" { return FISH; } */ static What lex(const char *YYCURSOR) { const char *YYMARKER; /*!re2c re2c:yyfill:enable = 0; re2c:define:YYCTYPE = char; !use:fish; !use:colors; * { return DUNNO; } */ } int main() { assert(lex("salmon") == FISH); assert(lex("what?") == DUNNO); return 0; } .ft P .fi .UNINDENT .UNINDENT .SS Example of a \fB/*!use:re2c ... */\fP block .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C // re2c $INPUT \-o $OUTPUT \-\-input\-encoding utf8 // This example supports multiple input encodings: UTF\-8 and UTF\-32. // Both lexers are generated from the same rules block, and the use // blocks add only encoding\-specific configurations. #include <assert.h> #include <stdint.h> /*!rules:re2c re2c:yyfill:enable = 0; "∀x ∃y" { return 0; } * { return 1; } */ static int lex_utf8(const uint8_t *YYCURSOR) { const uint8_t *YYMARKER; /*!use:re2c re2c:define:YYCTYPE = uint8_t; re2c:flags:8 = 1; */ } static int lex_utf32(const uint32_t *YYCURSOR) { const uint32_t *YYMARKER; /*!use:re2c re2c:define:YYCTYPE = uint32_t; re2c:flags:8 = 0; re2c:flags:u = 1; */ } int main() { static const uint8_t s8[] = // UTF\-8 { 0xe2, 0x88, 0x80, 0x78, 0x20, 0xe2, 0x88, 0x83, 0x79 }; static const uint32_t s32[] = // UTF32 { 0x00002200, 0x00000078, 0x00000020, 0x00002203, 0x00000079 }; assert(lex_utf8(s8) == 0); assert(lex_utf32(s32) == 0); return 0; } .ft P .fi .UNINDENT .UNINDENT .SH ENCODING SUPPORT .sp Speaking of encodings, it is necessary to understand the difference between code points and code units. Code point is an abstract symbol. Code unit is the smallest atomic unit of storage in the encoded text. A single code point may be represented with one or more code units. In a fixed\-length encoding all code points are represented with the same number of code units. In a variable\-length encoding code points may be represented with a different number of code units. Note that the "any" rule \fB[^]\fP matches any code point, but not necessarily any code unit. The only way to match any code unit regardless of the encoding it the default rule \fB*\fP\&. \fBYYCTYPE\fP size should be equal to the size of code unit. .sp Re2c supports the following encodings: ASCII, EBCDIC, UCS2, UTF8, UTF16 and UTF32. .INDENT 0.0 .IP \(bu 2 ASCII is enabled by default. It is a fixed\-length encoding with code space [0\-255] and 1\-byte code points and code units. .IP \(bu 2 EBCDIC is enabled with \fB\-e, \-\-ecb\fP option. It a fixed\-length encoding with code space [0\-255] and 1\-byte code points and code units. .IP \(bu 2 UCS2 is enabled with \fB\-w, \-\-wide\-chars\fP option. It is a fixed\-length encoding with code space [0\-0xFFFF] and 2\-byte code points and code units. .IP \(bu 2 UTF8 is enabled with \fB\-8, \-\-utf\-8\fP option. It is a variable\-length Unicode encoding with code space [0\-0x10FFFF]. Code points are represented with one, two, three or four 1\-byte code units. .IP \(bu 2 UTF16 is enabled with \fB\-x, \-\-utf\-16\fP option. It is a variable\-length Unicode encoding with code space [0\-0x10FFFF]. Code points are represented with one or two 2\-byte code units. .IP \(bu 2 UTF32 is enabled with \fB\-u, \-\-unicode\fP option. It is a fixed\-length Unicode encoding with code space [0\-0x10FFFF] and 4\-byte code points and code units. .UNINDENT .sp Encodings can also be set or unset using \fBre2c:flags\fP configuration, for example \fBre2c:flags:8 = 1;\fP enables UTF8. .sp Include file \fBinclude/unicode_categories.re\fP provides re2c definitions for the standard Unicode categories. .sp Option \fB\-\-input\-encoding utf8\fP enables Unicode literals in regular expressions. .sp Option \fB\-\-encoding\-policy <fail | substitute | ignore>\fP specifies the way re2c handles Unicode surrogates: code points in the range [0xD800\-0xDFFF]. .SS Example .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C // re2c $INPUT \-o $OUTPUT \-8 \-\-case\-ranges \-i // // Simplified "Unicode Identifier and Pattern Syntax" // (see https://unicode.org/reports/tr31) #include <assert.h> #include <stdint.h> /*!include:re2c "unicode_categories.re" */ static int lex(const char *YYCURSOR) { const char *YYMARKER; /*!re2c re2c:define:YYCTYPE = \(aqunsigned char\(aq; re2c:yyfill:enable = 0; id_start = L | Nl | [$_]; id_continue = id_start | Mn | Mc | Nd | Pc | [\eu200D\eu05F3]; identifier = id_start id_continue*; identifier { return 0; } * { return 1; } */ } int main() { assert(lex("_Ыдентификатор") == 0); return 0; } .ft P .fi .UNINDENT .UNINDENT .SH START CONDITIONS .sp Conditions are enabled with \fB\-c\fP \fB\-\-conditions\fP\&. This option allows one to encode multiple interrelated lexers within the same re2c block. .sp Each lexer corresponds to a single condition. It starts with a label of the form \fByyc_name\fP, where \fBname\fP is condition name and \fByyc\fP prefix can be adjusted with configuration \fBre2c:condprefix\fP\&. Different lexers are separated with a comment \fB/* *********************************** */\fP which can be adjusted with configuration \fBre2c:cond:divider\fP\&. .sp Furthermore, each condition has a unique identifier of the form \fByycname\fP, where \fBname\fP is condition name and \fByyc\fP prefix can be adjusted with configuration \fBre2c:condenumprefix\fP\&. Identifiers have the type \fBYYCONDTYPE\fP and should be generated with \fB/*!types:re2c*/\fP directive or \fB\-t\fP \fB\-\-type\-header\fP option. Users shouldn\(aqt define these identifiers manually, as the order of conditions is not specified. .sp Before all conditions re2c generates entry code that checks the current condition identifier and transfers control flow to the start label of the active condition. After matching some rule of this condition, lexer may either transfer control flow back to the entry code (after executing the associated action and optionally setting another condition with \fB=>\fP), or use \fB:=>\fP shortcut and transition directly to the start label of another condition (skipping the action and the entry code). Configuration \fBre2c:cond:goto\fP allows one to change the default behavior. .sp Syntactically each rule must be preceded with a list of comma\-separated condition names or a wildcard \fB*\fP enclosed in angle brackets \fB<\fP and \fB>\fP\&. Wildcard means "any condition" and is semantically equivalent to listing all condition names. Here \fBregexp\fP is a regular expression, \fBdefault\fP refers to the default rule \fB*\fP, and \fBaction\fP is a block of code. .INDENT 0.0 .IP \(bu 2 \fB<conditions\-or\-wildcard> regexp\-or\-default action\fP .IP \(bu 2 \fB<conditions\-or\-wildcard> regexp\-or\-default => condition action\fP .IP \(bu 2 \fB<conditions\-or\-wildcard> regexp\-or\-default :=> condition\fP .UNINDENT .sp Rules with an exclamation mark \fB!\fP in front of condition list have a special meaning: they have no regular expression, and the associated action is merged as an entry code to actions of normal rules. This might be a convenient place to peform a routine task that is common to all rules. .INDENT 0.0 .IP \(bu 2 \fB<!conditions\-or\-wildcard> action\fP .UNINDENT .sp Another special form of rules with an empty condition list \fB<>\fP and no regular expression allows one to specify an "entry condition" that can be used to execute code before entering the lexer. It is semantically equivalent to a condition with number zero, name \fB0\fP and an empty regular expression. .INDENT 0.0 .IP \(bu 2 \fB<> action\fP .IP \(bu 2 \fB<> => condition action\fP .IP \(bu 2 \fB<> :=> condition\fP .UNINDENT .SS Example .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C // re2c $INPUT \-o $OUTPUT \-ci #include <stdint.h> #include <limits.h> #include <assert.h> static const uint64_t ERROR = ~0lu; /*!types:re2c*/ template<int BASE> static void adddgt(uint64_t &u, unsigned int d) { u = u * BASE + d; if (u > UINT32_MAX) u = ERROR; } static uint64_t parse_u32(const char *s) { const char *YYMARKER; int c = yycinit; uint64_t u = 0; /*!re2c re2c:yyfill:enable = 0; re2c:api:style = free\-form; re2c:define:YYCTYPE = char; re2c:define:YYCURSOR = s; re2c:define:YYGETCONDITION = "c"; re2c:define:YYSETCONDITION = "c = @@;"; <*> * { return ERROR; } <init> \(aq0b\(aq / [01] :=> bin <init> "0" :=> oct <init> "" / [1\-9] :=> dec <init> \(aq0x\(aq / [0\-9a\-fA\-F] :=> hex <bin, oct, dec, hex> "\ex00" { return u; } <bin> [01] { adddgt<2> (u, s[\-1] \- \(aq0\(aq); goto yyc_bin; } <oct> [0\-7] { adddgt<8> (u, s[\-1] \- \(aq0\(aq); goto yyc_oct; } <dec> [0\-9] { adddgt<10>(u, s[\-1] \- \(aq0\(aq); goto yyc_dec; } <hex> [0\-9] { adddgt<16>(u, s[\-1] \- \(aq0\(aq); goto yyc_hex; } <hex> [a\-f] { adddgt<16>(u, s[\-1] \- \(aqa\(aq + 10); goto yyc_hex; } <hex> [A\-F] { adddgt<16>(u, s[\-1] \- \(aqA\(aq + 10); goto yyc_hex; } */ } int main() { assert(parse_u32("1234567890") == 1234567890); assert(parse_u32("0b1101") == 13); assert(parse_u32("0x7Fe") == 2046); assert(parse_u32("0644") == 420); assert(parse_u32("9999999999") == ERROR); assert(parse_u32("") == ERROR); return 0; } .ft P .fi .UNINDENT .UNINDENT .SH SKELETON PROGRAMS .sp With the \fB\-S, \-\-skeleton\fP option, re2c ignores all non\-re2c code and generates a self\-contained C program that can be further compiled and executed. The program consists of lexer code and input data. For each constructed DFA (block or condition) re2c generates a standalone lexer and two files: an \fB\&.input\fP file with strings derived from the DFA and a \fB\&.keys\fP file with expected match results. The program runs each lexer on the corresponding \fB\&.input\fP file and compares results with the expectations. Skeleton programs are very useful for a number of reasons: .INDENT 0.0 .IP \(bu 2 They can check correctness of various re2c optimizations (the data is generated early in the process, before any DFA transformations have taken place). .IP \(bu 2 Generating a set of input data with good coverage may be useful for both testing and benchmarking. .IP \(bu 2 Generating self\-contained executable programs allows one to get minimized test cases (the original code may be large or have a lot of dependencies). .UNINDENT .sp The difficulty with generating input data is that for all but the most trivial cases the number of possible input strings is too large (even if the string length is limited). Re2c solves this difficulty by generating sufficiently many strings to cover almost all DFA transitions. It uses the following algorithm. First, it constructs a skeleton of the DFA. For encodings with 1\-byte code unit size (such as ASCII, UTF\-8 and EBCDIC) skeleton is just an exact copy of the original DFA. For encodings with multibyte code units skeleton is a copy of DFA with certain transitions omitted: namely, re2c takes at most 256 code units for each disjoint continuous range that corresponds to a DFA transition. The chosen values are evenly distributed and include range bounds. Instead of trying to cover all possible paths in the skeleton (which is infeasible) re2c generates sufficiently many paths to cover all skeleton transitions, and thus trigger the corresponding conditional jumps in the lexer. The algorithm implementation is limited by ~1Gb of transitions and consumes constant amount of memory (re2c writes data to file as soon as it is generated). .SH VISUALIZATION AND DEBUG .sp With the \fB\-D, \-\-emit\-dot\fP option, re2c does not generate code. Instead, it dumps the generated DFA in DOT format. One can convert this dump to an image of the DFA using Graphviz or another library. Note that this option shows the final DFA after it has gone through a number of optimizations and transformations. Earlier stages can be dumped with various debug options, such as \fB\-\-dump\-nfa\fP, \fB\-\-dump\-dfa\-raw\fP etc. (see the full list of options). .SH SEE ALSO .sp You can find more information about re2c at the official website: \fI\%http://re2c.org\fP\&. Similar programs are flex(1), lex(1), quex(\fI\%http://quex.sourceforge.net\fP). .SH AUTHORS .sp Re2c was originaly written by Peter Bumbulis in 1993. Since then it has been developed and maintained by multiple volunteers; mots notably, Brain Young, Marcus Boerger, Dan Nuffer and Ulya Trofimovich. .\" Generated by docutils manpage writer. .