aboutsummaryrefslogtreecommitdiffstats
path: root/gnuwin32/man/cat1
diff options
context:
space:
mode:
Diffstat (limited to 'gnuwin32/man/cat1')
-rw-r--r--gnuwin32/man/cat1/bison.1.txt188
-rw-r--r--gnuwin32/man/cat1/flex.1.txt3013
-rw-r--r--gnuwin32/man/cat1/gperf.1.txt226
-rw-r--r--gnuwin32/man/cat1/iconv.1.txt48
-rw-r--r--gnuwin32/man/cat1/yacc.1.txt42
5 files changed, 3517 insertions, 0 deletions
diff --git a/gnuwin32/man/cat1/bison.1.txt b/gnuwin32/man/cat1/bison.1.txt
new file mode 100644
index 00000000..2c2cbe75
--- /dev/null
+++ b/gnuwin32/man/cat1/bison.1.txt
@@ -0,0 +1,188 @@
+BISON(1) User Commands BISON(1)
+
+
+
+NAME s
+ bison - GNU Project parser generator (yacc replacement)
+ n
+SYNOPSIS 2
+ j:l. [OPTION]... FILE
+ 4
+DESCRIPTION
+ Bison is a parser generator in the style of yacc(1). It
+ should be upwardly compatible with input files designed
+ for yacc.
+
+ Input files should follow the yacc convention of ending
+ in .y. Unlike yacc, the generated files do not have
+ fixed names, but instead use the prefix of the input
+ file. Moreover, if you need to put C++ code in the
+ input file, you can end his name by a C++-like extension
+ (.ypp or .y++), then bison will follow your extension to
+ name the output file (.cpp or .c++). For instance, a
+ grammar description file named parse.yxx would produce
+ the generated parser in a file named parse.tab.cxx,
+ instead of yacc's y.tab.c or old Bison version's
+ parse.tab.c.
+
+ This description of the options that can be given to
+ bison is adapted from the node Invocation in the
+ bison.texinfo manual, which should be taken as authori-
+ tative.
+
+ Bison supports both traditional single-letter options
+ and mnemonic long option names. Long option names are
+ indicated with -- instead of -. Abbreviations for
+ option names are allowed as long as they are unique.
+ When a long option takes an argument, like --file-pre-
+ fix, connect the option name and the argument with =.
+
+ Generate LALR(1) and GLR parsers.
+
+
+ Mandatory arguments to long options are mandatory for
+ short options too. The same is true for optional argu-
+ ments.
+
+
+ Operation modes:
+
+ -h, --help
+ display this help and exit
+
+ -V, --version
+ output version information and exit
+
+ --print-localedir
+ output directory containing locale-dependent data
+
+ --print-datadir
+ output directory containing skeletons and XSLT
+
+ -y, --yacc
+ emulate POSIX Yacc
+
+ -W, --warnings=[CATEGORY]
+ report the warnings falling in CATEGORY
+
+
+ Parser:
+
+ -L, --language=LANGUAGE
+ specify the output programming language (this is
+ an experimental feature)
+
+ -S, --skeleton=FILE
+ specify the skeleton to use
+
+ -t, --debug
+ instrument the parser for debugging
+
+ --locations
+ enable locations computation
+
+ -p, --name-prefix=PREFIX
+ prepend PREFIX to the external symbols
+
+ -l, --no-lines
+ don't generate `#line' directives
+
+ -k, --token-table
+ include a table of token names
+
+
+ Output:
+
+ --defines[=FILE]
+ also produce a header file
+
+ -d likewise but cannot specify FILE (for POSIX Yacc)
+
+ -r, --report=THINGS
+ also produce details on the automaton
+
+ --report-file=FILE
+ write report to FILE
+
+ -v, --verbose
+ same as `--report=state'
+
+ -b, --file-prefix=PREFIX
+ specify a PREFIX for output files
+
+ -o, --output=FILE
+ leave output to FILE
+
+ -g, --graph[=FILE]
+ also output a graph of the automaton
+
+ -x, --xml[=FILE]
+ also output an XML report of the automaton (the
+ XML schema is experimental)
+
+
+ Warning categories include:
+
+ `midrule-values'
+ unset or unused midrule values
+
+ `yacc' incompatibilities with POSIX YACC
+
+ `all' all the warnings
+
+ `no-CATEGORY'
+ turn off warnings in CATEGORY
+
+ `none' turn off all the warnings
+
+ `error'
+ treat warnings as errors
+
+
+ THINGS is a list of comma separated words that can
+ include:
+
+ `state'
+ describe the states
+
+ `itemset'
+ complete the core item sets with their closure
+
+ `lookahead'
+ explicitly associate lookahead tokens to items
+
+ `solved'
+ describe shift/reduce conflicts solving
+
+ `all' include all the above information
+
+ `none' disable the report
+
+
+
+AUTHOR
+ Written by Robert Corbett and Richard Stallman.
+
+
+ Copyright (C) 2008 Free Software Foundation, Inc. This
+ is free software; see the source for copying conditions.
+ There is NO warranty; not even for MERCHANTABILITY or
+ FITNESS FOR A PARTICULAR PURPOSE.
+
+REPORTING BUGS
+ Report bugs to <bug-bison@gnu.org>.
+
+SEE ALSO
+ lex(1), flex(1), yacc(1).
+
+ The full documentation for bison is maintained as a Tex-
+ info manual. If the info and bison programs are prop-
+ erly installed at your site, the command
+
+ info bison
+
+ should give you access to the complete manual.
+
+
+
+bison 2.4.1 December 2008 BISON(1)
diff --git a/gnuwin32/man/cat1/flex.1.txt b/gnuwin32/man/cat1/flex.1.txt
new file mode 100644
index 00000000..fe54aecf
--- /dev/null
+++ b/gnuwin32/man/cat1/flex.1.txt
@@ -0,0 +1,3013 @@
+FLEX(1) FLEX(1)
+
+
+
+
+
+NAME
+ flex - fast lexical analyzer generator
+
+SYNOPSIS
+ flex [-bcdfhilnpstvwBFILTV78+? -C[aefFmr] -ooutput
+ -Pprefix -Sskeleton] [--help --version] [filename ...]
+
+OVERVIEW
+ This manual describes flex, a tool for generating pro-
+ grams that perform pattern-matching on text. The manual
+ includes both tutorial and reference sections:
+
+ Description
+ a brief overview of the tool
+
+ Some Simple Examples
+
+ Format Of The Input File
+
+ Patterns
+ the extended regular expressions used by flex
+
+ How The Input Is Matched
+ the rules for determining what has been matched
+
+ Actions
+ how to specify what to do when a pattern is matched
+
+ The Generated Scanner
+ details regarding the scanner that flex produces;
+ how to control the input source
+
+ Start Conditions
+ introducing context into your scanners, and
+ managing "mini-scanners"
+
+ Multiple Input Buffers
+ how to manipulate multiple input sources; how to
+ scan from strings instead of files
+
+ End-of-file Rules
+ special rules for matching the end of the input
+
+ Miscellaneous Macros
+ a summary of macros available to the actions
+
+ Values Available To The User
+ a summary of values available to the actions
+
+ Interfacing With Yacc
+ connecting flex scanners together with yacc parsers
+
+ Options
+ flex command-line options, and the "%option"
+ directive
+
+ Performance Considerations
+ how to make your scanner go as fast as possible
+
+ Generating C++ Scanners
+ the (experimental) facility for generating C++
+ scanner classes
+
+ Incompatibilities With Lex And POSIX
+ how flex differs from AT&T lex and the POSIX lex
+ standard
+
+ Diagnostics
+ those error messages produced by flex (or scanners
+ it generates) whose meanings might not be apparent
+
+ Files
+ files used by flex
+
+ Deficiencies / Bugs
+ known problems with flex
+
+ See Also
+ other documentation, related tools
+
+ Author
+ includes contact information
+
+
+DESCRIPTION
+ flex is a tool for generating scanners: programs which
+ recognized lexical patterns in text. flex reads the
+ given input files, or its standard input if no file
+ names are given, for a description of a scanner to gen-
+ erate. The description is in the form of pairs of regu-
+ lar expressions and C code, called rules. flex generates
+ as output a C source file, lex.yy.c, which defines a
+ routine yylex(). This file is compiled and linked with
+ the -lfl library to produce an executable. When the
+ executable is run, it analyzes its input for occurrences
+ of the regular expressions. Whenever it finds one, it
+ executes the corresponding C code.
+
+SOME SIMPLE EXAMPLES
+ First some simple examples to get the flavor of how one
+ uses flex. The following flex input specifies a scanner
+ which whenever it encounters the string "username" will
+ replace it with the user's login name:
+
+ %%
+ username printf( "%s", getlogin() );
+
+ By default, any text not matched by a flex scanner is
+ copied to the output, so the net effect of this scanner
+ is to copy its input file to its output with each occur-
+ rence of "username" expanded. In this input, there is
+ just one rule. "username" is the pattern and the
+ "printf" is the action. The "%%" marks the beginning of
+ the rules.
+
+ Here's another simple example:
+
+ int num_lines = 0, num_chars = 0;
+
+ %%
+ \n ++num_lines; ++num_chars;
+ . ++num_chars;
+
+ %%
+ main()
+ {
+ yylex();
+ printf( "# of lines = %d, # of chars = %d\n",
+ num_lines, num_chars );
+ }
+
+ This scanner counts the number of characters and the
+ number of lines in its input (it produces no output
+ other than the final report on the counts). The first
+ line declares two globals, "num_lines" and "num_chars",
+ which are accessible both inside yylex() and in the
+ main() routine declared after the second "%%". There
+ are two rules, one which matches a newline ("\n") and
+ increments both the line count and the character count,
+ and one which matches any character other than a newline
+ (indicated by the "." regular expression).
+
+ A somewhat more complicated example:
+
+ /* scanner for a toy Pascal-like language */
+
+ %{
+ /* need this for the call to atof() below */
+ #include <math.h>
+ %}
+
+ DIGIT [0-9]
+ ID [a-z][a-z0-9]*
+
+ %%
+
+ {DIGIT}+ {
+ printf( "An integer: %s (%d)\n", yytext,
+ atoi( yytext ) );
+ }
+
+ {DIGIT}+"."{DIGIT}* {
+ printf( "A float: %s (%g)\n", yytext,
+ atof( yytext ) );
+ }
+
+ if|then|begin|end|procedure|function {
+ printf( "A keyword: %s\n", yytext );
+ }
+
+ {ID} printf( "An identifier: %s\n", yytext );
+
+ "+"|"-"|"*"|"/" printf( "An operator: %s\n", yytext );
+
+ "{"[^}\n]*"}" /* eat up one-line comments */
+
+ [ \t\n]+ /* eat up whitespace */
+
+ . printf( "Unrecognized character: %s\n", yytext );
+
+ %%
+
+ main( argc, argv )
+ int argc;
+ char **argv;
+ {
+ ++argv, --argc; /* skip over program name */
+ if ( argc > 0 )
+ yyin = fopen( argv[0], "r" );
+ else
+ yyin = stdin;
+
+ yylex();
+ }
+
+ This is the beginnings of a simple scanner for a lan-
+ guage like Pascal. It identifies different types of
+ tokens and reports on what it has seen.
+
+ The details of this example will be explained in the
+ following sections.
+
+FORMAT OF THE INPUT FILE
+ The flex input file consists of three sections,
+ separated by a line with just %% in it:
+
+ definitions
+ %%
+ rules
+ %%
+ user code
+
+ The definitions section contains declarations of simple
+ name definitions to simplify the scanner specification,
+ and declarations of start conditions, which are
+ explained in a later section.
+
+ Name definitions have the form:
+
+ name definition
+
+ The "name" is a word beginning with a letter or an
+ underscore ('_') followed by zero or more letters, dig-
+ its, '_', or '-' (dash). The definition is taken to
+ begin at the first non-white-space character following
+ the name and continuing to the end of the line. The
+ definition can subsequently be referred to using
+ "{name}", which will expand to "(definition)". For
+ example,
+
+ DIGIT [0-9]
+ ID [a-z][a-z0-9]*
+
+ defines "DIGIT" to be a regular expression which matches
+ a single digit, and "ID" to be a regular expression
+ which matches a letter followed by zero-or-more letters-
+ or-digits. A subsequent reference to
+
+ {DIGIT}+"."{DIGIT}*
+
+ is identical to
+
+ ([0-9])+"."([0-9])*
+
+ and matches one-or-more digits followed by a '.' fol-
+ lowed by zero-or-more digits.
+
+ The rules section of the flex input contains a series of
+ rules of the form:
+
+ pattern action
+
+ where the pattern must be unindented and the action must
+ begin on the same line.
+
+ See below for a further description of patterns and
+ actions.
+
+ Finally, the user code section is simply copied to
+ lex.yy.c verbatim. It is used for companion routines
+ which call or are called by the scanner. The presence
+ of this section is optional; if it is missing, the sec-
+ ond %% in the input file may be skipped, too.
+
+ In the definitions and rules sections, any indented text
+ or text enclosed in %{ and %} is copied verbatim to the
+ output (with the %{}'s removed). The %{}'s must appear
+ unindented on lines by themselves.
+
+ In the rules section, any indented or %{} text appearing
+ before the first rule may be used to declare variables
+ which are local to the scanning routine and (after the
+ declarations) code which is to be executed whenever the
+ scanning routine is entered. Other indented or %{} text
+ in the rule section is still copied to the output, but
+ its meaning is not well-defined and it may well cause
+ compile-time errors (this feature is present for POSIX
+ compliance; see below for other such features).
+
+ In the definitions section (but not in the rules sec-
+ tion), an unindented comment (i.e., a line beginning
+ with "/*") is also copied verbatim to the output up to
+ the next "*/".
+
+PATTERNS
+ The patterns in the input are written using an extended
+ set of regular expressions. These are:
+
+ x match the character 'x'
+ . any character (byte) except newline
+ [xyz] a "character class"; in this case, the pattern
+ matches either an 'x', a 'y', or a 'z'
+ [abj-oZ] a "character class" with a range in it; matches
+ an 'a', a 'b', any letter from 'j' through 'o',
+ or a 'Z'
+ [^A-Z] a "negated character class", i.e., any character
+ but those in the class. In this case, any
+ character EXCEPT an uppercase letter.
+ [^A-Z\n] any character EXCEPT an uppercase letter or
+ a newline
+ r* zero or more r's, where r is any regular expression
+ r+ one or more r's
+ r? zero or one r's (that is, "an optional r")
+ r{2,5} anywhere from two to five r's
+ r{2,} two or more r's
+ r{4} exactly 4 r's
+ {name} the expansion of the "name" definition
+ (see above)
+ "[xyz]\"foo"
+ the literal string: [xyz]"foo
+ \X if X is an 'a', 'b', 'f', 'n', 'r', 't', or 'v',
+ then the ANSI-C interpretation of \x.
+ Otherwise, a literal 'X' (used to escape
+ operators such as '*')
+ \0 a NUL character (ASCII code 0)
+ \123 the character with octal value 123
+ \x2a the character with hexadecimal value 2a
+ (r) match an r; parentheses are used to override
+ precedence (see below)
+
+
+ rs the regular expression r followed by the
+ regular expression s; called "concatenation"
+
+
+ r|s either an r or an s
+
+
+ r/s an r but only if it is followed by an s. The
+ text matched by s is included when determining
+ whether this rule is the "longest match",
+ but is then returned to the input before
+ the action is executed. So the action only
+ sees the text matched by r. This type
+ of pattern is called trailing context".
+ (There are some combinations of r/s that flex
+ cannot match correctly; see notes in the
+ Deficiencies / Bugs section below regarding
+ "dangerous trailing context".)
+ ^r an r, but only at the beginning of a line (i.e.,
+ which just starting to scan, or right after a
+ newline has been scanned).
+ r$ an r, but only at the end of a line (i.e., just
+ before a newline). Equivalent to "r/\n".
+
+ Note that flex's notion of "newline" is exactly
+ whatever the C compiler used to compile flex
+ interprets '\n' as; in particular, on some DOS
+ systems you must either filter out \r's in the
+ input yourself, or explicitly use r/\r\n for "r$".
+
+
+ <s>r an r, but only in start condition s (see
+ below for discussion of start conditions)
+ <s1,s2,s3>r
+ same, but in any of start conditions s1,
+ s2, or s3
+ <*>r an r in any start condition, even an exclusive one.
+
+
+ <<EOF>> an end-of-file
+ <s1,s2><<EOF>>
+ an end-of-file when in start condition s1 or s2
+
+ Note that inside of a character class, all regular
+ expression operators lose their special meaning except
+ escape ('\') and the character class operators, '-',
+ ']', and, at the beginning of the class, '^'.
+
+ The regular expressions listed above are grouped accord-
+ ing to precedence, from highest precedence at the top to
+ lowest at the bottom. Those grouped together have equal
+ precedence. For example,
+
+ foo|bar*
+
+ is the same as
+
+ (foo)|(ba(r*))
+
+ since the '*' operator has higher precedence than con-
+ catenation, and concatenation higher than alternation
+ ('|'). This pattern therefore matches either the string
+ "foo" or the string "ba" followed by zero-or-more r's.
+ To match "foo" or zero-or-more "bar"'s, use:
+
+ foo|(bar)*
+
+ and to match zero-or-more "foo"'s-or-"bar"'s:
+
+ (foo|bar)*
+
+
+ In addition to characters and ranges of characters,
+ character classes can also contain character class
+ expressions. These are expressions enclosed inside [:
+ and :] delimiters (which themselves must appear between
+ the '[' and ']' of the character class; other elements
+ may occur inside the character class, too). The valid
+ expressions are:
+
+ [:alnum:] [:alpha:] [:blank:]
+ [:cntrl:] [:digit:] [:graph:]
+ [:lower:] [:print:] [:punct:]
+ [:space:] [:upper:] [:xdigit:]
+
+ These expressions all designate a set of characters
+ equivalent to the corresponding standard C isXXX func-
+ tion. For example, [:alnum:] designates those charac-
+ ters for which isalnum() returns true - i.e., any alpha-
+ betic or numeric. Some systems don't provide isblank(),
+ so flex defines [:blank:] as a blank or a tab.
+
+ For example, the following character classes are all
+ equivalent:
+
+ [[:alnum:]]
+ [[:alpha:][:digit:]
+ [[:alpha:]0-9]
+ [a-zA-Z0-9]
+
+ If your scanner is case-insensitive (the -i flag), then
+ [:upper:] and [:lower:] are equivalent to [:alpha:].
+
+ Some notes on patterns:
+
+ - A negated character class such as the example
+ "[^A-Z]" above will match a newline unless "\n"
+ (or an equivalent escape sequence) is one of the
+ characters explicitly present in the negated
+ character class (e.g., "[^A-Z\n]"). This is
+ unlike how many other regular expression tools
+ treat negated character classes, but unfortu-
+ nately the inconsistency is historically
+ entrenched. Matching newlines means that a pat-
+ tern like [^"]* can match the entire input unless
+ there's another quote in the input.
+
+ - A rule can have at most one instance of trailing
+ context (the '/' operator or the '$' operator).
+ The start condition, '^', and "<<EOF>>" patterns
+ can only occur at the beginning of a pattern,
+ and, as well as with '/' and '$', cannot be
+ grouped inside parentheses. A '^' which does not
+ occur at the beginning of a rule or a '$' which
+ does not occur at the end of a rule loses its
+ special properties and is treated as a normal
+ character.
+
+ The following are illegal:
+
+ foo/bar$
+ <sc1>foo<sc2>bar
+
+ Note that the first of these, can be written
+ "foo/bar\n".
+
+ The following will result in '$' or '^' being
+ treated as a normal character:
+
+ foo|(bar$)
+ foo|^bar
+
+ If what's wanted is a "foo" or a bar-followed-by-
+ a-newline, the following could be used (the spe-
+ cial '|' action is explained below):
+
+ foo |
+ bar$ /* action goes here */
+
+ A similar trick will work for matching a foo or a
+ bar-at-the-beginning-of-a-line.
+
+HOW THE INPUT IS MATCHED
+ When the generated scanner is run, it analyzes its input
+ looking for strings which match any of its patterns. If
+ it finds more than one match, it takes the one matching
+ the most text (for trailing context rules, this includes
+ the length of the trailing part, even though it will
+ then be returned to the input). If it finds two or more
+ matches of the same length, the rule listed first in the
+ flex input file is chosen.
+
+ Once the match is determined, the text corresponding to
+ the match (called the token) is made available in the
+ global character pointer yytext, and its length in the
+ global integer yyleng. The action corresponding to the
+ matched pattern is then executed (a more detailed
+ description of actions follows), and then the remaining
+ input is scanned for another match.
+
+ If no match is found, then the default rule is executed:
+ the next character in the input is considered matched
+ and copied to the standard output. Thus, the simplest
+ legal flex input is:
+
+ %%
+
+ which generates a scanner that simply copies its input
+ (one character at a time) to its output.
+
+ Note that yytext can be defined in two different ways:
+ either as a character pointer or as a character array.
+ You can control which definition flex uses by including
+ one of the special directives %pointer or %array in the
+ first (definitions) section of your flex input. The
+ default is %pointer, unless you use the -l lex compati-
+ bility option, in which case yytext will be an array.
+ The advantage of using %pointer is substantially faster
+ scanning and no buffer overflow when matching very large
+ tokens (unless you run out of dynamic memory). The dis-
+ advantage is that you are restricted in how your actions
+ can modify yytext (see the next section), and calls to
+ the unput() function destroys the present contents of
+ yytext, which can be a considerable porting headache
+ when moving between different lex versions.
+
+ The advantage of %array is that you can then modify
+ yytext to your heart's content, and calls to unput() do
+ not destroy yytext (see below). Furthermore, existing
+ lex programs sometimes access yytext externally using
+ declarations of the form:
+ extern char yytext[];
+ This definition is erroneous when used with %pointer,
+ but correct for %array.
+
+ %array defines yytext to be an array of YYLMAX charac-
+ ters, which defaults to a fairly large value. You can
+ change the size by simply #define'ing YYLMAX to a dif-
+ ferent value in the first section of your flex input.
+ As mentioned above, with %pointer yytext grows dynami-
+ cally to accommodate large tokens. While this means
+ your %pointer scanner can accommodate very large tokens
+ (such as matching entire blocks of comments), bear in
+ mind that each time the scanner must resize yytext it
+ also must rescan the entire token from the beginning, so
+ matching such tokens can prove slow. yytext presently
+ does not dynamically grow if a call to unput() results
+ in too much text being pushed back; instead, a run-time
+ error results.
+
+ Also note that you cannot use %array with C++ scanner
+ classes (the c++ option; see below).
+
+ACTIONS
+ Each pattern in a rule has a corresponding action, which
+ can be any arbitrary C statement. The pattern ends at
+ the first non-escaped whitespace character; the remain-
+ der of the line is its action. If the action is empty,
+ then when the pattern is matched the input token is sim-
+ ply discarded. For example, here is the specification
+ for a program which deletes all occurrences of "zap me"
+ from its input:
+
+ %%
+ "zap me"
+
+ (It will copy all other characters in the input to the
+ output since they will be matched by the default rule.)
+
+ Here is a program which compresses multiple blanks and
+ tabs down to a single blank, and throws away whitespace
+ found at the end of a line:
+
+ %%
+ [ \t]+ putchar( ' ' );
+ [ \t]+$ /* ignore this token */
+
+
+ If the action contains a '{', then the action spans till
+ the balancing '}' is found, and the action may cross
+ multiple lines. flex knows about C strings and comments
+ and won't be fooled by braces found within them, but
+ also allows actions to begin with %{ and will consider
+ the action to be all the text up to the next %} (regard-
+ less of ordinary braces inside the action).
+
+ An action consisting solely of a vertical bar ('|')
+ means "same as the action for the next rule." See below
+ for an illustration.
+
+ Actions can include arbitrary C code, including return
+ statements to return a value to whatever routine called
+ yylex(). Each time yylex() is called it continues pro-
+ cessing tokens from where it last left off until it
+ either reaches the end of the file or executes a return.
+
+ Actions are free to modify yytext except for lengthening
+ it (adding characters to its end--these will overwrite
+ later characters in the input stream). This however
+ does not apply when using %array (see above); in that
+ case, yytext may be freely modified in any way.
+
+ Actions are free to modify yyleng except they should not
+ do so if the action also includes use of yymore() (see
+ below).
+
+ There are a number of special directives which can be
+ included within an action:
+
+ - ECHO copies yytext to the scanner's output.
+
+ - BEGIN followed by the name of a start condition
+ places the scanner in the corresponding start
+ condition (see below).
+
+ - REJECT directs the scanner to proceed on to the
+ "second best" rule which matched the input (or a
+ prefix of the input). The rule is chosen as
+ described above in "How the Input is Matched",
+ and yytext and yyleng set up appropriately. It
+ may either be one which matched as much text as
+ the originally chosen rule but came later in the
+ flex input file, or one which matched less text.
+ For example, the following will both count the
+ words in the input and call the routine special()
+ whenever "frob" is seen:
+
+ int word_count = 0;
+ %%
+
+ frob special(); REJECT;
+ [^ \t\n]+ ++word_count;
+
+ Without the REJECT, any "frob"'s in the input
+ would not be counted as words, since the scanner
+ normally executes only one action per token.
+ Multiple REJECT's are allowed, each one finding
+ the next best choice to the currently active
+ rule. For example, when the following scanner
+ scans the token "abcd", it will write "abcdab-
+ caba" to the output:
+
+ %%
+ a |
+ ab |
+ abc |
+ abcd ECHO; REJECT;
+ .|\n /* eat up any unmatched character */
+
+ (The first three rules share the fourth's action
+ since they use the special '|' action.) REJECT
+ is a particularly expensive feature in terms of
+ scanner performance; if it is used in any of the
+ scanner's actions it will slow down all of the
+ scanner's matching. Furthermore, REJECT cannot
+ be used with the -Cf or -CF options (see below).
+
+ Note also that unlike the other special actions,
+ REJECT is a branch; code immediately following it
+ in the action will not be executed.
+
+ - yymore() tells the scanner that the next time it
+ matches a rule, the corresponding token should be
+ appended onto the current value of yytext rather
+ than replacing it. For example, given the input
+ "mega-kludge" the following will write "mega-
+ mega-kludge" to the output:
+
+ %%
+ mega- ECHO; yymore();
+ kludge ECHO;
+
+ First "mega-" is matched and echoed to the out-
+ put. Then "kludge" is matched, but the previous
+ "mega-" is still hanging around at the beginning
+ of yytext so the ECHO for the "kludge" rule will
+ actually write "mega-kludge".
+
+ Two notes regarding use of yymore(). First, yymore()
+ depends on the value of yyleng correctly reflecting the
+ size of the current token, so you must not modify yyleng
+ if you are using yymore(). Second, the presence of
+ yymore() in the scanner's action entails a minor perfor-
+ mance penalty in the scanner's matching speed.
+
+ - yyless(n) returns all but the first n characters
+ of the current token back to the input stream,
+ where they will be rescanned when the scanner
+ looks for the next match. yytext and yyleng are
+ adjusted appropriately (e.g., yyleng will now be
+ equal to n ). For example, on the input "foobar"
+ the following will write out "foobarbar":
+
+ %%
+ foobar ECHO; yyless(3);
+ [a-z]+ ECHO;
+
+ An argument of 0 to yyless will cause the entire
+ current input string to be scanned again. Unless
+ you've changed how the scanner will subsequently
+ process its input (using BEGIN, for example),
+ this will result in an endless loop.
+
+ Note that yyless is a macro and can only be used in the
+ flex input file, not from other source files.
+
+ - unput(c) puts the character c back onto the input
+ stream. It will be the next character scanned.
+ The following action will take the current token
+ and cause it to be rescanned enclosed in paren-
+ theses.
+
+ {
+ int i;
+ /* Copy yytext because unput() trashes yytext */
+ char *yycopy = strdup( yytext );
+ unput( ')' );
+ for ( i = yyleng - 1; i >= 0; --i )
+ unput( yycopy[i] );
+ unput( '(' );
+ free( yycopy );
+ }
+
+ Note that since each unput() puts the given char-
+ acter back at the beginning of the input stream,
+ pushing back strings must be done back-to-front.
+
+ An important potential problem when using unput() is
+ that if you are using %pointer (the default), a call to
+ unput() destroys the contents of yytext, starting with
+ its rightmost character and devouring one character to
+ the left with each call. If you need the value of
+ yytext preserved after a call to unput() (as in the
+ above example), you must either first copy it elsewhere,
+ or build your scanner using %array instead (see How The
+ Input Is Matched).
+
+ Finally, note that you cannot put back EOF to attempt to
+ mark the input stream with an end-of-file.
+
+ - input() reads the next character from the input
+ stream. For example, the following is one way to
+ eat up C comments:
+
+ %%
+ "/*" {
+ register int c;
+
+ for ( ; ; )
+ {
+ while ( (c = input()) != '*' &&
+ c != EOF )
+ ; /* eat up text of comment */
+
+ if ( c == '*' )
+ {
+ while ( (c = input()) == '*' )
+ ;
+ if ( c == '/' )
+ break; /* found the end */
+ }
+
+ if ( c == EOF )
+ {
+ error( "EOF in comment" );
+ break;
+ }
+ }
+ }
+
+ (Note that if the scanner is compiled using C++,
+ then input() is instead referred to as yyinput(),
+ in order to avoid a name clash with the C++
+ stream by the name of input.)
+
+ - YY_FLUSH_BUFFER flushes the scanner's internal
+ buffer so that the next time the scanner attempts
+ to match a token, it will first refill the buffer
+ using YY_INPUT (see The Generated Scanner,
+ below). This action is a special case of the
+ more general yy_flush_buffer() function,
+ described below in the section Multiple Input
+ Buffers.
+
+ - yyterminate() can be used in lieu of a return
+ statement in an action. It terminates the scan-
+ ner and returns a 0 to the scanner's caller,
+ indicating "all done". By default, yyterminate()
+ is also called when an end-of-file is encoun-
+ tered. It is a macro and may be redefined.
+
+THE GENERATED SCANNER
+ The output of flex is the file lex.yy.c, which contains
+ the scanning routine yylex(), a number of tables used by
+ it for matching tokens, and a number of auxiliary rou-
+ tines and macros. By default, yylex() is declared as
+ follows:
+
+ int yylex()
+ {
+ ... various definitions and the actions in here ...
+ }
+
+ (If your environment supports function prototypes, then
+ it will be "int yylex( void )".) This definition may be
+ changed by defining the "YY_DECL" macro. For example,
+ you could use:
+
+ #define YY_DECL float lexscan( a, b ) float a, b;
+
+ to give the scanning routine the name lexscan, returning
+ a float, and taking two floats as arguments. Note that
+ if you give arguments to the scanning routine using a
+ K&R-style/non-prototyped function declaration, you must
+ terminate the definition with a semi-colon (;).
+
+ Whenever yylex() is called, it scans tokens from the
+ global input file yyin (which defaults to stdin). It
+ continues until it either reaches an end-of-file (at
+ which point it returns the value 0) or one of its
+ actions executes a return statement.
+
+ If the scanner reaches an end-of-file, subsequent calls
+ are undefined unless either yyin is pointed at a new
+ input file (in which case scanning continues from that
+ file), or yyrestart() is called. yyrestart() takes one
+ argument, a FILE * pointer (which can be nil, if you've
+ set up YY_INPUT to scan from a source other than yyin),
+ and initializes yyin for scanning from that file.
+ Essentially there is no difference between just assign-
+ ing yyin to a new input file or using yyrestart() to do
+ so; the latter is available for compatibility with pre-
+ vious versions of flex, and because it can be used to
+ switch input files in the middle of scanning. It can
+ also be used to throw away the current input buffer, by
+ calling it with an argument of yyin; but better is to
+ use YY_FLUSH_BUFFER (see above). Note that yyrestart()
+ does not reset the start condition to INITIAL (see Start
+ Conditions, below).
+
+ If yylex() stops scanning due to executing a return
+ statement in one of the actions, the scanner may then be
+ called again and it will resume scanning where it left
+ off.
+
+ By default (and for purposes of efficiency), the scanner
+ uses block-reads rather than simple getc() calls to read
+ characters from yyin. The nature of how it gets its
+ input can be controlled by defining the YY_INPUT macro.
+ YY_INPUT's calling sequence is
+ "YY_INPUT(buf,result,max_size)". Its action is to place
+ up to max_size characters in the character array buf and
+ return in the integer variable result either the number
+ of characters read or the constant YY_NULL (0 on Unix
+ systems) to indicate EOF. The default YY_INPUT reads
+ from the global file-pointer "yyin".
+
+ A sample definition of YY_INPUT (in the definitions sec-
+ tion of the input file):
+
+ %{
+ #define YY_INPUT(buf,result,max_size) \
+ { \
+ int c = getchar(); \
+ result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \
+ }
+ %}
+
+ This definition will change the input processing to
+ occur one character at a time.
+
+ When the scanner receives an end-of-file indication from
+ YY_INPUT, it then checks the yywrap() function. If
+ yywrap() returns false (zero), then it is assumed that
+ the function has gone ahead and set up yyin to point to
+ another input file, and scanning continues. If it
+ returns true (non-zero), then the scanner terminates,
+ returning 0 to its caller. Note that in either case,
+ the start condition remains unchanged; it does not
+ revert to INITIAL.
+
+ If you do not supply your own version of yywrap(), then
+ you must either use %option noyywrap (in which case the
+ scanner behaves as though yywrap() returned 1), or you
+ must link with -lfl to obtain the default version of the
+ routine, which always returns 1.
+
+ Three routines are available for scanning from in-memory
+ buffers rather than files: yy_scan_string(),
+ yy_scan_bytes(), and yy_scan_buffer(). See the discus-
+ sion of them below in the section Multiple Input
+ Buffers.
+
+ The scanner writes its ECHO output to the yyout global
+ (default, stdout), which may be redefined by the user
+ simply by assigning it to some other FILE pointer.
+
+START CONDITIONS
+ flex provides a mechanism for conditionally activating
+ rules. Any rule whose pattern is prefixed with "<sc>"
+ will only be active when the scanner is in the start
+ condition named "sc". For example,
+
+ <STRING>[^"]* { /* eat up the string body ... */
+ ...
+ }
+
+ will be active only when the scanner is in the "STRING"
+ start condition, and
+
+ <INITIAL,STRING,QUOTE>\. { /* handle an escape ... */
+ ...
+ }
+
+ will be active only when the current start condition is
+ either "INITIAL", "STRING", or "QUOTE".
+
+ Start conditions are declared in the definitions (first)
+ section of the input using unindented lines beginning
+ with either %s or %x followed by a list of names. The
+ former declares inclusive start conditions, the latter
+ exclusive start conditions. A start condition is acti-
+ vated using the BEGIN action. Until the next BEGIN
+ action is executed, rules with the given start condition
+ will be active and rules with other start conditions
+ will be inactive. If the start condition is inclusive,
+ then rules with no start conditions at all will also be
+ active. If it is exclusive, then only rules qualified
+ with the start condition will be active. A set of rules
+ contingent on the same exclusive start condition
+ describe a scanner which is independent of any of the
+ other rules in the flex input. Because of this, exclu-
+ sive start conditions make it easy to specify "mini-
+ scanners" which scan portions of the input that are syn-
+ tactically different from the rest (e.g., comments).
+
+ If the distinction between inclusive and exclusive start
+ conditions is still a little vague, here's a simple
+ example illustrating the connection between the two.
+ The set of rules:
+
+ %s example
+ %%
+
+ <example>foo do_something();
+
+ bar something_else();
+
+ is equivalent to
+
+ %x example
+ %%
+
+ <example>foo do_something();
+
+ <INITIAL,example>bar something_else();
+
+ Without the <INITIAL,example> qualifier, the bar pattern
+ in the second example wouldn't be active (i.e., couldn't
+ match) when in start condition example. If we just used
+ <example> to qualify bar, though, then it would only be
+ active in example and not in INITIAL, while in the first
+ example it's active in both, because in the first exam-
+ ple the example startion condition is an inclusive (%s)
+ start condition.
+
+ Also note that the special start-condition specifier <*>
+ matches every start condition. Thus, the above example
+ could also have been written;
+
+ %x example
+ %%
+
+ <example>foo do_something();
+
+ <*>bar something_else();
+
+
+ The default rule (to ECHO any unmatched character)
+ remains active in start conditions. It is equivalent
+ to:
+
+ <*>.|\n ECHO;
+
+
+ BEGIN(0) returns to the original state where only the
+ rules with no start conditions are active. This state
+ can also be referred to as the start-condition "INI-
+ TIAL", so BEGIN(INITIAL) is equivalent to BEGIN(0).
+ (The parentheses around the start condition name are not
+ required but are considered good style.)
+
+ BEGIN actions can also be given as indented code at the
+ beginning of the rules section. For example, the fol-
+ lowing will cause the scanner to enter the "SPECIAL"
+ start condition whenever yylex() is called and the
+ global variable enter_special is true:
+
+ int enter_special;
+
+ %x SPECIAL
+ %%
+ if ( enter_special )
+ BEGIN(SPECIAL);
+
+ <SPECIAL>blahblahblah
+ ...more rules follow...
+
+
+ To illustrate the uses of start conditions, here is a
+ scanner which provides two different interpretations of
+ a string like "123.456". By default it will treat it as
+ three tokens, the integer "123", a dot ('.'), and the
+ integer "456". But if the string is preceded earlier in
+ the line by the string "expect-floats" it will treat it
+ as a single token, the floating-point number 123.456:
+
+ %{
+ #include <math.h>
+ %}
+ %s expect
+
+ %%
+ expect-floats BEGIN(expect);
+
+ <expect>[0-9]+"."[0-9]+ {
+ printf( "found a float, = %f\n",
+ atof( yytext ) );
+ }
+ <expect>\n {
+ /* that's the end of the line, so
+ * we need another "expect-number"
+ * before we'll recognize any more
+ * numbers
+ */
+ BEGIN(INITIAL);
+ }
+
+ [0-9]+ {
+ printf( "found an integer, = %d\n",
+ atoi( yytext ) );
+ }
+
+ "." printf( "found a dot\n" );
+
+ Here is a scanner which recognizes (and discards) C com-
+ ments while maintaining a count of the current input
+ line.
+
+ %x comment
+ %%
+ int line_num = 1;
+
+ "/*" BEGIN(comment);
+
+ <comment>[^*\n]* /* eat anything that's not a '*' */
+ <comment>"*"+[^*/\n]* /* eat up '*'s not followed by '/'s */
+ <comment>\n ++line_num;
+ <comment>"*"+"/" BEGIN(INITIAL);
+
+ This scanner goes to a bit of trouble to match as much
+ text as possible with each rule. In general, when
+ attempting to write a high-speed scanner try to match as
+ much possible in each rule, as it's a big win.
+
+ Note that start-conditions names are really integer val-
+ ues and can be stored as such. Thus, the above could be
+ extended in the following fashion:
+
+ %x comment foo
+ %%
+ int line_num = 1;
+ int comment_caller;
+
+ "/*" {
+ comment_caller = INITIAL;
+ BEGIN(comment);
+ }
+
+ ...
+
+ <foo>"/*" {
+ comment_caller = foo;
+ BEGIN(comment);
+ }
+
+ <comment>[^*\n]* /* eat anything that's not a '*' */
+ <comment>"*"+[^*/\n]* /* eat up '*'s not followed by '/'s */
+ <comment>\n ++line_num;
+ <comment>"*"+"/" BEGIN(comment_caller);
+
+ Furthermore, you can access the current start condition
+ using the integer-valued YY_START macro. For example,
+ the above assignments to comment_caller could instead be
+ written
+
+ comment_caller = YY_START;
+
+ Flex provides YYSTATE as an alias for YY_START (since
+ that is what's used by AT&T lex).
+
+ Note that start conditions do not have their own name-
+ space; %s's and %x's declare names in the same fashion
+ as #define's.
+
+ Finally, here's an example of how to match C-style
+ quoted strings using exclusive start conditions, includ-
+ ing expanded escape sequences (but not including check-
+ ing for a string that's too long):
+
+ %x str
+
+ %%
+ char string_buf[MAX_STR_CONST];
+ char *string_buf_ptr;
+
+
+ \" string_buf_ptr = string_buf; BEGIN(str);
+
+ <str>\" { /* saw closing quote - all done */
+ BEGIN(INITIAL);
+ *string_buf_ptr = '\0';
+ /* return string constant token type and
+ * value to parser
+ */
+ }
+
+ <str>\n {
+ /* error - unterminated string constant */
+ /* generate error message */
+ }
+
+ <str>\\[0-7]{1,3} {
+ /* octal escape sequence */
+ int result;
+
+ (void) sscanf( yytext + 1, "%o", &result );
+
+ if ( result > 0xff )
+ /* error, constant is out-of-bounds */
+
+ *string_buf_ptr++ = result;
+ }
+
+ <str>\\[0-9]+ {
+ /* generate error - bad escape sequence; something
+ * like '\48' or '\0777777'
+ */
+ }
+
+ <str>\\n *string_buf_ptr++ = '\n';
+ <str>\\t *string_buf_ptr++ = '\t';
+ <str>\\r *string_buf_ptr++ = '\r';
+ <str>\\b *string_buf_ptr++ = '\b';
+ <str>\\f *string_buf_ptr++ = '\f';
+
+ <str>\\(.|\n) *string_buf_ptr++ = yytext[1];
+
+ <str>[^\\\n\"]+ {
+ char *yptr = yytext;
+
+ while ( *yptr )
+ *string_buf_ptr++ = *yptr++;
+ }
+
+
+ Often, such as in some of the examples above, you wind
+ up writing a whole bunch of rules all preceded by the
+ same start condition(s). Flex makes this a little eas-
+ ier and cleaner by introducing a notion of start condi-
+ tion scope. A start condition scope is begun with:
+
+ <SCs>{
+
+ where SCs is a list of one or more start conditions.
+ Inside the start condition scope, every rule automati-
+ cally has the prefix <SCs> applied to it, until a '}'
+ which matches the initial '{'. So, for example,
+
+ <ESC>{
+ "\\n" return '\n';
+ "\\r" return '\r';
+ "\\f" return '\f';
+ "\\0" return '\0';
+ }
+
+ is equivalent to:
+
+ <ESC>"\\n" return '\n';
+ <ESC>"\\r" return '\r';
+ <ESC>"\\f" return '\f';
+ <ESC>"\\0" return '\0';
+
+ Start condition scopes may be nested.
+
+ Three routines are available for manipulating stacks of
+ start conditions:
+
+ void yy_push_state(int new_state)
+ pushes the current start condition onto the top
+ of the start condition stack and switches to
+ new_state as though you had used BEGIN new_state
+ (recall that start condition names are also inte-
+ gers).
+
+ void yy_pop_state()
+ pops the top of the stack and switches to it via
+ BEGIN.
+
+ int yy_top_state()
+ returns the top of the stack without altering the
+ stack's contents.
+
+ The start condition stack grows dynamically and so has
+ no built-in size limitation. If memory is exhausted,
+ program execution aborts.
+
+ To use start condition stacks, your scanner must include
+ a %option stack directive (see Options below).
+
+MULTIPLE INPUT BUFFERS
+ Some scanners (such as those which support "include"
+ files) require reading from several input streams. As
+ flex scanners do a large amount of buffering, one cannot
+ control where the next input will be read from by simply
+ writing a YY_INPUT which is sensitive to the scanning
+ context. YY_INPUT is only called when the scanner
+ reaches the end of its buffer, which may be a long time
+ after scanning a statement such as an "include" which
+ requires switching the input source.
+
+ To negotiate these sorts of problems, flex provides a
+ mechanism for creating and switching between multiple
+ input buffers. An input buffer is created by using:
+
+ YY_BUFFER_STATE yy_create_buffer( FILE *file, int size )
+
+ which takes a FILE pointer and a size and creates a
+ buffer associated with the given file and large enough
+ to hold size characters (when in doubt, use YY_BUF_SIZE
+ for the size). It returns a YY_BUFFER_STATE handle,
+ which may then be passed to other routines (see below).
+ The YY_BUFFER_STATE type is a pointer to an opaque
+ struct yy_buffer_state structure, so you may safely ini-
+ tialize YY_BUFFER_STATE variables to ((YY_BUFFER_STATE)
+ 0) if you wish, and also refer to the opaque structure
+ in order to correctly declare input buffers in source
+ files other than that of your scanner. Note that the
+ FILE pointer in the call to yy_create_buffer is only
+ used as the value of yyin seen by YY_INPUT; if you rede-
+ fine YY_INPUT so it no longer uses yyin, then you can
+ safely pass a nil FILE pointer to yy_create_buffer. You
+ select a particular buffer to scan from using:
+
+ void yy_switch_to_buffer( YY_BUFFER_STATE new_buffer )
+
+ switches the scanner's input buffer so subsequent tokens
+ will come from new_buffer. Note that
+ yy_switch_to_buffer() may be used by yywrap() to set
+ things up for continued scanning, instead of opening a
+ new file and pointing yyin at it. Note also that
+ switching input sources via either yy_switch_to_buffer()
+ or yywrap() does not change the start condition.
+
+ void yy_delete_buffer( YY_BUFFER_STATE buffer )
+
+ is used to reclaim the storage associated with a buffer.
+ ( buffer can be nil, in which case the routine does
+ nothing.) You can also clear the current contents of a
+ buffer using:
+
+ void yy_flush_buffer( YY_BUFFER_STATE buffer )
+
+ This function discards the buffer's contents, so the
+ next time the scanner attempts to match a token from the
+ buffer, it will first fill the buffer anew using
+ YY_INPUT.
+
+ yy_new_buffer() is an alias for yy_create_buffer(), pro-
+ vided for compatibility with the C++ use of new and
+ delete for creating and destroying dynamic objects.
+
+ Finally, the YY_CURRENT_BUFFER macro returns a
+ YY_BUFFER_STATE handle to the current buffer.
+
+ Here is an example of using these features for writing a
+ scanner which expands include files (the <<EOF>> feature
+ is discussed below):
+
+ /* the "incl" state is used for picking up the name
+ * of an include file
+ */
+ %x incl
+
+ %{
+ #define MAX_INCLUDE_DEPTH 10
+ YY_BUFFER_STATE include_stack[MAX_INCLUDE_DEPTH];
+ int include_stack_ptr = 0;
+ %}
+
+ %%
+ include BEGIN(incl);
+
+ [a-z]+ ECHO;
+ [^a-z\n]*\n? ECHO;
+
+ <incl>[ \t]* /* eat the whitespace */
+ <incl>[^ \t\n]+ { /* got the include file name */
+ if ( include_stack_ptr >= MAX_INCLUDE_DEPTH )
+ {
+ fprintf( stderr, "Includes nested too deeply" );
+ exit( 1 );
+ }
+
+ include_stack[include_stack_ptr++] =
+ YY_CURRENT_BUFFER;
+
+ yyin = fopen( yytext, "r" );
+
+ if ( ! yyin )
+ error( ... );
+
+ yy_switch_to_buffer(
+ yy_create_buffer( yyin, YY_BUF_SIZE ) );
+
+ BEGIN(INITIAL);
+ }
+
+ <<EOF>> {
+ if ( --include_stack_ptr < 0 )
+ {
+ yyterminate();
+ }
+
+ else
+ {
+ yy_delete_buffer( YY_CURRENT_BUFFER );
+ yy_switch_to_buffer(
+ include_stack[include_stack_ptr] );
+ }
+ }
+
+ Three routines are available for setting up input
+ buffers for scanning in-memory strings instead of files.
+ All of them create a new input buffer for scanning the
+ string, and return a corresponding YY_BUFFER_STATE han-
+ dle (which you should delete with yy_delete_buffer()
+ when done with it). They also switch to the new buffer
+ using yy_switch_to_buffer(), so the next call to yylex()
+ will start scanning the string.
+
+ yy_scan_string(const char *str)
+ scans a NUL-terminated string.
+
+ yy_scan_bytes(const char *bytes, int len)
+ scans len bytes (including possibly NUL's) start-
+ ing at location bytes.
+
+ Note that both of these functions create and scan a copy
+ of the string or bytes. (This may be desirable, since
+ yylex() modifies the contents of the buffer it is scan-
+ ning.) You can avoid the copy by using:
+
+ yy_scan_buffer(char *base, yy_size_t size)
+ which scans in place the buffer starting at base,
+ consisting of size bytes, the last two bytes of
+ which must be YY_END_OF_BUFFER_CHAR (ASCII NUL).
+ These last two bytes are not scanned; thus, scan-
+ ning consists of base[0] through base[size-2],
+ inclusive.
+
+ If you fail to set up base in this manner (i.e.,
+ forget the final two YY_END_OF_BUFFER_CHAR
+ bytes), then yy_scan_buffer() returns a nil
+ pointer instead of creating a new input buffer.
+
+ The type yy_size_t is an integral type to which
+ you can cast an integer expression reflecting the
+ size of the buffer.
+
+END-OF-FILE RULES
+ The special rule "<<EOF>>" indicates actions which are
+ to be taken when an end-of-file is encountered and
+ yywrap() returns non-zero (i.e., indicates no further
+ files to process). The action must finish by doing one
+ of four things:
+
+ - assigning yyin to a new input file (in previous
+ versions of flex, after doing the assignment you
+ had to call the special action YY_NEW_FILE; this
+ is no longer necessary);
+
+ - executing a return statement;
+
+ - executing the special yyterminate() action;
+
+ - or, switching to a new buffer using
+ yy_switch_to_buffer() as shown in the example
+ above.
+
+ <<EOF>> rules may not be used with other patterns; they
+ may only be qualified with a list of start conditions.
+ If an unqualified <<EOF>> rule is given, it applies to
+ all start conditions which do not already have <<EOF>>
+ actions. To specify an <<EOF>> rule for only the ini-
+ tial start condition, use
+
+ <INITIAL><<EOF>>
+
+
+ These rules are useful for catching things like unclosed
+ comments. An example:
+
+ %x quote
+ %%
+
+ ...other rules for dealing with quotes...
+
+ <quote><<EOF>> {
+ error( "unterminated quote" );
+ yyterminate();
+ }
+ <<EOF>> {
+ if ( *++filelist )
+ yyin = fopen( *filelist, "r" );
+ else
+ yyterminate();
+ }
+
+
+MISCELLANEOUS MACROS
+ The macro YY_USER_ACTION can be defined to provide an
+ action which is always executed prior to the matched
+ rule's action. For example, it could be #define'd to
+ call a routine to convert yytext to lower-case. When
+ YY_USER_ACTION is invoked, the variable yy_act gives the
+ number of the matched rule (rules are numbered starting
+ with 1). Suppose you want to profile how often each of
+ your rules is matched. The following would do the
+ trick:
+
+ #define YY_USER_ACTION ++ctr[yy_act]
+
+ where ctr is an array to hold the counts for the differ-
+ ent rules. Note that the macro YY_NUM_RULES gives the
+ total number of rules (including the default rule, even
+ if you use -s), so a correct declaration for ctr is:
+
+ int ctr[YY_NUM_RULES];
+
+
+ The macro YY_USER_INIT may be defined to provide an
+ action which is always executed before the first scan
+ (and before the scanner's internal initializations are
+ done). For example, it could be used to call a routine
+ to read in a data table or open a logging file.
+
+ The macro yy_set_interactive(is_interactive) can be used
+ to control whether the current buffer is considered
+ interactive. An interactive buffer is processed more
+ slowly, but must be used when the scanner's input source
+ is indeed interactive to avoid problems due to waiting
+ to fill buffers (see the discussion of the -I flag
+ below). A non-zero value in the macro invocation marks
+ the buffer as interactive, a zero value as non-interac-
+ tive. Note that use of this macro overrides %option
+ always-interactive or %option never-interactive (see
+ Options below). yy_set_interactive() must be invoked
+ prior to beginning to scan the buffer that is (or is
+ not) to be considered interactive.
+
+ The macro yy_set_bol(at_bol) can be used to control
+ whether the current buffer's scanning context for the
+ next token match is done as though at the beginning of a
+ line. A non-zero macro argument makes rules anchored
+ with
+
+ The macro YY_AT_BOL() returns true if the next token
+ scanned from the current buffer will have '^' rules
+ active, false otherwise.
+
+ In the generated scanner, the actions are all gathered
+ in one large switch statement and separated using
+ YY_BREAK, which may be redefined. By default, it is
+ simply a "break", to separate each rule's action from
+ the following rule's. Redefining YY_BREAK allows, for
+ example, C++ users to #define YY_BREAK to do nothing
+ (while being very careful that every rule ends with a
+ "break" or a "return"!) to avoid suffering from unreach-
+ able statement warnings where because a rule's action
+ ends with "return", the YY_BREAK is inaccessible.
+
+VALUES AVAILABLE TO THE USER
+ This section summarizes the various values available to
+ the user in the rule actions.
+
+ - char *yytext holds the text of the current token.
+ It may be modified but not lengthened (you cannot
+ append characters to the end).
+
+ If the special directive %array appears in the
+ first section of the scanner description, then
+ yytext is instead declared char yytext[YYLMAX],
+ where YYLMAX is a macro definition that you can
+ redefine in the first section if you don't like
+ the default value (generally 8KB). Using %array
+ results in somewhat slower scanners, but the
+ value of yytext becomes immune to calls to
+ input() and unput(), which potentially destroy
+ its value when yytext is a character pointer.
+ The opposite of %array is %pointer, which is the
+ default.
+
+ You cannot use %array when generating C++ scanner
+ classes (the -+ flag).
+
+ - int yyleng holds the length of the current token.
+
+ - FILE *yyin is the file which by default flex
+ reads from. It may be redefined but doing so
+ only makes sense before scanning begins or after
+ an EOF has been encountered. Changing it in the
+ midst of scanning will have unexpected results
+ since flex buffers its input; use yyrestart()
+ instead. Once scanning terminates because an
+ end-of-file has been seen, you can assign yyin at
+ the new input file and then call the scanner
+ again to continue scanning.
+
+ - void yyrestart( FILE *new_file ) may be called to
+ point yyin at the new input file. The switch-
+ over to the new file is immediate (any previously
+ buffered-up input is lost). Note that calling
+ yyrestart() with yyin as an argument thus throws
+ away the current input buffer and continues scan-
+ ning the same input file.
+
+ - FILE *yyout is the file to which ECHO actions are
+ done. It can be reassigned by the user.
+
+ - YY_CURRENT_BUFFER returns a YY_BUFFER_STATE han-
+ dle to the current buffer.
+
+ - YY_START returns an integer value corresponding
+ to the current start condition. You can subse-
+ quently use this value with BEGIN to return to
+ that start condition.
+
+INTERFACING WITH YACC
+ One of the main uses of flex is as a companion to the
+ yacc parser-generator. yacc parsers expect to call a
+ routine named yylex() to find the next input token. The
+ routine is supposed to return the type of the next token
+ as well as putting any associated value in the global
+ yylval. To use flex with yacc, one specifies the -d
+ option to yacc to instruct it to generate the file
+ y.tab.h containing definitions of all the %tokens
+ appearing in the yacc input. This file is then included
+ in the flex scanner. For example, if one of the tokens
+ is "TOK_NUMBER", part of the scanner might look like:
+
+ %{
+ #include "y.tab.h"
+ %}
+
+ %%
+
+ [0-9]+ yylval = atoi( yytext ); return TOK_NUMBER;
+
+
+OPTIONS
+ flex has the following options:
+
+ -b Generate backing-up information to lex.backup.
+ This is a list of scanner states which require
+ backing up and the input characters on which they
+ do so. By adding rules one can remove backing-up
+ states. If all backing-up states are eliminated
+ and -Cf or -CF is used, the generated scanner
+ will run faster (see the -p flag). Only users
+ who wish to squeeze every last cycle out of their
+ scanners need worry about this option. (See the
+ section on Performance Considerations below.)
+
+ -c is a do-nothing, deprecated option included for
+ POSIX compliance.
+
+ -d makes the generated scanner run in debug mode.
+ Whenever a pattern is recognized and the global
+ yy_flex_debug is non-zero (which is the default),
+ the scanner will write to stderr a line of the
+ form:
+
+ --accepting rule at line 53 ("the matched text")
+
+ The line number refers to the location of the
+ rule in the file defining the scanner (i.e., the
+ file that was fed to flex). Messages are also
+ generated when the scanner backs up, accepts the
+ default rule, reaches the end of its input buffer
+ (or encounters a NUL; at this point, the two look
+ the same as far as the scanner's concerned), or
+ reaches an end-of-file.
+
+ -f specifies fast scanner. No table compression is
+ done and stdio is bypassed. The result is large
+ but fast. This option is equivalent to -Cfr (see
+ below).
+
+ -h generates a "help" summary of flex's options to
+ stdout and then exits. -? and --help are syn-
+ onyms for -h.
+
+ -i instructs flex to generate a case-insensitive
+ scanner. The case of letters given in the flex
+ input patterns will be ignored, and tokens in the
+ input will be matched regardless of case. The
+ matched text given in yytext will have the pre-
+ served case (i.e., it will not be folded).
+
+ -l turns on maximum compatibility with the original
+ AT&T lex implementation. Note that this does not
+ mean full compatibility. Use of this option
+ costs a considerable amount of performance, and
+ it cannot be used with the -+, -f, -F, -Cf, or
+ -CF options. For details on the compatibilities
+ it provides, see the section "Incompatibilities
+ With Lex And POSIX" below. This option also
+ results in the name YY_FLEX_LEX_COMPAT being
+ #define'd in the generated scanner.
+
+ -n is another do-nothing, deprecated option included
+ only for POSIX compliance.
+
+ -p generates a performance report to stderr. The
+ report consists of comments regarding features of
+ the flex input file which will cause a serious
+ loss of performance in the resulting scanner. If
+ you give the flag twice, you will also get com-
+ ments regarding features that lead to minor per-
+ formance losses.
+
+ Note that the use of REJECT, %option yylineno,
+ and variable trailing context (see the Deficien-
+ cies / Bugs section below) entails a substantial
+ performance penalty; use of yymore(), the ^ oper-
+ ator, and the -I flag entail minor performance
+ penalties.
+
+ -s causes the default rule (that unmatched scanner
+ input is echoed to stdout) to be suppressed. If
+ the scanner encounters input that does not match
+ any of its rules, it aborts with an error. This
+ option is useful for finding holes in a scanner's
+ rule set.
+
+ -t instructs flex to write the scanner it generates
+ to standard output instead of lex.yy.c.
+
+ -v specifies that flex should write to stderr a sum-
+ mary of statistics regarding the scanner it gen-
+ erates. Most of the statistics are meaningless
+ to the casual flex user, but the first line iden-
+ tifies the version of flex (same as reported by
+ -V), and the next line the flags used when gener-
+ ating the scanner, including those that are on by
+ default.
+
+ -w suppresses warning messages.
+
+ -B instructs flex to generate a batch scanner, the
+ opposite of interactive scanners generated by -I
+ (see below). In general, you use -B when you are
+ certain that your scanner will never be used
+ interactively, and you want to squeeze a little
+ more performance out of it. If your goal is
+ instead to squeeze out a lot more performance,
+ you should be using the -Cf or -CF options (dis-
+ cussed below), which turn on -B automatically
+ anyway.
+
+ -F specifies that the fast scanner table representa-
+ tion should be used (and stdio bypassed). This
+ representation is about as fast as the full table
+ representation (-f), and for some sets of pat-
+ terns will be considerably smaller (and for oth-
+ ers, larger). In general, if the pattern set
+ contains both "keywords" and a catch-all, "iden-
+ tifier" rule, such as in the set:
+
+ "case" return TOK_CASE;
+ "switch" return TOK_SWITCH;
+ ...
+ "default" return TOK_DEFAULT;
+ [a-z]+ return TOK_ID;
+
+ then you're better off using the full table rep-
+ resentation. If only the "identifier" rule is
+ present and you then use a hash table or some
+ such to detect the keywords, you're better off
+ using -F.
+
+ This option is equivalent to -CFr (see below).
+ It cannot be used with -+.
+
+ -I instructs flex to generate an interactive scan-
+ ner. An interactive scanner is one that only
+ looks ahead to decide what token has been matched
+ if it absolutely must. It turns out that always
+ looking one extra character ahead, even if the
+ scanner has already seen enough text to disam-
+ biguate the current token, is a bit faster than
+ only looking ahead when necessary. But scanners
+ that always look ahead give dreadful interactive
+ performance; for example, when a user types a
+ newline, it is not recognized as a newline token
+ until they enter another token, which often means
+ typing in another whole line.
+
+ Flex scanners default to interactive unless you
+ use the -Cf or -CF table-compression options (see
+ below). That's because if you're looking for
+ high-performance you should be using one of these
+ options, so if you didn't, flex assumes you'd
+ rather trade off a bit of run-time performance
+ for intuitive interactive behavior. Note also
+ that you cannot use -I in conjunction with -Cf or
+ -CF. Thus, this option is not really needed; it
+ is on by default for all those cases in which it
+ is allowed.
+
+ You can force a scanner to not be interactive by
+ using -B (see above).
+
+ -L instructs flex not to generate #line directives.
+ Without this option, flex peppers the generated
+ scanner with #line directives so error messages
+ in the actions will be correctly located with
+ respect to either the original flex input file
+ (if the errors are due to code in the input
+ file), or lex.yy.c (if the errors are flex's
+ fault -- you should report these sorts of errors
+ to the email address given below).
+
+ -T makes flex run in trace mode. It will generate a
+ lot of messages to stderr concerning the form of
+ the input and the resultant non-deterministic and
+ deterministic finite automata. This option is
+ mostly for use in maintaining flex.
+
+ -V prints the version number to stdout and exits.
+ --version is a synonym for -V.
+
+ -7 instructs flex to generate a 7-bit scanner, i.e.,
+ one which can only recognized 7-bit characters in
+ its input. The advantage of using -7 is that the
+ scanner's tables can be up to half the size of
+ those generated using the -8 option (see below).
+ The disadvantage is that such scanners often hang
+ or crash if their input contains an 8-bit charac-
+ ter.
+
+ Note, however, that unless you generate your
+ scanner using the -Cf or -CF table compression
+ options, use of -7 will save only a small amount
+ of table space, and make your scanner consider-
+ ably less portable. Flex's default behavior is
+ to generate an 8-bit scanner unless you use the
+ -Cf or -CF, in which case flex defaults to gener-
+ ating 7-bit scanners unless your site was always
+ configured to generate 8-bit scanners (as will
+ often be the case with non-USA sites). You can
+ tell whether flex generated a 7-bit or an 8-bit
+ scanner by inspecting the flag summary in the -v
+ output as described above.
+
+ Note that if you use -Cfe or -CFe (those table
+ compression options, but also using equivalence
+ classes as discussed see below), flex still
+ defaults to generating an 8-bit scanner, since
+ usually with these compression options full 8-bit
+ tables are not much more expensive than 7-bit
+ tables.
+
+ -8 instructs flex to generate an 8-bit scanner,
+ i.e., one which can recognize 8-bit characters.
+ This flag is only needed for scanners generated
+ using -Cf or -CF, as otherwise flex defaults to
+ generating an 8-bit scanner anyway.
+
+ See the discussion of -7 above for flex's default
+ behavior and the tradeoffs between 7-bit and
+ 8-bit scanners.
+
+ -+ specifies that you want flex to generate a C++
+ scanner class. See the section on Generating C++
+ Scanners below for details.
+
+ -C[aefFmr]
+ controls the degree of table compression and,
+ more generally, trade-offs between small scanners
+ and fast scanners.
+
+ -Ca ("align") instructs flex to trade off larger
+ tables in the generated scanner for faster per-
+ formance because the elements of the tables are
+ better aligned for memory access and computation.
+ On some RISC architectures, fetching and manipu-
+ lating longwords is more efficient than with
+ smaller-sized units such as shortwords. This
+ option can double the size of the tables used by
+ your scanner.
+
+ -Ce directs flex to construct equivalence
+ classes, i.e., sets of characters which have
+ identical lexical properties (for example, if the
+ only appearance of digits in the flex input is in
+ the character class "[0-9]" then the digits '0',
+ '1', ..., '9' will all be put in the same equiva-
+ lence class). Equivalence classes usually give
+ dramatic reductions in the final table/object
+ file sizes (typically a factor of 2-5) and are
+ pretty cheap performance-wise (one array look-up
+ per character scanned).
+
+ -Cf specifies that the full scanner tables should
+ be generated - flex should not compress the
+ tables by taking advantages of similar transition
+ functions for different states.
+
+ -CF specifies that the alternate fast scanner
+ representation (described above under the -F
+ flag) should be used. This option cannot be used
+ with -+.
+
+ -Cm directs flex to construct meta-equivalence
+ classes, which are sets of equivalence classes
+ (or characters, if equivalence classes are not
+ being used) that are commonly used together.
+ Meta-equivalence classes are often a big win when
+ using compressed tables, but they have a moderate
+ performance impact (one or two "if" tests and one
+ array look-up per character scanned).
+
+ -Cr causes the generated scanner to bypass use of
+ the standard I/O library (stdio) for input.
+ Instead of calling fread() or getc(), the scanner
+ will use the read() system call, resulting in a
+ performance gain which varies from system to sys-
+ tem, but in general is probably negligible unless
+ you are also using -Cf or -CF. Using -Cr can
+ cause strange behavior if, for example, you read
+ from yyin using stdio prior to calling the scan-
+ ner (because the scanner will miss whatever text
+ your previous reads left in the stdio input
+ buffer).
+
+ -Cr has no effect if you define YY_INPUT (see The
+ Generated Scanner above).
+
+ A lone -C specifies that the scanner tables
+ should be compressed but neither equivalence
+ classes nor meta-equivalence classes should be
+ used.
+
+ The options -Cf or -CF and -Cm do not make sense
+ together - there is no opportunity for meta-
+ equivalence classes if the table is not being
+ compressed. Otherwise the options may be freely
+ mixed, and are cumulative.
+
+ The default setting is -Cem, which specifies that
+ flex should generate equivalence classes and
+ meta-equivalence classes. This setting provides
+ the highest degree of table compression. You can
+ trade off faster-executing scanners at the cost
+ of larger tables with the following generally
+ being true:
+
+ slowest & smallest
+ -Cem
+ -Cm
+ -Ce
+ -C
+ -C{f,F}e
+ -C{f,F}
+ -C{f,F}a
+ fastest & largest
+
+ Note that scanners with the smallest tables are
+ usually generated and compiled the quickest, so
+ during development you will usually want to use
+ the default, maximal compression.
+
+ -Cfe is often a good compromise between speed and
+ size for production scanners.
+
+ -ooutput
+ directs flex to write the scanner to the file
+ output instead of lex.yy.c. If you combine -o
+ with the -t option, then the scanner is written
+ to stdout but its #line directives (see the -L
+ option above) refer to the file output.
+
+ -Pprefix
+ changes the default yy prefix used by flex for
+ all globally-visible variable and function names
+ to instead be prefix. For example, -Pfoo changes
+ the name of yytext to footext. It also changes
+ the name of the default output file from lex.yy.c
+ to lex.foo.c. Here are all of the names
+ affected:
+
+ yy_create_buffer
+ yy_delete_buffer
+ yy_flex_debug
+ yy_init_buffer
+ yy_flush_buffer
+ yy_load_buffer_state
+ yy_switch_to_buffer
+ yyin
+ yyleng
+ yylex
+ yylineno
+ yyout
+ yyrestart
+ yytext
+ yywrap
+
+ (If you are using a C++ scanner, then only yywrap
+ and yyFlexLexer are affected.) Within your scan-
+ ner itself, you can still refer to the global
+ variables and functions using either version of
+ their name; but externally, they have the modi-
+ fied name.
+
+ This option lets you easily link together multi-
+ ple flex programs into the same executable.
+ Note, though, that using this option also renames
+ yywrap(), so you now must either provide your own
+ (appropriately-named) version of the routine for
+ your scanner, or use %option noyywrap, as linking
+ with -lfl no longer provides one for you by
+ default.
+
+ -Sskeleton_file
+ overrides the default skeleton file from which
+ flex constructs its scanners. You'll never need
+ this option unless you are doing flex maintenance
+ or development.
+
+ flex also provides a mechanism for controlling options
+ within the scanner specification itself, rather than
+ from the flex command-line. This is done by including
+ %option directives in the first section of the scanner
+ specification. You can specify multiple options with a
+ single %option directive, and multiple directives in the
+ first section of your flex input file.
+
+ Most options are given simply as names, optionally pre-
+ ceded by the word "no" (with no intervening whitespace)
+ to negate their meaning. A number are equivalent to
+ flex flags or their negation:
+
+ 7bit -7 option
+ 8bit -8 option
+ align -Ca option
+ backup -b option
+ batch -B option
+ c++ -+ option
+
+ caseful or
+ case-sensitive opposite of -i (default)
+
+ case-insensitive or
+ caseless -i option
+
+ debug -d option
+ default opposite of -s option
+ ecs -Ce option
+ fast -F option
+ full -f option
+ interactive -I option
+ lex-compat -l option
+ meta-ecs -Cm option
+ perf-report -p option
+ read -Cr option
+ stdout -t option
+ verbose -v option
+ warn opposite of -w option
+ (use "%option nowarn" for -w)
+
+ array equivalent to "%array"
+ pointer equivalent to "%pointer" (default)
+
+ Some %option's provide features otherwise not available:
+
+ always-interactive
+ instructs flex to generate a scanner which always
+ considers its input "interactive". Normally, on
+ each new input file the scanner calls isatty() in
+ an attempt to determine whether the scanner's
+ input source is interactive and thus should be
+ read a character at a time. When this option is
+ used, however, then no such call is made.
+
+ main directs flex to provide a default main() program
+ for the scanner, which simply calls yylex().
+ This option implies noyywrap (see below).
+
+ never-interactive
+ instructs flex to generate a scanner which never
+ considers its input "interactive" (again, no call
+ made to isatty()). This is the opposite of
+ always-interactive.
+
+ stack enables the use of start condition stacks (see
+ Start Conditions above).
+
+ stdinit
+ if set (i.e., %option stdinit) initializes yyin
+ and yyout to stdin and stdout, instead of the
+ default of nil. Some existing lex programs
+ depend on this behavior, even though it is not
+ compliant with ANSI C, which does not require
+ stdin and stdout to be compile-time constant.
+
+ yylineno
+ directs flex to generate a scanner that maintains
+ the number of the current line read from its
+ input in the global variable yylineno. This
+ option is implied by %option lex-compat.
+
+ yywrap if unset (i.e., %option noyywrap), makes the
+ scanner not call yywrap() upon an end-of-file,
+ but simply assume that there are no more files to
+ scan (until the user points yyin at a new file
+ and calls yylex() again).
+
+ flex scans your rule actions to determine whether you
+ use the REJECT or yymore() features. The reject and
+ yymore options are available to override its decision as
+ to whether you use the options, either by setting them
+ (e.g., %option reject) to indicate the feature is indeed
+ used, or unsetting them to indicate it actually is not
+ used (e.g., %option noyymore).
+
+ Three options take string-delimited values, offset with
+ '=':
+
+ %option outfile="ABC"
+
+ is equivalent to -oABC, and
+
+ %option prefix="XYZ"
+
+ is equivalent to -PXYZ. Finally,
+
+ %option yyclass="foo"
+
+ only applies when generating a C++ scanner ( -+ option).
+ It informs flex that you have derived foo as a subclass
+ of yyFlexLexer, so flex will place your actions in the
+ member function foo::yylex() instead of
+ yyFlexLexer::yylex(). It also generates a
+ yyFlexLexer::yylex() member function that emits a run-
+ time error (by invoking yyFlexLexer::LexerError()) if
+ called. See Generating C++ Scanners, below, for addi-
+ tional information.
+
+ A number of options are available for lint purists who
+ want to suppress the appearance of unneeded routines in
+ the generated scanner. Each of the following, if unset
+ (e.g., %option nounput ), results in the corresponding
+ routine not appearing in the generated scanner:
+
+ input, unput
+ yy_push_state, yy_pop_state, yy_top_state
+ yy_scan_buffer, yy_scan_bytes, yy_scan_string
+
+ (though yy_push_state() and friends won't appear anyway
+ unless you use %option stack).
+
+PERFORMANCE CONSIDERATIONS
+ The main design goal of flex is that it generate high-
+ performance scanners. It has been optimized for dealing
+ well with large sets of rules. Aside from the effects
+ on scanner speed of the table compression -C options
+ outlined above, there are a number of options/actions
+ which degrade performance. These are, from most expen-
+ sive to least:
+
+ REJECT
+ %option yylineno
+ arbitrary trailing context
+
+ pattern sets that require backing up
+ %array
+ %option interactive
+ %option always-interactive
+
+ '^' beginning-of-line operator
+ yymore()
+
+ with the first three all being quite expensive and the
+ last two being quite cheap. Note also that unput() is
+ implemented as a routine call that potentially does
+ quite a bit of work, while yyless() is a quite-cheap
+ macro; so if just putting back some excess text you
+ scanned, use yyless().
+
+ REJECT should be avoided at all costs when performance
+ is important. It is a particularly expensive option.
+
+ Getting rid of backing up is messy and often may be an
+ enormous amount of work for a complicated scanner. In
+ principal, one begins by using the -b flag to generate a
+ lex.backup file. For example, on the input
+
+ %%
+ foo return TOK_KEYWORD;
+ foobar return TOK_KEYWORD;
+
+ the file looks like:
+
+ State #6 is non-accepting -
+ associated rule line numbers:
+ 2 3
+ out-transitions: [ o ]
+ jam-transitions: EOF [ \001-n p-\177 ]
+
+ State #8 is non-accepting -
+ associated rule line numbers:
+ 3
+ out-transitions: [ a ]
+ jam-transitions: EOF [ \001-` b-\177 ]
+
+ State #9 is non-accepting -
+ associated rule line numbers:
+ 3
+ out-transitions: [ r ]
+ jam-transitions: EOF [ \001-q s-\177 ]
+
+ Compressed tables always back up.
+
+ The first few lines tell us that there's a scanner state
+ in which it can make a transition on an 'o' but not on
+ any other character, and that in that state the cur-
+ rently scanned text does not match any rule. The state
+ occurs when trying to match the rules found at lines 2
+ and 3 in the input file. If the scanner is in that
+ state and then reads something other than an 'o', it
+ will have to back up to find a rule which is matched.
+ With a bit of headscratching one can see that this must
+ be the state it's in when it has seen "fo". When this
+ has happened, if anything other than another 'o' is
+ seen, the scanner will have to back up to simply match
+ the 'f' (by the default rule).
+
+ The comment regarding State #8 indicates there's a prob-
+ lem when "foob" has been scanned. Indeed, on any char-
+ acter other than an 'a', the scanner will have to back
+ up to accept "foo". Similarly, the comment for State #9
+ concerns when "fooba" has been scanned and an 'r' does
+ not follow.
+
+ The final comment reminds us that there's no point going
+ to all the trouble of removing backing up from the rules
+ unless we're using -Cf or -CF, since there's no perfor-
+ mance gain doing so with compressed scanners.
+
+ The way to remove the backing up is to add "error"
+ rules:
+
+ %%
+ foo return TOK_KEYWORD;
+ foobar return TOK_KEYWORD;
+
+ fooba |
+ foob |
+ fo {
+ /* false alarm, not really a keyword */
+ return TOK_ID;
+ }
+
+
+ Eliminating backing up among a list of keywords can also
+ be done using a "catch-all" rule:
+
+ %%
+ foo return TOK_KEYWORD;
+ foobar return TOK_KEYWORD;
+
+ [a-z]+ return TOK_ID;
+
+ This is usually the best solution when appropriate.
+
+ Backing up messages tend to cascade. With a complicated
+ set of rules it's not uncommon to get hundreds of mes-
+ sages. If one can decipher them, though, it often only
+ takes a dozen or so rules to eliminate the backing up
+ (though it's easy to make a mistake and have an error
+ rule accidentally match a valid token. A possible
+ future flex feature will be to automatically add rules
+ to eliminate backing up).
+
+ It's important to keep in mind that you gain the bene-
+ fits of eliminating backing up only if you eliminate
+ every instance of backing up. Leaving just one means
+ you gain nothing.
+
+ Variable trailing context (where both the leading and
+ trailing parts do not have a fixed length) entails
+ almost the same performance loss as REJECT (i.e., sub-
+ stantial). So when possible a rule like:
+
+ %%
+ mouse|rat/(cat|dog) run();
+
+ is better written:
+
+ %%
+ mouse/cat|dog run();
+ rat/cat|dog run();
+
+ or as
+
+ %%
+ mouse|rat/cat run();
+ mouse|rat/dog run();
+
+ Note that here the special '|' action does not provide
+ any savings, and can even make things worse (see Defi-
+ ciencies / Bugs below).
+
+ Another area where the user can increase a scanner's
+ performance (and one that's easier to implement) arises
+ from the fact that the longer the tokens matched, the
+ faster the scanner will run. This is because with long
+ tokens the processing of most input characters takes
+ place in the (short) inner scanning loop, and does not
+ often have to go through the additional work of setting
+ up the scanning environment (e.g., yytext) for the
+ action. Recall the scanner for C comments:
+
+ %x comment
+ %%
+ int line_num = 1;
+
+ "/*" BEGIN(comment);
+
+ <comment>[^*\n]*
+ <comment>"*"+[^*/\n]*
+ <comment>\n ++line_num;
+ <comment>"*"+"/" BEGIN(INITIAL);
+
+ This could be sped up by writing it as:
+
+ %x comment
+ %%
+ int line_num = 1;
+
+ "/*" BEGIN(comment);
+
+ <comment>[^*\n]*
+ <comment>[^*\n]*\n ++line_num;
+ <comment>"*"+[^*/\n]*
+ <comment>"*"+[^*/\n]*\n ++line_num;
+ <comment>"*"+"/" BEGIN(INITIAL);
+
+ Now instead of each newline requiring the processing of
+ another action, recognizing the newlines is "distrib-
+ uted" over the other rules to keep the matched text as
+ long as possible. Note that adding rules does not slow
+ down the scanner! The speed of the scanner is indepen-
+ dent of the number of rules or (modulo the considera-
+ tions given at the beginning of this section) how com-
+ plicated the rules are with regard to operators such as
+ '*' and '|'.
+
+ A final example in speeding up a scanner: suppose you
+ want to scan through a file containing identifiers and
+ keywords, one per line and with no other extraneous
+ characters, and recognize all the keywords. A natural
+ first approach is:
+
+ %%
+ asm |
+ auto |
+ break |
+ ... etc ...
+ volatile |
+ while /* it's a keyword */
+
+ .|\n /* it's not a keyword */
+
+ To eliminate the back-tracking, introduce a catch-all
+ rule:
+
+ %%
+ asm |
+ auto |
+ break |
+ ... etc ...
+ volatile |
+ while /* it's a keyword */
+
+ [a-z]+ |
+ .|\n /* it's not a keyword */
+
+ Now, if it's guaranteed that there's exactly one word
+ per line, then we can reduce the total number of matches
+ by a half by merging in the recognition of newlines with
+ that of the other tokens:
+
+ %%
+ asm\n |
+ auto\n |
+ break\n |
+ ... etc ...
+ volatile\n |
+ while\n /* it's a keyword */
+
+ [a-z]+\n |
+ .|\n /* it's not a keyword */
+
+ One has to be careful here, as we have now reintroduced
+ backing up into the scanner. In particular, while we
+ know that there will never be any characters in the
+ input stream other than letters or newlines, flex can't
+ figure this out, and it will plan for possibly needing
+ to back up when it has scanned a token like "auto" and
+ then the next character is something other than a new-
+ line or a letter. Previously it would then just match
+ the "auto" rule and be done, but now it has no "auto"
+ rule, only a "auto\n" rule. To eliminate the possibil-
+ ity of backing up, we could either duplicate all rules
+ but without final newlines, or, since we never expect to
+ encounter such an input and therefore don't how it's
+ classified, we can introduce one more catch-all rule,
+ this one which doesn't include a newline:
+
+ %%
+ asm\n |
+ auto\n |
+ break\n |
+ ... etc ...
+ volatile\n |
+ while\n /* it's a keyword */
+
+ [a-z]+\n |
+ [a-z]+ |
+ .|\n /* it's not a keyword */
+
+ Compiled with -Cf, this is about as fast as one can get
+ a flex scanner to go for this particular problem.
+
+ A final note: flex is slow when matching NUL's, particu-
+ larly when a token contains multiple NUL's. It's best
+ to write rules which match short amounts of text if it's
+ anticipated that the text will often include NUL's.
+
+ Another final note regarding performance: as mentioned
+ above in the section How the Input is Matched, dynami-
+ cally resizing yytext to accommodate huge tokens is a
+ slow process because it presently requires that the
+ (huge) token be rescanned from the beginning. Thus if
+ performance is vital, you should attempt to match
+ "large" quantities of text but not "huge" quantities,
+ where the cutoff between the two is at about 8K charac-
+ ters/token.
+
+GENERATING C++ SCANNERS
+ flex provides two different ways to generate scanners
+ for use with C++. The first way is to simply compile a
+ scanner generated by flex using a C++ compiler instead
+ of a C compiler. You should not encounter any compila-
+ tions errors (please report any you find to the email
+ address given in the Author section below). You can
+ then use C++ code in your rule actions instead of C
+ code. Note that the default input source for your scan-
+ ner remains yyin, and default echoing is still done to
+ yyout. Both of these remain FILE * variables and not
+ C++ streams.
+
+ You can also use flex to generate a C++ scanner class,
+ using the -+ option (or, equivalently, %option c++),
+ which is automatically specified if the name of the flex
+ executable ends in a '+', such as flex++. When using
+ this option, flex defaults to generating the scanner to
+ the file lex.yy.cc instead of lex.yy.c. The generated
+ scanner includes the header file FlexLexer.h, which
+ defines the interface to two C++ classes.
+
+ The first class, FlexLexer, provides an abstract base
+ class defining the general scanner class interface. It
+ provides the following member functions:
+
+ const char* YYText()
+ returns the text of the most recently matched
+ token, the equivalent of yytext.
+
+ int YYLeng()
+ returns the length of the most recently matched
+ token, the equivalent of yyleng.
+
+ int lineno() const
+ returns the current input line number (see
+ %option yylineno), or 1 if %option yylineno was
+ not used.
+
+ void set_debug( int flag )
+ sets the debugging flag for the scanner, equiva-
+ lent to assigning to yy_flex_debug (see the
+ Options section above). Note that you must build
+ the scanner using %option debug to include debug-
+ ging information in it.
+
+ int debug() const
+ returns the current setting of the debugging
+ flag.
+
+ Also provided are member functions equivalent to
+ yy_switch_to_buffer(), yy_create_buffer() (though the
+ first argument is an istream* object pointer and not a
+ FILE*), yy_flush_buffer(), yy_delete_buffer(), and
+ yyrestart() (again, the first argument is a istream*
+ object pointer).
+
+ The second class defined in FlexLexer.h is yyFlexLexer,
+ which is derived from FlexLexer. It defines the follow-
+ ing additional member functions:
+
+ yyFlexLexer( istream* arg_yyin = 0, ostream* arg_yyout =
+ 0 )
+ constructs a yyFlexLexer object using the given
+ streams for input and output. If not specified,
+ the streams default to cin and cout, respec-
+ tively.
+
+ virtual int yylex()
+ performs the same role is yylex() does for ordi-
+ nary flex scanners: it scans the input stream,
+ consuming tokens, until a rule's action returns a
+ value. If you derive a subclass S from
+ yyFlexLexer and want to access the member func-
+ tions and variables of S inside yylex(), then you
+ need to use %option yyclass="S" to inform flex
+ that you will be using that subclass instead of
+ yyFlexLexer. In this case, rather than generat-
+ ing yyFlexLexer::yylex(), flex generates
+ S::yylex() (and also generates a dummy
+ yyFlexLexer::yylex() that calls
+ yyFlexLexer::LexerError() if called).
+
+ virtual void switch_streams(istream* new_in = 0,
+ ostream* new_out = 0) reassigns yyin to new_in
+ (if non-nil) and yyout to new_out (ditto), delet-
+ ing the previous input buffer if yyin is reas-
+ signed.
+
+ int yylex( istream* new_in, ostream* new_out = 0 )
+ first switches the input streams via
+ switch_streams( new_in, new_out ) and then
+ returns the value of yylex().
+
+ In addition, yyFlexLexer defines the following protected
+ virtual functions which you can redefine in derived
+ classes to tailor the scanner:
+
+ virtual int LexerInput( char* buf, int max_size )
+ reads up to max_size characters into buf and
+ returns the number of characters read. To indi-
+ cate end-of-input, return 0 characters. Note
+ that "interactive" scanners (see the -B and -I
+ flags) define the macro YY_INTERACTIVE. If you
+ redefine LexerInput() and need to take different
+ actions depending on whether or not the scanner
+ might be scanning an interactive input source,
+ you can test for the presence of this name via
+ #ifdef.
+
+ virtual void LexerOutput( const char* buf, int size )
+ writes out size characters from the buffer buf,
+ which, while NUL-terminated, may also contain
+ "internal" NUL's if the scanner's rules can match
+ text with NUL's in them.
+
+ virtual void LexerError( const char* msg )
+ reports a fatal error message. The default ver-
+ sion of this function writes the message to the
+ stream cerr and exits.
+
+ Note that a yyFlexLexer object contains its entire scan-
+ ning state. Thus you can use such objects to create
+ reentrant scanners. You can instantiate multiple
+ instances of the same yyFlexLexer class, and you can
+ also combine multiple C++ scanner classes together in
+ the same program using the -P option discussed above.
+
+ Finally, note that the %array feature is not available
+ to C++ scanner classes; you must use %pointer (the
+ default).
+
+ Here is an example of a simple C++ scanner:
+
+ // An example of using the flex C++ scanner class.
+
+ %{
+ int mylineno = 0;
+ %}
+
+ string \"[^\n"]+\"
+
+ ws [ \t]+
+
+ alpha [A-Za-z]
+ dig [0-9]
+ name ({alpha}|{dig}|\$)({alpha}|{dig}|[_.\-/$])*
+ num1 [-+]?{dig}+\.?([eE][-+]?{dig}+)?
+ num2 [-+]?{dig}*\.{dig}+([eE][-+]?{dig}+)?
+ number {num1}|{num2}
+
+ %%
+
+ {ws} /* skip blanks and tabs */
+
+ "/*" {
+ int c;
+
+ while((c = yyinput()) != 0)
+ {
+ if(c == '\n')
+ ++mylineno;
+
+ else if(c == '*')
+ {
+ if((c = yyinput()) == '/')
+ break;
+ else
+ unput(c);
+ }
+ }
+ }
+
+ {number} cout << "number " << YYText() << '\n';
+
+ \n mylineno++;
+
+ {name} cout << "name " << YYText() << '\n';
+
+ {string} cout << "string " << YYText() << '\n';
+
+ %%
+
+ int main( int /* argc */, char** /* argv */ )
+ {
+ FlexLexer* lexer = new yyFlexLexer;
+ while(lexer->yylex() != 0)
+ ;
+ return 0;
+ }
+ If you want to create multiple (different) lexer
+ classes, you use the -P flag (or the prefix= option) to
+ rename each yyFlexLexer to some other xxFlexLexer. You
+ then can include <FlexLexer.h> in your other sources
+ once per lexer class, first renaming yyFlexLexer as fol-
+ lows:
+
+ #undef yyFlexLexer
+ #define yyFlexLexer xxFlexLexer
+ #include <FlexLexer.h>
+
+ #undef yyFlexLexer
+ #define yyFlexLexer zzFlexLexer
+ #include <FlexLexer.h>
+
+ if, for example, you used %option prefix="xx" for one of
+ your scanners and %option prefix="zz" for the other.
+
+ IMPORTANT: the present form of the scanning class is
+ experimental and may change considerably between major
+ releases.
+
+INCOMPATIBILITIES WITH LEX AND POSIX
+ flex is a rewrite of the AT&T Unix lex tool (the two
+ implementations do not share any code, though), with
+ some extensions and incompatibilities, both of which are
+ of concern to those who wish to write scanners accept-
+ able to either implementation. Flex is fully compliant
+ with the POSIX lex specification, except that when using
+ %pointer (the default), a call to unput() destroys the
+ contents of yytext, which is counter to the POSIX
+ specification.
+
+ In this section we discuss all of the known areas of
+ incompatibility between flex, AT&T lex, and the POSIX
+ specification.
+
+ flex's -l option turns on maximum compatibility with the
+ original AT&T lex implementation, at the cost of a major
+ loss in the generated scanner's performance. We note
+ below which incompatibilities can be overcome using the
+ -l option.
+
+ flex is fully compatible with lex with the following
+ exceptions:
+
+ - The undocumented lex scanner internal variable
+ yylineno is not supported unless -l or %option
+ yylineno is used.
+
+ yylineno should be maintained on a per-buffer
+ basis, rather than a per-scanner (single global
+ variable) basis.
+
+ yylineno is not part of the POSIX specification.
+
+ - The input() routine is not redefinable, though it
+ may be called to read characters following what-
+ ever has been matched by a rule. If input()
+ encounters an end-of-file the normal yywrap()
+ processing is done. A ``real'' end-of-file is
+ returned by input() as EOF.
+
+ Input is instead controlled by defining the
+ YY_INPUT macro.
+
+ The flex restriction that input() cannot be rede-
+ fined is in accordance with the POSIX specifica-
+ tion, which simply does not specify any way of
+ controlling the scanner's input other than by
+ making an initial assignment to yyin.
+
+ - The unput() routine is not redefinable. This
+ restriction is in accordance with POSIX.
+
+ - flex scanners are not as reentrant as lex scan-
+ ners. In particular, if you have an interactive
+ scanner and an interrupt handler which long-jumps
+ out of the scanner, and the scanner is subse-
+ quently called again, you may get the following
+ message:
+
+ fatal flex scanner internal error--end of buffer missed
+
+ To reenter the scanner, first use
+
+ yyrestart( yyin );
+
+ Note that this call will throw away any buffered
+ input; usually this isn't a problem with an
+ interactive scanner.
+
+ Also note that flex C++ scanner classes are reen-
+ trant, so if using C++ is an option for you, you
+ should use them instead. See "Generating C++
+ Scanners" above for details.
+
+ - output() is not supported. Output from the ECHO
+ macro is done to the file-pointer yyout (default
+ stdout).
+
+ output() is not part of the POSIX specification.
+
+ - lex does not support exclusive start conditions
+ (%x), though they are in the POSIX specification.
+
+ - When definitions are expanded, flex encloses them
+ in parentheses. With lex, the following:
+
+ NAME [A-Z][A-Z0-9]*
+ %%
+ foo{NAME}? printf( "Found it\n" );
+ %%
+
+ will not match the string "foo" because when the
+ macro is expanded the rule is equivalent to
+ "foo[A-Z][A-Z0-9]*?" and the precedence is such
+ that the '?' is associated with "[A-Z0-9]*".
+ With flex, the rule will be expanded to "foo([A-
+ Z][A-Z0-9]*)?" and so the string "foo" will
+ match.
+
+ Note that if the definition begins with ^ or ends
+ with $ then it is not expanded with parentheses,
+ to allow these operators to appear in definitions
+ without losing their special meanings. But the
+ <s>, /, and <<EOF>> operators cannot be used in a
+ flex definition.
+
+ Using -l results in the lex behavior of no paren-
+ theses around the definition.
+
+ The POSIX specification is that the definition be
+ enclosed in parentheses.
+
+ - Some implementations of lex allow a rule's action
+ to begin on a separate line, if the rule's pat-
+ tern has trailing whitespace:
+
+ %%
+ foo|bar<space here>
+ { foobar_action(); }
+
+ flex does not support this feature.
+
+ - The lex %r (generate a Ratfor scanner) option is
+ not supported. It is not part of the POSIX spec-
+ ification.
+
+ - After a call to unput(), yytext is undefined
+ until the next token is matched, unless the scan-
+ ner was built using %array. This is not the case
+ with lex or the POSIX specification. The -l
+ option does away with this incompatibility.
+
+ - The precedence of the {} (numeric range) operator
+ is different. lex interprets "abc{1,3}" as
+ "match one, two, or three occurrences of 'abc'",
+ whereas flex interprets it as "match 'ab' fol-
+ lowed by one, two, or three occurrences of 'c'".
+ The latter is in agreement with the POSIX speci-
+ fication.
+
+ - The precedence of the ^ operator is different.
+ lex interprets "^foo|bar" as "match either 'foo'
+ at the beginning of a line, or 'bar' anywhere",
+ whereas flex interprets it as "match either 'foo'
+ or 'bar' if they come at the beginning of a
+ line". The latter is in agreement with the POSIX
+ specification.
+
+ - The special table-size declarations such as %a
+ supported by lex are not required by flex scan-
+ ners; flex ignores them.
+
+ - The name FLEX_SCANNER is #define'd so scanners
+ may be written for use with either flex or lex.
+ Scanners also include YY_FLEX_MAJOR_VERSION and
+ YY_FLEX_MINOR_VERSION indicating which version of
+ flex generated the scanner (for example, for the
+ 2.5 release, these defines would be 2 and 5
+ respectively).
+
+ The following flex features are not included in lex or
+ the POSIX specification:
+
+ C++ scanners
+ %option
+ start condition scopes
+ start condition stacks
+ interactive/non-interactive scanners
+ yy_scan_string() and friends
+ yyterminate()
+ yy_set_interactive()
+ yy_set_bol()
+ YY_AT_BOL()
+ <<EOF>>
+ <*>
+ YY_DECL
+ YY_START
+ YY_USER_ACTION
+ YY_USER_INIT
+ #line directives
+ %{}'s around actions
+ multiple actions on a line
+
+ plus almost all of the flex flags. The last feature in
+ the list refers to the fact that with flex you can put
+ multiple actions on the same line, separated with semi-
+ colons, while with lex, the following
+
+ foo handle_foo(); ++num_foos_seen;
+
+ is (rather surprisingly) truncated to
+
+ foo handle_foo();
+
+ flex does not truncate the action. Actions that are not
+ enclosed in braces are simply terminated at the end of
+ the line.
+
+DIAGNOSTICS
+ warning, rule cannot be matched indicates that the given
+ rule cannot be matched because it follows other rules
+ that will always match the same text as it. For exam-
+ ple, in the following "foo" cannot be matched because it
+ comes after an identifier "catch-all" rule:
+
+ [a-z]+ got_identifier();
+ foo got_foo();
+
+ Using REJECT in a scanner suppresses this warning.
+
+ warning, -s option given but default rule can be matched
+ means that it is possible (perhaps only in a particular
+ start condition) that the default rule (match any single
+ character) is the only one that will match a particular
+ input. Since -s was given, presumably this is not
+ intended.
+
+ reject_used_but_not_detected undefined or
+ yymore_used_but_not_detected undefined - These errors
+ can occur at compile time. They indicate that the scan-
+ ner uses REJECT or yymore() but that flex failed to
+ notice the fact, meaning that flex scanned the first two
+ sections looking for occurrences of these actions and
+ failed to find any, but somehow you snuck some in (via a
+ #include file, for example). Use %option reject or
+ %option yymore to indicate to flex that you really do
+ use these features.
+
+ flex scanner jammed - a scanner compiled with -s has
+ encountered an input string which wasn't matched by any
+ of its rules. This error can also occur due to internal
+ problems.
+
+ token too large, exceeds YYLMAX - your scanner uses
+ %array and one of its rules matched a string longer than
+ the YYLMAX constant (8K bytes by default). You can
+ increase the value by #define'ing YYLMAX in the defini-
+ tions section of your flex input.
+
+ scanner requires -8 flag to use the character 'x' - Your
+ scanner specification includes recognizing the 8-bit
+ character 'x' and you did not specify the -8 flag, and
+ your scanner defaulted to 7-bit because you used the -Cf
+ or -CF table compression options. See the discussion of
+ the -7 flag for details.
+
+ flex scanner push-back overflow - you used unput() to
+ push back so much text that the scanner's buffer could
+ not hold both the pushed-back text and the current token
+ in yytext. Ideally the scanner should dynamically
+ resize the buffer in this case, but at present it does
+ not.
+
+ input buffer overflow, can't enlarge buffer because
+ scanner uses REJECT - the scanner was working on match-
+ ing an extremely large token and needed to expand the
+ input buffer. This doesn't work with scanners that use
+ REJECT.
+
+ fatal flex scanner internal error--end of buffer missed
+ - This can occur in an scanner which is reentered after
+ a long-jump has jumped out (or over) the scanner's acti-
+ vation frame. Before reentering the scanner, use:
+
+ yyrestart( yyin );
+
+ or, as noted above, switch to using the C++ scanner
+ class.
+
+ too many start conditions in <> construct! - you listed
+ more start conditions in a <> construct than exist (so
+ you must have listed at least one of them twice).
+
+FILES
+ -lfl library with which scanners must be linked.
+
+ lex.yy.c
+ generated scanner (called lexyy.c on some sys-
+ tems).
+
+ lex.yy.cc
+ generated C++ scanner class, when using -+.
+
+ <FlexLexer.h>
+ header file defining the C++ scanner base class,
+ FlexLexer, and its derived class, yyFlexLexer.
+
+ flex.skl
+ skeleton scanner. This file is only used when
+ building flex, not when flex executes.
+
+ lex.backup
+ backing-up information for -b flag (called
+ lex.bck on some systems).
+
+DEFICIENCIES / BUGS
+ Some trailing context patterns cannot be properly
+ matched and generate warning messages ("dangerous trail-
+ ing context"). These are patterns where the ending of
+ the first part of the rule matches the beginning of the
+ second part, such as "zx*/xy*", where the 'x*' matches
+ the 'x' at the beginning of the trailing context. (Note
+ that the POSIX draft states that the text matched by
+ such patterns is undefined.)
+
+ For some trailing context rules, parts which are actu-
+ ally fixed-length are not recognized as such, leading to
+ the abovementioned performance loss. In particular,
+ parts using '|' or {n} (such as "foo{3}") are always
+ considered variable-length.
+
+ Combining trailing context with the special '|' action
+ can result in fixed trailing context being turned into
+ the more expensive variable trailing context. For exam-
+ ple, in the following:
+
+ %%
+ abc |
+ xyz/def
+
+
+ Use of unput() invalidates yytext and yyleng, unless the
+ %array directive or the -l option has been used.
+
+ Pattern-matching of NUL's is substantially slower than
+ matching other characters.
+
+ Dynamic resizing of the input buffer is slow, as it
+ entails rescanning all the text matched so far by the
+ current (generally huge) token.
+
+ Due to both buffering of input and read-ahead, you can-
+ not intermix calls to <stdio.h> routines, such as, for
+ example, getchar(), with flex rules and expect it to
+ work. Call input() instead.
+
+ The total table entries listed by the -v flag excludes
+ the number of table entries needed to determine what
+ rule has been matched. The number of entries is equal
+ to the number of DFA states if the scanner does not use
+ REJECT, and somewhat greater than the number of states
+ if it does.
+
+ REJECT cannot be used with the -f or -F options.
+
+ The flex internal algorithms need documentation.
+
+SEE ALSO
+ lex(1), yacc(1), sed(1), awk(1).
+
+ John Levine, Tony Mason, and Doug Brown, Lex & Yacc,
+ O'Reilly and Associates. Be sure to get the 2nd edi-
+ tion.
+
+ M. E. Lesk and E. Schmidt, LEX - Lexical Analyzer Gener-
+ ator
+
+ Alfred Aho, Ravi Sethi and Jeffrey Ullman, Compilers:
+ Principles, Techniques and Tools, Addison-Wesley (1986).
+ Describes the pattern-matching techniques used by flex
+ (deterministic finite automata).
+
+AUTHOR
+ Vern Paxson, with the help of many ideas and much inspi-
+ ration from Van Jacobson. Original version by Jef
+ Poskanzer. The fast table representation is a partial
+ implementation of a design done by Van Jacobson. The
+ implementation was done by Kevin Gong and Vern Paxson.
+
+ Thanks to the many flex beta-testers, feedbackers, and
+ contributors, especially Francois Pinard, Casey Leedom,
+ Robert Abramovitz, Stan Adermann, Terry Allen, David
+ Barker-Plummer, John Basrai, Neal Becker, Nelson H.F.
+ Beebe, benson@odi.com, Karl Berry, Peter A. Bigot, Simon
+ Blanchard, Keith Bostic, Frederic Brehm, Ian Brockbank,
+ Kin Cho, Nick Christopher, Brian Clapper, J.T. Conklin,
+ Jason Coughlin, Bill Cox, Nick Cropper, Dave Curtis,
+ Scott David Daniels, Chris G. Demetriou, Theo Deraadt,
+ Mike Donahue, Chuck Doucette, Tom Epperly, Leo Eskin,
+ Chris Faylor, Chris Flatters, Jon Forrest, Jeffrey
+ Friedl, Joe Gayda, Kaveh R. Ghazi, Wolfgang Glunz, Eric
+ Goldman, Christopher M. Gould, Ulrich Grepel, Peer
+ Griebel, Jan Hajic, Charles Hemphill, NORO Hideo, Jarkko
+ Hietaniemi, Scott Hofmann, Jeff Honig, Dana Hudes, Eric
+ Hughes, John Interrante, Ceriel Jacobs, Michal
+ Jaegermann, Sakari Jalovaara, Jeffrey R. Jones, Henry
+ Juengst, Klaus Kaempf, Jonathan I. Kamens, Terrence O
+ Kane, Amir Katz, ken@ken.hilco.com, Kevin B. Kenny,
+ Steve Kirsch, Winfried Koenig, Marq Kole, Ronald Lam-
+ precht, Greg Lee, Rohan Lenard, Craig Leres, John
+ Levine, Steve Liddle, David Loffredo, Mike Long, Mohamed
+ el Lozy, Brian Madsen, Malte, Joe Marshall, Bengt
+ Martensson, Chris Metcalf, Luke Mewburn, Jim Meyering,
+ R. Alexander Milowski, Erik Naggum, G.T. Nicol, Landon
+ Noll, James Nordby, Marc Nozell, Richard Ohnemus,
+ Karsten Pahnke, Sven Panne, Roland Pesch, Walter Pelis-
+ sero, Gaumond Pierre, Esmond Pitt, Jef Poskanzer, Joe
+ Rahmeh, Jarmo Raiha, Frederic Raimbault, Pat Rankin,
+ Rick Richardson, Kevin Rodgers, Kai Uwe Rommel, Jim
+ Roskind, Alberto Santini, Andreas Scherer, Darrell
+ Schiebel, Raf Schietekat, Doug Schmidt, Philippe Schnoe-
+ belen, Andreas Schwab, Larry Schwimmer, Alex Siegel,
+ Eckehard Stolz, Jan-Erik Strvmquist, Mike Stump, Paul
+ Stuart, Dave Tallman, Ian Lance Taylor, Chris Thewalt,
+ Richard M. Timoney, Jodi Tsai, Paul Tuinenga, Gary Weik,
+ Frank Whaley, Gerhard Wilhelms, Kent Williams, Ken Yap,
+ Ron Zellar, Nathan Zelle, David Zuhn, and those whose
+ names have slipped my marginal mail-archiving skills but
+ whose contributions are appreciated all the same.
+
+ Thanks to Keith Bostic, Jon Forrest, Noah Friedman, John
+ Gilmore, Craig Leres, John Levine, Bob Mulcahy, G.T.
+ Nicol, Francois Pinard, Rich Salz, and Richard Stallman
+ for help with various distribution headaches.
+
+ Thanks to Esmond Pitt and Earle Horton for 8-bit charac-
+ ter support; to Benson Margulies and Fred Burke for C++
+ support; to Kent Williams and Tom Epperly for C++ class
+ support; to Ove Ewerlid for support of NUL's; and to
+ Eric Hughes for support of multiple buffers.
+
+ This work was primarily done when I was with the Real
+ Time Systems Group at the Lawrence Berkeley Laboratory
+ in Berkeley, CA. Many thanks to all there for the sup-
+ port I received.
+
+ Send comments to vern@ee.lbl.gov.
+
+
+
+Version 2.5 April 1995 FLEX(1)
diff --git a/gnuwin32/man/cat1/gperf.1.txt b/gnuwin32/man/cat1/gperf.1.txt
new file mode 100644
index 00000000..d9e128c7
--- /dev/null
+++ b/gnuwin32/man/cat1/gperf.1.txt
@@ -0,0 +1,226 @@
+GPERF(1) FSF GPERF(1)
+
+
+
+
+
+NAME
+ gperf - generate a perfect hash function from a key set
+
+SYNOPSIS
+ gperf [OPTION]... [INPUT-FILE]
+
+DESCRIPTION
+ GNU 'gperf' generates perfect hash functions.
+
+ If a long option shows an argument as mandatory, then it
+ is mandatory for the equivalent short option also.
+
+ Output file location:
+ --output-file=FILE Write output to specified file.
+
+ The results are written to standard output if no output
+ file is specified or if it is -.
+
+ Input file interpretation:
+ -e, --delimiters=DELIMITER-LIST
+ Allow user to provide a string containing delim-
+ iters used to separate keywords from their
+ attributes. Default is ",".
+
+ -t, --struct-type
+ Allows the user to include a structured type dec-
+ laration for generated code. Any text before %%
+ is considered part of the type declaration. Key
+ words and additional fields may follow this, one
+ group of fields per line.
+
+ --ignore-case
+ Consider upper and lower case ASCII characters as
+ equivalent. Note that locale dependent case map-
+ pings are ignored.
+
+ Language for the output code:
+ -L, --language=LANGUAGE-NAME
+ Generates code in the specified language. Lan-
+ guages handled are currently C++, ANSI-C, C, and
+ KR-C. The default is C.
+
+ Details in the output code:
+ -K, --slot-name=NAME
+ Select name of the keyword component in the key-
+ word structure.
+
+ -F, --initializer-suffix=INITIALIZERS
+ Initializers for additional components in the
+ keyword structure.
+
+ -H, --hash-function-name=NAME
+ Specify name of generated hash function. Default
+ is 'hash'.
+
+ -N, --lookup-function-name=NAME
+ Specify name of generated lookup function.
+ Default name is 'in_word_set'.
+
+ -Z, --class-name=NAME
+ Specify name of generated C++ class. Default name
+ is 'Perfect_Hash'.
+
+ -7, --seven-bit
+ Assume 7-bit characters.
+
+ -l, --compare-lengths
+ Compare key lengths before trying a string com-
+ parison. This is necessary if the keywords con-
+ tain NUL bytes. It also helps cut down on the
+ number of string comparisons made during the
+ lookup.
+
+ -c, --compare-strncmp
+ Generate comparison code using strncmp rather
+ than strcmp.
+
+ -C, --readonly-tables
+ Make the contents of generated lookup tables con-
+ stant, i.e., readonly.
+
+ -E, --enum
+ Define constant values using an enum local to the
+ lookup function rather than with defines.
+
+ -I, --includes
+ Include the necessary system include file
+ <string.h> at the beginning of the code.
+
+ -G, --global-table
+ Generate the static table of keywords as a static
+ global variable, rather than hiding it inside of
+ the lookup function (which is the default behav-
+ ior).
+
+ -P, --pic
+ Optimize the generated table for inclusion in
+ shared libraries. This reduces the startup time
+ of programs using a shared library containing the
+ generated code.
+
+ -Q, --string-pool-name=NAME
+ Specify name of string pool generated by option
+ --pic. Default name is 'stringpool'.
+
+ --null-strings
+ Use NULL strings instead of empty strings for
+ empty keyword table entries.
+
+ -W, --word-array-name=NAME
+ Specify name of word list array. Default name is
+ 'wordlist'.
+
+ -S, --switch=COUNT
+ Causes the generated C code to use a switch
+ statement scheme, rather than an array lookup ta-
+ ble. This can lead to a reduction in both time
+ and space requirements for some keyfiles. The
+ COUNT argument determines how many switch state-
+ ments are generated. A value of 1 generates 1
+ switch containing all the elements, a value of 2
+ generates 2 tables with 1/2 the elements in each
+ table, etc. If COUNT is very large, say 1000000,
+ the generated C code does a binary search.
+
+ -T, --omit-struct-type
+ Prevents the transfer of the type declaration to
+ the output file. Use this option if the type is
+ already defined elsewhere.
+
+ Algorithm employed by gperf:
+ -k, --key-positions=KEYS
+ Select the key positions used in the hash func-
+ tion. The allowable choices range between 1-255,
+ inclusive. The positions are separated by com-
+ mas, ranges may be used, and key positions may
+ occur in any order. Also, the meta-character '*'
+ causes the generated hash function to consider
+ ALL key positions, and $ indicates the "final
+ character" of a key, e.g., $,1,2,4,6-10.
+
+ -D, --duplicates
+ Handle keywords that hash to duplicate values.
+ This is useful for certain highly redundant key-
+ word sets.
+
+ -m, --multiple-iterations=ITERATIONS
+ Perform multiple choices of the -i and -j values,
+ and choose the best results. This increases the
+ running time by a factor of ITERATIONS but does a
+ good job minimizing the generated table size.
+
+ -i, --initial-asso=N
+ Provide an initial value for the associate values
+ array. Default is 0. Setting this value larger
+ helps inflate the size of the final table.
+
+ -j, --jump=JUMP-VALUE
+ Affects the "jump value", i.e., how far to
+ advance the associated character value upon col-
+ lisions. Must be an odd number, default is 5.
+
+ -n, --no-strlen
+ Do not include the length of the keyword when
+ computing the hash function.
+
+ -r, --random
+ Utilizes randomness to initialize the associated
+ values table.
+
+ -s, --size-multiple=N
+ Affects the size of the generated hash table. The
+ numeric argument N indicates "how many times
+ larger or smaller" the associated value range
+ should be, in relationship to the number of keys,
+ e.g. a value of 3 means "allow the maximum asso-
+ ciated value to be about 3 times larger than the
+ number of input keys". Conversely, a value of 1/3
+ means "make the maximum associated value about 3
+ times smaller than the number of input keys". A
+ larger table should decrease the time required
+ for an unsuccessful search, at the expense of
+ extra table space. Default value is 1.
+
+ Informative output:
+ -h, --help
+ Print this message.
+
+ -v, --version
+ Print the gperf version number.
+
+ -d, --debug
+ Enables the debugging option (produces verbose
+ output to the standard error).
+
+AUTHOR
+ Written by Douglas C. Schmidt and Bruno Haible.
+
+REPORTING BUGS
+ Report bugs to <bug-gnu-gperf@gnu.org>.
+
+COPYRIGHT
+ Copyright (C) 1989-1998, 2000-2003 Free Software Founda-
+ tion, Inc.
+ This is free software; see the source for copying condi-
+ tions. There is NO warranty; not even for MERCHANTABIL-
+ ITY or FITNESS FOR A PARTICULAR PURPOSE.
+
+SEE ALSO
+ The full documentation for gperf is maintained as a Tex-
+ info manual. If the info and gperf programs are prop-
+ erly installed at your site, the command
+
+ info gperf
+
+ should give you access to the complete manual.
+
+
+
+GNU gperf 3.0.1 June 2003 GPERF(1)
diff --git a/gnuwin32/man/cat1/iconv.1.txt b/gnuwin32/man/cat1/iconv.1.txt
new file mode 100644
index 00000000..2089d32e
--- /dev/null
+++ b/gnuwin32/man/cat1/iconv.1.txt
@@ -0,0 +1,48 @@
+ICONV(1) Linux Programmer's Manual ICONV(1)
+
+
+
+
+
+NAME
+ iconv - character set conversion
+
+SYNOPSIS
+ iconv [-c] [-s] [-f encoding] [-t encoding] [inputfile ...]
+ iconv -l
+
+DESCRIPTION
+ The iconv program converts text from one encoding to
+ another encoding. More precisely, it converts from the
+ encoding given for the -f option to the encoding given
+ for the -t option. Either of these encodings defaults to
+ the encoding of the current locale. All the inputfiles
+ are read and converted in turn; if no inputfile is
+ given, the standard input is used. The converted text is
+ printed to standard output.
+
+ When option -c is given, characters that cannot be con-
+ verted are silently discarded, instead of leading to a
+ conversion error.
+
+ When option -s is given, error messages about invalid or
+ unconvertible characters are omitted, but the actual
+ converted text is unaffected.
+
+ The encodings permitted are system dependent. For the
+ libiconv implementation, they are listed in the
+ iconv_open(3) manual page.
+
+ The iconv -l command lists the names of the supported
+ encodings, in a system dependent format. For the libi-
+ conv implementation, the names are printed in upper
+ case, separated by whitespace, and alias names of an
+ encoding are listed on the same line as the encoding
+ itself.
+
+SEE ALSO
+ iconv_open(3), locale(7)
+
+
+
+GNU January 13, 2002 ICONV(1)
diff --git a/gnuwin32/man/cat1/yacc.1.txt b/gnuwin32/man/cat1/yacc.1.txt
new file mode 100644
index 00000000..ada6a5c1
--- /dev/null
+++ b/gnuwin32/man/cat1/yacc.1.txt
@@ -0,0 +1,42 @@
+YACC(1) User Commands YACC(1)
+
+
+
+NAME
+ yacc - GNU Project parser generator
+
+SYNOPSIS
+ yacc [OPTION]... FILE
+
+DESCRIPTION
+ Yacc (Yet Another Compiler Compiler) is a parser genera-
+ tor. This version is a simple wrapper around bison(1).
+ It passes option -y, --yacc to activate the upward com-
+ patibility mode. See bison(1) for more information.
+
+AUTHOR
+ Written by Paul Eggert.
+
+REPORTING BUGS
+ Report bugs to <bug-bison@gnu.org>.
+
+COPYRIGHT
+ Copyright © 2008 Free Software Foundation, Inc.
+ This is free software; see the source for copying condi-
+ tions. There is NO warranty; not even for MERCHANTABIL-
+ ITY or FITNESS FOR A PARTICULAR PURPOSE.
+
+SEE ALSO
+ lex(1), flex(1), bison(1).
+
+ The full documentation for bison is maintained as a Tex-
+ info manual. If the info and bison programs are prop-
+ erly installed at your site, the command
+
+ info bison
+
+ should give you access to the complete manual.
+
+
+
+GNU Bison 2.4.1 November 2007 YACC(1)