Man Pages

lex(1p) - phpMan lex(1p) - phpMan

Command: man perldoc info search(apropos)  


LEX(1P)                    POSIX Programmer's Manual                   LEX(1P)



PROLOG
       This manual page is part of the POSIX Programmer's Manual.  The Linux implementation of this interface may dif-
       fer (consult the corresponding Linux manual page for details of Linux behavior), or the interface  may  not  be
       implemented on Linux.

NAME
       lex - generate programs for lexical tasks (DEVELOPMENT)

SYNOPSIS
       lex [-t][-n|-v][file ...]

DESCRIPTION
       The  lex utility shall generate C programs to be used in lexical processing of character input, and that can be
       used as an interface to yacc. The C programs shall be generated from lex source code and conform to  the  ISO C
       standard. Usually, the lex utility shall write the program it generates to the file lex.yy.c; the state of this
       file is unspecified if lex exits with a non-zero exit status. See the EXTENDED DESCRIPTION section for  a  com-
       plete description of the lex input language.

OPTIONS
       The  lex  utility  shall  conform to the Base Definitions volume of IEEE Std 1003.1-2001, Section 12.2, Utility
       Syntax Guidelines.

       The following options shall be supported:

       -n     Suppress the summary of statistics usually written with the -v option. If no table sizes  are  specified
              in the lex source code and the -v option is not specified, then -n is implied.

       -t     Write the resulting program to standard output instead of lex.yy.c.

       -v     Write a summary of lex statistics to the standard output. (See the discussion of lex table sizes in Def-
              initions in lex .) If the -t option is specified and -n is not specified, this report shall  be  written
              to  standard  error.  If  table  sizes are specified in the lex source code, and if the -n option is not
              specified, the -v option may be enabled.


OPERANDS
       The following operand shall be supported:

       file   A pathname of an input file. If more than one such file is specified, all files shall be concatenated to
              produce  a single lex program. If no file operands are specified, or if a file operand is '-', the stan-
              dard input shall be used.


STDIN
       The standard input shall be used if no file operands are specified, or if a file operand is  '-'  .  See  INPUT
       FILES.

INPUT FILES
       The  input  files shall be text files containing lex source code, as described in the EXTENDED DESCRIPTION sec-
       tion.

ENVIRONMENT VARIABLES
       The following environment variables shall affect the execution of lex:

       LANG   Provide a default value for the internationalization variables that are unset or  null.  (See  the  Base
              Definitions  volume  of IEEE Std 1003.1-2001, Section 8.2, Internationalization Variables for the prece-
              dence of internationalization variables used to determine the values of locale categories.)

       LC_ALL If set to a non-empty string value, override the values of all the other internationalization variables.

       LC_COLLATE

              Determine  the  locale  for  the  behavior of ranges, equivalence classes, and multi-character collating
              elements within regular expressions. If this variable is not set to the POSIX locale,  the  results  are
              unspecified.

       LC_CTYPE
              Determine  the locale for the interpretation of sequences of bytes of text data as characters (for exam-
              ple, single-byte as opposed to multi-byte characters in arguments and input files), and the behavior  of
              character  classes  within  regular  expressions.   If this variable is not set to the POSIX locale, the
              results are unspecified.

       LC_MESSAGES
              Determine the locale that should be used to affect the format and contents of diagnostic messages  writ-
              ten to standard error.

       NLSPATH
              Determine the location of message catalogs for the processing of LC_MESSAGES .


ASYNCHRONOUS EVENTS
       Default.

STDOUT
       If  the  -t option is specified, the text file of C source code output of lex shall be written to standard out-
       put.

       If the -t option is not specified:

        * Implementation-defined informational, error, and warning messages concerning the contents of lex source code
          input shall be written to either the standard output or standard error.


        * If  the  -v  option is specified and the -n option is not specified, lex statistics shall also be written to
          either the standard output or standard error, in an implementation-defined format. These statistics may also
          be  generated if table sizes are specified with a '%' operator in the Definitions section, as long as the -n
          option is not specified.


STDERR
       If the -t option is specified, implementation-defined informational, error, and warning messages concerning the
       contents of lex source code input shall be written to the standard error.

       If the -t option is not specified:

        1. Implementation-defined  informational,  error,  and  warning messages concerning the contents of lex source
           code input shall be written to either the standard output or standard error.


        2. If the -v option is specified and the -n option is not specified, lex statistics shall also be  written  to
           either  the  standard  output  or standard error, in an implementation-defined format. These statistics may
           also be generated if table sizes are specified with a '%' operator in the Definitions section, as  long  as
           the -n option is not specified.


OUTPUT FILES
       A  text  file containing C source code shall be written to lex.yy.c, or to the standard output if the -t option
       is present.

EXTENDED DESCRIPTION
       Each input file shall contain lex source code, which is a  table  of  regular  expressions  with  corresponding
       actions in the form of C program fragments.

       When lex.yy.c is compiled and linked with the lex library (using the -l l operand with c99), the resulting pro-
       gram shall read character input from the standard input and shall partition it  into  strings  that  match  the
       given expressions.

       When an expression is matched, these actions shall occur:

        * The  input  string that was matched shall be left in yytext as a null-terminated string; yytext shall either
          be an external character array or a pointer to a character string. As explained in Definitions in  lex,  the
          type  can  be  explicitly selected using the %array or %pointer declarations, but the default is implementa-
          tion-defined.


        * The external int yyleng shall be set to the length of the matching string.


        * The expression's corresponding program fragment, or action, shall be executed.


       During pattern matching, lex shall search the set of patterns for the  single  longest  possible  match.  Among
       rules that match the same number of characters, the rule given first shall be chosen.

       The general format of lex source shall be:


              Definitions
              %%
              Rules
              %%
              UserSubroutines

       The  first  "%%"  is  required to mark the beginning of the rules (regular expressions and actions); the second
       "%%" is required only if user subroutines follow.

       Any line in the Definitions section beginning with a <blank> shall be assumed to be a C  program  fragment  and
       shall  be  copied to the external definition area of the lex.yy.c file.  Similarly, anything in the Definitions
       section included between delimiter lines containing only "%{" and "%}" shall also be copied  unchanged  to  the
       external definition area of the lex.yy.c file.

       Any such input (beginning with a <blank> or within "%{" and "%}" delimiter lines) appearing at the beginning of
       the Rules section before any rules are specified shall be written to lex.yy.c after the declarations  of  vari-
       ables  for  the  yylex()  function  and before the first line of code in yylex(). Thus, user variables local to
       yylex() can be declared here, as well as application code to execute upon entry to yylex().

       The action taken by lex when encountering any input beginning with a <blank> or within "%{" and "%}"  delimiter
       lines  appearing  in  the  Rules  section but coming after one or more rules is undefined. The presence of such
       input may result in an erroneous definition of the yylex() function.

   Definitions in lex
       Definitions appear before the first "%%" delimiter. Any line in this section not  contained  between  "%{"  and
       "%}" lines and not beginning with a <blank> shall be assumed to define a lex substitution string. The format of
       these lines shall be:


              name substitute

       If a name does not meet the requirements for identifiers in the ISO C standard, the result  is  undefined.  The
       string  substitute  shall replace the string { name} when it is used in a rule. The name string shall be recog-
       nized in this context only when the braces are provided and when it does not appear within a bracket expression
       or within double-quotes.

       In  the Definitions section, any line beginning with a '%' (percent sign) character and followed by an alphanu-
       meric word beginning with either 's' or 'S' shall define a set of start conditions. Any line beginning  with  a
       '%'  followed by a word beginning with either 'x' or 'X' shall define a set of exclusive start conditions. When
       the generated scanner is in a %s state, patterns with no state specified shall be also active; in a  %x  state,
       such patterns shall not be active. The rest of the line, after the first word, shall be considered to be one or
       more <blank>-separated names of start conditions. Start condition names shall be constructed in the same way as
       definition  names.  Start conditions can be used to restrict the matching of regular expressions to one or more
       states as described in Regular Expressions in lex .

       Implementations shall accept either of the following two mutually-exclusive  declarations  in  the  Definitions
       section:

       %array Declare the type of yytext to be a null-terminated character array.

       %pointer
              Declare the type of yytext to be a pointer to a null-terminated character string.


       The default type of yytext is implementation-defined. If an application refers to yytext outside of the scanner
       source file (that is, via an extern), the application shall include the appropriate %array or %pointer declara-
       tion in the scanner source file.

       Implementations  shall accept declarations in the Definitions section for setting certain internal table sizes.
       The declarations are shown in the following table.

                                            Table: Table Size Declarations in lex

                               Declaration  Description                         Minimum Value
                               %p n         Number of positions                 2500
                               %n n         Number of states                    500
                               %a n         Number of transitions               2000
                               %e n         Number of parse tree nodes          1000
                               %k n         Number of packed character classes  1000
                               %o n         Size of the output array            3000

       In the table, n represents a positive decimal integer, preceded by one or more <blank>s. The exact  meaning  of
       these  table size numbers is implementation-defined. The implementation shall document how these numbers affect
       the lex utility and how they are related to any output that may be generated by the implementation should limi-
       tations be encountered during the execution of lex. It shall be possible to determine from this output which of
       the table size values needs to be modified to permit lex to successfully generate tables  for  the  input  lan-
       guage.   The  values  in  the column Minimum Value represent the lowest values conforming implementations shall
       provide.

   Rules in lex
       The rules in lex source files are a table in which the left column contains regular expressions and  the  right
       column contains actions (C program fragments) to be executed when the expressions are recognized.


              ERE action
              ERE action...

       The  extended regular expression (ERE) portion of a row shall be separated from action by one or more <blank>s.
       A regular expression containing <blank>s shall be recognized under one of the following conditions:

        * The entire expression appears within double-quotes.


        * The <blank>s appear within double-quotes or square brackets.


        * Each <blank> is preceded by a backslash character.


   User Subroutines in lex
       Anything in the user subroutines section shall be copied to lex.yy.c following yylex().

   Regular Expressions in lex
       The lex utility shall support the set of extended regular expressions  (see  the  Base  Definitions  volume  of
       IEEE Std 1003.1-2001,  Section  9.4, Extended Regular Expressions), with the following additions and exceptions
       to the syntax:

       "..."  Any string enclosed in double-quotes shall represent the characters within the  double-quotes  as  them-
              selves,  except  that  backslash escapes (which appear in the following table) shall be recognized.  Any
              backslash-escape sequence shall be terminated by the closing quote. For example, "\01" "1" represents  a
              single string: the octal value 1 followed by the character '1' .

       <state>r, <state1,state2,...>r

              The  regular expression r shall be matched only when the program is in one of the start conditions indi-
              cated by state, state1, and so on; see Actions in lex . (As an exception to  the  typographical  conven-
              tions  of  the  rest  of  this volume of IEEE Std 1003.1-2001, in this case <state> does not represent a
              metavariable, but the literal angle-bracket characters surrounding a symbol.) The start condition  shall
              be recognized as such only at the beginning of a regular expression.

       r/x    The  regular expression r shall be matched only if it is followed by an occurrence of regular expression
              x ( x is the instance of trailing context, further defined below).  The token returned in  yytext  shall
              only  match r. If the trailing portion of r matches the beginning of x, the result is unspecified. The r
              expression cannot include further trailing context or the '$'  (match-end-of-line)  operator;  x  cannot
              include the '^' (match-beginning-of-line) operator, nor trailing context, nor the '$' operator. That is,
              only one occurrence of trailing context is allowed in a lex regular expression,  and  the  '^'  operator
              only can be used at the beginning of such an expression.

       {name} When  name  is  one  of the substitution symbols from the Definitions section, the string, including the
              enclosing braces, shall be replaced by the substitute value. The substitute value shall  be  treated  in
              the  extended regular expression as if it were enclosed in parentheses. No substitution shall occur if {
              name} occurs within a bracket expression or within double-quotes.


       Within an ERE, a backslash character shall be considered to begin an escape sequence as specified in the  table
       in  the  Base  Definitions  volume of IEEE Std 1003.1-2001, Chapter 5, File Format Notation ( '\\', '\a', '\b',
       '\f', '\n', '\r', '\t', '\v' ). In addition, the escape sequences in the following table shall be recognized.

       A literal <newline> cannot occur within an ERE; the escape sequence '\n' can be used to represent a  <newline>.
       A <newline> shall not be matched by a period operator.

                                               Table: Escape Sequences in lex

                           Escape
                           Sequence Description                    Meaning
                           \digits  A backslash character followed The character whose encoding
                                    by the longest sequence of     is represented by the one,
                                    one, two, or three octal-digit two, or three-digit octal
                                    characters (01234567). If all  integer. If the size of a byte
                                    of the digits are 0 (that is,  on the system is greater than
                                    representation of the NUL      nine bits, the valid escape
                                    character), the behavior is    sequence used to represent a
                                    undefined.                     byte is implementation-
                                                                   defined. Multi-byte characters
                                                                   require multiple, concatenated
                                                                   escape sequences of this type,
                                                                   including the leading '\' for
                                                                   each byte.
                           \xdigits A backslash character followed The character whose encoding
                                    by the longest sequence of     is represented by the hexadec-
                                    hexadecimal-digit characters   imal integer.
                                    (01234567abcdefABCDEF). If all
                                    of the digits are 0 (that is,
                                    representation of the NUL
                                    character), the behavior is
                                    undefined.
                           \c       A backslash character followed The character 'c',  unchanged.
                                    by any character not described
                                    in this table or in the table
                                    in the Base Definitions volume
                                    of IEEE Std 1003.1-2001, Chap-
                                    ter 5, File Format Notation (
                                    '\\', '\a', '\b', '\f', '\n',
                                    '\r', '\t', '\v' ).

       Note:  If a '\x' sequence needs to be immediately followed by a hexadecimal digit character, a sequence such as
              "\x1" "1" can be used, which represents a character containing the value 1, followed  by  the  character
              '1' .


       The  order  of precedence given to extended regular expressions for lex differs from that specified in the Base
       Definitions volume of IEEE Std 1003.1-2001, Section 9.4, Extended Regular Expressions. The order of  precedence
       for lex shall be as shown in the following table, from high to low.

       Note:  The  escaped  characters  entry is not meant to imply that these are operators, but they are included in
              the table to show their relationships to the true operators. The start condition, trailing context,  and
              anchoring  notations have been omitted from the table because of the placement restrictions described in
              this section; they can only appear at the beginning or ending of an ERE.



                                                    Table: ERE Precedence in lex

                                      Extended Regular Expression        Precedence
                                      collation-related bracket symbols  [= =] [: :] [. .]
                                      escaped characters                 \<special character>
                                      bracket expression                 [ ]
                                      quoting                            "..."
                                      grouping                           ( )
                                      definition                         {name}
                                      single-character RE duplication    * + ?
                                      concatenation
                                      interval expression                {m,n}
                                      alternation                        |

       The ERE anchoring operators '^' and '$' do not appear in the table. With lex regular expressions, these  opera-
       tors  are  restricted  in  their  use:  the '^' operator can only be used at the beginning of an entire regular
       expression, and the '$' operator only at the end. The operators apply to the entire regular  expression.  Thus,
       for  example,  the  pattern  "(^abc)|(def$)" is undefined; it can instead be written as two separate rules, one
       with the regular expression "^abc" and one with "def$", which share a common action via the special '|'  action
       (see below). If the pattern were written "^abc|def$", it would match either "abc" or "def" on a line by itself.

       Unlike the general ERE rules, embedded anchoring is not allowed by  most  historical  lex  implementations.  An
       example  of embedded anchoring would be for patterns such as "(^| )foo( |$)" to match "foo" when it exists as a
       complete word. This functionality can be obtained using existing lex features:


              ^foo/[ \n]      |
              " foo"/[ \n]    /* Found foo as a separate word. */

       Note also that '$' is a form of trailing context (it is equivalent to "/\n" ) and as such cannot be  used  with
       regular  expressions containing another instance of the operator (see the preceding discussion of trailing con-
       text).

       The additional regular expressions trailing-context operator '/' can be used as an ordinary character  if  pre-
       sented  within double-quotes, "/" ; preceded by a backslash, "\/" ; or within a bracket expression, "[/]" . The
       start-condition '<' and '>' operators shall be special only in a start condition at the beginning of a  regular
       expression; elsewhere in the regular expression they shall be treated as ordinary characters.

   Actions in lex
       The  action  to  be  taken  when an ERE is matched can be a C program fragment or the special actions described
       below; the program fragment can contain one or more C statements, and can also  include  special  actions.  The
       empty  C  statement ';' shall be a valid action; any string in the lex.yy.c input that matches the pattern por-
       tion of such a rule is effectively ignored or skipped. However, the absence of an action shall  not  be  valid,
       and the action lex takes in such a condition is undefined.

       The specification for an action, including C statements and special actions, can extend across several lines if
       enclosed in braces:


              ERE <one or more blanks> { program statement
                                         program statement }

       The default action when a string in the input to a lex.yy.c program is not matched by any expression  shall  be
       to  copy  the  string  to the output. Because the default behavior of a program generated by lex is to read the
       input and copy it to the output, a minimal lex source program that has just "%%" shall  generate  a  C  program
       that simply copies the input to the output unchanged.

       Four special actions shall be available:


              |   ECHO;   REJECT;   BEGIN

       |      The  action  '|'  means  that the action for the next rule is the action for this rule. Unlike the other
              three actions, '|' cannot be enclosed in braces or be semicolon-terminated; the application shall ensure
              that it is specified alone, with no other actions.

       ECHO;  Write the contents of the string yytext on the output.

       REJECT;
              Usually  only  a  single expression is matched by a given string in the input. REJECT means "continue to
              the next expression that matches the current input", and shall cause whatever rule was the second choice
              after  the  current rule to be executed for the same input. Thus, multiple rules can be matched and exe-
              cuted for one input string or overlapping input strings.  For example,  given  the  regular  expressions
              "xyz"  and  "xy"  and  the  input "xyz", usually only the regular expression "xyz" would match. The next
              attempted match would start after z. If the last action in the "xyz" rule is REJECT, both this rule  and
              the  "xy"  rule  would  be executed. The REJECT action may be implemented in such a fashion that flow of
              control does not continue after it, as if it were equivalent to a goto to another part of  yylex().  The
              use of REJECT may result in somewhat larger and slower scanners.

       BEGIN  The action:


              BEGIN newstate;

       switches  the state (start condition) to newstate. If the string newstate has not been declared previously as a
       start condition in the Definitions section, the results are unspecified. The initial state is indicated by  the
       digit '0' or the token INITIAL.


       The  functions  or macros described below are accessible to user code included in the lex input. It is unspeci-
       fied whether they appear in the C code output of lex, or are accessible only through the -l l  operand  to  c99
       (the lex library).

       int  yylex(void)

              Performs  lexical  analysis on the input; this is the primary function generated by the lex utility. The
              function shall return zero when the end of input is reached; otherwise, it shall return non-zero  values
              (tokens) determined by the actions that are selected.

       int  yymore(void)

              When  called,  indicates that when the next input string is recognized, it is to be appended to the cur-
              rent value of yytext rather than replacing it; the value in yyleng shall be adjusted accordingly.

       int  yyless(int  n)

              Retains n initial characters in yytext, NUL-terminated, and treats the remaining characters as  if  they
              had not been read; the value in yyleng shall be adjusted accordingly.

       int  input(void)

              Returns  the  next  character  from  the  input, or zero on end-of-file.  It shall obtain input from the
              stream pointer yyin, although possibly via an intermediate buffer. Thus, once scanning  has  begun,  the
              effect  of  altering  the value of yyin is undefined. The character read shall be removed from the input
              stream of the scanner without any processing by the scanner.

       int  unput(int  c)

              Returns the character 'c' to the input; yytext and yyleng are undefined until  the  next  expression  is
              matched. The result of using unput() for more characters than have been input is unspecified.


       The  following  functions  shall  appear  only in the lex library accessible through the -l l operand; they can
       therefore be redefined by a conforming application:

       int  yywrap(void)

              Called by yylex() at end-of-file; the default  yywrap()  shall  always  return  1.  If  the  application
              requires yylex() to continue processing with another source of input, then the application can include a
              function yywrap(), which associates another file with the external variable FILE * yyin and shall return
              a value of zero.

       int  main(int  argc, char *argv[])

              Calls  yylex()  to  perform  lexical  analysis,  then exits. The user code can contain main() to perform
              application-specific operations, calling yylex() as applicable.


       Except for input(), unput(), and main(), all external and static names generated by lex shall  begin  with  the
       prefix yy or YY.

EXIT STATUS
       The following exit values shall be returned:

        0     Successful completion.

       >0     An error occurred.


CONSEQUENCES OF ERRORS
       Default.

       The following sections are informative.

APPLICATION USAGE
       Conforming  applications  are warned that in the Rules section, an ERE without an action is not acceptable, but
       need not be detected as erroneous by lex. This may result in compilation or runtime errors.

       The purpose of input() is to take characters off the input stream and discard them as far as the lexical analy-
       sis  is  concerned.  A common use is to discard the body of a comment once the beginning of a comment is recog-
       nized.

       The lex utility is not fully internationalized in its treatment of regular expressions in the lex  source  code
       or  generated  lexical  analyzer.  It  would  seem desirable to have the lexical analyzer interpret the regular
       expressions given in the lex source according to the environment specified when the lexical  analyzer  is  exe-
       cuted,  but  this  is not possible with the current lex technology. Furthermore, the very nature of the lexical
       analyzers produced by lex must be closely tied  to  the  lexical  requirements  of  the  input  language  being
       described,  which  is  frequently  locale-specific  anyway.  (For example, writing an analyzer that is used for
       French text is not automatically useful for processing other languages.)

EXAMPLES
       The following is an example of a lex program that implements a rudimentary scanner for a Pascal-like syntax:


              %{
              /* Need this for the call to atof() below. */
              #include <math.h>
              /* Need this for printf(), fopen(), and stdin below. */
              #include <stdio.h>
              %}


              DIGIT    [0-9]
              ID       [a-z][a-z0-9]*


              %%


              {DIGIT}+ {
                  printf("An integer: %s (%d)\n", yytext,
                      atoi(yytext));
                  }


              {DIGIT}+"."{DIGIT}*        {
                  printf("A float: %s (%g)\n", yytext,
                      atof(yytext));
                  }


              if|then|begin|end|procedure|function        {
                  printf("A keyword: %s\n", yytext);
                  }


              {ID}    printf("An identifier: %s\n", yytext);


              "+"|"-"|"*"|"/"        printf("An operator: %s\n", yytext);


              "{"[^}\n]*"}"    /* Eat up one-line comments. */


              [ \t\n]+        /* Eat up white space. */


              .  printf("Unrecognized character: %s\n", yytext);


              %%


              int main(int argc, char *argv[])
              {
                  ++argv, --argc;  /* Skip over program name. */
                  if (argc > 0)
                      yyin = fopen(argv[0], "r");
                  else
                      yyin = stdin;


                  yylex();
              }

RATIONALE
       Even though the -c option and references to the C language are retained in this description, lex may be  gener-
       alized  to other languages, as was done at one time for EFL, the Extended FORTRAN Language. Since the lex input
       specification is essentially language-independent, versions of this utility could be written  to  produce  Ada,
       Modula-2, or Pascal code, and there are known historical implementations that do so.

       The current description of lex bypasses the issue of dealing with internationalized EREs in the lex source code
       or generated lexical analyzer. If it follows the model used by awk (the source code is assumed to be  presented
       in  the  POSIX locale, but input and output are in the locale specified by the environment variables), then the
       tables in the lexical analyzer produced by lex would interpret EREs specified in the lex source in terms of the
       environment variables specified when lex was executed. The desired effect would be to have the lexical analyzer
       interpret the EREs given in the lex source according to the environment specified when the lexical analyzer  is
       executed, but this is not possible with the current lex technology.

       The  description  of  octal and hexadecimal-digit escape sequences agrees with the ISO C standard use of escape
       sequences. See the RATIONALE for ed for a discussion of bytes larger than 9 bits  being  represented  by  octal
       values.  Hexadecimal values can represent larger bytes and multi-byte characters directly, using as many digits
       as required.

       There is no detailed output format specification. The observed behavior of lex under four different  historical
       implementations  was  that  none  of these implementations consistently reported the line numbers for error and
       warning messages.  Furthermore, there was a desire that lex be allowed to  output  additional  diagnostic  mes-
       sages.  Leaving  message formats unspecified avoids these formatting questions and problems with international-
       ization.

       Although the %x specifier for exclusive start conditions is not historical practice, it is  believed  to  be  a
       minor  change to historical implementations and greatly enhances the usability of lex programs since it permits
       an application to obtain the expected functionality with fewer statements.

       The %array and %pointer declarations were added as a compromise between historical systems. The System  V-based
       lex  copies  the  matched  text  to  a yytext array. The flex program, supported in BSD and GNU systems, uses a
       pointer. In the latter case, significant performance improvements are available for some scanners. Most histor-
       ical  programs  should  require no change in porting from one system to another because the string being refer-
       enced is null-terminated in both cases. (The method used by flex in its case is to null-terminate the token  in
       place  by  remembering the character that used to come right after the token and replacing it before continuing
       on to the next scan.) Multi-file programs with external references to yytext outside the  scanner  source  file
       should  continue  to  operate  on their historical systems, but would require one of the new declarations to be
       considered strictly portable.

       The description of EREs avoids unnecessary duplication of ERE details because their meanings within a  lex  ERE
       are the same as that for the ERE in this volume of IEEE Std 1003.1-2001.

       The  reason  for  the undefined condition associated with text beginning with a <blank> or within "%{" and "%}"
       delimiter lines appearing in the Rules section is historical practice. Both the BSD and System V lex  copy  the
       indented (or enclosed) input in the Rules section (except at the beginning) to unreachable areas of the yylex()
       function (the code is written directly after a break statement). In some cases, the System V lex  generates  an
       error message or a syntax error, depending on the form of indented input.

       The  intention  in breaking the list of functions into those that may appear in lex.yy.c versus those that only
       appear in libl.a is that only those functions in libl.a can be reliably redefined by a conforming  application.

       The  descriptions  of standard output and standard error are somewhat complicated because historical lex imple-
       mentations chose to issue diagnostic messages to standard output (unless -t  was  given).  IEEE Std 1003.1-2001
       allows  this  behavior,  but  leaves  an  opening  for  the  more expected behavior of using standard error for
       diagnostics. Also, the System V behavior of writing the statistics when any table sizes are given  is  allowed,
       while BSD-derived systems can avoid it. The programmer can always precisely obtain the desired results by using
       either the -t or -n options.

       The OPERANDS section does not mention the use of - as a synonym for standard input; not all  historical  imple-
       mentations support such usage for any of the file operands.

       A  description of the translation table was deleted from early proposals because of its relatively low usage in
       historical applications.

       The change to the definition of the input() function that allows buffering of input  presents  the  opportunity
       for major performance gains in some applications.

       The  following examples clarify the differences between lex regular expressions and regular expressions appear-
       ing elsewhere in this volume of IEEE Std 1003.1-2001. For regular expressions of the  form  "r/x",  the  string
       matching  r  is always returned; confusion may arise when the beginning of x matches the trailing portion of r.
       For example, given the regular expression "a*b/cc" and the input "aaabcc",  yytext  would  contain  the  string
       "aaab"  on this match. But given the regular expression "x*/xy" and the input "xxxy", the token xxx, not xx, is
       returned by some implementations because xxx matches "x*" .

       In the rule "ab*/bc", the "b*" at the end of r extends r's match into the beginning of the trailing context, so
       the  result  is unspecified. If this rule were "ab/bc", however, the rule matches the text "ab" when it is fol-
       lowed by the text "bc" . In this latter case, the matching of r cannot extend into the beginning of x,  so  the
       result is specified.

FUTURE DIRECTIONS
       None.

SEE ALSO
       c99, ed, yacc

COPYRIGHT
       Portions of this text are reprinted and reproduced in electronic form from IEEE Std 1003.1, 2003 Edition, Stan-
       dard for Information Technology -- Portable Operating System Interface (POSIX), The Open Group Base  Specifica-
       tions  Issue  6,  Copyright (C) 2001-2003 by the Institute of Electrical and Electronics Engineers, Inc and The
       Open Group. In the event of any discrepancy between this version and the original IEEE and The Open Group Stan-
       dard,  the  original  IEEE  and  The  Open Group Standard is the referee document. The original Standard can be
       obtained online at http://www.opengroup.org/unix/online.html .



IEEE/The Open Group                  2003                              LEX(1P)