.TH "Lexing" 3o source: 2019-01-25 OCamldoc "OCaml library"
.SH NAME
Lexing \- The run-time library for lexers generated by ocamllex.
.SH Module
Module   Lexing
.SH Documentation
.sp
Module
.BI "Lexing"
 : 
.B sig  end

.sp
The run\-time library for lexers generated by 
.B ocamllex
\&.

.sp

.sp
.sp

.PP

.B === 
.B Positions
.B  ===

.PP
.I type position 
= {
 pos_fname : 
.B string
;
 pos_lnum : 
.B int
;
 pos_bol : 
.B int
;
 pos_cnum : 
.B int
;
 }

.sp
A value of type 
.B position
describes a point in a source file\&.
.B pos_fname
is the file name; 
.B pos_lnum
is the line number;
.B pos_bol
is the offset of the beginning of the line (number
of characters between the beginning of the lexbuf and the beginning
of the line); 
.B pos_cnum
is the offset of the position (number of
characters between the beginning of the lexbuf and the position)\&.
The difference between 
.B pos_cnum
and 
.B pos_bol
is the character
offset within the line (i\&.e\&. the column number, assuming each
character is one column wide)\&.
.sp
See the documentation of type 
.B lexbuf
for information about
how the lexing engine will manage positions\&.

.sp

.I val dummy_pos 
: 
.B position
.sp
A value of type 
.B position
, guaranteed to be different from any
valid position\&.

.sp

.PP

.B === 
.B Lexer buffers
.B  ===

.PP
.I type lexbuf 
= {
 refill_buff : 
.B lexbuf -> unit
;

.B mutable 
lex_buffer : 
.B bytes
;

.B mutable 
lex_buffer_len : 
.B int
;

.B mutable 
lex_abs_pos : 
.B int
;

.B mutable 
lex_start_pos : 
.B int
;

.B mutable 
lex_curr_pos : 
.B int
;

.B mutable 
lex_last_pos : 
.B int
;

.B mutable 
lex_last_action : 
.B int
;

.B mutable 
lex_eof_reached : 
.B bool
;

.B mutable 
lex_mem : 
.B int array
;

.B mutable 
lex_start_p : 
.B position
;

.B mutable 
lex_curr_p : 
.B position
;
 }

.sp
The type of lexer buffers\&. A lexer buffer is the argument passed
to the scanning functions defined by the generated scanners\&.
The lexer buffer holds the current state of the scanner, plus
a function to refill the buffer from the input\&.
.sp
At each token, the lexing engine will copy 
.B lex_curr_p
to
.B lex_start_p
, then change the 
.B pos_cnum
field
of 
.B lex_curr_p
by updating it with the number of characters read
since the start of the 
.B lexbuf
\&.  The other fields are left
unchanged by the lexing engine\&.  In order to keep them
accurate, they must be initialised before the first use of the
lexbuf, and updated by the relevant lexer actions (i\&.e\&. at each
end of line \-\- see also 
.B new_line
)\&.

.sp

.I val from_channel 
: 
.B Pervasives.in_channel -> lexbuf
.sp
Create a lexer buffer on the given input channel\&.
.B Lexing\&.from_channel inchan
returns a lexer buffer which reads
from the input channel 
.B inchan
, at the current reading position\&.

.sp

.I val from_string 
: 
.B string -> lexbuf
.sp
Create a lexer buffer which reads from
the given string\&. Reading starts from the first character in
the string\&. An end\-of\-input condition is generated when the
end of the string is reached\&.

.sp

.I val from_function 
: 
.B (bytes -> int -> int) -> lexbuf
.sp
Create a lexer buffer with the given function as its reading method\&.
When the scanner needs more characters, it will call the given
function, giving it a byte sequence 
.B s
and a byte
count 
.B n
\&. The function should put 
.B n
bytes or fewer in 
.B s
,
starting at index 0, and return the number of bytes
provided\&. A return value of 0 means end of input\&.

.sp

.PP

.B === 
.B Functions for lexer semantic actions
.B  ===

.PP

.PP

.B === The following functions can be called from the semantic actions
.B    of lexer definitions (the ML code enclosed in braces that
.B    computes the value returned by lexing functions)\&. They give
.B    access to the character string matched by the regular expression
.B    associated with the semantic action\&. These functions must be
.B    applied to the argument lexbuf, which, in the code generated by
.B    ocamllex, is bound to the lexer buffer passed to the parsing
.B    function\&. ===

.PP

.I val lexeme 
: 
.B lexbuf -> string
.sp

.B Lexing\&.lexeme lexbuf
returns the string matched by
the regular expression\&.

.sp

.I val lexeme_char 
: 
.B lexbuf -> int -> char
.sp

.B Lexing\&.lexeme_char lexbuf i
returns character number 
.B i
in
the matched string\&.

.sp

.I val lexeme_start 
: 
.B lexbuf -> int
.sp

.B Lexing\&.lexeme_start lexbuf
returns the offset in the
input stream of the first character of the matched string\&.
The first character of the stream has offset 0\&.

.sp

.I val lexeme_end 
: 
.B lexbuf -> int
.sp

.B Lexing\&.lexeme_end lexbuf
returns the offset in the input stream
of the character following the last character of the matched
string\&. The first character of the stream has offset 0\&.

.sp

.I val lexeme_start_p 
: 
.B lexbuf -> position
.sp
Like 
.B lexeme_start
, but return a complete 
.B position
instead
of an offset\&.

.sp

.I val lexeme_end_p 
: 
.B lexbuf -> position
.sp
Like 
.B lexeme_end
, but return a complete 
.B position
instead
of an offset\&.

.sp

.I val new_line 
: 
.B lexbuf -> unit
.sp
Update the 
.B lex_curr_p
field of the lexbuf to reflect the start
of a new line\&.  You can call this function in the semantic action
of the rule that matches the end\-of\-line character\&.

.sp
.B "Since"
3.11.0

.sp

.PP

.B === 
.B Miscellaneous functions
.B  ===

.PP

.I val flush_input 
: 
.B lexbuf -> unit
.sp
Discard the contents of the buffer and reset the current
position to 0\&.  The next use of the lexbuf will trigger a
refill\&.

.sp