Root Zanli
Home
Console
Upload
information
Create File
Create Folder
About
Tools
:
/
usr
/
share
/
doc
/
re2c
/
manual
/
fill
/
Filename :
fill.rst_
back
Copy
The need for buffering arises when the input cannot be mapped in memory all at once: either it is too large, or it comes in a streaming fashion (like reading from a socket). The usual technique in such cases is to allocate a fixed-sized memory buffer and process input in chunks that fit into the buffer. When the current chunk is processed, it is moved out and new data is moved in. In practice it is somewhat more complex, because lexer state consists not of a single input position, but a set of interrelated posiitons: - cursor: the next input character to be read (``YYCURSOR`` in default API or ``YYSKIP``/``YYPEEK`` in generic API) - limit: the position after the last available input character (``YYLIMIT`` in default API, implicitly handled by ``YYLESSTHAN`` in generic API) - marker: the position of the most recent match, if any (``YYMARKER`` in default API or ``YYBACKUP``/``YYRESTORE`` in generic API) - token: the start of the current lexeme (implicit in re2c API, as it is not needed for the normal lexer operation and can be defined and updated by the user) - context marker: the position of the trailing context (``YYCTXMARKER`` in default API or ``YYBACKUPCTX``/``YYRESTORECTX`` in generic API) - tag variables: submatch positions (defined with ``/*!stags:re2c*/`` and ``/*!mtags:re2c*/`` directives and ``YYSTAGP``/``YYSTAGN``/``YYMTAGP``/``YYMTAGN`` in generic API) Not all these are used in every case, but if used, they must be updated by ``YYFILL``. All active positions are contained in the segment between token and cursor, therefore everything between buffer start and token can be discarded, the segment from token and up to limit should be moved to the beginning of buffer, and the free space at the end of buffer should be filled with new data. In order to avoid frequent ``YYFILL`` calls it is best to fill in as many input characters as possible (even though fewer characters might suffice to resume the lexer). The details of ``YYFILL`` implementation are slightly different depending on which EOF handling method is used: the case of EOF rule is somewhat simpler than the case of bounds-checking with padding. Also note that if ``-f --storable-state`` option is used, ``YYFILL`` has slightly different semantics (desrbed in the section about storable state).