DynGenPar
Dynamic Generalized Parser
Public Member Functions | Protected Member Functions | Protected Attributes | List of all members
DynGenPar::TokenSource Class Referenceabstract
Inheritance diagram for DynGenPar::TokenSource:
Inheritance graph

Public Member Functions

 TokenSource ()
 
virtual ~TokenSource ()
 
Cat nextToken ()
 get the next token from the input, increment current position, save parse tree More...
 
Node parseTree ()
 get the parse tree for the last shifted token More...
 
virtual bool matchParseTree (const Node &treeToMatch)
 match the parse tree for the last shifted token against the given tree More...
 
int currentPosition ()
 get the current input position More...
 
virtual bool rewindTo (int pos, const LexerState &=LexerState())
 rewind to an older position (requires buffering) More...
 
virtual LexerState saveState ()
 saves the current lexer state, by default a null LexerState More...
 

Protected Member Functions

virtual Cat readToken ()=0
 get the next token from the input, to be implemented by subclasses More...
 
bool simpleRewind (int pos, bool rewindOnly=false)
 basic implementation of rewindTo for subclasses which support it More...
 

Protected Attributes

int currPos
 
Node tree
 sub-parse-tree for hierarchical parsing More...
 

Detailed Description

Definition at line 804 of file dyngenpar.h.

Constructor & Destructor Documentation

◆ TokenSource()

DynGenPar::TokenSource::TokenSource ( )
inline

Definition at line 806 of file dyngenpar.h.

◆ ~TokenSource()

virtual DynGenPar::TokenSource::~TokenSource ( )
inlinevirtual

Definition at line 807 of file dyngenpar.h.

Member Function Documentation

◆ currentPosition()

int DynGenPar::TokenSource::currentPosition ( )
inline

get the current input position

Definition at line 840 of file dyngenpar.h.

◆ matchParseTree()

virtual bool DynGenPar::TokenSource::matchParseTree ( const Node treeToMatch)
inlinevirtual

match the parse tree for the last shifted token against the given tree

Returns
true if they should be considered identical for the purposes of a PMCFG rule using the same variable twice, false otherwise

The default implementation compares only the category. Other token sources may want to also look at the data and children fields.

Definition at line 836 of file dyngenpar.h.

◆ nextToken()

Cat DynGenPar::TokenSource::nextToken ( )
inline

get the next token from the input, increment current position, save parse tree

An epsilon token is returned and currPos is not incremented if the end of input was reached.

Definition at line 813 of file dyngenpar.h.

◆ parseTree()

Node DynGenPar::TokenSource::parseTree ( )
inline

get the parse tree for the last shifted token

Definition at line 825 of file dyngenpar.h.

◆ readToken()

virtual Cat DynGenPar::TokenSource::readToken ( )
protectedpure virtual

get the next token from the input, to be implemented by subclasses

Implemented in DynGenPar::ListTokenSource, DynGenPar::TextByteTokenSource, DynGenPar::ByteTokenSource, and DynGenPar::FlexLexerTokenSource.

◆ rewindTo()

virtual bool DynGenPar::TokenSource::rewindTo ( int  pos,
const LexerState = LexerState() 
)
inlinevirtual

rewind to an older position (requires buffering)

Returns
true if successful, false otherwise

in all cases, destroys the saved parse tree

By default, only succeeds if the position is the current one, otherwise always returns false. Can be overridden by subclasses.

Some subclasses (e.g., file-backed ones) can wind both forward and backward, some can only rewind, some cannot wind at all. The method will return false if seeking to the new position is not possible.

Reimplemented in DynGenPar::ListTokenSource, DynGenPar::TextByteTokenSource, and DynGenPar::ByteTokenSource.

Definition at line 853 of file dyngenpar.h.

◆ saveState()

virtual LexerState DynGenPar::TokenSource::saveState ( )
inlinevirtual

saves the current lexer state, by default a null LexerState

Reimplemented in DynGenPar::TextByteTokenSource.

Definition at line 858 of file dyngenpar.h.

◆ simpleRewind()

bool DynGenPar::TokenSource::simpleRewind ( int  pos,
bool  rewindOnly = false 
)
inlineprotected

basic implementation of rewindTo for subclasses which support it

some subclasses may need additional processing

Definition at line 864 of file dyngenpar.h.

Member Data Documentation

◆ currPos

int DynGenPar::TokenSource::currPos
protected

Definition at line 871 of file dyngenpar.h.

◆ tree

Node DynGenPar::TokenSource::tree
protected

sub-parse-tree for hierarchical parsing

This variable can be set by readToken to a subtree produced by a hierarchical parser, which will be attached to the resulting parse tree in lieu of a leaf node.

If this variable is left unset, a leaf node is automatically produced.

Definition at line 879 of file dyngenpar.h.


The documentation for this class was generated from the following file: