DynGenPar::TokenSource Class Reference

Inheritance diagram for DynGenPar::TokenSource:

Public Member Functions
	TokenSource ()
virtual	~TokenSource ()
Cat	nextToken ()
	get the next token from the input, increment current position, save parse tree More...
Node	parseTree ()
	get the parse tree for the last shifted token More...
virtual bool	matchParseTree (const Node &treeToMatch)
	match the parse tree for the last shifted token against the given tree More...
int	currentPosition ()
	get the current input position More...
virtual bool	rewindTo (int pos, const LexerState &=LexerState())
	rewind to an older position (requires buffering) More...
bool	rewindTo (int pos, const Node &parseTree, const LexerState &lexerState=LexerState())
	rewind to an older position (requires buffering) and restore the parse tree (needed for lookahead) More...
virtual LexerState	saveState ()
	saves the current lexer state, by default a null LexerState More...

Protected Member Functions
virtual Cat	readToken ()=0
	get the next token from the input, to be implemented by subclasses More...
bool	simpleRewind (int pos, bool rewindOnly=false)
	basic implementation of rewindTo for subclasses which support it More...

Protected Attributes
int	currPos
Node	tree
	sub-parse-tree for hierarchical parsing More...

Detailed Description

Definition at line 813 of file dyngenpar.h.

Constructor & Destructor Documentation

TokenSource()

DynGenPar::TokenSource::TokenSource ( )

inline

Definition at line 815 of file dyngenpar.h.

~TokenSource()

virtual DynGenPar::TokenSource::~TokenSource ( )

inlinevirtual

Definition at line 816 of file dyngenpar.h.

Member Function Documentation

currentPosition()

int DynGenPar::TokenSource::currentPosition ( )

inline

get the current input position

Definition at line 849 of file dyngenpar.h.

matchParseTree()

virtual bool DynGenPar::TokenSource::matchParseTree ( const Node &  treeToMatch )

inlinevirtual

match the parse tree for the last shifted token against the given tree

Returns

true if they should be considered identical for the purposes of a PMCFG rule using the same variable twice, false otherwise

The default implementation compares only the category. Other token sources may want to also look at the data and children fields.

Definition at line 845 of file dyngenpar.h.

nextToken()

Cat DynGenPar::TokenSource::nextToken ( )

inline

get the next token from the input, increment current position, save parse tree

An epsilon token is returned and currPos is not incremented if the end of input was reached.

Definition at line 822 of file dyngenpar.h.

parseTree()

Node DynGenPar::TokenSource::parseTree ( )

inline

get the parse tree for the last shifted token

Definition at line 834 of file dyngenpar.h.

readToken()

virtual Cat DynGenPar::TokenSource::readToken ( )

protectedpure virtual

get the next token from the input, to be implemented by subclasses

Implemented in DynGenPar::ListTokenSource, DynGenPar::TextByteTokenSource, DynGenPar::ByteTokenSource, and DynGenPar::FlexLexerTokenSource.

rewindTo() [1/2]

virtual bool DynGenPar::TokenSource::rewindTo ( int  pos, const LexerState &  = LexerState()  )

inlinevirtual

rewind to an older position (requires buffering)

Returns

true if successful, false otherwise

in all cases, destroys the saved parse tree

By default, only succeeds if the position is the current one, otherwise always returns false. Can be overridden by subclasses.

Some subclasses (e.g., file-backed ones) can wind both forward and backward, some can only rewind, some cannot wind at all. The method will return false if seeking to the new position is not possible.

Reimplemented in DynGenPar::ListTokenSource, DynGenPar::TextByteTokenSource, and DynGenPar::ByteTokenSource.

Definition at line 862 of file dyngenpar.h.

rewindTo() [2/2]

bool DynGenPar::TokenSource::rewindTo ( int  pos, const Node &  parseTree, const LexerState &  lexerState = LexerState()  )

inline

rewind to an older position (requires buffering) and restore the parse tree (needed for lookahead)

Returns

true if successful, false otherwise

in all cases, restores the saved parse tree to the passed parseTree

Relies on the virtual rewindTo to do its actual work.

Definition at line 874 of file dyngenpar.h.

saveState()

virtual LexerState DynGenPar::TokenSource::saveState ( )

inlinevirtual

saves the current lexer state, by default a null LexerState

Reimplemented in DynGenPar::TextByteTokenSource.

Definition at line 881 of file dyngenpar.h.

simpleRewind()

bool DynGenPar::TokenSource::simpleRewind ( int  pos, bool  rewindOnly = false  )

inlineprotected

basic implementation of rewindTo for subclasses which support it

some subclasses may need additional processing

Definition at line 887 of file dyngenpar.h.

Member Data Documentation

currPos

int DynGenPar::TokenSource::currPos

protected

Definition at line 894 of file dyngenpar.h.

tree

Node DynGenPar::TokenSource::tree

protected

sub-parse-tree for hierarchical parsing

This variable can be set by readToken to a subtree produced by a hierarchical parser, which will be attached to the resulting parse tree in lieu of a leaf node.

If this variable is left unset, a leaf node is automatically produced.

Definition at line 902 of file dyngenpar.h.

---

The documentation for this class was generated from the following file:

• dyngenpar.h