DynGenPar::TokenSource Class Reference

Inheritance diagram for DynGenPar::TokenSource:

List of all members.

Public Member Functions
	TokenSource ()
virtual	~TokenSource ()
Cat	nextToken ()
	get the next token from the input, increment current position, save parse tree
Node	parseTree ()
	get the parse tree for the last shifted token
virtual bool	matchParseTree (const Node &treeToMatch)
	match the parse tree for the last shifted token against the given tree
int	currentPosition ()
	get the current input position
virtual bool	rewindTo (int pos, const LexerState &=LexerState())
	rewind to an older position (requires buffering)
virtual LexerState	saveState ()
	saves the current lexer state, by default a null LexerState
Protected Member Functions
virtual Cat	readToken ()=0
	get the next token from the input, to be implemented by subclasses
bool	simpleRewind (int pos)
	basic implementation of rewindTo for subclasses which support it
Protected Attributes
int	currPos
Node	tree
	sub-parse-tree for hierarchical parsing

---

Detailed Description

Definition at line 696 of file dyngenpar.h.

---

Constructor & Destructor Documentation

DynGenPar::TokenSource::TokenSource

(

)

[inline]

Definition at line 698 of file dyngenpar.h.

virtual DynGenPar::TokenSource::~TokenSource

(

)

[inline, virtual]

Definition at line 699 of file dyngenpar.h.

---

Member Function Documentation

int DynGenPar::TokenSource::currentPosition

(

)

[inline]

get the current input position

Definition at line 732 of file dyngenpar.h.

virtual bool DynGenPar::TokenSource::matchParseTree

(

const Node &

treeToMatch

)

[inline, virtual]

match the parse tree for the last shifted token against the given tree

Returns:

true if they should be considered identical for the purposes of a PMCFG rule using the same variable twice, false otherwise

The default implementation compares only the category. Other token sources may want to also look at the data and children fields.

Definition at line 728 of file dyngenpar.h.

Cat DynGenPar::TokenSource::nextToken

(

)

[inline]

get the next token from the input, increment current position, save parse tree

An epsilon token is returned and currPos is not incremented if the end of input was reached.

Definition at line 705 of file dyngenpar.h.

Node DynGenPar::TokenSource::parseTree

(

)

[inline]

get the parse tree for the last shifted token

Definition at line 717 of file dyngenpar.h.

virtual Cat DynGenPar::TokenSource::readToken

(

)

[protected, pure virtual]

get the next token from the input, to be implemented by subclasses

Implemented in DynGenPar::ByteTokenSource, DynGenPar::TextByteTokenSource, DynGenPar::ListTokenSource, and DynGenPar::FlexLexerTokenSource.

virtual bool DynGenPar::TokenSource::rewindTo	(	int	pos,
		const LexerState &	= LexerState()
	)		[inline, virtual]

rewind to an older position (requires buffering)

Returns:

true if successful, false otherwise

in all cases, destroys the saved parse tree

By default, only succeeds if the position is the current one, otherwise always returns false. Can be overridden by subclasses.

Reimplemented in DynGenPar::ByteTokenSource, DynGenPar::TextByteTokenSource, and DynGenPar::ListTokenSource.

Definition at line 741 of file dyngenpar.h.

virtual LexerState DynGenPar::TokenSource::saveState

(

)

[inline, virtual]

saves the current lexer state, by default a null LexerState

Reimplemented in DynGenPar::TextByteTokenSource.

Definition at line 746 of file dyngenpar.h.

bool DynGenPar::TokenSource::simpleRewind

(

int

pos

)

[inline, protected]

basic implementation of rewindTo for subclasses which support it

some subclasses may need additional processing

Definition at line 752 of file dyngenpar.h.

---

Member Data Documentation

int DynGenPar::TokenSource::currPos [protected]

Definition at line 759 of file dyngenpar.h.

Node DynGenPar::TokenSource::tree [protected]

sub-parse-tree for hierarchical parsing

This variable can be set by readToken to a subtree produced by a hierarchical parser, which will be attached to the resulting parse tree in lieu of a leaf node.

If this variable is left unset, a leaf node is automatically produced.

Definition at line 767 of file dyngenpar.h.

---

The documentation for this class was generated from the following file:

• dyngenpar.h