public class PythonTokenMaker extends AbstractJFlexTokenMaker
| Modifier and Type | Field and Description |
|---|---|
static int |
LONG_STRING_1 |
static int |
LONG_STRING_2 |
static int |
YYEOF
This character denotes the end of file
|
static int |
YYINITIAL
lexical states
|
offsetShift, s, startcurrentToken, firstToken, previousToken| Constructor and Description |
|---|
PythonTokenMaker()
Constructor.
|
PythonTokenMaker(java.io.InputStream in)
Creates a new scanner.
|
PythonTokenMaker(java.io.Reader in)
Creates a new scanner
There is also a java.io.InputStream version of this constructor.
|
| Modifier and Type | Method and Description |
|---|---|
void |
addToken(char[] array,
int start,
int end,
int tokenType,
int startOffset)
Adds the token specified to the current linked list of tokens.
|
java.lang.String[] |
getLineCommentStartAndEnd(int languageIndex)
Returns the text to place at the beginning and end of a
line to "comment" it in this programming language.
|
Token |
getTokenList(javax.swing.text.Segment text,
int initialTokenType,
int startOffset)
Returns the first token in the linked list of tokens generated
from
text. |
void |
yybegin(int newState)
Enters a new lexical state
|
char |
yycharat(int pos)
Returns the character at position pos from the
matched text.
|
void |
yyclose()
Closes the input stream.
|
int |
yylength()
Returns the length of the matched text region.
|
Token |
yylex()
Resumes scanning until the next regular expression is matched,
the end of input is encountered or an I/O-Error occurs.
|
void |
yypushback(int number)
Pushes the specified amount of characters back into the input stream.
|
void |
yyreset(java.io.Reader reader)
Resets the scanner to read from a new input stream.
|
int |
yystate()
Returns the current lexical state.
|
java.lang.String |
yytext()
Returns the text matched by the current regular expression.
|
yybeginaddNullToken, addToken, addToken, createOccurrenceMarker, getClosestStandardTokenTypeForInternalType, getCurlyBracesDenoteCodeBlocks, getInsertBreakAction, getLanguageIndex, getLastTokenTypeOnLine, getMarkOccurrencesOfTokenType, getOccurrenceMarker, getShouldIndentNextLineAfter, isIdentifierChar, isMarkupLanguage, resetTokenList, setLanguageIndexpublic static final int YYEOF
public static final int YYINITIAL
public static final int LONG_STRING_2
public static final int LONG_STRING_1
public PythonTokenMaker()
public PythonTokenMaker(java.io.Reader in)
in - the java.io.Reader to read input from.public PythonTokenMaker(java.io.InputStream in)
in - the java.io.Inputstream to read input from.public void addToken(char[] array,
int start,
int end,
int tokenType,
int startOffset)
addToken in interface TokenMakeraddToken in class TokenMakerBasearray - The character array.start - The starting offset in the array.end - The ending offset in the array.tokenType - The token's type.startOffset - The offset in the document at which this token
occurs.public java.lang.String[] getLineCommentStartAndEnd(int languageIndex)
getLineCommentStartAndEnd in interface TokenMakergetLineCommentStartAndEnd in class TokenMakerBaselanguageIndex - The language index at the offset in question.
Since some TokenMakers effectively have nested
languages (such as JavaScript in HTML), this parameter tells the
TokenMaker what sub-language to look at.null value for either means there
is no string to add for that part. A value of
null for the array means this language
does not support commenting/uncommenting lines.public Token getTokenList(javax.swing.text.Segment text, int initialTokenType, int startOffset)
text. This method must be implemented by
subclasses so they can correctly implement syntax highlighting.text - The text from which to get tokens.initialTokenType - The token type we should start with.startOffset - The offset into the document at which
text starts.Token in a linked list representing
the syntax highlighted text.public final void yyreset(java.io.Reader reader)
reader - the new input streampublic final void yyclose()
throws java.io.IOException
java.io.IOExceptionpublic final int yystate()
public final void yybegin(int newState)
yybegin in class AbstractJFlexTokenMakernewState - the new lexical statepublic final java.lang.String yytext()
public final char yycharat(int pos)
pos - the position of the character to fetch.
A value from 0 to yylength()-1.public final int yylength()
public void yypushback(int number)
number - the number of characters to be read again.
This number must not be greater than yylength()!public Token yylex() throws java.io.IOException
java.io.IOException - if any I/O-Error occurs