Class Tokenizer

java.lang.Object
com.codename1.ui.editor.Tokenizer
All Implemented Interfaces:
SyntaxHighlighter
public class Tokenizer extends Object implements SyntaxHighlighter

A small, allocation light, stateful lexer for the pure code editor. It tokenizes a single line at a time given the lexer state carried over from the previous line, and returns the state to carry into the next line. This makes rehighlighting incremental: after an edit only the changed line and any following lines whose entry state changed need to be retokenized.

Only the spans that carry color (keyword, string, comment, number) are emitted; the gaps between them are drawn in the default text color.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final int
    COMMENT
    Token kind: a comment.
    static final int
    KEYWORD
    Token kind: a language keyword.
    static final int
    NUMBER
    Token kind: a numeric literal.
    static final int
    PROPERTY
    Token kind: an attribute name or object property.
    static final int
    STATE_BLOCK_COMMENT
    Lexer state: inside a block comment that opened on a previous line.
    static final int
    STATE_CSS_COMMENT_DECLARATION
    Lexer state: inside a CSS comment opened from a declaration block.
    static final int
    STATE_CSS_DECLARATION
    Lexer state: inside a CSS declaration block.
    static final int
    STATE_NORMAL
    Lexer state: normal code.
    static final int
    STATE_TEMPLATE
    Lexer state: inside a backtick template string that opened on a previous line.
    static final int
    STATE_TRIPLE_DOUBLE
    Lexer state: inside a Python triple-double-quoted string.
    static final int
    STATE_TRIPLE_SINGLE
    Lexer state: inside a Python triple-single-quoted string.
    static final int
    STATE_XML_COMMENT
    Lexer state: inside an XML/HTML comment.
    static final int
    STRING
    Token kind: a string or character literal.
    static final int
    TYPE
    Token kind: an XML/HTML tag or a declared type.

  • Constructor Summary

    Constructors
    Constructor
    Description
    Tokenizer(LanguageDef def)
    Creates a tokenizer for the given language.

  • Method Summary

    Modifier and Type
    Method
    Description
    SyntaxHighlightResult
    tokenize(String line, int startState)
    Tokenizes one line.

    Methods inherited from class Object

    clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • KEYWORD

      public static final int KEYWORD
      Token kind: a language keyword.
      See Also:

    • STRING

      public static final int STRING
      Token kind: a string or character literal.
      See Also:

    • COMMENT

      public static final int COMMENT
      Token kind: a comment.
      See Also:

    • NUMBER

      public static final int NUMBER
      Token kind: a numeric literal.
      See Also:

    • TYPE

      public static final int TYPE
      Token kind: an XML/HTML tag or a declared type.
      See Also:

    • PROPERTY

      public static final int PROPERTY
      Token kind: an attribute name or object property.
      See Also:

    • STATE_NORMAL

      public static final int STATE_NORMAL
      Lexer state: normal code.
      See Also:

    • STATE_BLOCK_COMMENT

      public static final int STATE_BLOCK_COMMENT
      Lexer state: inside a block comment that opened on a previous line.
      See Also:

    • STATE_TEMPLATE

      public static final int STATE_TEMPLATE
      Lexer state: inside a backtick template string that opened on a previous line.
      See Also:

    • STATE_XML_COMMENT

      public static final int STATE_XML_COMMENT
      Lexer state: inside an XML/HTML comment.
      See Also:

    • STATE_TRIPLE_SINGLE

      public static final int STATE_TRIPLE_SINGLE
      Lexer state: inside a Python triple-single-quoted string.
      See Also:

    • STATE_TRIPLE_DOUBLE

      public static final int STATE_TRIPLE_DOUBLE
      Lexer state: inside a Python triple-double-quoted string.
      See Also:

    • STATE_CSS_DECLARATION

      public static final int STATE_CSS_DECLARATION
      Lexer state: inside a CSS declaration block.
      See Also:

    • STATE_CSS_COMMENT_DECLARATION

      public static final int STATE_CSS_COMMENT_DECLARATION
      Lexer state: inside a CSS comment opened from a declaration block.
      See Also:

  • Constructor Details

    • Tokenizer

      public Tokenizer(LanguageDef def)
      Creates a tokenizer for the given language.

  • Method Details

    • tokenize

      public SyntaxHighlightResult tokenize(String line, int startState)

      Tokenizes one line.

      Parameters

      • line: the line text (without its trailing newline)

      • startState: the lexer state carried from the previous line

      Returns

      the colored spans and exit state

      Specified by:
      tokenize in interface SyntaxHighlighter