Class Tokenizer

Tokenizer, which split the code into single tokens

Inheritance
System.Object
Tokenizer
Inherited Members
System.Object.ToString()
System.Object.Equals(System.Object)
System.Object.Equals(System.Object, System.Object)
System.Object.ReferenceEquals(System.Object, System.Object)
System.Object.GetHashCode()
System.Object.GetType()
System.Object.MemberwiseClone()
Namespace: Simplic.Lexer
Assembly: Simplic.Lexer.dll
Syntax
public class Tokenizer

Constructors

| Improve this Doc View Source

Tokenizer(ILexerConstants)

Create

Declaration
public Tokenizer(ILexerConstants lexerConstants)
Parameters
Type Name Description
ILexerConstants lexerConstants

Properties

| Improve this Doc View Source

Tokens

Tokens for the current tokenizer

Declaration
public Dequeue<string> Tokens { get; set; }
Property Value
Type Description
Dequeue<System.String>

Methods

| Improve this Doc View Source

Parse(String)

Parse script into tokens

Declaration
public void Parse(string Code)
Parameters
Type Name Description
System.String Code