yoh2 / libbf-rs Goto Github PK

Add tokenizer that allows tokens to be described in regular expressions.

This is a preparation to make token string to available to use various scenes.

e.g. ParseError::UnexpectedEndOfFile(usize) → ParseError::UnexpectedEndOfFile{ pos_in_chars: usize }

Introduce new type of program which has token information corresponding to each instruction.
The features of the new type (tentatively called FatProgram) are following:

• FatProgram is constructed by a sequence of pairs of an instruction and several token information corresponding to the instruction.
• Introduce Nop Instruction to be able to have omitted tokens. (Token omitting causes in case of a token sequence consisting of equal numbers of pointer increments and pointer decrements. The same is true to data increments/decrements)
• FatProgram has a method to convert into Program by stripping token information.

Some changes have breaking changes.

• #11
• #12
• #14
• documentation fixes
• #18