A simple tokenizing module for the Jai programming language. You can check the code in module.jai and a few examples of how to use them in example.jai.

The idea is to take text and transform it to a set of tokens that are easier to work with for parsing purposes. To start you'd initialize a Tokenizer which just contains the running data used as we generate the tokens:

text := "1 123.123 abc";
tokenizer : Tokenizer;
init_tokenizer(*tokenizer, text);

Then you can retrieve an array of all tokens by calling tokenize_all_tokens:

tokens := tokenize_all_tokens(*tokenizer);
for tokens do_something_with(token);

Alternatively you could request one token at a time and handle them as they come:

while true
{
    token, error := tokenize_one_token(*tokenizer);
    if token.type == .INVALID || token.type == .END then break;
    do_something_with(token);
}

Some running examples of this in example.jai, including some toy parser for some code.

I want people to be able to just use this without thinking about licensing, although give me a shout if you do use it and attribution is appreciated 😊. Replicating STB's (https://github.com/nothings/stb) licensing stuff where you can choose to use public domain or MIT.