toktok

Generic tokenizer written in Nim language 👑 Powered by Nim's Macros

Pure Nim score 15/100 · tests present · no docs generated

Summary

Latest Version	Unknown
License	MIT
CI Status	Failing
Downloads	0
Last Indexed	2026-07-21 05:25

Installation

nimble install toktok

choosenim install toktok

git clone https://github.com/openpeeps/toktok

OS Compatibility

Platform	Linux	macOS	Windows	FreeBSD	OpenBSD	NetBSD	Android	iOS	WASM	Embedded
toktok	✓	✓	✓	-	-	-	-	-	-	-

Source

Repository	https://github.com/openpeeps/toktok
Homepage	https://github.com/openpeeps/toktok
Registry Source	nimble_official

README

Generic tokenizer written in Nim language, powered by Nim's Macros 👑

nimble install toktok

API reference

😍 Key Features

✨ Powered by Nim's Macros
🪄 Based on std/lexbase / Zero Regular Expression
Compile-time generation using macro-based TokenKind enum, lexbase
Runtime generation using TokenKind tables, lexbase
Open Source | MIT

[!NOTE] This is a generic Lexer, based on std/ streams, lexbase and macros. It is meant to be used by higher level parsers for writing any kind of tools or programs.

[!NOTE] Compile with -d:toktokdebug to inspect the generated code.

Quick Example

# Register your custom handlers
handlers:
  proc handleImport(lex: var Lexer, kind: TokenKind) =
    # tokenize `import x, y, z`
    lex.kind = kind

# Register your tokens
registerTokens defaultSettings:
  `const` = "const"
  `echo` = "hello"
  asgn = '=':   # `=` is tkAsgn
    eq = '='    # `==` is tkEQ
  excl = '!':
    ne = '='
  at = '@':
    import = tokenizer(handleImport, "import") 


# Tokenizing...
var
  tok = lexer.init(sample)
  prev: TokenTuple
  curr: TokenTuple = tok.getToken
  next: TokenTuple = tok.getToken 

proc walk(tok: var Lexer) =
  prev = curr
  curr = next
  next = tok.getToken

while likely(curr.kind != tkEOF):
  if tok.hasError: break
  echo curr # use `getToken` consumer to get token by token
  walk tok

TODO

[ ] Runtime Token generation using tables/critbits

❤ Contributions & Support

🐛 Found a bug? Create a new Issue
👋 Wanna help? Fork it!