boogie

A suite of embedded data stores in Nim with
write-ahead log (WAL) support for durability and crash recovery

nimble install boogie

API reference

😍 Key Features

• BTrees storage and Hash Tables for fast lookups
• Write-ahead log (WAL) for durability and crash recovery, with batched group commits, CRC32 record framing, and O(1) LSN recovery
• Simple API for inserting, updating, deleting, and querying records
• Configurable options for performance tuning, such as batch sizes and flush intervals
• In-memory or On-disk storage modes
• Primitive data types (string, int, float, bool, json, null)

What's included?

1. Key/Value Store - A simple Key-Value store implementation with WAL support, plus an optional lazy-read mode that keeps values on disk
2. RDBMS Store - A relational WAL-based database with support for schemas, primary keys, typed columns and foreign keys with RESTRICT delete actions
3. Vector Store - Vector store implementation with WAL support and nearest-neighbor search (cosine / dot / L2), including named partitions that bound a search to a locality scope
4. Columnar Store - Columnar storage engine for analytics workloads with WAL support and in-memory column caching
5. Graph Store - A simple graph database with support for nodes, relationships, and basic graph queries (e.g., neighbors, BFS) with WAL support
6. Document Store - A flexible document store on top of the WAL, storing JSON (or BSON) documents
7. SQL Driver - A db_connector-compatible driver (boogie/db_boogie) that compiles SQL statements to vancode bytecode and evaluates them against the RDBMS store

[!NOTE] Boogie is an experimental project mostly made with the chatbot for fun and learning. It is still in early stages, so expect data loss and breaking changes. Use at your own risk.

This can be used as a simple embedded database for your Nim applications. If you want, you can use openpeeps/e2ee to encrypt the data before inserting it into Boogie database.

Examples

Every store supports an in-memory mode and a disk-backed mode with WAL, group commits, and periodic checkpoints. Benchmark numbers come from the bundled test suites (nim r -d:release tests/testX.nim) on the author's development machine.

Key/Value Store

import boogie/stores/kv
import std/options

let kvStore = newInMemoryKvStore()  # or newKvStore("mydb", ksmDisk, enableWal = true)
kvStore.put("greeting", "hello")
kvStore.put("user:1", "alice")
if kvStore.hasKey("greeting"):
  echo kvStore.get("greeting").get()   # "hello"
discard kvStore.delete("user:1")       # true
for k, v in kvStore.pairsUnordered:    # unordered iteration
  echo k, " -> ", v

For very large datasets, newKvStore("mydb", ksmDisk, enableWal = true, lazyReads = true) keeps only a key → offset index in memory and reads values back from the WAL on demand. Values are never resident in RAM (0 bytes retained for 50k×256B in the benchmark), at the cost of a disk seek+read per get instead of a memory lookup.

RDBMS Store

import boogie/stores/rdbms
import std/options

let store = newStore("blogdata", smDisk, enableWal = true, checkpointEveryOps = 10)

let users = newTable(
  name = "users",
  primaryKey = "id",
  columns = [
    newColumn("id", dtInt, false),
    newColumn("name", dtText, false),
    newColumn("email", dtText, true)
  ],
  primaryKeyMode = pkmSerial
)
store.createTableIfNotExist(users)

# Auto-increment primary key
let userPk = store.insertRow("users", row({
  "name": newTextValue("Alice"),
  "email": newTextValue("alice@example.com")
}))

# Explicit primary key
store.insertRow("users", "42", row({"name": newTextValue("Bob")}))

# Point lookup + indexed equality search
echo store.getRow("users", userPk).isSome
let usersT = store.getTable("users").get()
usersT.createIndex("name")
for (pk, r) in usersT.where("name", newTextValue("Alice")):
  echo pk, " -> ", r

discard store.deleteRow("users", userPk)
store.checkpoint()

Vector Store

import boogie/stores/vectorstore
import std/options

let vs = newInMemoryVectorStore()  # or newVectorStore("vecdb", smDisk, enableWal = true)
vs.createCollection(newCollection("embeddings", 3))
vs.insert("embeddings", "doc-1", @[0.1'f32, 0.2, 0.3], "tenant-7")
vs.insert("embeddings", "doc-2", @[0.9'f32, 0.8, 0.7], "tenant-7")

echo vs.get("embeddings", "doc-1").isSome
let q = @[0.11'f32, 0.19, 0.31]
# scope the search to a partition to bound the scanned candidate set
for (pk, score) in vs.nearest("embeddings", q, k = 2, dmCosine, "tenant-7"):
  echo pk, " (", score, ")"

Vectors can be grouped into named partitions (a locality scope, like a ring in KoutenDB). A partition-scoped nearest scores only that partition's rows: with 20k vectors across 100 partitions it scans 200 instead of 20,000 (99% reduction) and runs ~50x faster in the benchmark. collection.partitionSize("tenant-7") reports the bounded candidate set size.

Columnar Store

import boogie/stores/columnar
import std/json

var s = openColumnarStore("analytics")

s.createTable(TableSchema(
  name: "events",
  primaryKey: "id",
  rowCount: 0,
  columns: @[
    ColumnSchema(name: "id", kind: ctInt64, nullable: false, codec: ccNone),
    ColumnSchema(name: "user", kind: ctString, nullable: false, codec: ccNone),
    ColumnSchema(name: "amount", kind: ctFloat64, nullable: false, codec: ccNone)
  ]
))

s.insertBatch("events", @[
  %*{"id": 1, "user": "alice", "amount": 10.5},
  %*{"id": 2, "user": "bob", "amount": 25.0}
])

# Projection + filter
let rows = s.scan("events", @["id", "user"], filters = @[
  Filter(column: "amount", op: foGt, value: newJFloat(10.0))
])

# Aggregation
let ag = s.aggregate("events", @[
  AggregateSpec(column: "", kind: akCount, alias: "cnt"),
  AggregateSpec(column: "amount", kind: akAvg, alias: "avg_amount")
])
echo ag["cnt"].getInt()          # 2

Columns are parsed from disk once and cached in memory, so repeated scans/filters over the same columns are much faster than the first (cold) scan.

Graph Store

import boogie/stores/graphstore
import std/json

var gs = openGraphStore("graphdb")

var tx = beginTx(gs)
let alice = createNode(tx, @["Person"], %*{"name": "Alice"})
let bob   = createNode(tx, @["Person"], %*{"name": "Bob"})
discard createRelationship(tx, alice, bob, "KNOWS", %*{"since": 2024})
commit(tx)

echo gs.neighbors(alice)          # @[bob]
for r in gs.outgoing(alice, "KNOWS"):
  echo r.toId
for n in gs.findNodesByLabel("Person"):
  echo n.id
echo gs.traverseBfs(alice, maxDepth = 2).len

closeGraphStore(gs)

Document Store

import boogie/stores/docstore
import std/json, std/options

var store = openDocumentStore("doctest", name = "documents", defaultEncoding = deJson)
store.insert("k1", %*{"name": "Alice", "age": 30})
store.upsert("k1", %*{"name": "Alice", "age": 31})

if store.hasKey("k1"):
  echo store.get("k1").get()["age"]   # 31
discard store.delete("k1")
store.checkpoint()

SQL Driver (db_connector compatible)

boogie/db_boogie mirrors the db_connector/db_sqlite API, so applications can switch to Boogie by changing a single import. Statements are parsed with openparser/sql, lowered to vancode bytecode (WHERE predicates run as real VM loops using NULL-aware comparison opcodes), and evaluated against the RDBMS store, with WAL durability and crash recovery for free.

import boogie/db_boogie

let db = open("mydb", "", "", "app")  # or open(":memory:", "", "", "")
db.exec(sql"CREATE TABLE users (
  id INTEGER PRIMARY KEY,
  name TEXT NOT NULL,
  age INT DEFAULT 0)")

db.exec(sql"INSERT INTO users (name, age) VALUES (?, ?)", "alice", "30")
echo db.insertID(sql"INSERT INTO users (name) VALUES (?)", "bob")

for row in db.fastRows(sql"SELECT id, name FROM users WHERE age > ? ORDER BY name", "18"):
  echo row

# prepared statements with typed bindings
let ps = db.prepare("UPDATE users SET age = age + 1 WHERE id = ?")
ps.bindParam(1, 1'i64)
db.exec(ps)

echo db.getValue(sql"SELECT COUNT(*) FROM users")
db.close()

Supported subset: CREATE TABLE [IF NOT EXISTS] (typed columns, PRIMARY KEY, NOT NULL, DEFAULT), DROP TABLE [IF EXISTS], CREATE INDEX, multi-row INSERT with serial primary keys, SELECT with WHERE/ORDER BY/LIMIT/ OFFSET/DISTINCT, UPDATE ... SET ... WHERE, DELETE FROM ... WHERE, aggregates without GROUP BY (COUNT/SUM/AVG/MIN/MAX), and expressions with comparisons, AND/OR/NOT, arithmetic, IS [NOT] NULL and ?/$n placeholders. JOINs, GROUP BY/HAVING and subqueries are not supported yet.

Unlike the SQLite backend, statements here are precompiled bytecode cached per connection; interpolated exec() calls recompile when their text changes. Numeric comparisons follow SQLite column-affinity rules (text operands coerce when they parse) and NULL propagates per SQL semantics.

[!TIP] Run the full test + benchmark suites with nimble test -d:release (the -d:release flag is required for accurate benchmarks).

❤ Contributions & Support

• 🐛 Found a bug? Create a new Issue
• 👋 Wanna help? Fork it!

	Switch to Open-Source LLMs via OpenCode GO, choosing from a variety of powerful models such as DeepSeek, Qwen, Kimi, GLM-5, MiniMax, MiMo. 🍕 Use our referral link to get started!

🎩 License

LGPLv3 license. Made by Humans from OpenPeeps.
Copyright OpenPeeps & Contributors — All rights reserved.