narrow

Nim bindings for Apache Arrow, providing access to Arrow's columnar memory format and compute capabilities.

Overview

Narrow wraps the Apache Arrow GLib C API to provide Nim with:

• Columnar data structures - Arrays, chunked arrays, tables, and record batches
• I/O operations - Reading and writing Parquet, CSV, Feather, and IPC formats
• Compute operations - Expression-based filtering, aggregations, and the Acero execution engine
• Memory safety - Integration with Nim's ARC/ORC memory management via GObject reference counting

Documentation

• API docs — generated from source
• Cookbook — runnable recipes for common tasks

Requirements

• Nim 2.2.6 or later
• Apache Arrow GLib (arrow-glib) >= 24.0.0
• Apache Parquet GLib (parquet-glib) >= 24.0.0
• Apache Arrow Dataset GLib (arrow-dataset-glib) >= 24.0.0
• GLib 2.0 (glib-2.0)
• GObject 2.0 (gobject-2.0)
• pkg-config

Installing Dependencies

Ubuntu/Debian:

sudo apt install libarrow-glib-dev libparquet-glib-dev libarrow-dataset-glib-dev pkg-config

macOS:

brew install apache-arrow-glib

The apache-arrow-glib Homebrew formula includes all required components: - arrow-glib (Apache Arrow GLib) - parquet-glib (Apache Parquet GLib) - arrow-dataset-glib (Apache Arrow Dataset GLib) - GLib and GObject (dependencies of apache-arrow-glib)

Windows (MSYS2):

Install MSYS2, then open a MSYS2 MINGW64 terminal and run:

pacman -S mingw-w64-x86_64-gcc mingw-w64-x86_64-pkg-config mingw-w64-x86_64-glib2 mingw-w64-x86_64-arrow

All subsequent commands (nimble install, nimble test, etc.) must also be run from the MSYS2 MINGW64 terminal, as pkg-config and the compiler are only available there.

Installation

Add to your .nimble file:

requires "narrow >= 0.0.1"

Or install directly:

nimble install narrow

With atlas:

atlas use narrow

Note: Narrow wraps the Apache Arrow GLib C API. You must install the system libraries (arrow-glib, parquet-glib, arrow-dataset-glib) and link them via pkg-config when compiling your project. See Installing Dependencies below.

Usage

Creating Arrays

import narrow

# Create arrays from sequences
let intArr = newArray(@[1'i32, 2'i32, 3'i32])
let strArr = newArray(@["hello", "world"])

# Or use builders for more control
var builder = newArrayBuilder[int64]()
builder.append(1'i64)
builder.append(2'i64)
builder.appendNull()
let arr = builder.finish()

Creating Tables

import narrow
import narrow/column/metadata

# Define schema
let schema = newSchema([
  newField[int32]("id"),
  newField[string]("name"),
  newField[float64]("score")
])

# Create arrays
let ids = newArray(@[1'i32, 2'i32, 3'i32])
let names = newArray(@["Alice", "Bob", "Charlie"])
let scores = newArray(@[95.5'f64, 87.2'f64, 92.1'f64])

# Build table
let table = newArrowTable(schema, ids, names, scores)

Reading and Writing Parquet

import narrow/io/parquet

# Write table to Parquet
writeTable(table, "data.parquet")

# Read table from Parquet
let table = readTable("data.parquet")

# Read with filtering
import narrow/compute/expressions

let age = col("age")
let filtered = readTable("data.parquet", filter = age > 18)

Reading and Writing CSV

import narrow/io/csv

# Read CSV
let table = readCSV("data.csv")

# Read with custom delimiter
let table = readCSV("data.csv", newCsvReadOptions(delimiter = some(';')))

# Write table to CSV
writeCsv("output.csv", table)

Filtering Data

import narrow/compute/expressions

let name = col("name")
let age = col("age")
let active = col("active")

# Simple filters
let filter1 = age >= 18
let filter2 = name.contains("admin")
let filter3 = startsWith(name, "A")

# Combined filters
let complexFilter = (age >= 18) and (name.toLower().contains("admin")) and active.isValid()

let filtered = table.filter(complexFilter)

Project Status

This library is under active development. APIs may change until a stable release.

Current capabilities: - Primitive array types (int, float, bool, string) - Nested types (lists, structs, maps) - Temporal types (date, timestamp, duration) - CSV and Parquet I/O - Expression-based filtering - Acero execution engine bindings

Development

Building from Source

git clone https://github.com/BontaVlad/narrow.git
cd narrow
nimble install -y

Running Tests

just test

# or run tests in parallel
just test-debug-par

Or with nimble:

LSAN_OPTIONS="suppressions=lsan.supp:print_suppressions=0" nimble test -d:useSanitizers

Generating Bindings

The Arrow C API bindings are auto-generated using Futhark. This is only needed for development (updating bindings when Arrow C API changes), not for using the library:

nimble generate

Acknowledgments

This project relies on several open source libraries:

• Apache Arrow - Columnar data format and compute libraries
• Arrow GLib - GObject-based C bindings for Arrow
• Parquet GLib - Parquet format support for Arrow GLib
• Arrow Dataset GLib - Dataset API for Arrow GLib
• GLib - Core application building blocks
• GObject - Object system for C
• Futhark - Nim bindings generator for C libraries

License

MIT License - see LICENSE file for details.