GPU AToLL

Overview

GPU-AToLL (GPU Auspice Tech on LLVM) targets efficient and safe multi-tenancy on NVIDIA GPUs. It applies Software Fault Isolation (SFI) techniques at the NVVM IR level, enabling memory and fault isolation across tenant workloads that run on a single CUDA context. Combined with a restricted API for scheduling GPU tasks, GPU AToLL achieves:

• Safe spatial sharing of GPU resources for tenant workloads
• Minimal start-up costs (through avoiding CUDA context initialization)
• Close-to-native speed (by working at the NVVM IR level, after LLVM optimizations)

Core Infrastructure

GPU AToLL consists of two reusable layers:

• AToLL LLVM passes harden NVVM IR modules before they get compiled to PTX.
• atoll-runtime-core manages CUDA state, application loading, pipeline lifecycles, memory arenas, and toolchain invocation.

The client-server/ directory provides client-server infrastructure implemented on top of atoll-runtime-core.

Key Concepts

• Application: A set of CUDA kernels, provided as LLVM bitcode, that is hardened and compiled to PTX when loaded.
• Pipeline: An isolated execution context with its own CUDA stream and Device Arena. Device-buffer allocations made through the Pipeline are regions within that arena.
• Device Arena: A per-Pipeline GPU memory allocation. The runtime validates operations against its boundaries, while hardened kernels confine device-buffer accesses to it.

Build and Installation

Requirements

• CUDA Driver: 13.0+
• Rust: 2024 edition (1.86+)
• CMake: 3.20+
• Ninja

Build

From a fresh clone, first initialize the pinned dependencies and their required submodules:

cargo atoll init

Then build IREE and LLVM, the AToLL native libraries, and the Rust workspace:

cargo atoll build

Additional verification commands are available through the same Cargo entry point:

cargo atoll check-abi     # Verify shared C, C++, and Rust ABI values
cargo atoll test-passes   # Run GPU AToLL's LLVM pass regression suite

Installation

Build and install to the default ./install/ staging prefix, or select another writable prefix:

cargo atoll install
cargo atoll install --prefix ~/.local

For a system-wide installation, build and stage the complete prefix tree as your normal user, then copy only the finished artifacts with elevated permissions:

cargo atoll install
sudo cp -a install/. /usr/local/

Do not run Cargo itself through sudo; doing so can create root-owned build artifacts and may not find a user-managed Cargo installation. Pass --no-build to install already-built release artifacts.

API Documentation

Rust API documentation for atoll-runtime-core is generated by GitLab CI and published through GitLab Pages at https://gpu-atoll.gitlab.io/gpu-atoll/atoll-runtime-core/.

To build the same docs locally:

cargo doc -p atoll-runtime-core --no-deps --open

Components

AToLL LLVM Passes

Built as libAtollPasses.so from the passes/ directory. The plugin provides the atoll-mem pass (class AtollMem) and the promote-ptx-constant-globals pass. The atoll-mem pass:

1. Determines the memory location and concrete PTX state space for each instruction with memory side effects, rejecting accesses that cannot be classified safely.
   • Supports global, shared, local, and constant PTX memory spaces.
2. Preserves each operation's required access alignment when rewriting pointers, preventing misaligned tenant accesses from faulting the shared CUDA context.
3. Zero-initializes supported local and shared memory, including dynamic shared memory, at kernel entry to prevent residual data leakage.

Hardened GPU AToLL modules must not retain writable module-declared storage, because that storage would be shared across isolated module instances. The promote-ptx-constant-globals pass therefore moves eligible immutable NVPTX globals from the generic address space into PTX constant memory while enforcing its module-wide 64 KiB capacity.

atoll-runtime-core

atoll-runtime-core is the reusable runtime foundation for GPU AToLL implementations. It:

• Owns CUDA state and caches loaded modules.
• Validates and compiles LLVM bitcode to hardened PTX using the AToLL passes.
• Manages application and pipeline lifecycles.
• Allocates device-memory arenas and validates operations against their boundaries.
• Validates kernel argument types and executes kernel launches.

atoll-harden

A CLI tool for applying the AToLL passes directly to an LLVM bitcode or IR file. Useful for inspecting hardening output and debugging.

atoll-harden kernels.bc                     # Harden; output: kernels.hardened.ll
atoll-harden kernels.bc -o hardened.ll      # Specify output file
atoll-harden kernels.bc --verbose           # Print per-instruction analysis
atoll-harden kernels.bc --summarize         # Print the hardening analytics summary table
atoll-harden kernels.bc --clamping-mode modulo-based  # Use modulo-based hardening

Project Structure

gpu-atoll/
├── .cargo/config.toml   # Defines the `cargo atoll` command
├── assets/              # Images and architecture diagrams used throughout the project
├── atoll-runtime-core/  # Reusable CUDA runtime, application and pipeline core
├── atoll-types/         # Types shared across runtime-core and client-server crates
├── Cargo.toml           # Rust workspace manifest
├── client-server/       # Client-server infrastructure built on top of runtime core
├── external/            # External dependencies
├── passes/              # AToLL LLVM passes, plugin, and LIT tests
├── scripts/             # Developer/user helper scripts
├── tests/               # Project-wide live tests (require appropriate hardware)
└── tools/xtask/         # Cargo-based build, test, and install orchestration

Roadmap

Sound Device Code Validation

As is, the project only hardens against memory-related violations. For soundness, further guarantees have to be made to isolate all possible fault escapes:

• Ensure Control Flow Integrity
• Investigate stack exhaustion mitigations

Fair Resource Scheduling

CUDA Green Contexts: Currently, GPU AToLL uses CUDA streams for spatial execution, but integration of Green Contexts is planned to allow for more granular control over resource allocation and scheduling. As of CUDA 13.1, green contexts are supported by the Driver API, enabling us to integrate them without bringing in the CUDA runtime as a dependency.

Synchronization Timeouts and Fuel Metering: GPU AToLL currently limits resource exhaustion with per-application synchronization timeouts. This protection is cooperative: end users are trusted to configure reasonable timeout values and to call synchronization primitives from their host-side code. Fuel metering remains a future direction for enforcing compute budgets within kernels and potentially preempting over-budget or non-terminating work without relying on those assumptions.

Extended CUDA Feature Support

• Support additional NVVM IR memory operations, including Tensor Memory Accelerator (TMA) copies and Distributed Shared Memory (DSM) accesses within thread block clusters
• Allow calling device kernels (currently calling kernels from device code is unsupported, all such calls are expected to be inlined)
• Investigate potential support for heap allocations

Acknowledgments

This work was funded by the European Union, under the Horizon Europe grant 101092850 (project AERO).