Fearless Concurrency
on the GPU

An interactive guide to cuTile Rust, ownership-aware GPU kernels, and the execution model behind Grout.

Authors

  • Melih Elibol1
  • Jared Roesch1
  • Isaac Gelado1
  • Eric Buehler2
  • Michael Garland1

Affiliations

  1. NVIDIA, USA
  2. Hugging Face, USA

Published

Jun. 14, 2026

PDF

GPU speed, without leaving Rust’s safety model

Rust’s ownership rules are valuable precisely when many actors touch shared state. A GPU kernel is one of the hardest places to preserve them: host code launches asynchronous work, device programs execute in parallel, and a mistaken destination index can turn into a race with undefined behavior.

cuTile Rust narrows the problem. Programmers write tile operations with single-threaded semantics. Mutable output tensors are partitioned into disjoint regions before launch. Shared inputs stay immutable. The generated launch boundary carries those distinctions from host Rust into device-side tensor views.

The design in one sentence

Split mutable memory before the GPU sees it, preserve ordering inside each tile, and make every escape hatch explicit.

A typed path to the GPU
safe / typed / tiled

Make the race impossible.

cuTile Rust maps Rust ownership onto GPU tiles, then keeps the fast path close to Tile IR.

01 / rustborrow rules
02 / tileowned region
03 / gpuordered work
CUTILE RUSTunsafe stays explicit
The paper connects Rust source, a generated launcher, Tile IR, and a runtime-compiled cubin.

The result is not “Rust makes every GPU program safe.” The safe surface covers the tensor access patterns cuTile Rust can express. Raw pointers and unchecked access remain available inside explicit unsafe code when a kernel needs lower-level control.

Ownership crosses the launch boundary

The familiar Rust rule is aliasing XOR mutability: one &mut T, or many &T, but not both. cuTile Rust maps that distinction onto a tile grid.

A host program partitions a mutable output, then passes each tile program an exclusive sub-tensor. Immutable inputs are broadcast as shared tensor views. The generated launcher holds those values while work is in flight and recovers them after synchronization with the same ownership form.

One output, many disjoint owners
Drag the tile count. Each tile program receives a different mutable output region while shared inputs remain readable by all programs.

The paper’s element-wise add example contains no unsafe: host code makes 128-element output partitions, and the kernel loads matching input tiles before storing their sum. A proc macro generates the typed launch interface and enforces the accepted kernel parameter grammar at compile time.

What does not become safe automatically?

The model does not prove arbitrary per-thread protocols, unrestricted raw-pointer arithmetic, or every possible synchronization pattern. Unsupported patterns remain explicit unsafe opt-outs. The safety claim is about the accepted tensor and partition API under the paper’s launch and Tile IR assumptions.

Disjointness outside. Ordering inside.

Partitioning handles conflicts between tile programs. A second problem remains inside one tile program: GPU compilers may reorder operations unless the IR carries an ordering edge.

Tile IR uses tokens. Operations on an exclusive mutable view form a token chain, so a load followed by a store has a happens-before relationship. Reads through shared immutable views carry no such token dependency and can be reordered for performance.

t0loadt1storet2 t_0 \xrightarrow{\texttt{load}} t_1 \xrightarrow{\texttt{store}} t_2
Watch the ordering contract
Compare shared reads, which are free to move, with mutable load/store operations, which remain ordered.

The paper gives a concrete race: a cuTile Python attention permutation swaps two destination indices. Different tile programs then write the same destination, and 17–35% of elements differ across runs in the reported configurations. In cuTile Rust’s partition view, the programmer does not choose that destination index; the owned sub-tensor is the destination.

The index-swap bug becomes unconstructible
This conceptual view contrasts an indexed store with a partition-owned store. It illustrates the API boundary, not a compiler proof replay.
Theorem scope

The paper proves data-race freedom for kernels written in the safe cuTile Rust API, assuming the generated launcher and kernel entry realize the stated disjoint partitions and token threading. Unsafe opt-outs are excluded from that theorem.

Build once, choose how to execute

Host-side GPU work is represented by DeviceOp: a lazy, typed operation that owns or borrows its operands and composes before submission. The same operation can be driven synchronously, awaited by an async executor, or captured as a CUDA graph.

Choose the execution regime
Pipeline length amortizes async callback cost. Graph replay removes per-kernel launch overhead; async matters when the host has useful work to overlap.

The paper reports about 7.3 μs per operation for individually synchronized launches, about 3.4 μs for chained sync or amortized async, and about 0.8 μs for graph replay on the RTX 5090 experiment. Those are measurements for the described setup, not universal constants.

Why scoped graph capture helps borrowing

A recorded operation is added to the capture stream but does not execute as an independent kernel during capture. A borrow can therefore be released between record calls while the graph retains stream order. The implementation restricts graph capture to non-allocating operations so replay does not depend on unstable allocation addresses.

Safety without a visible tax

The central performance question is whether ownership-aware access costs more than manual pointer access. On the NVIDIA B200, the paper compares safe Rust, unsafe Rust, cuTile Python, and cuBLAS across memory-bound element-wise addition and compute-bound GEMM.

At N = 2^28, safe and unsafe Rust both reach 7.02 TB/s for element-wise add, while cuTile Python reaches 7.01 TB/s against a reported 7.68 TB/s peak. At M = N = K = 8192, safe Rust reaches 2.07 PFlop/s, 96.4% of cuBLAS; unsafe Rust is within 0.3% of it, and cuTile Python reaches 2.04 PFlop/s.

The safety overhead result
Reported B200 measurements from the paper. The chart separates same-backend comparisons from the vendor-library reference.

The end-to-end case study is Grout, a Qwen3 inference engine built on cuTile Rust. In the single-request sweep, Grout reaches 154.7 generated tokens/s for Qwen3-4B on an RTX 5090 at generation length 8192 and 80.1 tokens/s for Qwen3-32B on a B200, compared with 77.5 for vLLM and 76.5 for SGLang in that B200 configuration.

What the numbers do not say

The benchmarks use specific GPUs, model sizes, tile shapes, clocks, and batch-1 workloads. Grout is a specialized case study, not a general-purpose serving-stack replacement. The paper also reports GEMM gaps at some sizes and uses cuBLAS for model GEMMs.

The useful boundary

cuTile Rust’s correspondence is strongest where a kernel naturally looks like:

  1. partition an output into disjoint tiles;
  2. read shared inputs;
  3. perform tile operations;
  4. write the owned output view;
  5. compose the launch with an explicit execution mode.

It gives up some SIMT-level control in exchange for tile-level reasoning. Explicit warp primitives, shared-memory protocols, and patterns outside the tensor API still need unsafe code. The API is young, and the paper identifies broader safe coverage, async heterogeneous scheduling, and cross-device ownership as future work.

The durable idea is not that a type system erases GPU complexity. It is that a carefully chosen abstraction can move the most common race invariants into ordinary Rust ownership, keep the fast path close to the backend, and leave the remaining danger visible.

Read the source

Melih Elibol, Jared Roesch, Isaac Gelado, Eric Buehler, and Michael Garland. “Fearless Concurrency on the GPU.” arXiv:2606.15991, 2026. Paper · CUDA Tile · cuTile Python · Grout