Async DAGs vs Fibers: Stackless and Stackful Concurrency in C++

You need to run 100,000 outbound RPCs at once. You are not going to spawn 100,000 OS threads — that’s gigabytes of stacks and a scheduler meltdown. So the whole game of a high-concurrency server is: overlap latency without a thread per in-flight operation. There are two architectures for that, and they are duals of each other.

The first is the async DAG. You express the work as a graph of continuations: each async step returns a future/promise (or a reactive stream, or a sender), and you wire “do this after that” with .then / when_all / co_await. When a result resolves, its continuation fires and the graph advances. Nothing blocks a thread — a “wait” just registers a callback and returns. This is what a future, a promise, a reactive stream, and a C++26 sender all are underneath: the same async dependency DAG in different clothing.

The second is the fiber. You write ordinary sequential, blocking-looking code — and each concurrent task runs on a lightweight user-space thread with its own stack. When a fiber “blocks,” the scheduler saves its stack and switches to another; the OS thread never parks.

Both run a hundred thousand things on a handful of threads. The difference is entirely in how a paused piece of work is represented — a small heap object, or a whole stack — and that one decision cascades into memory, ergonomics, debuggability, and the single most important property of all: whether “async” infects your whole codebase.

Model A — the async DAG (stackless)

A paused computation is represented by a continuation object on the heap that holds only the state that must survive the wait. There is no stack sitting idle. When the awaited result arrives, “resuming” is just a function call into that continuation — no stack switch, no registers to swap.

The nodes are futures/promises; the edges are continuations. The same three shapes recur:

flowchart LR
  A[fetch user] -->|then| C[render]
  B[fetch prefs] -->|then| C
  C -->|then| D[write cache]
  C -->|then| E[log]

fetch user and fetch prefs are in flight simultaneously; render is a when_all join that fires when both resolve; then two independent continuations fan out. No thread was blocked at any node — each arrow is “when this future completes, schedule that continuation.”

In modern C++ you write that graph as straight-line code with stackless coroutines (co_await), and the compiler builds the continuation for you:

Task<Page> render_page(UserId id) {
  auto [user, prefs] = co_await when_all(fetch_user(id), fetch_prefs(id));  // join node
  Page p = render(user, prefs);
  co_await write_cache(p);       // suspension point → returns to the executor
  co_return p;
}

That reads sequential, but there is no stack held across the co_awaits. The compiler transforms render_page into a state machine and heap-allocates a coroutine frame that stores just the resume point and the locals still live at each suspension:

coroutine frame (heap, ~tens–hundreds of bytes)
┌───────────────────────────────┐
│ resume_index : 2              │  ← which co_await we're parked at
│ promise/return slot           │
│ live locals across suspend:   │
│   user, prefs, p              │  ← ONLY what must survive the wait
└───────────────────────────────┘
resume = coroutine_handle.resume()  →  a normal call back into the state machine

Under the hood a future/promise is a small shared state: a value slot, a continuation slot, a refcount. .then(f) stores f; fulfilling the promise schedules f on an executor (a thread pool whose ready-queue is — yes — a producer-consumer queue). A when_all node is a counter that fires downstream at zero. Two dialects:

• Eager — the future starts running the moment it’s created (folly::Future, std::future, JS Promise). f.thenValue(g) chains continuations.
• Lazy — you compose a description and it runs only when started (std::execution senders/receivers, C++26). Connecting a sender to a receiver yields an operation-state stored in place — no heap allocation — and start() kicks it; completion signals (set_value/set_error/set_stopped) walk back up the receiver chain. It’s a typed, structured, allocation-free async DAG:

using namespace std::execution;
sender auto pipe =
      when_all(fetch_user(id), fetch_prefs(id))
    | then([](User u, Prefs p){ return render(u, p); })
    | let_value([](Page p){ return write_cache(p) | then([&]{ return p; }); });
Page page = std::this_thread::sync_wait(std::move(pipe)).value();

The cost model. Memory is O(live state) per in-flight op — tens to hundreds of bytes — so millions of concurrent async operations fit comfortably. “Context switching” isn’t a switch at all; it’s a call to the next continuation. This is why event-loop / future-based servers scale to enormous concurrency on few threads.

Model B — fibers (stackful)

A paused computation is represented by an entire stack. A fiber is a user-space thread: its own contiguous stack (default 64 KB to a few MB), its own saved registers, cooperatively scheduled. You write normal sequential code; a “wait” on a fiber-aware primitive yields — the scheduler saves this fiber’s stack pointer + callee-saved registers and restores another fiber’s.

Page render_page(UserId id) {
  boost::fibers::future<User>  fu = fetch_user_async(id);   // launches work
  boost::fibers::future<Prefs> fp = fetch_prefs_async(id);
  User  user  = fu.get();     // "blocks" → YIELDS this fiber; OS thread runs others
  Prefs prefs = fp.get();
  Page p = render(user, prefs);
  write_cache(p).get();       // yields again; resumes right here, stack intact
  return p;                   // ordinary control flow, exceptions, loops all work
}

There is no .then, no co_await, no state machine — the stack itself is the continuation. Everything live across the wait (the whole call chain, not just a few locals) is preserved because the whole stack is preserved.

Under the hood the switch is boost::context’s jump_fcontext (hand-written assembly per architecture): save the current callee-saved registers and stack pointer into the old fiber, load the new fiber’s. No kernel, ~100 ns.

2 OS threads, a scheduler ready-queue, many fibers — each with a full stack:

  T0 ▶ [fiber A stack ~256 KB] ──fu.get()──▶ save SP+regs, jump ──▶ [fiber C]
                                                     │
  ready-queue (PC): [C][D][E] ◀── pushed when A's future resolves
                                                     │
  T1 ▶ [fiber B stack ~256 KB] ──runs to a yield──▶ next ready fiber

  a fiber can suspend ANYWHERE — even 20 frames deep inside a library call —
  because its whole stack is what gets parked.

The cost model. Memory is O(max stack depth) per fiber — KB to MB each — so “a million fibers” is real memory pressure in a way “a million futures” is not; fibers scale to thousands–hundreds of thousands, bounded by stack RAM. The switch is a genuine register/SP swap (~100 ns), cheap but not free. In exchange you get natural control flow and real, contiguous stack traces.

The architectural head-to-head

Everything below flows from heap continuation vs whole stack:

Notice the ready-queue line: both are, at bottom, a producer-consumer queue feeding worker threads. That plumbing is identical. The architecture diverges only in what sits in the queue — a tiny continuation, or a fiber owning a stack.

The one that actually decides it: function coloring

The deepest difference isn’t performance — it’s who has to know. Async is viral. The moment fetch_user returns a future (or is a coroutine), its caller must co_await/.then it, which makes the caller async, which makes its caller async, all the way up. Async “colors” every function on the path (What Color Is Your Function?). You cannot call an async function from an ordinary one and simply wait — not without blocking a thread and losing the whole point.

Fibers are transparent / colorless. Any ordinary synchronous function — including third-party code you can’t modify — can run inside a fiber, and if something 15 frames down calls a fiber-aware wait, the whole fiber suspends. No signatures change, no async keyword climbs the call stack. This is why fibers are the pragmatic way to make a large existing synchronous codebase concurrent without rewriting it into continuations, and why suspension “anywhere, even deep in a callee” is a genuine capability, not a footnote.

The flip side is the trap: because a fiber looks like a normal thread, calling a thread-blocking primitive by accident (a raw std::mutex, a blocking syscall, a non-fiber future.get) parks the OS thread and can deadlock the pool. Async DAGs don’t have that failure mode — nothing is ever “parked” because nothing holds a thread while it waits.

They are the same DAG, represented two ways

Squint and the duality is exact. Both express a graph of dependent async work and run it on a small thread pool via a producer-consumer ready-queue. The async model makes the dependencies explicit (the .then/when_all edges) and the control flow implicit (scattered across continuations); the fiber model makes the control flow explicit (sequential code) and the dependencies implicit (they’re just the order of your .get() calls). One stores a paused node’s state in a minimal heap frame; the other stores it in a whole stack. Every other trade-off — memory, switch cost, coloring, debuggability — is downstream of that single representational choice.

Where C++ is going, and the pragmatic split

• std::execution senders/receivers (C++26) is the standard betting on the stackless async DAG, and it’s worth understanding what it actually is, because it fixes the two things that historically made async DAGs painful.

  A sender is a lazy description of async work — it hasn’t started; it just describes what will happen and how it will finish. A receiver is the trio of callbacks a sender invokes on completion: set_value(results...) on success, set_error(e) on failure, set_stopped() on cancellation. Connecting a sender to a receiver produces an operation-state that is stored in place — no heap allocation — and calling start() on it kicks off the work. The algorithms (then, when_all, let_value) are themselves senders that wrap senders, so the entire async DAG becomes one typed value the compiler can see through and inline — no per-node allocation, no type-erased std::function continuations.

  On top of that it bakes in structured concurrency: async operations form a strict tree tied to lexical scope, exactly like function calls. A parent operation cannot complete until every child it spawned has completed — the same rule as “a function can’t return until its callees return.” That is what kills the classic async DAG hazard: because a child is bounded by its parent’s scope, a continuation can no longer outlive the data it captured by reference (the dangling-reference-in-a-lambda bug that plagues raw callback graphs), and cancellation propagates cleanly down the tree via set_stopped. It is, in short, the async DAG with composition and lifetimes finally solved at the type level.

• Fibers (boost::fiber, folly::fibers) remain the answer when you can’t or won’t color your code: a large synchronous codebase you need to make concurrent, deep/arbitrary suspension points, or a place where readable sequential logic and real stack traces beat squeezing per-op memory. Game engines go further and run a task graph on fibers, so a job can wait on its children without ever parking a thread.

Choosing

• Reach for the async DAG (futures / senders) when concurrency is huge and memory-bound — hundreds of thousands of in-flight RPCs — the per-op state is small, and you want tight memory bounds and composable, structured pipelines. This is the default for high-fanout serving.
• Reach for fibers when you need colorless concurrency over existing synchronous code, suspension deep inside call chains, or sequential readability and real stack traces — and you can afford a stack per in-flight task and are disciplined about never calling thread-blocking primitives.
• Fuse them when you have a dependency graph and jobs that must block/join mid-flight: a task DAG scheduled onto fibers (the game-engine pattern) buys core utilization and colorless blocking.

The mistake is arguing “callbacks vs coroutines” or “futures vs fibers” as a style preference. They’re two representations of the same async dependency graph, and the real question is a systems one: should a paused piece of work cost a heap frame or a whole stack? Answer that against your concurrency scale, your memory budget, and how much code you’re willing to paint async — and the architecture picks itself.