Fan-out / fan-in with durable per-item recovery in Rust on Resonate

A workflow that fans work out across several parallel branches and then joins on their results loses progress on every crash if each branch is just an in-process task — the items that had already finished get re-executed when the workflow restarts. Resonate gives each parallel branch its own durable promise, so completed branches are replayed from the log instead of re-executed and only pending branches actually run again. The example-fan-out-fan-in-rs repo shows this in roughly 50 lines: one workflow function that spawns three leaf functions in parallel, and the leaf function itself.

The shape of the solution

#[resonate::function]
async fn fan_out_fan_in(ctx: &Context, items: Vec<WorkItem>) -> Result<Vec<WorkResult>> {
    // Fan-out: spawn all three items in parallel
    let h1 = ctx.run(process_item, items[0].clone()).spawn().await?;
    let h2 = ctx.run(process_item, items[1].clone()).spawn().await?;
    let h3 = ctx.run(process_item, items[2].clone()).spawn().await?;
 
    // Fan-in: collect all results — each is individually durable
    let r1 = h1.await?;
    let r2 = h2.await?;
    let r3 = h3.await?;
 
    Ok(vec![r1, r2, r3])
}
// from example-fan-out-fan-in-rs/src/main.rs:23

The leaf function it spawns is a plain #[resonate::function] that returns a serializable result:

#[resonate::function]
async fn process_item(item: WorkItem) -> Result<WorkResult> {
    let output = format!("Processed: {} (item #{})", item.data, item.id);
    Ok(WorkResult {
        id: item.id,
        output,
    })
}
// from example-fan-out-fan-in-rs/src/main.rs:41

The durable primitives in play

• ctx.run(process_item, item) — builds a RunTask that will execute process_item as a child durable execution of the parent workflow. Source: src/main.rs:26–28. SDK: resonate-sdk-rs/resonate/src/context.rs:244 (pub fn run).
• .spawn().await? — eagerly creates the child's durable promise on the server and starts the task on a fresh tokio::spawn, returning a DurableFuture<T> handle. The promise exists before the parent moves on, so a crash after spawn().await? does not lose the fact that this branch was started. Source: src/main.rs:26–28. SDK: resonate-sdk-rs/resonate/src/context.rs:588 (pub async fn spawn).
• DurableFuture<T> — the handle returned by .spawn(). Awaiting it returns Result<T> once the child task resolves; if the child was already resolved when this worker took over (replay after crash), the await returns the cached value without re-running the child. Source: src/main.rs:31–33. SDK: resonate-sdk-rs/resonate/src/futures.rs:10 (struct) and resonate-sdk-rs/resonate/src/futures.rs:46 (IntoFuture impl that resolves from the recorded result).
• #[resonate::function] — registers the function as a durable execution target with the SDK; each invocation gets its own promise record on the server. Source: src/main.rs:23 (workflow) and src/main.rs:41 (leaf).
• Resonate::new(...) + resonate.register(fan_out_fan_in) + resonate.register(process_item) — wires the worker to the Resonate server at http://localhost:8001 and tells the server which functions this worker can execute. Source: src/main.rs:52–58.

What the SDK handles vs. what you write

What you write: the body of fan_out_fan_in reads like straight-line Rust — three spawn calls, three await calls, return the collected vector. The leaf function is just a Result-returning async fn. No retry loop, no checkpoint table, no per-item state machine, no idempotency keys in user code.

What the SDK handles: creating one durable promise per child before the child runs, persisting the child's resolved value on the server when it completes, returning the cached value on replay instead of re-executing the child, and dispatching the child body onto a fresh local tokio task in the same worker process. On a crash, when the workflow is picked up again, each ctx.run(...).spawn().await? short-circuits to a DurableFuture::resolved(...) for any child that had already completed (see resonate-sdk-rs/resonate/src/context.rs:607–611), and only pending children are re-spawned onto a new tokio task. Cross-worker routing is a separate primitive (ctx.rpc(...).spawn() / RemoteFuture) that this example does not use.

Failure modes covered

• Worker crashes after some children have completed. When the workflow resumes, .spawn() calls consume_promise_record (resonate-sdk-rs/resonate/src/context.rs:605) which returns the existing promise record for each child; for children in PromiseState::Resolved .spawn() returns DurableFuture::resolved(value) directly (resonate-sdk-rs/resonate/src/context.rs:607–611), so the leaf body never runs a second time. Only children still in PromiseState::Pending are re-spawned.
• Worker crashes before any children have completed. The parent workflow itself is durable; on restart, spawn().await? for each item creates the durable promise and starts the child task. The create-promise call is idempotent — if the promise already exists, the server returns the existing record (resonate-sdk-rs/resonate/src/effects.rs:35–41, invoked from .spawn() at context.rs:605). No duplicate work, because the promise IDs are derived from the parent.
• A child task panics or is dropped mid-flight. The DurableFuture await maps a dropped sender to Error::JoinError(format!("task {} was dropped", id)) (resonate-sdk-rs/resonate/src/futures.rs:63); the workflow's ? propagates that error out of fan_out_fan_in. The child's promise stays in pending or rejected state on the server and is the source of truth for what to do next.
• A child resolves to an error. PromiseState::Rejected / RejectedCanceled / RejectedTimedout produce a DurableFuture::rejected(...) (resonate-sdk-rs/resonate/src/context.rs:612–616); awaiting it returns Err, which the h1.await? line in the workflow propagates.

When to reach for this pattern

• If you need to run N independent work items in parallel and the cost of re-running any one of them on a crash is non-trivial.
• If the work items are heterogeneous in latency and you don't want a slow item to serialize fast ones, but you still want one durable identity per item rather than one big retry.
• If you want true parallelism (each .spawn() lands on a fresh tokio::spawn task, multiplexed across the Tokio runtime's worker threads) rather than the cooperative concurrency of tokio::join! over ctx.run(...).await. The SDK's own doc-comment frames it as "True parallelism via .spawn()" (resonate-sdk-rs/resonate/src/context.rs:240).
• If you want each child's result observable in the execution tree as its own promise (use resonate tree <id> against the parent ID per the repo README).
• If you have a fixed, small fan-out per workflow invocation; this example handles exactly three items by index. Variable-N fan-out is the same primitive in a loop, but this repo does not demonstrate that shape.

Sources

• Example repo: https://github.com/resonatehq-examples/example-fan-out-fan-in-rs
• Workflow function: src/main.rs:23–36
• Leaf function: src/main.rs:41–48
• Worker setup: src/main.rs:50–65
• Dependency pin: Cargo.toml:8 (resonate = { git = "https://github.com/resonatehq/resonate-sdk-rs", branch = "main" } — the repo tracks the SDK's main branch, not a published crate version)
• Resonate Rust SDK: https://github.com/resonatehq/resonate-sdk-rs
• ctx.run: resonate-sdk-rs/resonate/src/context.rs:244
• .spawn() for local children: resonate-sdk-rs/resonate/src/context.rs:588
• DurableFuture: resonate-sdk-rs/resonate/src/futures.rs:10, IntoFuture impl at :46
• create_promise idempotency contract: resonate-sdk-rs/resonate/src/effects.rs:35–41