Coming from Temporal: A Real Orchestrator (Saga + Approval) in Go

This brief maps a real production orchestrator — a durable Kubernetes node-drain workflow — onto the Temporal patterns it combines. There is no single 1:1 Temporal sample for this shape. It is a composite of two:

• temporalio/samples-go/saga — sequential steps with compensation, each run as a Temporal activity.
• temporalio/samples-go/await-signals — a workflow that parks on an external event, waiting for a signal from an operator or client process.

If you are porting either of those patterns, or combining them, the example-node-drain-orchestrator-go repo shows both in one codebase. The sibling briefs Saga / Compensation and Human-in-the-Loop cover each pattern individually; this one shows what it looks like when you compose them.

The shape

The orchestrator drains Kubernetes nodes one at a time. Two mechanics run the workflow:

1. Checkpointed steps (ctx.Run) — each drain attempt, the node-list fetch, and the timestamp recordings are all durable steps. On resume after a crash the SDK checks whether each child promise has already settled; if it has, the result is returned immediately without re-executing. This maps to the saga pattern: sequential activity executions, each recorded in the workflow's history.

2. Operator approval gate (ctx.Promise) — when a drain is blocked by a PodDisruptionBudget, the workflow creates a latent durable promise, logs its ID, and parks. An operator resolves the promise through the gateway HTTP process, and the workflow immediately resumes. This maps to the await-signals pattern: a workflow suspended until an external actor delivers data.

A separate gateway binary starts operations and settles decisions. It never runs workflow code. This maps to the Temporal pattern of a worker process plus a standalone client that calls SignalWorkflow.

Temporal: saga — checkpointed steps

The Temporal saga sample runs a sequence of activities, deferring compensating activities on failure:

// saga/workflow.go (temporalio/samples-go)
func TransferMoney(ctx workflow.Context, transferDetails TransferDetails) (err error) {
    options := workflow.ActivityOptions{
        StartToCloseTimeout: time.Minute,
        RetryPolicy: &temporal.RetryPolicy{
            InitialInterval:    time.Second,
            BackoffCoefficient: 2.0,
            MaximumAttempts:    3,
        },
    }
    ctx = workflow.WithActivityOptions(ctx, options)
 
    err = workflow.ExecuteActivity(ctx, Withdraw, transferDetails).Get(ctx, nil)
    if err != nil {
        return err
    }
 
    defer func() {
        if err != nil {
            errCompensation := workflow.ExecuteActivity(ctx, WithdrawCompensation, transferDetails).Get(ctx, nil)
            err = multierr.Append(err, errCompensation)
        }
    }()
 
    err = workflow.ExecuteActivity(ctx, Deposit, transferDetails).Get(ctx, nil)
    // ...
    return nil
}

Temporal: await-signals — external event gate

The await-signals sample parks the workflow on a channel condition, with a background goroutine receiving signals:

// await-signals/await_signals_workflow.go (temporalio/samples-go)
func (a *AwaitSignals) Listen(ctx workflow.Context) {
    for {
        selector := workflow.NewSelector(ctx)
        selector.AddReceive(workflow.GetSignalChannel(ctx, "Signal1"), func(c workflow.ReceiveChannel, more bool) {
            c.Receive(ctx, nil)
            a.Signal1Received = true
        })
        // ... Signal2, Signal3 ...
        selector.Select(ctx)
    }
}
 
func AwaitSignalsWorkflow(ctx workflow.Context) (err error) {
    var a AwaitSignals
    workflow.Go(ctx, a.Listen)          // background goroutine receives signals
 
    err = workflow.Await(ctx, func() bool {
        return a.Signal1Received
    })
    // ...
}

The external process that delivers the signal:

// await-signals/starter/main.go (temporalio/samples-go)
err = c.SignalWorkflow(context.Background(), we.GetID(), we.GetRunID(), "Signal1", nil)

Resonate: checkpointed steps (drain/orchestrator.go)

Each drain step is a plain Go function passed to ctx.Run. No activity registration, no ActivityOptions, no task-queue wiring. The function runs once; its result is recorded on a child durable promise and short-circuits on replay:

// from drain/orchestrator.go:144-168
func (o *Orchestrator) DrainAllNodes(ctx *resonate.Context, args Args) (Result, error) {
    startedAt, err := o.checkpointNow(ctx)
    if err != nil {
        return Result{}, err
    }
 
    nodesF, err := ctx.Run(o.getNodes, getNodesArgs{NodeSelector: args.NodeSelector})
    if err != nil {
        return Result{}, err
    }
    var nodes []k8s.NodeInfo
    if err := nodesF.Await(&nodes); err != nil {
        return Result{}, err
    }
 
    for _, node := range nodes {
        res, err := o.drainNode(ctx, node.Name, args.Options)
        if err != nil {
            return Result{}, err
        }
        // ...
    }
    // ...
}

drainNode is itself a thin ctx.Run wrapper around drainSingleNode:

// from drain/orchestrator.go:256-266
func (o *Orchestrator) drainNode(ctx *resonate.Context, nodeName string, opts Options) (NodeResult, error) {
    f, err := ctx.Run(o.drainSingleNode, nodeArgs{NodeName: nodeName, Options: opts})
    if err != nil {
        return NodeResult{}, err
    }
    var res NodeResult
    if err := f.Await(&res); err != nil {
        return NodeResult{}, err
    }
    return res, nil
}

drainSingleNode is a plain Go function that calls Kubernetes APIs with context.Background(). The only thing that makes it durable is being invoked via ctx.Run.

Resonate: operator approval gate (drain/orchestrator.go + cmd/gateway/main.go)

When a drain is blocked and force mode is off, the workflow creates a latent promise and parks on it. One call replaces the signal channel definition, the background goroutine, the shared boolean flag, and the condition function:

// from drain/orchestrator.go:179-228
decisionF, err := ctx.Promise()
if err != nil {
    return Result{}, err
}
log.Printf("[drain] node %s blocked — resolve promise %q with skip|retry|abort|force",
    node.Name, decisionF.ID())
 
var decision Decision
if err := decisionF.Await(&decision); err != nil {
    return Result{}, err
}
 
switch decision {
case DecisionAbort:
    // return early with failed status
case DecisionRetry:
    retry, err := o.drainNode(ctx, node.Name, args.Options)
    // ...
case DecisionForce:
    forced := args.Options
    forced.Force = true
    res, err := o.drainNode(ctx, node.Name, forced)
    // ...
case DecisionSkip:
    // leave the failed result, continue to the next node
}

The gateway settles the promise from outside the workflow:

// from cmd/gateway/main.go:249-253
rec, err := gw.r.Sender().PromiseSettle(ctx, resonate.PromiseSettleReq{
    ID:    promiseID,
    State: resonate.SettleStateResolved,
    Value: value,
})

The gateway starts operations with r.RPC and uses a distinct group ("drain-gateway") so the Resonate Server never routes workflow dispatches to it:

// from cmd/gateway/main.go:43-54
r, err := resonate.New(resonate.Config{
    Network: httpnet.NewHTTP(*url, httpnet.HTTPOptions{
        PID:   pid,
        Group: drain.GatewayGroup, // distinct group — never receives workflow dispatches
    }),
})
// ...
if _, err := gw.r.RPC(ctx, operationID, drain.FuncName, args, resonate.RPCOptions{
    Target: gw.target, // "poll://any@default"
}); err != nil { ... }

The worker joins the opposite group and raises the task TTL so long-running drain steps don't expire their lease:

// from cmd/worker/main.go:33-43
r, err := resonate.New(resonate.Config{
    Network: httpnet.NewHTTP(*url, httpnet.HTTPOptions{
        PID:   pid,
        Group: drain.WorkerGroup,
    }),
    TTL: 10 * time.Minute,
})

What maps to what

Porting it

Checkpointed steps (from saga):

1. Remove workflow.ActivityOptions and workflow.WithActivityOptions. Per-call options move to resonate.RunOpts{Timeout: d} as an optional variadic arg to ctx.Run.
2. Replace each workflow.ExecuteActivity(ctx, Fn, args).Get(ctx, &out) with:

   f, err := ctx.Run(fn, args)
   if err != nil { return ..., err }
   if err := f.Await(&out); err != nil { return ..., err }

   Always check err before calling f.Await — ctx.Run returns a nil *Future on error, and calling Await on nil panics.
3. There is no RegisterActivity equivalent. ctx.Run accepts functions in four forms: no args, context only, args only, or context plus args. Pass method values, closures, or package-level functions — all work.
4. Compensation defers are ordinary Go code. Replace the activity call inside the defer the same way as step 2; the defer mechanics are unchanged.
5. Remove RegisterWorkflow. Register once with resonate.Register(r, "name", fn). Handle both return values — a bare call discarding both compiles but silently drops the error and the *RegisteredFunc handle.

Operator approval gate (from await-signals):

6. Remove the signal channel definitions, the background workflow.Go goroutine, the shared boolean fields, and the workflow.Await / workflow.AwaitWithTimeout condition functions. Replace the whole group with:

   decisionF, err := ctx.Promise()
   log.Println("resolve:", decisionF.ID())
   var out MyType
   if err := decisionF.Await(&out); err != nil { ... }

7. In the external client (the gateway), replace client.SignalWorkflow with r.Sender().PromiseSettle(ctx, resonate.PromiseSettleReq{ID: promiseID, State: resonate.SettleStateResolved, Value: value}). See Notes & coverage for how to build value correctly.
8. The external client must use a distinct Resonate group from the worker. If the gateway shared the worker's group, the server would round-robin workflow dispatches to it and the gateway would drop them ("function not registered"). Set the group in httpnet.HTTPOptions{Group: "drain-gateway"}.

Worker TTL:

9. Raise Config.TTL if individual steps can take longer than the default 60-second task lease. This example uses TTL: 10 * time.Minute. The worker still releases the lease the moment it parks on a human-decision promise.

What's actually different

No workflow/activity distinction. Temporal marks the boundary between the deterministic sandbox (workflow.Context) and free I/O (context.Context). In Resonate the boundary is structural: the workflow function receives *resonate.Context and routes durable I/O through it; the step function (drainSingleNode) is a plain Go function that calls context.Background() freely for Kubernetes APIs. The SDK does not enforce determinism inside the step — keep non-idempotent I/O inside ctx.Run.

Promise ID is the idempotency key. Temporal threads a Workflow ID (stable, caller-supplied) and a Run ID (per attempt). Resonate's single promise ID plays the role of the Workflow ID — it is the stable identity of the execution. Submitting r.RPC with the same ID again is idempotent: the server returns the existing promise without dispatching a second execution.

Groups replace task queues. A Temporal task queue determines which workers pick up which work. Resonate groups do the same job. Target: "poll://any@default" routes the dispatch to any worker in the "default" group. The gateway uses "drain-gateway" to opt out of receiving dispatches.

Signals vs. durable promises. A Temporal signal is a named channel attached to a running workflow instance; delivering it requires the workflow ID (and optionally the run ID) plus a signal name. The server durably records the signal in the event history the moment it is sent — no worker needs to be online at that instant. A latent Resonate promise is a first-class server-side object: it can be settled before the workflow reaches Await — the value is durably stored and survives crashes and replays — though the ctx.Promise() call must execute before any settlement takes effect. There is no signal-name registry, no goroutine, and no condition function.

Replay model. Both systems re-execute the workflow function body from the top on resume. In Temporal the event history is authoritative; in Resonate, the settled child promises are. The practical rule is identical: completed work is never repeated, and side effects outside the checkpointed boundary re-execute. Keep non-idempotent I/O inside ctx.Run.

Notes & coverage

Value encoding for PromiseSettle. This is the roughest edge in the Go SDK today. resonate.NewValue(v) stores raw JSON without the base64 wrapping the SDK codec expects when it decodes the value inside f.Await. The gateway works around this with a manual encodeSettleValue helper: JSON → base64 → quoted JSON string, assigned to resonate.Value{Data: json.RawMessage(quoted)} (see cmd/gateway/main.go:281-292). A high-level r.Promises().Resolve(id, value) that folds encoding and the settle RPC into one call is tracked in resonate-sdk-go#28. Until that lands, follow the pattern in cmd/gateway/main.go — or use the resonate promises resolve CLI for ad-hoc resolution.

Compensation shape. The saga sample uses defer + a compensating ExecuteActivity. This example uses a different shape: drainSingleNode returns a structured NodeResult (never a Go error) when a PDB blocks, and the workflow branches into the approval gate. When the operator chooses abort the workflow returns early; retry and force re-invoke drainNode for the same node via a new child promise ID, so the retry step is itself checkpointed independently of the original failed attempt.

ctx.Run vs. r.RPC. This codebase uses two dispatch mechanisms. ctx.Run (used inside DrainAllNodes) executes a local Go function in the same process — the step runs on the worker that owns the workflow and the result is recorded on a child durable promise. r.RPC (used in the gateway's /drain handler) dispatches to a named registered function on a remote worker group; it is the top-level entry point that starts a new workflow execution. ctx.RPC also exists in the SDK for in-workflow cross-worker dispatch, but is not used in this example.

This brief is a composite. There is no single Temporal sample that corresponds 1:1 to example-node-drain-orchestrator-go. The checkpointed-steps shape comes from saga; the external-event gate comes from await-signals. If you only need one of the two patterns, the sibling briefs cover each independently.

Sources

• Example repo: github.com/resonatehq-examples/example-node-drain-orchestrator-go
• Temporal saga sample: github.com/temporalio/samples-go/saga
• Temporal await-signals sample: github.com/temporalio/samples-go/await-signals
• Concept-level guide, all SDKs: docs.resonatehq.io/evaluate/coming-from/temporal