Q074. How do you implement graceful shutdown and task supervision?

Short Answer

Graceful shutdown has three phases: detect a trigger, notify owned tasks to stop accepting work, then wait for them to finish cleanup within a deadline. Track every important task with JoinSet, stored JoinHandles, or another supervisor so panics and errors are observed. After the grace period, abort remaining async tasks and report what failed. Do not rely on dropping handles or runtime shutdown as the normal lifecycle.

Deep Dive

A production service usually needs to:

1. stop admission of new requests;
2. broadcast cancellation to workers;
3. let in-flight operations reach documented safe points;
4. flush or close resources that require async work;
5. join tasks and classify normal exits, errors, panics, and cancellations;
6. enforce a bounded shutdown deadline.

JoinSet makes a dynamic group observable, but completed task outputs must be drained. A shutdown signal can use watch, broadcast, or a cancellation token. The signal should be idempotent because several causes may race.

Supervision also applies before shutdown. Unexpected exit of a critical task may require restarting it, degrading readiness, or terminating the process rather than silently continuing.

Internal Model

A cooperative signal wakes waiting tasks, which exit through normal async control flow and run both synchronous destructors and awaited cleanup. Joining observes task completion.

If the deadline expires, abort requests cause remaining async futures to be dropped at scheduling boundaries. Blocking closures and external operations need their own cancellation strategy.

Example

use std::time::Duration;
use tokio::sync::watch;
use tokio::task::JoinSet;

async fn worker(mut shutdown: watch::Receiver<bool>) {
    loop {
        tokio::select! {
            changed = shutdown.changed() => {
                if changed.is_err() || *shutdown.borrow() {
                    break;
                }
            }
            _ = tokio::time::sleep(Duration::from_millis(1)) => {
                // Process one bounded unit of work.
            }
        }
    }
}

#[tokio::main]
async fn main() {
    let (shutdown_tx, shutdown_rx) = watch::channel(false);
    let mut tasks = JoinSet::new();
    tasks.spawn(worker(shutdown_rx));

    shutdown_tx.send(true).unwrap();

    while let Some(result) = tasks.join_next().await {
        result.unwrap();
    }
}

Common Mistakes

• Detaching important tasks and losing their errors or panics.
• Broadcasting shutdown without waiting for task completion.
• Waiting forever without a deadline or abort fallback.

Follow-ups

• What should happen when a critical background task exits unexpectedly?
• How can readiness checks participate in shutdown?
• Which cleanup operations must finish before the process exits?

Related

• Q066. How does JoinHandle behave on completion, panic, drop, and abort?
• Q069. How are futures and tasks cancelled in Rust?
• Q073. How do Tokio channels provide communication and backpressure?

References

• Tokio Documentation
• Tokio Tutorial