A deep dive into how TomorrowCentral detects agent failures, checkpoints state, and resumes execution — without duplicating work.

The problem with stateless agents

Most agent frameworks treat a run as atomic: it either completes or it doesn't. When a process crashes halfway through a 40-step reasoning chain, the entire run is lost. The agent restarts from scratch, re-calls tools it already called, re-fetches data it already has. This is wasteful at best and dangerous at worst — imagine an agent that sends a notification midway through a run, crashes, restarts, and sends the notification again.

AgentState was designed to make this impossible.

How checkpointing works

Every time an agent completes a tool call, AgentState writes a checkpoint to persistent storage before the next step begins. The checkpoint includes the tool name, inputs, outputs, and the agent's full key-value state at that moment.

The write is synchronous and confirmed before execution continues. This means that if the process crashes after step 12, the checkpoint for step 12 is guaranteed to exist.

When the agent resumes (via state.resume(thread_id=...)), it reads the checkpoint log, rebuilds its prior context, and continues from step 13. Steps 1–12 are not re-executed.

Deduplication by design

Each tool call within a thread is assigned a deterministic step number. When resuming, the agent checks whether a step has already been checkpointed before executing it. This prevents re-execution even if the crash happened during the write phase of the checkpoint itself.

This approach borrows from distributed systems patterns — specifically, write-ahead logging — applied to agent execution.

What this means for you

You can run long agents confidently. A 200-step data pipeline agent that crashes at step 150 resumes at step 151. Infrastructure hiccups don't cost you hours of LLM compute. And idempotent side effects (like file writes or API calls) happen exactly once per step.