A five-hour import that fails at hour four does not need to become a five-hour restart. Designing for recovery means the pipeline resumes from the failure point, not from row zero. That design is a small handful of patterns applied consistently. Skipping them turns every failure into a full re-run and every full re-run into a scheduling nightmare. The site's Bulk import time estimator factors resumability into ETA calculations. For the wider FHIR framing, the FHIR primer hub on this site has more.

The Failure Modes

• Worker crashes mid-batch
• Network partition to the server
• Server-side rate limit throttles
• Bad data in a specific batch
• Auth token expiration
• Whole-region outage

Each has a different recovery path. Design for the common cases; escalate the rare ones.

Idempotent Writes

Every write should be safe to repeat:

• Use INSERT ... ON CONFLICT DO UPDATE semantics
• Key on resourceType + id to detect duplicates
• Emit a "already-loaded" counter alongside the "loaded" counter

Idempotency is the foundation of every recovery pattern. Without it, restart produces duplicates.

For the parallelization side, parallelizing an import without corrupting references is the entry.

Checkpointing

Every N successful writes, checkpoint:

• Last successful row index
• Fleet-wide throughput
• Retry queue state

On failure, resume from the last checkpoint. On success, discard the checkpoint.

Checkpoint frequency is a trade-off: too often adds overhead, too rarely loses progress.

• Every 10,000 rows for high-throughput imports
• Every 60 seconds for time-based checkpointing

The Retry Queue

Failed writes go into a retry queue:

• Categorize by failure type
• Retry with backoff for transient failures
• Skip and log for structural failures
• Alert for auth failures

The queue drains as the import progresses. Successful retries reduce the queue; new failures add to it.

For the monitoring side, monitoring an in-flight import for the numbers that matter is the entry.

Row-Level Progress

Every row processed should record:

• Attempted
• Succeeded
• Failed with category
• Final disposition

That gives the operator a per-row picture on demand. Without it, "which rows didn't land" becomes a manual investigation.

The Partial-Success Report

At end of import:

• Total rows
• Successful writes
• Failed writes with categories and counts
• Rows requiring manual review

That report is what tells the operator what to do next. It also feeds the next import's estimate.

Recovery Playbook

Every operator running an import should have a written playbook:

• "The import stalled — do X"
• "The retry rate spiked — do Y"
• "A specific row failed repeatedly — do Z"

The playbook lives in version control next to the import code. Every run refines it.

Rollback vs Roll-Forward

• Rollback — undo everything, start clean
• Roll-forward — accept what landed, fix the rest

Rollback is safer but expensive. Roll-forward is faster but requires understanding what landed.

For most imports, roll-forward with idempotent retries wins. For imports with side effects (notifications, downstream indices), rollback may be needed.

For the schema-change side, designing a repeatable import that survives schema changes is the entry.

The Blast Radius

Every import has a blast radius:

• Just this table — small
• This table + indices + downstream views — medium
• This table + external notifications sent — large

Design imports to have a small blast radius. Do not send notifications from an import in flight; queue them for a post-import processor that runs after verification.

The Dry Run

Before recovery on production, dry-run the recovery on staging:

• Same command, same data
• Verify the recovery path works
• Verify no accidental double-processing

Dry-run catches recovery bugs that would otherwise show up in production.

The Short Version

Idempotent writes. Checkpoints. Retry queue. Row-level progress. Partial-success reports. Playbook. Roll-forward as default. Small blast radius. Dry-run before production. That is the recovery design that saves the 3 AM restart.

Sources

• HTML - HTML, HL7 FHIR - Transaction and Batch interactions