Reliable Result Delivery
Check results are never silently dropped, even through a network interruption between a Scout and the Snitcher server:
- The Scout writes every completed check result to its own local database the instant it's produced — before it's ever sent anywhere. If the connection to Snitcher is down, the result simply waits there; nothing is lost.
- Every 5 seconds, the Scout flushes its unsent results as one hashed batch and waits up to 10 seconds for Snitcher to acknowledge. If the ack doesn't arrive (timeout, or the connection drops mid-wait), it retries with exponential backoff, capped at 60 seconds, and resets to normal cadence once an ack succeeds.
- Snitcher, in turn, collects incoming batches from all Scouts and flushes them to the database every 5 seconds; a batch is acknowledged only after it has actually been persisted (and run through normal status/alert processing) — never before.
- Results are still bounded: anything older than
SCOUT__ResultRetentionHours(default 24 hours) is evicted from the Scout's local queue regardless of whether it was ever delivered, so a long-term outage doesn't grow the Scout's local database without bound.
This design trades a small amount of latency (the 5s batching window) for a strong guarantee: a Scout that's cut off from the server for minutes or hours delivers everything it queued the moment connectivity returns, rather than dropping results during the gap.