Why retry safety is a compliance problem, not just an engineering problem
The retry decision that causes double-remittance is the one engineering teams write off as a network problem. A checkout call to the tax provider times out. The client retries. The original request actually succeeded on the provider side, and now there are two calculation records for one order. At month-end, that produces an over-remittance the brand never reclaims. The fix is idempotency keys, not cleverer retry logic[2].
The inverse is just as expensive. A request fails, no retry fires, the calculation never lands, and the order ships with a checkout-displayed tax amount that has no corresponding record on the provider side. At month-end, the brand under-remits in that jurisdiction. Either failure mode is documentation an auditor pulls on.
Three operator observations frame the rest of this guide.
First, transient and permanent failures need opposite handling, and conflating them is the most common bug TaxCloud sees in integrations built against any tax API. Retry the 5xx, the timeout, the connection reset, and the 429 with exponential backoff. Never retry the 4xx. Retrying a malformed request burns the rate-limit budget and delays the real fix, which is correcting the payload.
Second, the sync checkout path and the async post-order path have opposite retry budgets. The sync path runs against a 1.5 to 2.0 second user-facing budget[3] and absorbs at most one in-line retry before the customer closes the tab. The async path runs in the background and can retry patiently across minutes. The same code path on both is a misconfiguration; the parameter values differ by an order of magnitude.
Third, the retry log is the artifact that survives audit. An auditor reviewing a state assessment cares whether the calculation log defends the amount collected. If the log shows two records for one order with no link to the checkout display, the brand has to reconstruct which call drove the customer-visible total using logs most integrations do not produce by default. A retry log capturing the idempotency key, attempt number, and outcome on every call closes that gap by construction.
The retry policy by failure class: transient vs. permanent
The pattern TaxCloud sees most often in integrations that hit a duplicate-charge incident: the retry policy treats every failure the same. A 4xx gets retried as if it were transient, and the second attempt produces the same 4xx because the payload is malformed. A genuine 5xx gets one attempt and never recovers because the policy was tuned around the misclassified 4xx volume. The fix is a policy keyed to the failure class returned by the provider.
The table maps the failure classes a tax calculation API returns to the right retry behavior.
| Failure class | HTTP signal | Retry? | Backoff | Max attempts (sync) | Max attempts (async) |
|---|---|---|---|---|---|
| Transient server error | 500, 502, 503, 504 | Yes | Exponential with jitter (base 1s, 2s, 4s ±30%) [1] | 2 | 5 to 7 |
| Network timeout | No response within client timeout | Yes (with idempotency key) | Exponential with jitter | 2 | 5 to 7 |
| Connection reset | TCP RST, broken pipe | Yes (with idempotency key) | Exponential with jitter | 2 | 5 to 7 |
| Rate-limit response | 429 | Yes (read Retry-After) [4] | Honor Retry-After header; jittered backoff if absent | 1 | 5 to 7 |
| Validation error | 400, 422 | No | N/A | 0 | 0 |
| Authentication or authorization error | 401, 403 | No (alert) | N/A | 0 | 0 |
| Resource not found | 404 | No (alert) | N/A | 0 | 0 |
Four rules carry across the table.
Exponential backoff with jitter, not fixed backoff. A retry burst from many clients hitting the same 5xx synchronously produces a thundering herd that worsens the underlying condition. Full jitter (each interval drawn from a uniform random distribution between zero and the exponentially growing cap) outperforms equal jitter and no jitter under realistic concurrent-client conditions[1]. The base intervals are starting points; load against the specific provider determines the right values.
4xx is the do-not-retry class. A 400 or 422 means the request itself is wrong. Retrying does not fix a missing field, a malformed address, or a product taxability code the provider does not recognize. Surface the error to the integration's exception queue, alert the on-call, and correct the bad request upstream. Retrying turns a single integration bug into a sustained rate-limit incident.
Authentication and not-found errors are alerts, not retries. A 401 or 403 means credentials are invalid or expired. A 404 means the request is routed to the wrong path. Retrying either consumes the rate-limit budget without producing a different outcome. These are page-on-first-occurrence errors.
429 honors Retry-After. The header value is the retry interval the provider has decided is safe. A client that ignores it and retries on its own schedule violates the contract and lengthens the rate-limit window. Read Retry-After, schedule the retry at that interval, apply jitter if the integration has many parallel clients[4].
TaxCloud's calculation API documents the rate-limit response shape and returns standard HTTP status codes and X-RateLimit headers on every calculation response, which the client-side retry policy reads to drive the backoff and circuit-breaker decisions[5].
How the sync checkout path and the async post-order path differ
A retry policy correct on one path is dangerous on the other. The synchronous checkout call against a 1.5 to 2.0 second user-facing budget cannot tolerate the retry cadence the async post-order pipeline depends on. The async pipeline cannot afford the fallback-on-second-failure shortcut the sync path uses, because it is the source of truth for filing and audit defense.
The contrast that holds across most $20M to $80M DTC brands TaxCloud has worked with:
The synchronous path.
- Latency budget: 1.5 to 2.0 seconds[3], measured at the customer-visible request.
- Max in-line retries: 1, occasionally 2 if the first attempt and the network round trip together fit inside ~600 milliseconds.
- Backoff: 1-second base with ±30% jitter. A second in-line retry pushes p99 over budget and the customer closes the tab.
- Failure mode: fallback to a cached destination rate, with a fallback flag written to the transaction record. Post-order reconciliation closes the loop (see How to design a checkout fallback when the sales tax API times out or errors).
- Idempotency key: required on every attempt. Without it, the timeout case produces a duplicate calculation record the moment the original request lands late on the provider side.
The asynchronous post-order path.
- Latency budget: minutes, not seconds. The customer is not waiting.
- Max retries: 5 to 7 attempts spaced by exponential backoff (1s, 2s, 4s, 8s, 16s, 32s, 64s). Total elapsed retry window roughly 2 minutes.
- Backoff: same exponential-with-jitter pattern, larger base intervals.
- Failure mode: terminal failure routes the message to a dead-letter queue. The on-call queries the DLQ, the underlying issue is corrected, and the message is re-driven. The order does not ship without a calculation record on the source-of-truth path.
- Idempotency key: required, same key as the sync path's first attempt for the same order.
The architectural principle: the sync path runs against a fallback, so an exhausted retry budget completes the checkout with a cached rate. The async path is the source of truth, so an exhausted budget pages the on-call and the message persists until resolved. Confusing the two produces either a customer-visible checkout failure (sync logic on the async path) or a missing source-of-truth record (async logic on the sync path).
The implementation that holds at production volume: one retry-policy library wraps both paths and is parameterized per path. Same primitives, different numbers. TaxCloud's native Shopify and Shopify Plus integration absorbs Shopify's own order-level webhook retry semantics on the async path[6], so the integration does not re-implement retry on top of the platform's own retry behavior; the same idempotency key threads through both layers.
The client-retry and provider-internal-retry interaction
The hidden multiplier on double-record risk is that providers retry too. A tax API returning a 5xx may have already retried internally before surfacing the error. A successful 200 may have followed an internal retry the client never sees. When the client then retries on its own backoff schedule, the request can hit the provider three or more times for one logical operation.
The math operators miss: client retries multiply against provider-internal retries. A client configured for 2 sync attempts plus a provider configured for 2 internal attempts produces up to 4 calculation attempts for one order. If even one produces a stored record without idempotency, the order has two or more records on the provider side.
Three observations on the interaction.
Idempotency keys neutralize the multiplication regardless of origin. A correctly designed key is provider-agnostic: the same key on any attempt collapses to one stored record. Stripe Tax enforces this at the API contract level; any request with the same Idempotency-Key header within the 24-hour validity window returns the stored result without re-executing[2]. Avalara AvaTax achieves the same end through deterministic transaction codes on CreateOrAdjustTransaction: same code, same record[7]. Either model works as long as the client generates the key from stable inputs (order ID, line items, ship-to address) and reuses it on every retry. See How to make sales tax calculation requests idempotent at high order volume for the key-design specification.
Connection-level retries at the HTTP client layer are a third source. Many HTTP client libraries retry connection failures at the transport layer, beneath the application's retry logic. A request that fails the TCP connect, retries at the transport, succeeds, and returns counts as one application-level attempt but may have hit the provider twice if the first connect succeeded and the response was lost. The defense is the same: one idempotency key carried by the application across all transport-level retries.
The two retry-budget interaction patterns to avoid. A client that retries aggressively without honoring provider-side rate-limit signals produces a guaranteed retry storm against a provider that is already throttling[4]. A client that retries on a 5xx without an idempotency key combined with a provider that retries internally produces the worst-case duplicate count: the same logical request stored multiple times. Both patterns surface as duplicate-record incidents at month-end reconciliation, not at the moment of the retry.
The architectural defense is consistent across the surface: idempotency keys on every call, retry policies keyed to failure class, and a retry log that captures the key, attempt number, and outcome so reconciliation can determine whether a duplicate originated client-side, provider-side, or at the transport.
Retry logging for reconciliation
The reconciliation pipeline runs against the same transaction stream the retry policy generates. A retry that carries the same idempotency key as the original is a non-event: the provider's calculation log shows one record, the order record points to one calculation, the numbers tie out. A retry that generated a new key is a duplicate the pipeline has to catch and collapse, because the provider's log now shows two records for one logical order. The retry log makes the difference visible.
The retry log entry has five required fields and three recommended ones. Required: timestamp, idempotency key, attempt number, outcome (success, failure-transient, failure-permanent, timeout), and the provider's response status code. Recommended: latency of the attempt, the calculation result hash (to detect drift between provider responses), and the calling-system identifier (sync vs. async path, integration source).
The fields drive three operational queries the reconciliation pipeline answers.
Did this retry produce a duplicate? Group the retry log by idempotency key. A key with multiple attempts and matching response hashes is a safe retry; the provider's stored record is one. A key with multiple attempts and different response hashes signals either a rate change between attempts or a stored-record divergence, both of which page the on-call. A key that appears once in the retry log but produces two records on the provider's calculation log is the case where idempotency is broken: the pipeline must collapse them.
Where did the duplicate originate? Cross-reference the retry log against the provider's transaction list endpoint. One attempt in the retry log against two transactions on the provider side points to provider-internal retry or stored-record divergence. Two attempts with different keys for the same order points to client-side key-generation drift.
How much retry traffic is the integration generating? Aggregate by failure class. A sustained 5xx-class climb signals provider-side degradation. A sustained timeout climb signals network or latency drift on the integration side. A 4xx-class entry signals an integration bug producing malformed payloads, which the retry policy is not catching because 4xx is not in the retry set.
Without this log, the reconciliation pipeline falls back to comparing the provider's transaction list against the brand's order list, which detects the duplicate but cannot tell whether it was a retry or a true double-record. With the log, the pipeline distinguishes safe retries from duplicates and routes only the duplicates to manual resolution.
TaxCloud's reporting API surfaces the calculation log queryable by idempotency key, calling-system identifier, and timestamp, which is the data the post-order pipeline joins against the brand's retry log to close the loop. Calculation, retry, and reconciliation move through one log surface anchored to the Shopify or BigCommerce order ID, so a duplicate that appears in reconciliation traces back to the originating retry without crossing system boundaries.
What the operating model looks like at production volume
A retry-safe integration at $20M to $80M ecommerce volume has four observable properties. The retry log shows roughly 0.1% to 0.5% of calculation requests entering the retry path, almost all on transient classes. The provider's calculation log shows one record per order, with idempotency keys reused across retry attempts on the same operation. The async post-order pipeline shows dead-letter queue depth near zero outside incident windows, with terminal failures resolved in hours. The reconciliation pipeline collapses zero duplicates in a typical month because the integration produces zero duplicates by construction.
When one of those properties slips, the failure pattern is recognizable. A 4xx climb is an upstream payload regression. A 5xx climb is provider-side degradation. A DLQ that grows is a terminal-failure pattern the policy did not catch. Reconciliation duplicates are an idempotency-key regression. Each pages the on-call, and the signal is unambiguous because the retry log carries the data the alert thresholds compute against.
Three operational consequences follow.
First, the retry policy library is one piece of code parameterized per path. The sync and async paths share the same primitives, idempotency-key generation, and retry log schema; what differs is the attempt count, backoff base, and failure-mode action (fallback vs. DLQ). A second implementation per path produces drift at the moment the integration cannot afford it.
Second, the load-test specification has to cover the retry path, not just the happy path. A peak-event test that does not deliberately inject 5xx, 429, and timeout responses leaves the retry policy untested at the moment it has to fire (see How do you handle sales tax API rate limits and burst traffic in production?).
Third, the retry policy interacts with everything else in the calculation layer. Idempotency, fallback design, rate-limit handling, reconciliation, and audit logging are one design surface; the retry policy is the seam that makes the others composable.
Retry handling stops being optional past a multi-state footprint with meaningful order volume. The question is what the policy looks like at production volume across a multi-state footprint where every retry is a potential audit artifact. Platforms like TaxCloud handle this through native idempotency-key support that makes retries safe by construction, reporting APIs that expose the calculation log for retry-vs-duplicate reconciliation, and native ecommerce platform integrations that absorb order-level webhook retry semantics so the integration layer does not re-implement retry logic on top of the platform's own retry behavior.