What does a realistic load-test plan look like for Black Friday and Cyber Week?
The load test has four stages: ramp, plateau, error injection, and ramp-down. Each targets a distinct failure class.
Ramp: 3 minutes from baseline to 20-30x peak
Three minutes is fast by standard load-testing convention. That is the point. Black Friday traffic does not arrive as a smooth linear ramp. When midnight hits Thursday or the promotional email lands at 8 AM Eastern, checkout volume spikes within minutes. A 3-minute ramp exposes two failure classes a slow ramp misses: rate-limit failures triggered by the speed of the traffic increase, and warm-up failures in provider-side connection pools or internal caches that haven't pre-allocated to peak capacity.
The target TPS is 20-30x the sustained daily baseline for the tax calculation step, consistent with the peak-to-baseline ratio documented in the TPS design reference for this category.[1] For a $40M Shopify Plus brand generating roughly 1,100 daily orders at a $100 average order value, tax calculation fires at 3-5x the order rate, putting the sustained API call rate around 0.15-0.19 calls per second. At the 20-30x peak multiplier, the load-test target is 3-5.7 calls per second at the tax endpoint.
Plateau: 10 minutes at peak TPS
The plateau is where sustained-throughput failures surface. Rate-limit failures that don't appear in the first 30 seconds of the ramp often appear after 5-7 minutes at sustained load, when per-minute rolling windows reset. Connection pool exhaustion on the client side appears in the same window. Ten minutes at sustained peak load covers two rolling-window resets for most rate-limit implementations, making it long enough to catch second-order failures that a shorter plateau hides.
Error injection: throughout the plateau
Run three injection scenarios concurrently during the plateau:
- 10% of requests with malformed payloads (invalid postal codes, missing required line-item fields, zero-quantity items)
- 5% of requests routed to a mock endpoint configured for a 5-second response delay, simulating a slow-responding destination
- 2% of requests with simulated network jitter (100-500ms packet loss delays, not just added latency)
Tools for this work: k6[2], Locust[3], and Gatling[4] all support custom traffic shaping with concurrent error injection through scenario-based scripting. k6's stages and scenarios configuration handles the ramp-plateau-ramp-down shape natively. Locust's StepLoadShape class handles the same through Python scripting. Gatling's throttle and rampUsers DSL handles the shape with JVM-based execution. Any of the three works; the choice depends on what the engineering team already runs in its CI pipeline.
Ramp-down: 2 minutes back to baseline
The ramp-down is not a formality. It tests whether the checkout flow recovers cleanly as the circuit breaker resets, connection pools drain, and the fallback rate table deactivates. A system that degrades gracefully under peak load but fails to recover cleanly afterward is still a production problem. It manifests as a slow checkout recovery window rather than a hard failure at peak, and it produces a degraded experience for the hour following the initial traffic surge.
Where to point the test: sandbox, vendor load environment, or local mock
The sandbox is the wrong target for throughput testing
Most tax provider sandboxes are shared environments with rate limits set at a fraction of production configuration, typically 10-20% of the production ceiling. Testing against the sandbox understates real production capacity. The result is a false all-clear: the sandbox passes at 3x baseline while production fails at 18x. The sandbox is the right tool for schema validation, integration smoke tests, and verifying that a new field mapping doesn't break response parsing. It is not the right tool for load testing.
Vendor load environments: production-like throughput without production data contamination
Some tax API providers offer dedicated load-test endpoints or isolated staging environments that mirror production throughput profiles without writing transaction records into the live filing ledger. When a provider offers this, it is the correct load-test target. Requests move through the same infrastructure, rate limits reflect production configuration, and response latency matches what Cyber Week traffic will see. Coordinating access typically requires a support request or account manager conversation 4-6 weeks before peak. Leaving this coordination until two weeks out is a planning failure, not a vendor failure.
Local mock: highest fidelity when built correctly
A local mock removes the vendor dependency entirely and lets the engineering team run the load test at any scale, at any time, without external coordination. The investment is in building a mock that faithfully mirrors the provider's response shapes, error distribution, and timing behavior. A mock that returns 200 in 20ms on every request is worse than useless: it conditions the client to expect behavior production will not provide, and it makes the load test an expensive exercise that produces no useful signal.
A faithful local mock captures three dimensions:
- Response shapes. The actual JSON structure of success responses, 429 responses, validation error responses, and 5xx responses, including every field the client reads and acts on.
- Error distribution. A realistic mix of response codes under sustained load, not just happy-path 200s.
- Timing behavior. Response latency that matches the production p50/p95/p99 distribution, including the long tail.
TaxCloud's calculation API exposes a single endpoint for rate lookup and calculation, which makes the mock surface predictable. The API's documented throughput profile gives the engineering team a concrete ceiling to test toward, not a number to reverse-engineer from sandbox behavior that doesn't reflect production rate-limit configuration. Building the mock against a single well-documented API surface is substantially faster than building against a multi-endpoint provider that requires separate mocks for calculation, validation, and transaction recording.
What to test beyond happy-path throughput
Happy-path throughput confirms the integration works at volume. It does not confirm the integration fails safely. Three failure classes require explicit test design.
Rate-limit behavior
The rate-limit test validates four behaviors in sequence:
- The HTTP client receives a 429 response and enters a backoff cycle rather than flooding the retry queue with immediate retries.
- The backoff uses exponential backoff with jitter,[5] not fixed intervals. Fixed intervals cause synchronized retry bursts that worsen the rate-limit condition at the provider.
- The circuit breaker opens after the configured threshold of rate-limit responses, typically 5 consecutive 429s or a 429 error rate above 20% over a 10-second window.
- The fallback rate table activates when the circuit breaker is open, orders continue to complete, and each fallback completion is flagged on the transaction record for post-event reconciliation.
To test this: configure the load test to burst above the provider's documented rate limit for a 90-second window and observe the client's response. If retries arrive without backoff, the circuit breaker never opens, and no fallback activates, the integration is not production-safe for Cyber Week.
Retry behavior
Three properties matter:
- Retries carry the original idempotency key, so the provider can detect and deduplicate a retry from a new request. An integration that generates a new idempotency key on retry will double-record the transaction if the original request succeeded silently before the client-side timeout fired.
- Retries give up after the configured cap. Three to five attempts is a reasonable ceiling for a synchronous checkout path; the combined per-retry timeout budget must not allow five retries to exceed the user-facing checkout latency budget in aggregate.
- Retries use exponential backoff with jitter. The jitter component prevents retry storms when multiple concurrent requests hit the same transient error simultaneously.
Timeout behavior
The checkout path has a latency budget, and the tax calculation step gets a defined slice of it.[6] The timeout test validates three behaviors: the synchronous tax call times out at the configured budget (not 5 seconds later when the provider finally drains a slow queue); the order proceeds with a fallback rate when the call times out; and the fallback rate gets flagged on the transaction record so post-event reconciliation can identify every order that completed with a fallback value and verify the correct tax amount.
| Failure class | What to inject | What to observe |
|---|---|---|
| Rate-limit (429) | Burst above published API limit for 90 seconds | Backoff cycle initiates; circuit breaker opens; fallback activates; orders continue completing |
| Timeout | 5% of requests routed to slow-responding mock (5s delay) | Timeout fires at configured budget; order completes with fallback; fallback flag written to transaction record |
| Network jitter | 2% of requests with 100-500ms simulated packet loss | Retry path activates; idempotency key preserved; no double-record in transaction log |
| Malformed payload | 10% of requests with invalid postal codes or missing fields | 4xx response handled without retry; error logged; checkout not blocked |
| Provider 5xx | Configured percentage of 500/503 responses | Circuit-breaker threshold logic fires; retry policy for 5xx treated separately from 429 policy |
Metrics to capture during the load test
The metrics that separate a passing test from a meaningful test are the ones that reveal failure-mode behavior, not just throughput capacity. Aggregate request count and average latency conceal more than they reveal at Cyber Week scale.
Latency at the tax-calculation step, not at the checkout boundary
Measure latency at the tax API call boundary, not at checkout completion. End-to-end checkout latency aggregates a dozen service calls. What matters for this test is the tax-calculation step's contribution specifically. Representative targets for a well-tuned integration against a well-provisioned provider: p50 at 30-50ms, p95 at 80-120ms, p99 at 150-200ms. During the plateau, any p99 above 300ms warrants investigation. It signals either a provider throughput constraint or a client-side issue such as connection pool exhaustion or a synchronous retry blocking the thread pool.
TaxCloud's native Shopify and Shopify Plus integration runs the tax calculation inline with the Shopify checkout extensibility layer, without a middleware hop. The p95 and p99 numbers from the load test translate directly to what the Shopify Plus checkout flow experiences in production, making the test results a reliable predictor of peak-day behavior.
Error rate by class, not by aggregate
Aggregate error rate obscures failure mode. A 2% error rate composed entirely of 429s handled correctly by the backoff cycle is acceptable behavior under load. A 2% error rate composed of unhandled 5xx responses surfacing in the checkout UI is not. Log errors by class: timeout, 429, 5xx, 4xx validation error, network error. Each class maps to a different remediation path, and conflating them delays diagnosis during a live incident.
The business-level gate: end-to-end checkout completion rate
All technical metrics resolve to one business number: what fraction of checkout attempts completed successfully during the plateau? A load test showing p99 latency at 400ms and a 1% fallback activation rate may still produce a 99.5% checkout completion rate, an acceptable Cyber Week outcome if the fallback is properly instrumented and flagged. A test showing p99 at 200ms but a 3% checkout abandonment rate from a misconfigured circuit-breaker reset interval fails even when the raw latency looks acceptable.
| Metric | Flag during plateau if | Why it matters |
|---|---|---|
| p50 latency (tax-calculation step) | Above 50ms | Suggests provider queue buildup earlier than expected |
| p95 latency (tax-calculation step) | Above 150ms | Checkout latency budget risk |
| p99 latency (tax-calculation step) | Above 300ms | Checkout abandonment risk at the tail |
| Error rate, timeout class | Above 0.5% | Timeout budget misconfigured or provider overloaded under sustained load |
| Error rate, rate-limit class (429) | Above 0 without corresponding backoff | Client-side rate handling not implemented |
| Circuit-breaker opens | More than 2 per 5-minute window | Breaker oscillation rather than clean open-then-recover cycle |
| Fallback rate-table activations | Above 1% of requests | Acceptable if flagged on record; unacceptable if silent |
| End-to-end checkout completion rate | Drop above 0.5% from baseline | Business-level failure regardless of how individual technical metrics appear |
The sign-off cadence from 8 weeks out to final all-clear
The load test has three gates. Each gate produces a documented decision. An undocumented "it looked fine" is not a gate result.
Gate 1: Initial baseline test, 8-10 weeks before peak
The goal is to surface architectural problems while there is still time to fix them. Rate-limit ceilings that require provider escalation take 3-6 weeks to resolve through support channels. Fallback paths that haven't been built take 2-4 weeks to implement, instrument, and retest. Circuit-breaker configurations that haven't been set require less time but must still be validated with a dedicated load cycle after the fact. Discovering these problems 10 days before Black Friday produces a crisis. Discovering them 9 weeks out produces a sprint plan.
Gate 1 must produce a documented artifact: peak TPS achieved, plateau error rate by class, p99 at peak, whether the circuit breaker opened, whether the fallback activated, and a specific list of findings with assigned owners and timelines.
Gate 2: Iteration cycles, 4-8 weeks before peak
Run a load test after each material change: client code refactor, timeout budget adjustment, fallback implementation, provider rate-limit increase confirmation, infrastructure scaling event. The minimum cadence is once per two-week sprint; the right cadence is once per significant change that could affect throughput behavior.
Each iteration cycle produces an updated result against the Gate 1 baseline: did the change improve or degrade the metric, and by how much? Document the delta. If Gate 2 reveals that a client-side change degraded p99 by 40ms, that is a regression requiring investigation before Gate 3. If it reveals that a provider rate-limit increase added 25% headroom above the 20x target, that is a confidence increase that belongs in the sign-off record.
Gate 3: Final sign-off test, 2-3 weeks before peak
The final test runs against the exact code that will be in production on Black Friday. No "we'll merge the last change after sign-off." Gate 3 locks the build and documents three things: test results against the pass criteria established at Gate 1 (not invented retrospectively at Gate 3); who ran the test, who reviewed the results, and who approved; and any known gaps between results and pass criteria, with a documented decision for each gap.
If the Gate 3 test fails, the decision tree has four branches:
- Fix the root cause within 48-72 hours and run a re-test against the same pass criteria.
- Escalate to the provider for an emergency rate-limit increase or access to the vendor load environment.
- Ship the fallback path as the primary operating mode for peak, accepting post-event reconciliation overhead in exchange for checkout safety.
- Document the known risk explicitly and brief the on-call team with a specific incident playbook: the failure mode, the detection signal, and the response procedure.
Shipping without a documented decision from one of these four branches produces an undocumented production problem on the highest-revenue day of the year.
Post-peak retrospective: closing the loop for next year
Each retrospective converts this year's production surprises into next year's test requirements. Brands that run this loop annually arrive at each Cyber Week with a test plan calibrated to their actual production behavior, not assumptions built on two-year-old traffic patterns.
Capture real peak traffic data
Pull two sources immediately after the Cyber Week window closes: the provider's calculation logs for the full peak period and the application's own observability layer (request logs, circuit-breaker state changes, fallback activation counts). Both are required. The provider's logs show what the tax engine received and how it responded. The application logs show what the client did with those responses.
Compare against the load-test prediction
Three comparisons matter. First: was the actual peak TPS higher or lower than the load-test target? If lower, the test was conservative and the ramp multiplier should be revised upward for next year. If higher, the test was insufficient and next year's plateau target must increase. Second: did any failure modes surface in production that didn't appear in the load test? If yes, what differed between test and production conditions: traffic distribution, request payload shape, provider behavior under live load, time-of-day infrastructure state? Third: did the fallback rate table activate? If yes, what fraction of orders completed with fallback rates, and did the reconciliation process successfully identify and resolve every flagged transaction?
Document and feed forward
The retrospective produces three outputs for next year's test design: an updated ramp multiplier and plateau duration based on actual peak data; a revised error injection scenario list reflecting failure modes that actually surfaced in production; and a verdict on mock fidelity, specifically whether the mock's response timing distribution matched what production delivered.
TaxCloud's reporting API is one example of a provider-held calculation log that supports this workflow: it exposes timestamped calculation logs for the full Cyber Week window, including every tax call, the calculated amounts, and fallback flags for any order that completed with a fallback rate. That log serves as the ground truth for comparing load-test prediction against production reality, the source record for reconciling every fallback-rated order, and the input that feeds next year's test design. Running the retrospective against a complete, provider-held calculation log rather than application-side approximations is what makes the year-over-year comparison reliable and the annual planning loop worth running.