Resilient Third-Party API Integration

Their outage should not become yours

Every product depends on someone else's API: payments, telephony, shipping, email, an AI provider, a partner's system. A third-party API integration usually works on the first day. It breaks later, on the vendor's schedule: a slow region, a rate limit you did not know about, a field that used to be a number and is now a string, a maintenance window at 2 a.m.

The goal is not to prevent those failures. It is to make sure they show up as a delayed background job and a clear alert, not as a frozen checkout or a stack trace in front of a customer.

One client per vendor

The first rule is structural: all calls to a vendor go through one client module in your codebase. Not scattered requests.post calls across views and tasks, but one place that knows the base URL, authentication, timeouts, retry policy, and how to turn the vendor's errors into your own.

That one decision makes everything else possible. When the vendor changes an endpoint, there is one place to update. When you need metrics per vendor, there is one place to measure. When a key has to be rotated, nothing else in the code knows where it lives.

Timeouts on every call

The default timeout in many HTTP libraries is "wait forever". Set an explicit connect timeout and read timeout on every call, sized to how long the caller can afford to wait. A call made while a user waits needs a short timeout and a plan for what to show when it expires. A call from a background job can wait longer.

A missing timeout is a common way a slow vendor takes down an otherwise healthy service: requests pile up waiting, workers run out, and everything else queues behind them.

Retry only what is safe to retry

Retries help with brief failures and make permanent ones worse. A sensible policy:

• Retry network errors, timeouts, 429 Too Many Requests, and 5xx responses.
• Do not retry 4xx validation errors. The same request will fail the same way.
• Back off exponentially, with jitter, so a recovering service is not hit by every client at the same moment.
• Respect Retry-After when the vendor sends it.
• Retry writes only if they are idempotent. If the vendor supports idempotency keys, send one derived from your own operation id. If it does not, check whether the first attempt went through before trying again.

A queue for anything that does not need an answer now

Most integration work does not need to happen inside the user's request: syncing a customer to a CRM, sending a confirmation, pushing an order to a fulfilment partner, processing a webhook. Put it on a queue and let workers handle it.

With Celery on RabbitMQ, retry behaviour can live in the task definition: autoretry_for for the exception types worth retrying, retry_backoff and retry_jitter for spacing, and max_retries so a permanently failing task ends up somewhere visible instead of retrying forever. Pair it with acks_late and idempotent task code, so a worker that dies mid-task can safely run it again.

Some work cannot be queued. Our voice AI assistant checks application status through the same APIs the support agents use, while the caller waits on the line. A call like that has to complete in time or fail clearly; it cannot be pushed to a background job.

Rate limits are your problem too

Vendors publish rate limits, and some enforce limits they do not publish. Keep a client-side limit below theirs, shared across all workers (a counter in Redis is the usual tool), so a burst of jobs is smoothed out on your side instead of turning into a wall of 429 responses and retries that make it worse.

Stop calling a service that is down

When a vendor is clearly down, hammering it with retries wastes resources and delays recovery. A circuit breaker stops calls for a short period after a run of failures, fails fast in the meantime, and lets a single trial request through to check whether the service is back. Interactive features then fall back to a degraded mode (cached data, "we will email you when this is ready") instead of waiting on a dead dependency.

Inbound: webhooks from the vendor

Many integrations also receive events back. The same rules apply in reverse: verify the signature, store the event, acknowledge quickly, process in a worker, and deduplicate on the vendor's event id. Our article on Stripe webhooks walks through that pattern in detail.

Know when it is broken before customers do

Measure each vendor separately: error rate, latency, retries, and the depth of its queue. Alert on changes, not only on hard failures; rising latency from a vendor can be an early warning of an outage. And reconcile: a scheduled job that compares your records with the vendor's catches the events that never arrived and the writes that silently failed.

Where this stops being right

• A prototype or an internal tool. A direct call with a timeout is fine while you are proving the idea. The full setup pays off once real users or real money depend on the integration.
• Vendors with an official SDK that already handles retries and idempotency. Use it inside your client module rather than rebuilding what it does, but keep your own timeouts, metrics, and fallbacks around it.
• Strictly synchronous flows. Some steps, such as charging a card at checkout, need an answer before the user moves on. There the work is choosing timeouts and clear failure messages, not queueing.

FAQ

How many retries are enough? For interactive calls, usually one quick retry at most. For background jobs, several retries with backoff over minutes or hours, then a dead-letter queue and an alert.

Do we need a circuit breaker for every vendor? For vendors on the critical path of user-facing features, yes. For background-only integrations, retries with backoff and a capped queue are usually enough.

What if the vendor has poor documentation? Start with a short spike against the real API to find the undocumented behaviour, record real responses as test fixtures, and estimate the full integration after that.

Our support agent works through order and returns APIs, which is exactly where these failure modes show up. For the integration work itself, see API and SDK integration.