Connecting two systems through an API is often fairly straightforward. Keeping that connection working reliably is where things become more interesting.
It's easy to build an integration that works when both services are available, requests arrive in the expected order and nothing goes wrong. In practice, APIs time out, webhooks are delivered more than once, credentials expire and services occasionally become unavailable. None of these are particularly unusual events, yet they can cause significant problems when an integration hasn't been designed to account for them.
I think the important distinction is that an integration shouldn't simply work under ideal conditions. It should also behave predictably when those conditions change.
Start with what the integration needs to do
Before writing any code, it helps to understand what each system is responsible for and what should happen when information moves between them.
Consider an application receiving payment notifications from a billing provider. A successful payment might activate a subscription, update an invoice and send a confirmation email. Those actions sound simple enough, but there are a few questions worth answering first.
What happens if the same notification arrives twice? What if the notification arrives before another event it depends on? What if the payment provider sends the event successfully but our application fails halfway through processing it?
The answers influence how we design the integration. They also determine whether the application can recover from failures without somebody having to intervene manually.
Treat webhooks as notifications, not guarantees
Webhooks are useful because they allow one service to notify another when something changes, without the receiving application having to continually request updates.
However, webhook delivery is generally designed around at least once delivery rather than exactly once processing. Providers commonly retry events when they don't receive a successful response, meaning the same event may arrive multiple times.
For that reason, a webhook handler needs to be idempotent. Processing an event twice should have the same effect as processing it once.
A common approach is to store the unique event identifier supplied by the provider and check whether it has already been processed. The check and the work it protects need to be designed carefully: two copies of the same event can arrive at almost the same time. A database uniqueness constraint and transactional processing are more reliable than relying on a simple read-before-write check.
There is another distinction worth making here. Receiving an event doesn't necessarily mean it has been processed.
Where possible, I prefer to keep the webhook endpoint's responsibility small: validate the request, record the event durably and acknowledge it. The actual business logic can then run separately through a queue or worker. This makes the endpoint less likely to time out and gives us more control over retries and failures.
Of course, acknowledgement should only happen once the event has been safely recorded. Returning a successful response before doing that risks losing the event altogether.
Plan for retries and partial failures
Network requests fail for all sorts of reasons. Sometimes the remote service is unavailable. Sometimes it responds too slowly. Sometimes it completes the requested action but the response never reaches us.
That last case is particularly important. A timeout doesn't necessarily mean nothing happened.
When making requests to an external API, I generally consider:
- Timeouts: Every outbound request should have sensible connection and response timeouts.
- Retries: Temporary failures can be retried with exponential backoff and a degree of randomness, rather than immediately sending the same request again.
- Idempotency: Where an API supports idempotency keys, use them for operations that must not be repeated, such as creating a payment or order.
- Permanent failures: Invalid credentials, malformed requests and many validation errors need investigation or correction, not endless retries.
- Retry limits: A failing integration should not be allowed to retry indefinitely without visibility.
Retries are helpful, but they shouldn't simply move the problem elsewhere. If a third-party service is struggling, repeatedly increasing the number of requests can make matters worse.
A bounded retry strategy, followed by a failed-job queue or another recoverable state, provides a much better foundation.
Don't assume events will arrive in order
Another easy assumption to make is that events will arrive in the same order in which they occurred.
Imagine receiving a subscription cancellation notification before the event confirming that the subscription was created. If the application simply applies events in the order they arrive, it might end up with an incorrect subscription state.
Some providers include sequence numbers or timestamps that can help identify stale events. Others recommend retrieving the current resource from their API rather than treating the webhook payload as the complete source of truth.
The right approach depends on the provider and the domain. In some cases we can safely ignore an outdated event; in others we need to fetch the latest state and reconcile our own records against it.
The point is to make this a deliberate decision rather than depending on delivery order.
Security needs to be part of the design
An endpoint that accepts incoming webhooks is publicly accessible, so we need to establish that requests genuinely came from the expected provider.
Many services sign webhook payloads using a shared secret. We should verify that signature against the raw request body, using the provider's documented algorithm and a timing-safe comparison. Where supported, timestamp checks also help reduce the risk of replay attacks.
Beyond signatures, there are a few other sensible precautions. Store API credentials securely, keep permissions as narrow as possible and support secret rotation. Avoid writing tokens or sensitive customer information into logs. Validate incoming data even when its signature is correct.
A valid signature confirms that the request was signed with the expected secret; it doesn't guarantee that every field is suitable for our application to trust without checking.
Make failures visible and recoverable
One of the more frustrating integration problems is when everything appears to be working until somebody notices missing or incorrect data several days later.
A reliable integration needs to be observable. We should be able to answer fairly basic questions: when was an event received, what happened to it, how many times was it attempted and why did it fail?
I find it useful to keep a record of each event's lifecycle, with states such as received, processing, completed and failed. Logs should include a correlation or event identifier so we can follow an individual request without exposing its sensitive contents.
Metrics are useful too. A sudden increase in failed events, unusually long processing times or a backlog of unprocessed messages can indicate a problem before it becomes visible to customers.
Just as importantly, there should be a way to retry or replay failed events safely. Once processing is idempotent, recovering from an outage becomes considerably easier.
Design for the people maintaining it
It's tempting to think of integration reliability purely as a technical concern. In reality, it also affects the people responsible for running the application.
If an integration stops working, can somebody identify the cause? Can they tell which records were affected? Can they recover without editing database tables or asking a developer to run commands on a server?
These are worth considering early on, even if the first version only provides a simple administration view of incoming events and their processing status.
There is no need to build an elaborate integration platform for every small application. The complexity should be proportionate to the consequences of failure. But the underlying principles remain much the same.
Reliability is part of the feature
An API integration isn't finished simply because it successfully sends or receives a request.
It needs to handle duplicate messages, temporary outages, unexpected ordering and incomplete operations. It needs appropriate security, useful monitoring and a sensible path to recovery.
Building these considerations into the original design usually takes less effort than untangling them once an integration is being relied upon in production.
Ultimately, the aim is quite straightforward: systems should be able to exchange information without us having to worry about whether a temporary failure will leave them in an inconsistent state.