Implementation Guides

Handling webhooks

Webhooks are a relatively modern way of sending information across two systems. Instead of a client requesting a server for updated information, webhooks allow the server to inform the client when they have updated information. While, technically speaking, this is also just a standard request and response, the critical distinction is that the originator and responder roles are reversed. That allows webhooks a few benefits that a typical client-server model doesn't.

The most common scenario for using webhooks is to avoid polling. Asynchronous actions often happen in different parts of the system or outside the system, which the server can or has to keep track of. But there is no mechanism for a client to do so. So how does the client get updated info? Usually by using a technique called "polling", asking the server at pre-decided time intervals if it has some new information. This approach has some challenges.

  • It requires resources - since continuous requests are going out

  • It requires the client to be alive or run on a cron - which means customer-side implementations, serverless architectures, etc. face challenges in implementing polling efficiently

  • It is not efficient - since the server does not promise when the updates will be available, the client would spend time and resources, and face challenging implementations, all without the guarantee of receiving an update

Webhooks solve all these problems to a large extent.

How webhooks work

Webhooks work by exposing an API endpoint that the server can call with or without authentication. The client registers this endpoint with the server, typically with a set of events the client wants to get notified about. When a relevant update is available, the server calls this API endpoint and delivers the updates per a pre-negotiated contract.

sequenceDiagram
    autonumber
    Client->>Server: Register webhook URL
    Server-->>Server: Has an update through any internal or external process
    Server->>Client: Webhook notification request
    Client->>Server: Acknowledgment

The server defines the request's format, the data sent through it, the acknowledgment response it expects, and how the client should process the information received through the webhook notifications.

How InsightIQ delivers webhooks

We at InsightIQ do a lot of asynchronous data processing that we need to tell our developers about.

Some updates, like profile data under Identity, are straightforward. We tell you the account ID for which the profile info got updated, and you can fetch the relevant profile from InsightIQ.

Some other updates, like content information under Engagement or transaction information under Income, are a bit tricky. They come in bulk - tens, hundreds, and maybe even thousands. In these cases, too, we share the account ID and content IDs of the items that have updated information, and you can fetch the relevant data from InsightIQ.

You must be wondering - why do we send IDs and not the updated information directly? This is a deliberate design choice. We could send you the profile information directly over a webhook since the data set is small, but we don't do so to maintain consistency across our webhooks. We won't be able to do it for the other data sets. Not that it is technically not possible, but the webhooks need to be fast due to the nature of their utility, and keeping the payload light is one of the ways we can ensure this.

Processing webhook data

So now, what is the best approach to get data from these IDs once you receive them via our webhook? The most straightforward process would be to hit our GET endpoints that respond to IDs. While this works, it is not the most efficient approach. The processing load on your systems would be high; it would take linear time in proportion to the number of updates, and sooner or later, you would run into rate-limiting issues.

To overcome this, we have come up with another endpoint that can get you the data for up to 100 IDs at once. Here is what the request would look like.

  • HTTP Method: POST

  • Enpoint structure: <base URL>/<resource path>/search

  • Request body structure:

    {
  "ids": [
    <array of IDs>
  ]
}

  • Response body structure:

    {
  "data": [
    <array of objects corresponding to the IDs>
  ]
}

Using this API, this is what we think a typical webhook workflow would look like for you.

  1. Receive payload from an incoming webhook.

  2. Verify the webhook signature.

  3. Return 200 OK to acknowledge that you have received the webhook.

  4. Save the entire payload to a database (or you may want to push it to a message queue if the loads are high or if it makes sense for your use case).

  5. From the database or queue, send the entire payload to the bulk retrieval APIs in a single go.

  6. Save the response from bulk retrieval API to your database.

General good practices

While these are not rules, these are a general set of good practices followed by most webhook implementations. You should keep them in mind while implementing a webhook consumer service, along with any changes or tweaks according to the specifications of the webhook provider.

  • Webhooks are delivered via POST HTTP method, so the URL exposed to receive must be able to process POST requests.

  • Webhooks are usually supposed to be asynchronous, unless specified. This results in multiple behaviors.

    • The order of webhook events is not guaranteed. You may receive them out of order, and you should build the consumer in such a way that ordering does not affect the processing of information.

    • Webhooks may be delayed. Sometimes by a few milliseconds, sometimes a few seconds, sometimes more. Hence it might be okay to build a business sensitive usecase dependent on a webhook, but not a time sensitive usecase. For that, polling might be a better idea (which usually results in hitting rate limits etc. which is the exact problem webhooks try to solve). Make a choice here that is suitable according to your business flow.

    • What this means is that you should acknowledge the receipt of a webhook as soon as you get it (usually by sending a 200 OK response) and then do all the processing of the webhook payload in a separate thread or process. You shouldn't block a webhook for payload processing, this quickly chokes up the webhook provider system.

  • Owing to the above fact, providers usually keep a shorter timeout window for webhook requests at their end, as compared to normal requests. If you do not respond with a successful acknowledgment within that window, the webhook delivery is considered to be failed. So again, it is important to acknowledge the webhook delivery as soon as it hits your system. We keep this timeout window as 5 seconds.

  • We talked about a "failed webhook delivery" in the above scenarios. Usually webhook providers have a retry mechanism where they repeat the same webhook request. The time windows and number of retries vary according to the provider policy. We retry a failed webhook 3 times - after 1 minute, after 5 minutes and after 6 hours.

  • Triggering the retry mechanism can lead to the following two scenarios.

    • If the webhook consumer system is genuinely not working as expected and resulting in repeated failures, there is no point for the provider to keep sending the same request or even other requests to the same webhook. In this case, the provider may disable the webhook and alert you through a human mode of communication about this. Once you figure out the reason for the failures and fix it, you can get the webhook re-enabled.

    • If the webhook consumer is actually working but fails to send a successful acknowledgment, the webhook requests will still go into retry mode. Now because you are still processing those requests, you will get the same data again and again. You should make sure the webhook system is idempotent - which means you should not alter the state in different ways or do a different kind of processing upon receiving the same payload multiple times.

  • Webhooks can be offered with certain security mechanisms, such as signature verification, IP allow-listing, auth based security, etc. A webhook without auth is inherently insecure, because that means anyone can send data to your system and you might process that without any checks. It is up to you to check with your provider what all do they offer, and implement those mechanisms at your end (or collaborate with the provider to get them enabled). We offer a signature verification and IP allow-listing mecahnism.

Path forward

While we currently recommend using this API in association with webhooks, we are looking to make filtering using more attributes easier via the same API in the future. You might find another use case for it too. Let us know what you are using it for and how we can help you by writing to us at [email protected].

Was this helpful?
Was this helpful?