Webhook Security Testing Beyond Signature Checks

Webhook Security Testing Beyond Signature Checks

Webhook security testing beyond signature verification means testing what your application does after the signature passes: replays, duplicate deliveries, concurrent handlers, out-of-order events, tenant mapping, secret rotation and worker authorisation. A valid signature proves who sent the bytes. It does not prove the state change those bytes trigger is safe.

Most teams get the signature check right, because the provider documentation walks them through it. The bugs live one layer down, in the handler, the queue and the worker. This guide covers that gap on inbound SaaS webhook receivers, with a failure-mode table, a test matrix that defines expected outcomes beyond HTTP status, and the evidence to keep. It is written for security and backend engineers who receive events from payment, source-control or identity providers and let those events change application state. All fixtures below are illustrative and belong in an isolated, authorised environment with fake accounts and non-financial state.

What a valid signature establishes, and what it leaves open

A valid signature establishes that someone holding the shared secret produced a MAC over the exact bytes you received. That is all. GitHub documents this as an HMAC of the payload delivered in the X-Hub-Signature-256 header, to be compared with a secure comparison function (GitHub Docs).

What the signature leaves open is most of the interesting behaviour:

• Freshness. A captured, validly signed delivery stays validly signed. Whether it is rejected later depends on a timestamp scheme, a delivery ID store, or both. Provider semantics differ, so don't assume a universal tolerance window. Read what your provider actually signs.
• Uniqueness. Providers retry and redeliver. GitHub notes that a replay attacker can re-send an intercepted delivery, and recommends the X-GitHub-Delivery header as a unique identifier; redelivered requests retain that ID (GitHub Docs). Treat that header as unsigned metadata unless your provider says it is covered by the signature. It is a dedupe key, not proof of authenticity.
• Order. Events may arrive in a different order from the one they were generated in, and duplicates can occur.
• Meaning. The signature says nothing about which of your tenants the event belongs to, or whether the action it requests is something the downstream worker should perform.

So the testing question changes. It is no longer "does a bad signature get rejected?" It is "for every validly signed input, including ones the sender never meant to send twice, late or in that order, is the resulting state change one we are happy to own?"

Failure modes at a glance: signature check versus the extra test

The table maps each failure mode to what a signature check gives you, what it does not, and the additional test that closes the gap. Use it as the scoping document for a webhook test plan.

The rest of this guide walks the rows in the order you would test them.

Test the ingress boundary: raw body, invalid signatures and fail-closed behaviour

Start with the boundary, because every other test assumes it holds. The goal is a receiver that fails closed: any request that does not authenticate is rejected before it reaches a queue, a database or a business handler.

Raw body handling. Frameworks love to parse JSON before your code sees it. If verification runs against a re-serialised object rather than the original bytes, you get one of two bugs: legitimate events fail (an outage), or the team "fixes" it by loosening verification (a hole). Signature verification relies on checking the payload exactly as sent. Test that your receiver does the same by altering whitespace or key order in a signed body and confirming rejection.

Invalid and missing signatures. Send requests with no signature header, an empty one, a malformed one and a well-formed signature from the wrong secret. The expected outcome is rejection with a non-2xx response. The evidence matters more than the status code: no job enqueued, no row written, no outbound call made. A 401 that still enqueued the payload is a failure.

Constant-time comparison. GitHub's guidance calls for a secure comparison function rather than plain string equality (GitHub Docs). Confirm in code review that the comparison helper is used. Timing attacks over a network are hard to demonstrate, so this is a code-informed check, not a black-box one.

Check the event type before acting. GitHub recommends checking the event type and action before processing (GitHub Docs). Test that an authenticated event of an unexpected type or action is ignored cleanly, not routed to a default handler that does something.

Payload validation. A signed payload is still untrusted input. Validate it against a schema before use, and test that oversized bodies, wrong types, missing required fields and unexpected nested structures are rejected or quarantined without side effects. Values that flow into queries, templates or downstream API calls deserve the same input-handling tests as any other request field. The sender's signature sanitises nothing.

Rate limits, allowlists and mTLS. GitHub suggests allowing its published IP addresses (GitHub Docs). Treat IP allowlisting and mTLS as defence in depth, never as a replacement for signature verification. Rate limiting is worth testing for a different reason: a flood of invalid signatures should be cheap to reject and should not starve legitimate deliveries. Measure that valid events still process under invalid-traffic load.

Replay, duplicates and concurrency: test the side effect, not the status code

The most common webhook bug in review is a handler that returns 200 on a duplicate and performs the side effect again. The HTTP status looks perfect. The ledger is wrong.

Define the dedupe key before you test it. There are two different identifiers, and mixing them up causes bugs. A delivery ID identifies one attempt to send; a business-event ID identifies the thing that happened. A redelivery may keep the delivery ID (GitHub documents this for X-GitHub-Delivery) while a retried business action may produce a different one. Build the dedupe key from trusted provider, account and event context appropriate to your provider, not from a bare header value that an unrelated sender could collide with. Scope the key per integration so one tenant's event ID cannot suppress another's.

Replay test. In an isolated environment with a fake account, capture one valid delivery and submit it again unchanged. Where the scheme includes a signed timestamp, also submit it after your tolerance window. Expected outcome: the second submission is recognised as already handled. Evidence: effect counter unchanged, dedupe record shows both receipts, audit log distinguishes "duplicate, acknowledged" from "processed".

Preserve legitimate retries. Rejecting duplicates with an error can trigger the provider's retry machinery for an event you already handled. The right behaviour is usually to acknowledge the duplicate without repeating the effect. Test both: the duplicate is not re-executed, and the provider is not told something failed.

Concurrency test. Send the same event several times in parallel. A check-then-insert dedupe (read the table, find nothing, then write) has a window in which two workers both pass the check. The fix is an atomic operation, such as a unique constraint or a conditional insert inside the same transaction as the effect. Correct behaviour looks like one committed effect, with the remaining attempts observed as conflicts or duplicates. Don't call the result "exactly-once". What you can test is that the side effect is idempotent under duplicate and concurrent delivery.

What to inspect. Check the effect itself: balance, entitlement, role, record count. For a fake non-financial fixture, that might be a counter on a test subscription or a seat count. If the only assertion is on the response, the test isn't testing the risk.

Ordering, delays and crashes after acceptance

Providers don't promise your handler sees events in the order things happened. A receiver that treats each event as the new truth will, sooner or later, apply an old truth on top of a new one.

Out-of-order test. Build a small state machine fixture: a record that moves from created to active to cancelled. Deliver the cancelled event, then the older active event, both validly signed. The expected outcome depends on your design, and that's the point. Write it down before testing. Reasonable designs ignore events older than the current state's version or timestamp, or fetch current state from the provider's API instead of trusting the payload order. Evidence: the final state matches the newest real-world state, and the stale event is logged as superseded.

Delayed delivery. Hold a signed event for longer than your normal processing window and then deliver it. If the handler uses "now" rather than the event's own time when applying changes, delayed events can reopen things that were closed. Test it.

Crash after durable acceptance. Most robust receivers acknowledge quickly and process asynchronously. GitHub recommends responding within 10 seconds (GitHub Docs), which pushes work onto a queue. That creates a new boundary: the event is accepted and stored, but the effect has not happened. Kill the worker at three points: before it starts, mid-effect, and after the effect but before it records completion. Expected outcomes:

• Before start: the job is picked up again and runs once.
• Mid-effect: the transaction rolls back or the effect is resumable, with no half-applied state.
• After effect, before completion record: the retry detects the effect already happened through the same idempotency key, and does not repeat it.

Teams miss the third case because the dedupe record and the effect live in different places. If they aren't committed together, a crash between them produces either a lost event or a double effect.

Reconciliation. GitHub advises redelivering missed deliveries after downtime (GitHub Docs). Test the path where a backlog of old, validly signed events arrives all at once after an outage. Then confirm that a periodic reconciliation job can find records whose local state disagrees with the provider's and correct them. Webhooks are a notification channel. A receiver with no way to repair stale state treats them as a source of truth, which they are not.

Tenant mapping, secret rotation and worker authorisation

These three checks sit furthest from the signature and are the ones most likely to be missing from a test plan.

Tenant and account mapping. A multi-tenant receiver has to decide which of its customers an event belongs to. The mapping must come from trusted integration configuration, such as the endpoint, the secret or the stored provider account linked to a tenant, and not from an ownership claim inside the payload. A payload that says "this belongs to account X" is the sender's assertion. If your handler trusts it blindly, a sender who is legitimate for tenant A can name tenant B. Test with a fake pair of tenants: send an event signed with tenant A's secret that references tenant B's provider account or object IDs. Expected outcome: rejected or quarantined, with no write to either tenant. Evidence: the audit log records the mismatch with both identifiers. For the wider method of proving tenant boundaries hold, see tenant isolation testing; this article only covers the webhook-specific mapping step.

Secret rotation and revocation. Rotation is a security control only if the old secret actually stops working. Test it in three steps. First, rotate the secret and send a delivery signed with the old one after any overlap window you intend to allow: it should fail. Second, confirm the overlap window itself is bounded and logged, since an indefinite grace period is a standing second secret. Third, revoke an integration while events sit in the queue. Expected outcome: queued work for the revoked integration is discarded or re-authorised at execution time, not run on the strength of a check made minutes earlier. Authentication at ingress is a point-in-time decision.

Worker authorisation. Passing ingress validation doesn't mean the worker should perform the action. Ask what the worker can do when it receives a validated event. If it runs with a broad service credential and the event says "grant role admin to user 123", then the signature has become the only control between a third party and your authorisation model. Trace each event type to its effect and ask three questions. Is the actor in the event allowed to cause this effect for this tenant? Does the worker re-check current permissions and tenant ownership when it acts? Is the effect bounded to what the integration is supposed to do? Test it with a validly signed event requesting something the integration should never be able to cause. The event type check from earlier belongs here too: an allowlist of event types mapped to narrow handlers is safer than a generic dispatcher.

If a finding comes out of these tests and you decide not to fix it immediately, record it properly. Our note on accepted-risk findings covers owner, expiry and compensating control.

Evidence to keep, and how to judge a code-informed test

A webhook test that ends at "returned 200" or "returned 401" proves very little. Keep evidence that shows the state change, or the absence of one.

A test matrix with expected outcomes. Each row names the input, the application-specific expected outcome, and the evidence that confirms it:

Judging a code-informed test. If a tester or a tool says it has tested your webhook receiver with source access, ask one question: what did reading the code contribute to the verified test? A good answer is specific. The code showed the dedupe check and the insert are two separate statements, so the concurrent test targeted that window. The code showed the worker trusts an account ID from the payload, so the mismatch test used exactly that field. The code showed the event dispatcher falls through to a default handler, so the unexpected-type test hit it. A weak answer is a generic list of webhook risks with no reference to your handlers. The reverse also matters: the running system should confirm what the code suggested, so a reported issue is a demonstrated state change, not an inference. We cover that distinction in verified security findings.

Make the tests durable. Once a webhook fix lands, turn the failing case into an automated test that runs on every change to the handler. The approach is the same as for authorisation fixes: the test should fail if the fix is reverted, and assert on state, not status.

Treat the signature as step one of the test plan

A signature check is necessary and nowhere near sufficient. The risk in an inbound webhook receiver sits in what happens after authentication: whether a duplicate repeats an effect, whether a stale event overwrites a newer one, whether the payload decides which tenant gets written to, and whether a worker acts on authority nobody checked. If you are evaluating application testing for your SaaS, ask how it handles state-changing flows like these, and ask what the code contributed to each verified result. Borg provides continuous offensive security that runs code-informed application testing and verifies findings with evidence and reproduction steps. Book a Demo to see how that evidence looks on your own application.