Securing IDP-Observ's Webhooks: Two Vendors, Two Different Auth Models

Previous post: Measuring Lead Time When No Tool Sees the Whole Path

That post ended with an honesty list — things the thesis’s proof of concept didn’t cover. Two of them: the trace ID was plain MD5, and the webhook endpoints accepted anything. Both read like small, well-scoped fixes. Building them showed one wasn’t as small as it looked.

Two problems wearing one name

The original limitations note said, cleanly: “add HMAC signature verification on inbound webhooks.” That phrasing assumes one mechanism. There isn’t one — there are two webhook sources, and they don’t authenticate the same way.

GitLab sends a static secret back in an X-Gitlab-Token header. You set the secret once, in the webhook config; GitLab echoes it on every request; you compare strings. It’s not a signature over the payload — a signature would let you prove the body wasn’t tampered with in transit. This is simpler: proof that the sender knows a shared secret, nothing more.

Jira Cloud’s built-in webhook feature has no equivalent at all. No signing header, no secret field, nothing. If you want any verification on a plain Jira webhook, you have to build the mechanism yourself — the practical route is a Jira Automation rule (“Send web request”) that adds a custom header with a value only you and Jira know.

So “add HMAC signature verification” turned into two separate, source-specific checks — a token compare for GitLab, a custom shared-secret header for Jira — and neither of them is actually an HMAC signature. The corrected version of that limitations note, if I’d known this going in, would have said: verify each webhook source using whatever authentication mechanism it actually supports. Less punchy. More true.

What changed

Each source gets its own verifier:

def verify_gitlab_token(request: Request) -> bool:
    """Compare GitLab's `X-Gitlab-Token` header against the configured secret."""
    expected = os.getenv("GITLAB_WEBHOOK_SECRET", "gitlab_webhook_secret_change_me")
    received = request.headers.get("X-Gitlab-Token", "")
    return hmac.compare_digest(received, expected)


def verify_jira_secret(request: Request) -> bool:
    """Jira has no native webhook signing — this header is added manually
    via a Jira Automation rule that calls this endpoint."""
    expected = os.getenv("JIRA_WEBHOOK_SECRET", "jira_webhook_secret_change_me")
    received = request.headers.get("X-Webhook-Secret", "")
    return hmac.compare_digest(received, expected)

Both routers now reject with 401 before doing anything else if the check fails. hmac.compare_digest matters here — a plain == on secrets leaks timing information about how many characters matched, which is a real (if narrow) side channel.

The other limitation — MD5 for the trace ID — got a separate, unrelated fix: keyed HMAC-SHA256 instead of a plain hash.

secret = os.getenv("TRACE_ID_HMAC_SECRET", ...).encode("utf-8")
digest = hmac.new(secret, normalized_key.encode("utf-8"), hashlib.sha256).hexdigest()
return digest[:32]  # truncate to 128 bits — the width OTel expects

Plain MD5 of the Jira key means anyone who knows (or guesses) the key format can precompute every trace ID. Keying the hash with a secret means the trace ID is only derivable by whoever holds that secret — the same “unified trace” property, minus the guessability.

Does it work

Both changes have a specific, falsifiable test, not just “the endpoint returns 200 now”:

Auth: each source gets a rejects-missing / rejects-wrong / accepts-correct triplet via TestClient, hitting the real router with real headers.
Keying: the same Jira key, hashed under two different secrets, must produce two different trace IDs. If it doesn’t, the hash isn’t actually keyed — it’s decoration.

def test_trace_id_is_keyed_not_just_hashed(self, monkeypatch):
    monkeypatch.setenv("TRACE_ID_HMAC_SECRET", "secret-one")
    trace_id_secret_one = generate_trace_id("TEST-42")

    monkeypatch.setenv("TRACE_ID_HMAC_SECRET", "secret-two")
    trace_id_secret_two = generate_trace_id("TEST-42")

    assert trace_id_secret_one != trace_id_secret_two

All pre-existing tests still pass; nothing about the correlation logic itself changed — only the algorithm underneath it.

The tradeoff

This doesn’t make the system production-ready, and it introduces a smaller caveat of its own:

One shared secret per source, no rotation. If GITLAB_WEBHOOK_SECRET leaks, every webhook from that source is spoofable until you rotate it — and rotation today means a manual redeploy, not a live key-rotation path.
The Jira fix is a workaround, not a platform feature. It depends on someone configuring the Automation rule correctly and keeping the header value in sync with the middleware’s env var. If Jira ever ships native webhook signing, that should replace this.
State is still in-memory, the study is still local-only. Those limitations are unrelated to this change and still open.

The next step

The generalizable part isn’t the crypto — hmac.compare_digest and a keyed hash are both standard-library one-liners. It’s the instinct to check: when a plan says “add X” across more than one integration, does every integration actually support X the same way? Here, “add webhook signature verification” quietly assumed GitHub-style signing that neither of my two actual vendors implements. The fix took twenty minutes once I’d read each vendor’s docs instead of assuming from the first one.

Code and full write-up: github.com/NewerKey/idp-observ. Part of an ongoing series on hardening a thesis project into something closer to production.

Next: keythread: When the Only Reader Stops Being You