Previous post: Securing IDP-Observ’s Webhooks: Two Vendors, Two Different Auth Models
The first two posts in this series hardened IDP-Observ as a thesis project — proving the technique, then closing the gaps its own limitations list admitted to. Both times, the audience was still me, working in one repo. This post is about what changes when the audience becomes a team that has never seen your code and has no reason to trust it yet: keythread, a standalone package extracted from IDP-Observ’s correlation technique.
A technique isn’t a package
IDP-Observ’s core idea — hash a business key into a deterministic OpenTelemetry trace ID so async webhooks land in one trace — doesn’t need Jira or GitLab specifically. Any tool that fires a webhook with a business identifier in it qualifies. But the thesis code didn’t reflect that: routing was an if/elif chain keyed on source name, and adding a third tool meant editing that chain.
The first real change was making that generalization explicit instead of implicit. SourceAdapter is a typing.Protocol — three methods (verify_request, extract_correlation_key, map_event), no subclassing required:
@runtime_checkable
class SourceAdapter(Protocol):
def verify_request(self, headers: Mapping[str, str], body: bytes) -> bool: ...
def extract_correlation_key(self, payload: dict) -> str | None: ...
def map_event(self, payload: dict) -> "UnifiedEvent | None": ...
Structural typing instead of an ABC matters here for a reason bigger than style: it means an adapter someone else writes, in a package I’ve never seen, satisfies the interface just by having the right three methods — no import from my package required at the type level. GitHub became the adapter that proved this wasn’t hypothetical: unlike Jira and GitLab, it signs the whole request body (X-Hub-Signature-256) instead of comparing a static secret, and its event type lives in a header the adapter never receives. Writing it against the Protocol, without touching the Protocol, was the actual test.
The harder question: does this follow OpenTelemetry’s own rules?
Generalizing the interface answers “can a new source be added.” It doesn’t answer a different question: is this actually built the way OpenTelemetry says extensible things should be built, or does it just look extensible from the inside?
OpenTelemetry’s own docs name five concrete ways it’s designed to be extended: a custom Collector receiver, loading instrumentation libraries into an SDK, building a distribution of an SDK, a custom exporter, a custom propagator. I checked keythread against each — not from memory, against the actual code — and found a real gap and a real miscategorization risk, sitting next to two things it correctly doesn’t do at all.
The gap: OTel’s own instrumentation libraries are discovered via Python entry points — opentelemetry-bootstrap scans installed packages and auto-registers whatever matches, no PR into OTel’s own source required. keythread’s adapter registry didn’t do this. Its own docstring said so outright: “there is no entry-points/plugin-discovery mechanism in v1.” Adding a Linear or Jenkins adapter meant either forking the repo or hand-wiring registration in your own app — a real gap against a package whose whole pitch is “any team can adopt it regardless of tooling.”
The fix mirrors OTel’s own pattern directly:
ENTRY_POINT_GROUP = "keythread_adapters"
def discover_adapters() -> None:
for ep in entry_points(group=ENTRY_POINT_GROUP):
try:
adapter = ep.load()()
except Exception as e:
logger.warning(f"Skipping adapter plugin '{ep.name}': {e}")
continue
if isinstance(adapter, SourceAdapter):
register_adapter(ep.name, adapter)
A third-party package now publishes [project.entry-points.keythread_adapters] pointing at a zero-argument factory, and it’s picked up at startup — no PR required. A plugin that fails to import, or fails to construct because its secret isn’t configured, is logged and skipped rather than taking the whole app down.
The miscategorization risk: it would be easy to describe keythread’s core trick as “a custom propagator” — it does correlate context across process boundaries, after all. But an OTel Propagator specifically injects and extracts an already-existing SpanContext across a live call between two processes. Jira and GitLab webhooks carry no trace context to extract, and the two sides never call each other. What keythread actually built is a custom IdGenerator: both sides derive the same ID independently from a shared key, instead of one side handing a value to the other. Different SDK extension point, different mechanism, and worth naming precisely rather than letting “propagator” become the informal shorthand — so the docstrings say so now, explicitly, in OTel’s own terms.
Two of the five axes needed nothing at all, and saying so explicitly turned out to matter as much as fixing the gap: keythread doesn’t build a Collector receiver (it sits in front of the Collector, not inside it) and doesn’t build a custom exporter (every backend it targets already speaks OTLP). Neither is a missing feature. Writing that down is what tells the difference between a deliberate boundary and an oversight — otherwise both look identical from the outside.
Does it work
148 tests, 92% coverage, lint and format clean. The entry-point discovery got four new tests specifically for failure modes, since a plugin mechanism that only tests the happy path isn’t tested:
def test_skips_entry_point_whose_factory_raises(self, monkeypatch):
def factory():
raise RuntimeError("required secret not configured")
monkeypatch.setattr(
"keythread.adapters.registry.entry_points",
lambda *, group: [_FakeEntryPoint("misconfigured", factory=factory)],
)
discover_adapters() # must not raise
assert get_adapter("misconfigured") is None
That one matters more than it looks: a plugin ecosystem where one broken or unconfigured package takes down every adapter, built-in or not, isn’t one anyone will trust with a second package.
The tradeoff
- Not on PyPI yet. Install is source-only for now (
pip install git+...) — publishing needs a name reservation and a release workflow, tracked as its own piece of work rather than rushed. - Three adapters, not the five-plus the README invites. Jira, GitLab, GitHub are built in; Linear, Azure DevOps, Jenkins, CircleCI, Bitbucket are named as wanted but not written.
- Some design calls are still provisional, on purpose. The state manager’s TTL default, the closed
EventTypeenum, dropping IDP-Observ’sCORRELATION_STRATEGYfallback toggle — judgment calls made with the information available, flagged in the README as open to revisiting once real usage pushes back.
The next step
The generalizable part isn’t the entry-points snippet — that’s a few lines once you know the pattern exists. It’s checking a new package against the platform’s own documented extension points before assuming it’s extensible, instead of after. “Can someone add a new source” and “does this follow the framework’s own rules for adding one” turned out to have different answers, and the gap between them was sitting in a docstring that already admitted it, unread until it was time to check.
Code: github.com/NewerKey/keythread. Extracted from github.com/NewerKey/idp-observ, my B.Sc. thesis project.