Skip to main content

Monitoring OPAL

There are multiple ways you can monitor your OPAL deployment:

  • Logs - Using the structured logs outputted to stderr by both the OPAL-servers and OPAL-clients
  • Health-checks - OPAL exposes HTTP health check endpoints (See below)
  • Callbacks - Using the callback webhooks feature - having OPAL-clients report their updates
  • Statistics - Using the built-in statistics feature in OPAL (See below)
  • DataDog APM - OPAL supports DataDog APM tracing (requires ddtrace v4.8.2+; OPAL itself requires Python 3.10+)

Health checks​

OPAL Server​

opal-server exposes http health check endpoints on / & /healthcheck.
Currently it returns 200 OK as long as server is up.

OPAL Client​

opal-client exposes 2 types of http health checks:

  • Readiness - available on /ready.
    Returns 200 OK if the client have loaded policy & data to OPA at least once (from either server, or local backup file), otherwise 503 Unavailable is returned.

  • Liveness - available on /, /healthcheck & /healthy.
    Returns 200 OK if the last attempts to load policy & data to OPA were successful, otherwise 503 Unavailable is returned

Notice: if you don't except your opal-client to load any data into OPA, set OPAL_DATA_UPDATER_ENABLED: False, so opal-client could report being healthy.

You can also configure opal-client to store dynamic health status as a document in OPA, Learn more here

OPAL Statistics​

By enabling OPAL_STATISTICS_ENABLED=true (on both servers and clients), the OPAL-Server would start maintaining a unified state of all the clients and which topics they've subscribed to. The state can then be retrieved as a JSON object by calling the /statistics api route on the server

Code Reference:​

DataDog APM Monitoring​

OPAL supports DataDog APM (Application Performance Monitoring) for observing your OPAL deployment's performance and traces.

Requirements​

  • ddtrace v4.8.2 or higher, below 5 - 4.8.2 is the first release with the fix for CVE-2026-50271 (GHSA-mw54-j2v2-42hr)
  • Python 3.10 or higher - OPAL's own floor (ddtrace v4 itself supports 3.9+)

Configuration​

To enable DataDog APM monitoring:

  1. Set the ENABLE_DATADOG_APM=true environment variable for both OPAL server and client
  2. (Optional) To disable APM when needed, set DD_TRACE_ENABLED=false

Breaking Changes in ddtrace v3.0.0​

The following changes were made to support ddtrace v3.0.0:

  • The ddtrace.filters.TraceFilter import has been moved to ddtrace.trace.TraceFilter
  • Programmatic APM disabling via tracer.configure(enabled=False) is no longer supported
  • APM should now be disabled using the DD_TRACE_ENABLED=false environment variable

For more information about ddtrace v3.0.0 changes, see the official migration guide.

ddtrace v4​

ddtrace v4 traces every websocket message by default; v3 did not. OPAL's pub/sub runs over websockets, so that would turn each published update into spans. OPAL keeps the v3 behaviour: the Docker images set DD_TRACE_WEBSOCKET_MESSAGES_ENABLED=false, and configure_apm applies the same default to the ASGI, Starlette and FastAPI integrations when neither DD_TRACE_WEBSOCKET_MESSAGES_ENABLED nor the older DD_ASGI_TRACE_WEBSOCKET is set, including under ddtrace-run. Set DD_TRACE_WEBSOCKET_MESSAGES_ENABLED=true to trace websocket messages; in the Docker images it is the only variable that does, since the image's own setting takes precedence over the older name.

OPAL uses none of the APIs v4 removed (Pin, Span.set_tag_str, ddtrace.settings, the opentracer package and others); its only v4-specific change is the websocket default above. If you import ddtrace yourself next to OPAL, for example from a custom fetch provider, check your own code against the v4.0.0 release notes.

OPAL Client tracker (EAP)​

Alternative implementation for the Statistics feature, OPAL-Server tracks all OPAL-clients connected through websocket. Gathered information includes connection details (client's source host and port), connection time, and subscribed topics. Available through /pubsub_client_info api route on the server.

Caveats:​

  • When UVICORN_NUM_WORKERS > 1, retrieved information would only include clients connected to the replying server process.
  • The values of query parameters with auth-like names (e.g. token, api_key, password) in a client's websocket URL are reported as ***.
  • This is an early access feature and is likely to change. Backward compatibility is not garaunteed.