Monitoring Basics - Smile CDR Documentation

Monitoring Basics

This page outlines methods for monitoring Smile CDR.

Smile CDR provides several hooks that are suitable for integration with an external monitoring system.

HTTP Endpoint Health Check

All HTTP Servers supply an HTTP listener at the path /endpoint-health that can be used to query for the current availability of the endpoint. This endpoint will return an HTTP 200 with a simple JSON response payload as long at the endpoint is enabled and operational.

This is useful for systems requiring a simple way to test whether an endpoint is available, such as load balancers that perform automated failover based on a status test. The Endpoint Health Check is very lightweight and may be called frequently without causing extra load on the server.

The Endpoint Healh Check is available on any endpoint modules that supply an HTTP server (e.g. FHIR REST Endpoint or Web Admin Console. It will typically be served at /endpoint-health, but note that it will respect the context path if one is set. So for example if the context path is set to /admin, the Endpoint Health Check will be served at /admin/endpoint-health.

The following example shows a simple HTTP query against the Endpoint Health Check.

GET /endpoint-health

Copy

The server will respond with the following response:

200 OK

{"status":"OPERATIONAL"}

Copy

If any health checks are failing on the given module, the server will return a response which includes the string NOT_OPERATIONAL as well as the IDs of any failing health checks. For example:

200 OK

{
  "status" : "NOT_OPERATIONAL",
  "failingHealthChecks" : [ "connection_pool" ]
}

Copy

By default, the endpoint will return an HTTP 200 OK regardless of whether any health checks are failing. A different status code can be returned by adjusting the Unhealthy Status Code setting.

Process Status Endpoints

Alongside the per-module Endpoint Health Check above, every Smile CDR process can serve two process-scoped endpoints on a dedicated status port. They are off by default: set node.status.port in the node configuration properties file to turn them on.

Endpoint Reports Healthy Unhealthy
/startup Whether this process has finished booting 200 503 while still starting
/readiness Whether health checks across every running module are passing 200 503 while the process is still starting, when a check is failing, when a module failed to start, or when the process is shutting down

These differ from /endpoint-health in three ways. They are process-scoped, aggregating every module in the process rather than reporting one module's own listener. They come up early in the boot sequence, before any module has started, so they can answer while the process is still starting. And they are unauthenticated and served on their own port, outside any module's HTTP server, so they do not respect a module's context path.

They are shaped for orchestrator startup and readiness probes, and Kubernetes Startup and Readiness Probes covers the startupProbe and readinessProbe configuration. Nothing ties them to Kubernetes, though. They are plain HTTP, so any monitoring tool can poll them.

Because the status port is unauthenticated by design, access control is left to the network: restrict it with a firewall rule, security group, or network policy so it is reachable only by the systems that probe it. That is also why the endpoints are off until you ask for them: upgrading Smile CDR never opens a port you did not configure.

All three settings are node configuration properties, set in the same properties file as node.id:

node.status.port=9300

Copy

Response payloads

Both endpoints return pretty-printed JSON with a Content-Type of application/json. Null fields are omitted rather than serialized as null, so a field's absence is meaningful.

/startup carries only a boot status. While the process is booting it returns STARTING with HTTP 503; once boot completes it returns HTTP 200 and:

{
  "status" : "STARTED"
}

Copy

/readiness when healthy (HTTP 200). failingHealthChecks is absent, not empty:

{
  "status" : "READY",
  "nodeId" : "Master",
  "processId" : "Algonquin"
}

Copy

nodeId is the node.id from your configuration. processId is the name this process claimed from the Cluster Manager when it joined the cluster; it is absent until that has happened, so an early /readiness call during boot can legitimately return a body with no processId.

/readiness when not ready (HTTP 503). failingHealthChecks lists every check that is currently failing, each naming the module (the id you gave it in your configuration) and the healthCheck:

{
  "status" : "NOT_READY",
  "nodeId" : "Master",
  "processId" : "Algonquin",
  "failingHealthChecks" : [ {\
    "module" : "persistence",\
    "healthCheck" : "connection_pool"\
  }, {\
    "module" : "fhir_endpoint",\
    "healthCheck" : "dependent_fhir_storage_health"\
  } ]
}

Copy

A module that failed to start is reported against its own id:

  "failingHealthChecks" : [ {\
    "module" : "subscription",\
    "healthCheck" : "failedToStart"\
  } ]

Copy

Conditions that belong to the process rather than to any one module use the reserved module id _process. A process that is still booting reports starting, and no module health checks are run until boot completes:

  "failingHealthChecks" : [ {\
    "module" : "_process",\
    "healthCheck" : "starting"\
  } ]

Copy

A process that is shutting down reports shuttingDown as soon as shutdown begins, before modules have finished stopping:

  "failingHealthChecks" : [ {\
    "module" : "_process",\
    "healthCheck" : "shuttingDown"\
  } ]

Copy

Reading a failing /readiness check

Nothing is latched: a result is reused for at most one second, far below any sensible probe interval, so a transient failure such as a brief database outage clears on its own as soon as the underlying condition resolves. A request that arrives while shutdown is under way always gets a fresh answer, so draining is never delayed by that one-second window.

Most healthCheck values are reported by a module and name one of that module's own checks. Five are reported by the process itself and use the reserved module id _process. One more, healthCheckUnreadable, is reported against the module it concerns:

healthCheck Module What it means First thing to check
connection_pool / connection_pool_write the reporting module Cannot obtain a (writable) database connection Database is up, reachable, and not read-only
http_responses the reporting module Sustained HTTP 5xx on this listener Recent errors or a downstream dependency
dependent_fhir_storage_health the reporting module The FHIR storage this endpoint depends on is unhealthy The persistence module it points at
ft_indexing_status the reporting module Full-text indexing is behind or failing Elastic/Lucene index health
transaction_log_purge, stats_heartbeat, stats_cleanup_* the reporting module A background maintenance job is not running Scheduler / database
hl7v2_mllp_listener the reporting module HL7v2 MLLP listener is down Listener port / config
sp_sync_analysis / sp_sync_sync the reporting module Search parameter synchronization is failing Search-parameter config
custom (or a module-specific name) the reporting module A check owned by a specific module or integration That module's own documentation
failedToStart the module that failed That module did not start, and will not recover unattended The module's configuration; see below
starting _process The process has not finished booting, so its modules cannot serve traffic yet Nothing. This is expected during a restart or a cold start, and /startup reports the same window
shuttingDown _process The process is shutting down Nothing. This is expected during a restart or a planned stop
healthCheckTimeout _process The health checks did not finish within node.status.health_check_timeout_seconds A module whose check is blocked, e.g. on an unresponsive database; raise the timeout if the node simply has more checks than fit in the budget
healthCheckUnavailable _process No health check thread was free to run this probe. Either a check is stuck and ignoring cancellation, or more readiness requests arrived at once than there are health check threads Preceding healthCheckTimeout entries and the server log point at a stuck check; if there are none, reduce how many systems poll /readiness at once. Restart the process if a stuck check does not clear
healthCheckError _process The checks could not be evaluated at all, most often because a module's context was closing during a restart Whether a module is restarting; the server log carries the underlying error
healthCheckUnreadable the unreadable module The module is running but could not supply its health checks, most often because it is mid-restart Whether that module is restarting; it clears on the next probe once the module is back

One root cause often produces two entries. In the NOT_READY payload above the database is unreachable, so Persistence cannot obtain a connection and the FHIR endpoint that depends on Persistence reports its upstream storage as unhealthy. Fix the database and both clear on the next probe.

The status port reports only the module id and check name, never the underlying error. For the exact error, such as the actual database message, use the authenticated Runtime Status Health Checks operation, which returns every check with its message. For a module that failed to start, the startup log has the exception.

failedToStart Unlike a failing health check, failedToStart does not clear on its own. Boot has finished, so /startup returns 200 while /readiness stays at 503 indefinitely. Every process running the same node configuration is affected, not just one.

Where you correct the configuration depends on node.propertysource, and the default is the less obvious of the two:

Runtime Health Checks

As it is running, Smile CDR maintains a number of "health checks", which are simple status monitors that monitor whether a particular component of the system is operational.

For example, health checks exist to test database availability and health, HTTP server availability, scheduled job status, etc.

These health checks may be queried by a monitoring tool using the Runtime Status Health Checks operation on the JSON Admin API.

Monitoring Basics

Operating

Monitoring Basics Externalized Metrics Smile CDR Observability Performance Tuning Troubleshooting

Externalized Metrics