# Monitoring Basics

This page outlines methods for monitoring Smile CDR.

Smile CDR provides several hooks that are suitable for integration with an external monitoring system.

# HTTP Endpoint Health Check

All HTTP Servers supply an HTTP listener at the path `/endpoint-health` that can be used to query for the current availability of the endpoint. This endpoint will return an HTTP 200 with a simple JSON response payload as long at the endpoint is enabled and operational.

This is useful for systems requiring a simple way to test whether an endpoint is available, such as load balancers that perform automated failover based on a status test. The Endpoint Health Check is very lightweight and may be called frequently without causing extra load on the server.

The Endpoint Healh Check is available on any endpoint modules that supply an HTTP server (e.g. _FHIR REST Endpoint_ or _Web Admin Console_. It will typically be served at `/endpoint-health`, but note that it will respect the [context path](/content/docs/product_configuration/http_server_setup.html#context-path/index.html) if one is set. So for example if the context path is set to `/admin`, the Endpoint Health Check will be served at `/admin/endpoint-health`.

The following example shows a simple HTTP query against the Endpoint Health Check.

```http
GET /endpoint-health
```

Copy

The server will respond with the following response:

```http
200 OK

{"status":"OPERATIONAL"}
```

Copy

If any health checks are failing on the given module, the server will return a response which includes the string `NOT_OPERATIONAL` as well as the IDs of any failing health checks. For example:

```http
200 OK

{
  "status" : "NOT_OPERATIONAL",
  "failingHealthChecks" : [ "connection_pool" ]
}
```

Copy

By default, the endpoint will return an HTTP 200 OK regardless of whether any health checks are failing. A different status code can be returned by adjusting the [Unhealthy Status Code](/content/docs/configuration_categories/http_listener.html#property-unhealthy-status-code/index.html) setting.

# Process Status Endpoints

Alongside the per-module Endpoint Health Check above, every Smile CDR process can serve two **process-scoped** endpoints on a dedicated **status port**. They are **off by default**: set
`node.status.port` in the node configuration properties file to turn them on.

| Endpoint | Reports | Healthy | Unhealthy |
| --- | --- | --- | --- |
| `/startup` | Whether this process has finished booting | `200` | `503` while still starting |
| `/readiness` | Whether health checks across every running module are passing | `200` | `503` while the process is still starting, when a check is failing, when a module failed to start, or when the process is shutting down |

These differ from `/endpoint-health` in three ways. They are **process-scoped**, aggregating every module in the process rather than reporting one module's own listener. They come up **early in the boot sequence**, before any module has started, so they can answer while the process is still starting. And they are **unauthenticated and served on their own port**, outside any module's HTTP server, so they do not respect a module's context path.

They are shaped for orchestrator startup and readiness probes, and [Kubernetes Startup and Readiness Probes](/content/docs/guide_installation/deploying_a_kubernetes_managed_cluster.html#healthprobes/index.html) covers the `startupProbe` and `readinessProbe` configuration. Nothing ties them to Kubernetes,
though. They are plain HTTP, so any monitoring tool can poll them.

Because the status port is unauthenticated by design, access control is left to the network: restrict it with a firewall rule, security group, or network policy so it is reachable only by the systems that probe it. That is also why the endpoints are off until you ask for them: upgrading Smile CDR never opens a port you did not configure.

All three settings are [node configuration properties](/content/docs/guide_installation/configuring_smile_cdr.html#additional-node-configuration/index.html),
set in the same properties file as `node.id`:

- `node.status.port` turns the endpoints on and chooses the port; leave it out and nothing binds.
Like every other server port on the node, `node.server_port_offset` is added to it, so several
nodes can share a host without colliding. The two together must still land in `1` to `65535`.
- `node.status.bind_address` restricts which interface the listener binds to. The default is every
interface, which is what a Kubernetes probe needs; set it to `127.0.0.1` to keep the endpoints off
the network when you poll them from the same host.
- `node.status.health_check_timeout_seconds` bounds how long `/readiness` waits for the health
checks. Checks run one after another, so this budget covers the total across every module, not the
slowest single check. It defaults to `5` and accepts values from `1` to `600`. If a node reports
`healthCheckTimeout` under normal load, raise this budget rather than treating the probe as a
genuine failure. Raise your probe's own client-side timeout to match, or the probe will give up
before the endpoint answers. A Kubernetes `readinessProbe` defaults `timeoutSeconds` to 1, well
under this budget; see
[Kubernetes Startup and Readiness Probes](/content/docs/guide_installation/deploying_a_kubernetes_managed_cluster.html#healthprobes/index.html).

```properties
node.status.port=9300
```

Copy

## Response payloads

Both endpoints return pretty-printed JSON with a `Content-Type` of `application/json`. Null fields are omitted rather than serialized as `null`, so a field's absence is meaningful.

**`/startup`** carries only a boot status. While the process is booting it returns `STARTING` with HTTP `503`; once boot completes it returns HTTP `200` and:

```json
{
  "status" : "STARTED"
}
```

Copy

**`/readiness` when healthy** (HTTP `200`). `failingHealthChecks` is absent, not empty:

```json
{
  "status" : "READY",
  "nodeId" : "Master",
  "processId" : "Algonquin"
}
```

Copy

`nodeId` is the `node.id` from your configuration. `processId` is the name this process claimed from the Cluster Manager when it joined the cluster; it is absent until that has happened, so an early
`/readiness` call during boot can legitimately return a body with no `processId`.

**`/readiness` when not ready** (HTTP `503`). `failingHealthChecks` lists every check that is currently failing, each naming the `module` (the id you gave it in your configuration) and the
`healthCheck`:

```json
{
  "status" : "NOT_READY",
  "nodeId" : "Master",
  "processId" : "Algonquin",
  "failingHealthChecks" : [ {\
    "module" : "persistence",\
    "healthCheck" : "connection_pool"\
  }, {\
    "module" : "fhir_endpoint",\
    "healthCheck" : "dependent_fhir_storage_health"\
  } ]
}
```

Copy

A module that failed to start is reported against its own id:

```json
  "failingHealthChecks" : [ {\
    "module" : "subscription",\
    "healthCheck" : "failedToStart"\
  } ]
```

Copy

Conditions that belong to the process rather than to any one module use the reserved module id
`_process`. A process that is still booting reports `starting`, and no module health checks are run until boot completes:

```json
  "failingHealthChecks" : [ {\
    "module" : "_process",\
    "healthCheck" : "starting"\
  } ]
```

Copy

A process that is shutting down reports `shuttingDown` as soon as shutdown begins, before modules have finished stopping:

```json
  "failingHealthChecks" : [ {\
    "module" : "_process",\
    "healthCheck" : "shuttingDown"\
  } ]
```

Copy

## Reading a failing `/readiness` check

Nothing is latched: a result is reused for at most one second, far below any sensible probe interval,
so a transient failure such as a brief database outage clears on its own as soon as the underlying condition resolves. A request that arrives while shutdown is under way always gets a fresh answer, so draining is never delayed by that one-second window.

Most `healthCheck` values are reported by a module and name one of that module's own checks. Five are reported by the process itself and use the reserved module id `_process`. One more,
`healthCheckUnreadable`, is reported against the module it concerns:

| `healthCheck` | Module | What it means | First thing to check |
| --- | --- | --- | --- |
| `connection_pool` / `connection_pool_write` | the reporting module | Cannot obtain a (writable) database connection | Database is up, reachable, and not read-only |
| `http_responses` | the reporting module | Sustained HTTP 5xx on this listener | Recent errors or a downstream dependency |
| `dependent_fhir_storage_health` | the reporting module | The FHIR storage this endpoint depends on is unhealthy | The persistence module it points at |
| `ft_indexing_status` | the reporting module | Full-text indexing is behind or failing | Elastic/Lucene index health |
| `transaction_log_purge`, `stats_heartbeat`, `stats_cleanup_*` | the reporting module | A background maintenance job is not running | Scheduler / database |
| `hl7v2_mllp_listener` | the reporting module | HL7v2 MLLP listener is down | Listener port / config |
| `sp_sync_analysis` / `sp_sync_sync` | the reporting module | Search parameter synchronization is failing | Search-parameter config |
| `custom` (or a module-specific name) | the reporting module | A check owned by a specific module or integration | That module's own documentation |
| `failedToStart` | the module that failed | That module did not start, and will not recover unattended | The module's configuration; see below |
| `starting` | `_process` | The process has not finished booting, so its modules cannot serve traffic yet | Nothing. This is expected during a restart or a cold start, and `/startup` reports the same window |
| `shuttingDown` | `_process` | The process is shutting down | Nothing. This is expected during a restart or a planned stop |
| `healthCheckTimeout` | `_process` | The health checks did not finish within `node.status.health_check_timeout_seconds` | A module whose check is blocked, e.g. on an unresponsive database; raise the timeout if the node simply has more checks than fit in the budget |
| `healthCheckUnavailable` | `_process` | No health check thread was free to run this probe. Either a check is stuck and ignoring cancellation, or more readiness requests arrived at once than there are health check threads | Preceding `healthCheckTimeout` entries and the server log point at a stuck check; if there are none, reduce how many systems poll `/readiness` at once. Restart the process if a stuck check does not clear |
| `healthCheckError` | `_process` | The checks could not be evaluated at all, most often because a module's context was closing during a restart | Whether a module is restarting; the server log carries the underlying error |
| `healthCheckUnreadable` | the unreadable module | The module is running but could not supply its health checks, most often because it is mid-restart | Whether that module is restarting; it clears on the next probe once the module is back |

One root cause often produces two entries. In the `NOT_READY` payload above the database is unreachable, so Persistence cannot obtain a connection and the FHIR endpoint that depends on Persistence reports its upstream storage as unhealthy. Fix the database and both clear on the next probe.

The status port reports only the module id and check name, never the underlying error. For the exact error, such as the actual database message, use the authenticated [Runtime Status Health Checks](/content/docs/guide_operating#runtime-health-checks/index.html) operation, which returns every check with its message. For a module that failed to start, the startup log has the exception.

**`failedToStart`**
Unlike a failing health check, `failedToStart` does not clear on its own. Boot has finished, so `/startup`
returns `200` while `/readiness` stays at `503` indefinitely. Every process running the same node configuration is affected, not just one.

**Where you correct the configuration depends on** **[`node.propertysource`](/content/docs/guide_installation/configuring_smile_cdr.html#module-property-source/index.html),**
and the default is the less obvious of the two:

- **`DATABASE` (the default)**. Only the Cluster Manager is configured from the properties
file. Every other module is configured from the Cluster Manager database after first startup, so
editing the properties file and restarting will **not** change a failed module's configuration.
Fix it in the Web Admin Console or through the
[Module Config endpoint](/content/docs/json_admin_endpoints/module_config_endpoint.html), then restart the
module or the process.
- **`PROPERTIES`**. Module configuration is read from the properties file on every startup.
Correct it there and restart the process.

# Runtime Health Checks

As it is running, Smile CDR maintains a number of "health checks", which are simple status monitors that monitor whether a particular component of the system is operational.

For example, health checks exist to test database availability and health, HTTP server availability, scheduled job status, etc.

These health checks may be queried by a monitoring tool using the [Runtime Status Health Checks](/content/docs/json_admin_endpoints/runtime_status_endpoint.html#runtime-status-health-checks/index.html) operation on the [JSON Admin API](/content/docs/json_admin_endpoints/json_admin_api.html).

Monitoring Basics

Operating

[Monitoring Basics](/content/docs/guide_operating/monitoring_basics.html)
[Externalized Metrics](/content/docs/guide_operating/externalized_metrics.html)
[Smile CDR Observability](/content/docs/guide_operating/smilecdr_observability_guide.html)
[Performance Tuning](/content/docs/guide_operating/performance_tuning.html)
[Troubleshooting](/content/docs/guide_operating/troubleshooting.html)

[Externalized Metrics](/content/docs/guide_operating/externalized_metrics.html)