External Object Storage - Smile CDR Documentation

External Object Storage

Smile CDR supports storing data in external object storage services including Amazon S3, Azure Blob Storage, and MinIO. This storage backend is used by:


AWS S3

In this mode, content is stored in an AWS S3 bucket. All data will be stored in one bucket, which can be named via the Blob Service Bucket / Container property. You can also configure the region by setting the Blob Service Region property.

On boot, Smile CDR will create a bucket if it does not already exist.

Authentication to S3 is done using the DefaultAwsCredentialsProviderChain. This means that credentials can be provided in a variety of ways, including:

However, you also have the option to provide your own credentials via the Blob Service S3 Access Key property and the Blob Service S3 Secret Key property. If credentials are provided in this fashion, they will be used instead of the default credentials.

The account you authenticate with will need permissions to create buckets, as well as to put/head/get/delete objects in the bucket.

MinIO

In this mode, content is stored in a MinIO server. All data will be stored in one bucket, which can be named via the Blob Service Bucket / Container property.

Authentication for MinIO must be provided via the Blob Service S3 Access Key and Blob Service S3 Secret Key properties.

Currently, MinIO is only recommended for development purposes.

Azure Blob Storage

In this mode, content is stored in an Azure Blob Storage container. All data will be stored in one container, which can be named via the Blob Service Bucket / Container property. You can also configure the account name by setting the Blob Service Azure Account property.

On boot, Smile CDR will create a container if it does not already exist.

There are three different supported authentication methods:

  1. Access Key
  2. Account-level Shared Access Signatures (SAS) token
  3. Azure Active Directory

The authenticated account will need permissions to create containers, as well as to PUT/HEAD/GET/DELETE objects in the container.


External Object Storage Performance Tuning

Smile CDR maintains a pool of persistent HTTP connections to blob storage services (S3, Azure Blob Storage, MinIO) to minimize latency and resource overhead. The following settings control the size and behavior of this pool and should be tuned according to throughput requirements and environment resources.

Maximum Connections

Maximum Connections defines the hard limit on the number of active HTTP connections to the storage service. The default value is 100. Increase this value if Externalized Binary Storage and/or Externalized Resource Body Storage are used extensively to allow more concurrent file transfers. Ensure that infrastructure (CPU/Network) can support the increased load. Setting this value excessively high may degrade performance or cause failures by exhausting local system resources and triggering rate-limiting errors from the storage service.

Connection Timeout

Connection Timeout defines the maximum time (in seconds) to wait for establishing a new TCP connection to the storage service. The default value is 30 seconds. Adjust this value based on network reliability and available system resources.

Maximum Pending Connection Requests

Maximum Pending Connection Requests defines the maximum number of requests that can be queued when all connections are busy. Requests exceeding this limit are rejected. The default value is 10000. This queue acts as a buffer to handle temporary traffic bursts. When increasing Maximum Connections, consider proportionally increasing this value to maintain the ~100x ratio between queue size and connection pool size. If the queue fills up frequently, it typically indicates that the Maximum Connections limit is too low for the sustained workload, or that the storage service is responding too slowly.

Connection Acquisition Timeout

Connection Acquisition Timeout defines the maximum time (in seconds) a request will wait in the pending queue for a connection to become available. The default value is 30 seconds. If requests frequently fail with this timeout, it indicates that the connection pool is fully utilized. Rather than increasing this timeout (which typically just masks the issue and increases latency), it is recommended to increase Maximum Connections to allow higher parallel throughput.

Binary Data

External Object Storage

FHIR Storage

FHIR Storage Modules FHIR Endpoint Module FHIRWeb Console OpenAPI / Swagger Support FHIR Endpoint Customization Resource IDs Search Parameters Search Parameter Features Phonetic Search Parameters Search Parameter Tuning Search Parameter Reindexing Searching for Data Creating Data Reading Data Updating Data Deleting Data Binary Data External Object Storage Externalized Resource Body Storage Request Tracing and Provenance Resource Versions and Versioned References Tags, Profiles, and Security Labels Partitioning and Multitenancy Patient ID Partitioning Modes Custom Resource Types Batch and Scheduled Jobs Tokenization Bulk Resource Modification

Externalized Resource Body Storage