Tokenization - Smile CDR Documentation

Tokenization and Field-Level Encryption

Some organizations, jurisdictions, or projects have policies requiring that specific fields within resources be obfuscated when at rest in the database.

For example, consider the following simple Patient resource:

{
    "resourceType": "Patient",
    "id": 123,
    "name": [ {\
        "family": "Simpson",\
        "given": [ "Homer" ]\
    } ],
    "birthDate": "1956-05-12",
    "gender": "male"
}

Copy

In the example above, the complete resource text will be stored in a text field within the database. In addition, for FHIR search indexing, specific strings are extracted from the resource and also stored in dedicated database tables designed to support searching.

Of course, database-level encryption and HTTPS transport security should generally already be used to protect this data, but sometimes additional security is needed for specific fields. Smile CDR Tokenization causes specific elements within the FHIR resource body to be extracted and replaced with "tokens", which are opaque strings serving as placeholders for these elements.

By using the tokenization capability, these strings can be replaced by opaque placeholder strings, which reduces the ability for someone to re-identify the data if they have access to the database.

These tokenized strings can take any form and do not need to match the format of the original datatype. For this reason, they are stored in an extension and the original value is removed. The following example shows a Patient.birthDate element with its value replaced by a tokenized string:

{
    "_birthDate": {
        "extension": [\
            {\
                "url": "https://smilecdr.com/fhir/ns/StructureDefinition/resource-tokenized-value",\
                "valueCode": "cce82123-748d-4597-b52b-9200646ab788"\
            }\
        ]
    }
}

Copy

Tokenization Algorithm

Smile CDR relies on a user-supplied algorithm for tokenization, and does not provide the tokenization capability (i.e. the actual algorithm used to convert between a plaintext string and a token) directly. This is because tokenization should be external to Smile CDR for separation of concerns.

The tokenization algorithm:

Tokenization Provider

The tokenization algorithm is supplied via an implementation of the ITokenizationProvider interface. This interface provides two methods, one for converting a string into a token and the other for performing the reverse.

You can use the General Purpose Interceptor Demo Project as a starting point for creating your own tokenization provider.

Example Tokenization Provider

The following example shows a tokenization provider:

/**
 * A simple tokenization provider which tokenizes a few PHI fields within the Patient resource. This spec uses
 * the (completely insecure!) ROT-13 algorithm for tokenization and is intended for demonstration
 * purposes only.
 */
public class ExampleTokenizationProvider implements ITokenizationProvider {

/**
    * This method is called in order to tokenize one or more source strings. The system tries
    * to provide batches of strings for tokenization so that if an external tokenization
    * service is used, and it supports batching, this capability can be leveraged.
    *
    * @param theRequestDetails The request associated with the tokenization. This object
    *                          contains one or more strings for tokenization.
    * @param theRequests       The requests, which include the specific rule as well as the object being tokenized
    */
   @Override
   public TokenizationResults tokenize(RequestDetails theRequestDetails, TokenizationRequests theRequests) {
      TokenizationResults retVal = new TokenizationResults();
      for (TokenizationRequest request : theRequests) {
         String source = request.getObjectAsString();
         String token = rot13(source);
         retVal.addResult(request, token);
      }
      return retVal;
   }

/**
    * This method is called in order to convert one or more tokenized strings back into their
    * original source value. This method must return exactly the same value as was originally
    * provided for tokenization. Method is only called if one or more of the configured
    * tokenization rules declare support for de-tokenization.
    */
   @Override
   public DetokenizationResults detokenize(RequestDetails theRequestDetails, DetokenizationRequests theRequests) {
      DetokenizationResults retVal = new DetokenizationResults();
      for (DetokenizationRequest request : theRequests) {
         String token = request.getToken();
         String source = rot13(token);
         retVal.addResult(request, source);
      }
      return retVal;
   }

/**
    * Implementation of ROT13 obfuscation, based on a solution found
    * here: https://stackoverflow.com/questions/8981296/rot-13-function-in-java
    * This is not intended to be a suitable production tokenization algorithm,
    * it is simply provided as an easy way to demonstrate the concept!
    */
   public static String rot13(String theInput) {
      StringBuilder sb = new StringBuilder();
      for (int i = 0; i < theInput.length(); i++) {
         char c = theInput.charAt(i);
         if (c >= 'a' && c <= 'm') c += 13;
         else if (c >= 'A' && c <= 'M') c += 13;
         else if (c >= 'n' && c <= 'z') c -= 13;
         else if (c >= 'N' && c <= 'Z') c -= 13;
         else if (c >= '0' && c <= '4') c += 5;
         else if (c >= '5' && c <= '9') c -= 5;
         sb.append(c);
      }
      return sb.toString();
   }
}

Copy

Tokenization Rules

The configured tokenization rules define the set of FHIR data elements which will be tokenized. Essentially, they are a collection of FHIRPath Expressions which will be extracted from resources being stored in the repository and replaced with equivalent tokens.

The rules are configured using the Tokenization Rules Text or Tokenization Rules File settings. The value is a JSON document using the TokenizationRules model.

Each rule must contain a FHIRPath expression, beginning with a resource type. For example, the expression Patient.name.family instructs the module that when storing a Patient resource, each repetition of the Patient.name element must have the family name extracted and replaced with a token.

If a given expression corresponds to a search parameter which is active on the server, that search parameter must also be declared in the rule. See Searching and Tokenization below.

The following example shows a rule collection with several active rules for the Patient resource.

{
    "rules" : [ {\
        "description" : "Rule for a path including a search parameter",\
        "path" : "Patient.identifier",\
        "searchParameter" : "identifier",\
        "searchValueNormalization" : "IDENTIFIER",\
        "status" : "ACTIVE"\
    }, {\
        "description" : "Another rule for a path including a search parameter",\
        "path" : "Patient.name.family",\
        "searchParameter" : "family",\
        "searchValueNormalization" : "STRING",\
        "status" : "ACTIVE"\
    }, {\
        "description" : "Rule for a path with no associated search parameter",\
        "path" : "Patient.maritalStatus",\
        "status" : "ACTIVE"\
    } ]
}

Copy

Searching and Tokenization

When an element in a resource is tokenized and that element is also used as a search parameter expression, declaring the search parameter as a part of the Tokenization Rule causes the search index to also be tokenized.

For example, this means that if you have chosen to tokenize the Patient.name.family element (which is used to support the family Search Parameter) the tokenized string will be indexed instead of the original value. Suppose the configured tokenization algorithm tokenizes the value "Smith" with the token "ABCDEFG". When performing a search using this parameter, the value being searched for will also be tokenized to ensure that values can still be found. In other words, the string "Smith" which was supplied by the client will be tokenized to "ABCDEFG" and this will be the string used to search the database, matching the tokenized value that was actually stored.

To make this work, Smile CDR automatically creates internal SearchParameter resources with the same name as the original SearchParameter but with the suffix -tokenized. Therefore, if a FHIR client performs a search for Patient?family=smith, the search term will be automatically tokenized and the search will be treated as Patient?family-tokenized=ABCDEFG.

Defining Tokenization Rules to Support Searching

To define a rule which supports searching, you must include the following elements in the rule definition:

An example rule which supports searching is shown below:

{
    "rules" : [ {\
        "path" : "Patient.identifier",\
        "searchParameter" : "identifier",\
        "searchValueNormalization" : "IDENTIFIER"\
    } ]
}

Copy

Search Normalization

If you need to support searching on a tokenized value, you will need to declare a normalization format corresponding to the datatype of the element being indexed. For example, if you are indexing the Patient.identifier element, which is an element with the FHIR Identifier datatype, you will need to declare a normalization format of IDENTIFIER.

Only a specific subset of datatypes are supported for search indexing when they are tokenized. Some datatypes may not be reasonable to support for searching (e.g. Quantity datatypes are typically searched using a comparator or a range of values, which is not practical to search against a tokenized index). Other datatypes have simply not been added because they have not been requested as supported options. If you need support for other datatypes, please get in touch.

The following normalization formats are available:

Mode Supported Datatypes for Indexing Description
NONE Use this value if no searching will be performed on the element being tokenized by the rule.
CODE code This mode should be selected if the expression points to an element of type *code* where the element has a vocabulary [Binding Strength](https://hl7.org/fhir/terminologies.html#strength) of **Required**, and the search parameter is of type *token*. For example, the expression `Patient.gender` or `Encounter.status`.


Using the `CODE` normalization allows searches to be performed using either the raw code (e.g. `Patient?gender=male`) or the qualified system and code (e.g. `Patient?gender=http://hl7.org/fhir/administrative-gender|male\`).
CODEABLECONCEPT CodeableConcept This mode should be selected if the expression points to an element of type *CodeableConcept* and the search parameter is of type *token*. For example, the expression `Observation.code`.
CONTACTPOINT ContactPoint This mode should be selected if the expression points to an element of type *ContactPoint* and the search parameter is of type *token*. For example, the expression `Patient.telecom`.
DATE date Tokenize a date value, such as `Patient.birthDate`. Only exact matches are supported, meaning that searches with a given date value in the URL will only match resources with the same level of precision in the actual value.

In other words, if a resource has a date string of `2021-01-01`, then a search for `Patient?birthDate=2021-01-01` will match that resource, but a search for `Patient?birthDate=2021-01` will not. If the resource has lower precision in the stored date, only a search containing the lower precision will match.
IDENTIFIER Identifier This mode should be selected if the expression points to an element of type *Identifier* and the search parameter is of type *token*. For example, the expression `Patient.identifier`.
STRING string The search index string is normalized according to the FHIR string normalization rules (case-insensitive, diacritics are removed, etc.). This mode should only be used if the expression refers to a primitive string, and the search parameter is of type string. For example, the expression `Patient.name.family`.

Limitations

Note the following limitations on searches which use tokenized values:

Tokenization and Validation

When using either Repository Validation or Endpoint Validation with this feature, resources are validated prior to tokenization.

This means that tokenization will not cause validation failures if a mandatory data element is then removed and tokenized. However, if a non-reversible tokenization algorithm is chosen it could mean that the resource will no longer meet the same requirements when it is returned.

Other Limitations

The following limitations apply to the use of Tokenization. These limitations could be removed in future versions of Smile CDR, please get in touch if you require any of these features.

Updating Tokenization Rules

If it is necessary to change your tokenization rules, you may make changes to your rules file and restart your FHIR storage module at any time. However, this may cause issues if you have existing data stored in your repository which was tokenized under the previous rules.

Smile CDR provides the Update Tokenization operation ($sdh.update-tokenization) that can be used to apply the new rules to any existing data in the repository.

Adding New Tokenization Rules

When a new tokenization rule is added, any resources that are added or modified will be tokenized using the new rules. Any existing data will still be available and will be returned with the original non-tokenized values. New data will be detokenized on access and will be returned with the original non-tokenized values as well (assuming the tokenization algorithm is reversible and the rule is configured to actually detokenize).

If the new rule includes a search parameter, searches using this parameter will only find resources added or modified after the rule was added.

The Update Tokenization operation can be used to apply the new rules to existing data, after which all data will be tokenized, and search parameters will be able to find all applicable resources.

Removing Tokenization Rules

If a tokenization rule is no longer needed, it should generally not be immediately removed if any data is already stored with tokenized data. Once a rule is removed, existing tokenized values will be returned without being de-tokenized.

In order to remove a rule, you should use the following steps:

Update Tokenization Operation ($sdh.update-tokenization)

Initiating an Update Tokenization job requires the FHIR_OP_UPDATE_TOKENIZATION permission.

The Update Tokenization operation can be used to apply the new rules to any existing data in the repository. This operation is invoked by executing an HTTP POST request to the $sdh.update-tokenization operation.

This operation follows the FHIR Asynchronous Interaction Request Pattern. This means that the operation will return immediately, and will return a response containing a URL to poll for the status of the operation. It can also be monitored using the Batch Job Endpoint or through the Web Admin Console.

Input Parameters

The Update Tokenization operation accepts a Parameters resource as input. All Common Bulk Modification Request Parameters are supported.

In addition to the common parameters:

Example Request

The following example shows an HTTP POST request to the $sdh.update-tokenization operation:

POST /$sdh.update-tokenization

{
    "resourceType": "Parameters",
    "parameter": [\
        {\
            "name": "url",\
            "valueString": "Patient?"\
        }\
    ]
}

Copy

Example Response

The response will contain a URL to poll for the status of the operation:

202 Accepted

{
    "resourceType":"OperationOutcome",
    "issue":[\
        {\
            "severity":"information",\
            "code":"informational",\
            "diagnostics":"$sdh.update-tokenization job has been accepted. Poll for status at the following URL: http://localhost:8000/$sdh.update-tokenization-status?_jobId=33b280a7-1715-4d38-a7d4-6f5d1c705551"\
        }\
    ]
}

Copy

Batch and Scheduled Jobs

Tokenization

FHIR Storage

FHIR Storage Modules FHIR Endpoint Module FHIRWeb Console OpenAPI / Swagger Support FHIR Endpoint Customization Resource IDs Search Parameters Search Parameter Features Phonetic Search Parameters Search Parameter Tuning Search Parameter Reindexing Searching for Data Creating Data Reading Data Updating Data Deleting Data Binary Data External Object Storage Externalized Resource Body Storage Request Tracing and Provenance Resource Versions and Versioned References Tags, Profiles, and Security Labels Partitioning and Multitenancy Patient ID Partitioning Modes Custom Resource Types Batch and Scheduled Jobs Tokenization Bulk Resource Modification

Bulk Resource Modification