# Realtime Export Rules Definition [LMA](/content/docs/hdp_product/maturity_model.html)

The Realtime Export Rules Definition controls the behaviour of the Realtime Export module. It is composed of some settings, and a collection of transformers, and can be configured either via the [Script Text](/content/docs/configuration_categories/realtime_export.html#property-realtime-export-script-text/index.html), or the [Script File](/content/docs/configuration_categories/realtime_export.html#property-realtime-export-script-file/index.html).
This is an example Rules Definition:

```json
{
   "retainAllHistory": false,
   "concatenationDelimiter": " - ",
   "overflowStrategy": "TRUNCATE",
   "namedTransformers": {
      "simple-humanname-transformer": {
         "columns": [\
            {\
               "columnName": "name_use",\
               "fhirPath": "use",\
               "columnType": "STRING"\
            },\
            {\
               "columnName": "given_names",\
               "fhirPath": "given",\
               "columnType": "STRING",\
               "multiplePrimitiveStrategy": "CONCAT",\
               "maximumSize": 20\
            },\
            {\
               "columnName": "family_name",\
               "fhirPath": "family",\
               "columnType": "STRING"\
            }\
         ]
      }
   },
   "transformers": [\
      {\
         "resourceType": "Patient",\
         "tableName": "rte_patient",\
         "columns": [\
            {\
               "columnName": "id",\
               "fhirPath": "Patient.id",\
               "columnType": "STRING"\
            },\
            {\
               "columnName": "first_name",\
               "fhirPath": "Patient.name.given.first()",\
               "columnType": "STRING"\
            },\
            {\
               "columnName": "version",\
               "fhirPath": "Patient.meta.versionId",\
               "columnType": "INT"\
            }\
         ],\
         "childTables": [\
            {\
               "fhirPath": "Patient.name",\
               "tableName": "rte_patient_name",\
               "childTransformerName": "simple-humanname-transformer"\
            }\
         ]\
      }\
   ]
}
```

Copy

1. `retainAllHistory` – When disabled, only the current version of each resources is tracked in the remote database. When enabled, historical versions of every table are stored in tables suffixed with `_history`. For example, if history was enabled, and the remote table to be updated was `rte_patient_table`, then the RTE module would also try to update the table `rte_patient_table_history`. The history table never receives delete queries, and so will retain the full history of any resources exported.
2. `concatenationDelimiter` – Any fields with `multiplePrimitiveStrategy` set to `CONCAT` will use the delimiter defined here.
3. `overflowStrategy` – If the `maximumSize` is set on a column, and the value to be stored in it exceeds that length, then the overflow strategy is applied to the data before storage. The options are `DROP` and `TRUNCATE`. This setting is only meaningful for JDBC targets, where remote column widths (e.g. `VARCHAR(n)`) are fixed by the database schema. The Delta Lake export target stores strings without a width limit, so `maximumSize` and `overflowStrategy` have no effect there unless you set `maximumSize` explicitly to cap column length for downstream consumers.
4. `transformers` – A collection of objects which define resource transformation rules.
5. `namedTransformers` – A common set of transformers that can be used in multiple other transformers.

Transformers are the primary component that you will interact with when configuring the Realtime Export module. Each transformer contains a set of rules and metadata which inform the module as to how to process any given incoming resource type. The remainder of this page discusses transformers in detail.

# Transformers [LMA](/content/docs/hdp_product/maturity_model.html)

```json
{
   "resourceType": "Patient",
   "tableName": "remote_patient",
   "columns": [\
      {\
         "columnName": "id",\
         "fhirPath": "Patient.id",\
         "columnType": "STRING"\
      },\
      {\
         "columnName": "family_name",\
         "fhirPath": "Patient.name.family.first()",\
         "columnType": "STRING",\
         "maximumSize": 40\
      },\
      {\
         "columnName": "gender",\
         "fhirPath": "Patient.gender",\
         "columnType": "STRING"\
      }\
   ]
}
```

Copy

Transformers are configured as part of the Realtime Export Rules Definition and must each minimally have:

1. `resourceType` – This is the type of resource which the transformer is applied to. When a message comes in on a channel matching the resource type, the transformer will be applied. Multiple transformers may each refer to the same resource type.
2. `tableName` – The name of the table in the remote database where the transformed data will be exported to. Note that each transformer can move data into exactly one remote table.
3. `columns` – A list of columns in the remote table which need to be populated by this resource.

Here is the anatomy of the `column` element:

1. `columnName` – The name of the column in the remote table.
2. `fhirPath` – The [FHIRPath](http://hl7.org/fhirpath/) expression to be applied to the incoming resource, in
order to retrieve the needed information.
3. `columnType` – The database column type in the remote table. This is currently limited to the values defined
in [ColumnTypeEnum](https://github.com/jamesagnew/hapi-fhir/blob/master/hapi-fhir-sql-migrate/src/main/java/ca/uhn/fhir/jpa/migrate/taskdef/ColumnTypeEnum.java)
.
4. `maximumSize`: This should be set to correspond to the maximum length of the column set in the database for STRING
fields. This field is optional, but if set, will use the top-level `overflowStrategy` defined in the rules. This
setting targets JDBC backends with fixed column widths; Delta Lake strings are unbounded, so it only matters for
Delta Lake if you want to cap values before they reach downstream consumers.

Let's use this transformer to show a small example of how Realtime Export will operate if a Patient resource is created in the database.

Take for example this Patient:

```json
{
   "resourceType":"Patient",
   "id":"123",
   "meta":{
      "versionId":"1",
      "lastUpdated":"2020-09-15T18:38:07.182+00:00"
   },
   "identifier":[\
      {\
         "system": "http://example.org/mrn",\
         "value": "123"\
      },\
      {\
         "system": "http://example.org/person-id",\
         "value": "456"\
      }\
   ],
   "gender":"male",
   "name":[\
      {\
         "family":"Scott",\
         "given":[\
            "Michael",\
            "Gary"\
         ]\
      },\
      {\
         "family":"Levinson",\
         "given":[\
            "Astrid"\
         ]\
      }\
   ],
   "address":[\
      {\
         "use":"home",\
         "line":[\
            "#4 Privet Drive",\
            "Under the staircase"\
         ],\
         "city":"Little Whinging",\
         "district":"Surrey",\
         "postalCode":"ABC 123"\
      }\
   ]
}
```

Copy

When this Patient is received in the Realtime Export module, the following will happen:

1. First, the module will check all known transformers against the incoming resource.
2. Finding that our above transformer matches this resource type (the resource is a Patient, and the resourceType of the transformer is set to Patient), processing will begin for this transformer.
3. For each column in the column list, apply the fhirPath expression against the resource. For example, for the column named `family_name`, the fhirPath expression `Patient.name.family.first()` will be applied. This will return the value `Scott`.
4. After this has occurred for each of the columns, the module will generate a sql statement to insert the found data into the remote database, which will look something like the following:

```sql
INSERT INTO remote_patient (id, first_name, gender, source_resource_id, version) VALUES ('Patient/123', 'Scott', 'male', 'Patient/123', 1)
```

Copy

5. The generated SQL is then executed.

You will notice that the generated SQL includes two elements that were not considered in the transformer. Both `version` and `source_resource_id` fields are automatically included in every transformer, as they are required for internal logic of updates and deletes.

# Named Transformers [LMA](/content/docs/hdp_product/maturity_model.html)

There are many times when you may want to reuse common transformation logic. For example, you may want to process a Patient, and have all of its names be stored in a child table. At the same time, you have a transformer for Practitioners, and you also want their names to be stored in their own child table.
Instead of repeating the transformer multiple times, you can instead create a `namedTransformer`, and make use of it in other transformers. The following example shows a full set of Realtime Export rules which fulfill these exact conditions that were just laid out:

```json
{
   "concatenationDelimiter": "|",
   "overflowStrategy": "TRUNCATE",
   "namedTransformers": {
      "simple-humanname-transformer": {
         "columns": [\
            {\
               "columnName": "name_use",\
               "fhirPath": "use",\
               "columnType": "STRING"\
            },\
            {\
               "columnName": "given_names",\
               "fhirPath": "given",\
               "columnType": "STRING",\
               "multiplePrimitiveStrategy": "CONCAT",\
            },\
            {\
               "columnName": "family_name",\
               "fhirPath": "family",\
               "columnType": "STRING"\
            }\
         ]
      }
   },
   "transformers": [\
      {\
         "resourceType": "Patient",\
         "tableName": "remote_patient",\
         "columns": [\
            {\
               "columnName": "id",\
               "fhirPath": "Patient.id",\
               "columnType": "STRING"\
            },\
         ],\
         "childTables": [\
            {\
               "fhirPath": "Patient.name",\
               "tableName": "remote_patient_name",\
               "childTransformerName": "simple-humanname-transformer"\
            }\
         ]\
      },\
      {\
         "resourceType": "Practitioner",\
         "tableName": "remote_practitioner",\
         "columns": [\
            {\
               "columnName": "id",\
               "fhirPath": "Practitioner.id",\
               "columnType": "STRING"\
            },\
         ],\
         "childTables": [\
            {\
               "fhirPath": "Practitioner.name",\
               "tableName": "remote_practitioner_name",\
               "childTransformerName": "simple-humanname-transformer"\
            }\
         ]\
      }\
   ]
}
```

Copy

For this example to be relevant, we must also include a sample practitioner, seen below.

```json
{
   "resourceType":"Practitioner",
   "id":"321",
   "meta":{
      "versionId":"1",
      "lastUpdated":"2020-10-28T18:38:07.182+00:00"
   },
   "gender":"female",
   "name":[\
      {\
         "family":"Davar",\
         "given":[\
            "Shallan"\
         ]\
      }\
   ],
   "address":[\
      {\
         "use":"home",\
         "city":"Vedenar",\
         "district":"Jah Keved"\
      }\
   ]
}
```

Copy

In this example, the Realtime Export module will process Patient and Practitioner names using the same rules, despite the end data being stored in separate tables, like so:

# Managing high-cardinality elements [LMA](/content/docs/hdp_product/maturity_model.html)

The previous example shows a very simple case, where all the FHIRPath expressions evaluate down to a single element which can be stored in a database, but the reality of FHIR is more complex. For example, a Patient can have 0..\* names. In order to handle this, Realtime Export supports multiple strategies for handling high-cardinality
data elements.

**Multiple-primitive strategy**
One option for high-cardinality data is to simply perform extra processing to reduce the elements down to a single element. This can be done by setting the `multiplePrimitiveStrategy` in the column json. The following is an example

```json
{
   "columnName": "all_first_names",
   "fhirPath": "Patient.name.given",
   "multiplePrimitiveStrategy": "CONCAT"
}
```

Copy

In this example, the FHIRPath expression `Patient.name.given` has a good chance of returning multiple entries. the `multiplePrimitiveStrategy` tells Realtime Export which behaviour to use if it encounters multiple values here. In this case, it is set to `CONCAT`, which will simply concatenate all the values into a single string. Using the example Patient above, this would end up evaluating to
`Michael|Gary|Astrid`. The delimiter used is set in the top-level JSON, in a field called `concatenationDelimiter`

An alternative to the `CONCAT` strategy is the `FIRST` strategy, which would simply return `Michael` in this case.

## Child tables

It is not always desired to use a simple multiple primitive strategy, and you would instead like to use foreign key relationships to manage this data, as one normally does with relational data.

For example, this transformer will add an entry to the `remote_patients` table, but also add two entries to the `remote_patient_names` table for our example Patient.

```json
{
   "resourceType": "Patient",
   "tableName": "remote_patient",
   "columns": [\
      {\
         "columnName": "id",\
         "fhirPath": "Patient.id",\
         "columnType": "STRING"\
      },\
      {\
         "columnName": "gender",\
         "fhirPath": "Patient.gender",\
         "columnType": "STRING"\
      }\
   ],
   "childTables": [\
      {\
         "tableName": "remote_patient_name",\
         "fhirPath": "Patient.name",\
         "childTransformer": {\
            "columns": [\
               {\
                  "columnName": "given_names",\
                  "fhirPath": "given",\
                  "columnType": "STRING",\
                  "multiplePrimitiveStrategy": "CONCAT"\
               },\
               {\
                  "columnName": "family_name",\
                  "fhirPath": "family",\
                  "columnType": "STRING"\
               }\
            ]\
         }\
      }\
   ]
}
```

Copy

Note that the definition of the child table contains:

1. `tableName` – This is much the same as the top-level transformer `tableName` field, in that it defines the name of the remote table.
2. `fhirPath` – The fhirPath which returns the collection of elements. In this case `Patient.name` will return a collection of [HumanNames](https://www.hl7.org/fhir/datatypes.html#HumanName).
3. `childTransformer` – This field contains the transformation rules for each element being iterated over. Can contain `columns` and also `childTables`.

Given a child table, the Realtime Export module will apply the child transformer over every element in the collection defined by the `fhirPath`.

This will generate the following entries in the remote database:

You will notice some entries here that were not in the child transformer. Namely, `id` and `parent_reference`. The `id` is an automatically generated UUID, and the `parent_reference` is automatically added to the resource to be used as a foreign key on the parent table.

This concept of child tables can be repeatedly nested. For example, you could have a transformer that had a child element for each address,
and that address element can have a child entry for each address-line. An example of this is shown below:

```json
{
   "resourceType":"Patient",
   "tableName":"remote_patient",
   "columns":[\
      {\
         "columnName":"id",\
         "fhirPath":"Patient.id",\
         "columnType":"STRING"\
      },\
      {\
         "columnName":"gender",\
         "fhirPath":"Patient.gender",\
         "columnType":"STRING"\
      }\
   ],
   "childTables":[\
      {\
         "fhirPath":"Patient.address",\
         "tableName":"remote_patient_address",\
         "childTransformer":{\
            "columns":[\
               {\
                  "columnName":"address_use",\
                  "fhirPath":"use",\
                  "columnType":"STRING"\
               },\
               {\
                  "columnName":"address_postal",\
                  "fhirPath":"postalCode",\
                  "columnType":"STRING"\
               },\
            ],\
            "childTables":[\
               {\
                  "fhirPath":"line",\
                  "tableName":"remote_patient_address_line",\
                  "childTransformer":{\
                     "columns":[\
                        {\
                           "columnName":"line",\
                           "fhirPath":"$this",\
                           "columnType":"STRING"\
                        }\
                     ]\
                  }\
               }\
            ]\
         }\
      }\
   ]
}
```

Copy

There is one special thing to note in this transformer. The nested child table uses a special FHIRPath expression called `$this`, which refers to the element itself, whatever it happens to be.
This transformer will generate the following database entries.

[Bootstrapping RTE Data](/content/docs/realtime_export/bootstrap_loading_rte.html)

Realtime Export Rules Definition

Realtime Export

[Realtime Export Overview](/content/docs/realtime_export/realtime_export_overview.html)
[Bootstrapping RTE Data](/content/docs/realtime_export/bootstrap_loading_rte.html)
[Realtime Export Rules Definition](/content/docs/realtime_export/realtime_export_rules_definition.html)
[Using FHIRPath](/content/docs/realtime_export/using_fhirpath.html)
[Delta Lake Export --EAP--](/content/docs/realtime_export/delta_lake_export.html)
[Bulk Batch Replication --EAP--](/content/docs/realtime_export/bulk_batch_replication.html)

[Using FHIRPath](/content/docs/realtime_export/using_fhirpath.html)