Update layer configurations
You can use the Data API to reconfigure data layers through a series of REST requests. The process of reconfiguring a layer varies depending on the type of layer you want to update.
Updates to layers and catalogs are asynchronuous and will not cause any downtime. For more information, see Create a catalog
Warning
It is important to note that making certain layer configuration changes after data is stored in the layer and/or after the parent catalog is available in Marketplace is very risky and can lead to irrecoverable impacts on data. Such changes should be made with great caution and understanding of the ramifications. Learn more here: Mutable properties by layer type.
- Obtain an authorization token. For more information, see the Identity & Access Management Guide.
- Use the API Lookup service to get the API endpoint for the
config
v1
API. For more information, see the API Lookup Developer's Guide. -
Use this request to the config
API to update the catalog and layer. This request updates the catalog with the new fields.
Note
For complete information on using the config
service, see the API Reference.
PUT /<Base path for the config API from the API Lookup Service>/catalogs/<Catalog HRN> HTTP/1.1
Host: <Hostname for the config API from the API Lookup Service>
Content-Type: application/json
Authorization: Bearer <Authorization Token>
Cache-Control: no-cache
{
"description": "<Describe the content and purpose of the catalog>",
"id": "<A unique identifier>",
"name": "<Catalog name>",
"notifications": {
"enabled": false
},
"summary": "<A short description of the content and purpose of the catalog>",
"layers": [
{
"id": "<A unique ID for the layer>",
"name": "<A user-friendly name to use for display purposes>",
"summary": "<A short description of the content and purpose of the layer>",
"description": "<A long description of the content and purpose of the layer>",
"schema": {
"hrn": "<The schema's HERE Resource Name>"
},
"contentType": "<The media type>",
"contentEncoding": "<Whether to compress data>",
"partitioning": {
"scheme": "<The method to use to divide data in the layer into partitions>",
},
"layerType": "<The type of layer: index, versioned, volatile, or stream>",
"volume": {
"volumeType": <durable, volatile>,
"maxMemoryPolicy": "failOnWrite"
}
}
]
}
Partial update to a layer configuration
You can use the Data API to reconfigure data layers through REST requests using the PATCH method. See also: PATCH Method for HTTP
- Obtain an authorization token. For more information, see the Identity & Access Management Guide.
- Use the API Lookup service to get the API endpoint for the
config
v1
API. For more information, see the API Lookup Developer's Guide. -
Use this request to the config
API to update layer only. This request updates the layer for mutable fields.
Note
For more information on using the config
service, see the API Reference.
PATCH /<Base path for the config API from the API Lookup Service>/catalogs/<Catalog HRN>/layers/<Layer ID> HTTP/1.1
Host: <Hostname for the config API from the API Lookup Service>
Content-Type: application/json
Authorization: Bearer <Authorization Token>
Cache-Control: no-cache
{
"schema": {
"hrn": "<The new schema's HERE Resource Name>"
}
}
Mutable properties by layer type
Index layer
It is important to understand the ramifications of layer configuration updates you make after data has already been stored in a layer. For an index layer example, updating the schema association could adversely impact downstream consumer processes if those processes are not also updated to leverage the new schema. Also, only one schema can be associated with a layer so it is important that any schema change not only represent new data stored to the layer but also all pre-existing data stored in the layer. If not, consumers of the data will no longer be able to effectively read the data.
Furthermore, the schema configuration update will not be dynamically picked up by a running application or pipeline. Downstream pipelines will need to be restarted to pick up these changes.
Schema association
An update to the schema association is possible, but high risk. See the above index layer schema change example for more detail.
Digest
The digest field is immutable once set due to the severity of the data corruption possible with such a change. The server will ignore further attempts to update this configuration.
TTL
This configuration is mutable. Changes will take effect immediately once saved. Index layer TTL changes will be applied to all stored objects.
Versioned layer
Updates to a versioned layer will be in effect immediately upon server confirmation. Configuration changes to versioned layers are very limited to prevent data corruption and wider impacts across multiple dependent versioned layers.
Schema association
An update to the schema association is possible, but high risk. Only one schema can be associated with a layer and that schema must represent all data in the layer. Schema changes will break downstream processes and services until those downstream dependencies are also updated.
Digest
The digest field is immutable once set due to the severity of the data corruption possible with such a change. The server will ignore further attempts to update this configuration.
Stream layer
parallelization
Parallelization is implemented as Kafka topic partition count. It's immutable once set. To change it, the layer must be recreated.
dataInThroughputKbps
This configuration is mutable. Changes will take effect immediately once saved.
dataOutThroughputKbps
This configuration is mutable. Changes will take effect immediately once saved.
TTL
This configuration is mutable. Changes will take effect immediately once saved. Stream layer TTL changes will be applied to all stored objects.
ContentType & ContentEncoding
ContentType and Encoding configurations do not affect Kafka topic handling in any way. These configurations are mutable, enabling you to correct simple layer configuration mistakes after a layer is created.
If data is already saved in a stream layer before these changes are made, updates to these configurations will cause breaking changes to downstream consumers of the data. Data already saved in the layer prior to these changes will be saved using the old configuration settings. New data stored in the layer after these changes are saved using the new configuration settings so the overall data set will contain a mix. It is important that you realize the significant impact of these changes on your data before making them. Given the nature of stream layers (continuous feed of messages), these impacts are not permanent but you will need to wait for the messages encoded via the old layer configuration settings to be purged from the stream before restarting downstream pipelines and services.
Schema association
An update to the schema association is possible, but high risk. Only one schema can be associated with a layer and that schema must represent all data in the layer. Schema changes will break downstream processes and services until those downstream dependencies are also updated.
Interactive map layer
Searchable Properties
Searchable Properties is a list of strings, containing the properties you want to get indexed for faster searching. Searchable properties can be updated at any time.
If the layer encompasses less than 10,000 features, all properties are searchable.
Note
The maximum number of searchable properties is eight. This is the maximum number of user-added and automatically indexed properties combined.