Catalog Operations

In this section we review the most used catalog-level operations provided by the SDK. Full details on all methods can be found in the API documentation.

• Creating, updating, and deleting catalogs is done at the platform level. See Platform Operations section for details.

For all code examples below, assume the following was executed prior, with HRN set to a catalog for which you have Read/Write/Manage permissions.

from here.platform import Platform
platform = Platform()
hrn = "hrn:here:data::olp-here:your-catalog"
catalog = platform.get_catalog(hrn=hrn)

Get Catalog Details

Get catalog details given HRN of the catalog.

catalog_details = catalog.get_details()

Get Catalog First Version

Gets the first version of the catalog available on the platform.

catalog_first_version = catalog.first_version()

Get Catalog Latest Version

Gets the latest version of the catalog available on the platform.

catalog_latest_version = catalog.latest_version()

List Catalog Versions

Gets the list of all catalog versions available on the platform.

catalog_versions = catalog.list_versions()

Details of single version

Gets the details of specified version of the catalog.

version = catalog.get_version()

Catalog Compatible Versions

Given a list of HRNs and versions provided, returns compatible versions iterator which yields list of Versions of this catalog for which the listed HRN are either present in the direct or indirect dependencies with the same version, or are not present. Each Version object contains version and shared_dependencies attribute. catalog_dependencies: list of HRNs and versions limit: number of versions at each yield

catalog_compatible_versions = catalog.compatible_versions(catalog_dependencies=catalog_dependencies, limit=limit)

Catalog Has Layer

Check if a catalog has a layer of given layer ID. It will return True or False depending on whether the catalog has that layer.

result = catalog.has_layer(layer_id=layer_id)

Catalog Get Layer

Gets a Layer object for the given layer ID.

layer = catalog.get_layer(layer_id=layer_id)

Catalog Add Layer

Adds a single new layer to existing catalog.

new_layer = catalog.add_layer(
    id="volatile-layer-1",
    layer_type="volatile",
    name="vol-layer-name",
    summary="vol-layer-summary",
    description="vol-layer-desc",
    content_type="application/x-protobuf",
    partitioning={"scheme": "generic"}
)
# Refresh your catalog
catalog = platform.get_catalog(catalog.hrn)

Catalog Update Layer

Updates a single layer.

catalog.update_layer(
    layer_id="volatile-layer-1",
    layer_type="volatile",
    name="new-vol-layer-name",
    summary="new-vol-layer-summary",
    description="new-vol-layer-desc",
    content_type="application/x-protobuf",
    partitioning={"scheme": "generic"}
)
# Refresh your catalog
catalog = platform.get_catalog(catalog.hrn)

Catalog Delete Layer

Deletes the layer of the catalog.

catalog.delete_layer(layer_id=layer_id)

Catalog Grant Access

Grants access to a catalog to an entity.

entity_type = "sample-entity-type"
entity_id = "sample-entity-id"
action = "sample-action"
catalog.grant_access(entity_type=entity_type,entity_id=entity_id,action=action)

Catalog Revoke Access

Revokes access to a catalog from an entity.

entity_type = "sample-entity-type"
entity_id = "sample-entity-id"
action = "sample-action"
catalog.revoke_access(entity_type=entity_type,entity_id=entity_id,action=action)

Share Catalog

Share a catalog with an entity.

entity_type = "sample-entity-type"
entity_id = "sample-entity-id"
catalog.share(entity_type=entity_type,entity_id=entity_id)

Open Catalog in Portal

Open the catalog page on the HERE platform portal.

catalog.open_in_portal()

Write to Catalog Layers

The SDK provides separate write methods tailored to the type of layer being written to:

• write_index_layer
• write_stream_layers
• write_versioned_layers
• write_volatile_layers

In all cases except index layer, writing data simultaneously to multiple layers of the same type in the same catalog is supported. To allow this, and to maintain consistency for versioned layers, these "write multiple layer" methods are surfaced as catalog operations. If you need only to write to a single layer you should use the equivalent layer level methods. Note that for Interactive Map layers only layer level methods are provided.

The layers_write_info Structure

When writing data to stream, versioned, or volatile layers, you will need to construct a nested dict containing layer_ids, partition_ids, and files or data in bytes to be uploaded.

# layers_write_info for writing single partition to single layer
layer_id = 'my-test-layer'
partition_data = {'23300689': path_to_local_data_file_or_reference_to_in_memory_binary_data}
my_layer_info = {layer_id: partition_data}

# layers_write_info for writing multiple partitions to single layer
layer_id = 'my-test-layer'
partition_data = {'23300689': some_path_or_bytes, '23300347': some_other_path_or_bytes}
my_layer_info = {layer_id: partition_data}

# layers_write_info for writing to partition in two different layers
layer_id_1 = 'layer-1'
layer_id_2 = 'layer-2'
partition_data_1 = {'23300689': some_path_or_bytes}
partition_data_2 = {'23300689': some_other_path_or_bytes}
my_layer_info = {layer_id_1: partition_data_1, layer_id_2: partition_data_2}

Examples below show only required parameters for write methods. See the API documentation for details of optional parameters applicable to each method.

Write to Stream Layer

from here.platform import Platform

catalog = platform.get_catalog("HRN-OF-MY-CATALOG")
catalog.write_stream_layers(layers_write_info=my_layer_info)

Write to Versioned Layer

from here.platform import Platform
catalog = platform.get_catalog("HRN-OF-MY-CATALOG")
catalog.write_versioned_layers(layers_write_info=my_layer_info)

Write to Volatile Layer

from here.platform import Platform
catalog = platform.get_catalog("HRN-OF-MY-CATALOG")
catalog.write_volatile_layers(layers_write_info=my_layer_info)

Write to Index Layer

Writing to an index layer is different than other layer types, and there are additional required method arguments (fields). Below is a simple example of the syntax. Consult the API documentation for full details.

from here.platform import Platform
catalog = platform.get_catalog("HRN-OF-MY-CATALOG")
layer_id = "sample_index_layer"
file_path = "~/sample_data.parquet"
fields = {"ingestionTime": 1114, "partition_id": 343433}
catalog.write_index_layer(layer_id, path_or_data=file_path, fields=fields)