Catalogs, layers & partitions

Data Organization

For an overview of Data Organization, see the following video.

The data storage hierarchy in the HERE platform consists of the following parts:

• Catalog: At the top level of the hierarchy, a catalog is an organizational structure that allows you to group together data sets with similar contents. Data sharing occurs at the catalog level and allows access to the entire content.

• Layers: A layer allows you to group together similar types of data. It also allows you to specify configurations that restrict the type, size, and frequency of data being stored.

There are two categories of layers as follows:

• Short term storage (useful for quick access)

  • Stream layer: Good for real-time sensor data. Often used for submitting time sensitive data, such as a vehicle sensor stream in a hazard warning scenario.
  • Volatile layer: Used for data that must be updated or accessed quickly without requiring the entire data set. Inbound data overwrites any existing data stored in partitions. There is no long term data retention, making it suitable for applications such as real time updates to traffic and weather.

• Long term storage (better for applications where data retention is required)

  • Object store layer: An object / file system storage solution that enables you to store raw data objects in the HERE platform without any kind of HERE proprietary partitioning. It is useful for integration with other systems.
  • Versioned layer: A version control system designed specifically for data. Data is partitioned and updates can be applied across multiple partitions to maintain version consistency. HERE's Data Processing Library allows you to build your data processing pipelines, which ensures data layers are consistent across all partitions within a single catalog.
  • Index layer: Allows you to define multiple indexes making it easy to query your data. The layer is designed to be used as an archival system for time series data. It is suitable for applications such as historical analysis.
  • Interactive map layer: Useful for storing and interacting with data in more detail. Partitions are not used and the user interacts directly with individual geographical features. Interactive map layers combine storage with processing allowing you to query and manipulate individual features similar to how you use a database. For example, although data isn't partitioned, you can query data to be grouped by tile allowing you to display it in rendering applications.

• Schemas: Schemas define the organization of data in each partition of a layer, both the structure of the data and its content. In the HERE platform, schemas are defined using Protocol Buffers. If you are reading or writing data to a layer, use the layer's schema to understand the structure of the data you receive, and the way your data should be structured to write it to the layer. Schemas are also used by pipelines to operate on data.

If you are creating your own catalog and layers, including a schema for each layer enables you to share data with others by defining how others should consume the data.

• Partitions: Catalogs are divided into layers which in turn are divided into partitions, making partitions the smallest unit of data in the system. Partitions are key values within a catalog and layers are a namespace for partitions. You can either partition your data generically or according to the HERE tiling model.

For information on catalogs, layers, and partitions, see the Data API.

Data Formats

For an overview of Data Formats, see the following video.

You can upload data to the platform using a variety of file formats. These include:

• CSV
• Protocol buffers (protobufs)
• JSON
• Image
• Video

You can also upload your own files in the format of your choice.

If you want to visualize your data in the platform Data Inspector, you can format your data using GeoJSON or protobuf. GeoJSON renders in the Data Inspector, but protobuf requires a rendering plugin. You can, however, use a HERE protobuf Schema if it suits your use case. These schemas include:

• Schemas for HERE Map Content
• SENSORIS
• HERE Realtime Traffic

You can also use HERE tools to build your own custom renderer.

By attaching a schema to your layers containing data in a protobuf format, consumers are able to download the schema for decoding by using the generated language bindings in Java and Scala applications. This also allows you to drill down into the contents in the inspector tab.

You can choose a HERE schema for your use, or develop your own using HERE's Maven archetype schema generator and upload them yourself. If you want to work with your own custom schema, you can build a rendering plugin to visualization your data on the map.

Note:

If you use HERE's visualization tools, you must specify HERE tile as your partitioning format.