Organize your work in projects

Objectives: Create a project and a catalog, add this catalog to the project, and link a public catalog to the project with the OLP Command Line Interface (CLI).

Complexity: Beginner

Time to complete: 30 min

Prerequisites: Verify your credentials

Source code: Download

This example demonstrates how to create projects, and how to manage project members and resources using the OLP CLI.

A project is a collection of resources (catalogs, pipelines, and schemas) with controlled access. The usage of any compute, storage, and transfer operations on a resource associated with the project is automatically accounted for by project ID in the Credit Usage report on the HERE platform. As a project admin, you can grant other users, apps, and groups access to the project. Users with project access rights can create and manage project resources.

To follow this tutorial, you need to have the CLI installed and have both platform and repository credentials set up on your local machine.

To verify that you set up your platform credentials correctly, make sure that the Verify your credentials tutorial returns the expected result.

Create a project

Give your project a unique identifier and name.

Replace {{YOUR_PROJECT_ID}} and {{YOUR_PROJECT_NAME}} with your own ID and name in the command below and execute it:

olp project create {{YOUR_PROJECT_ID}} {{YOUR_PROJECT_NAME}}

The OLP CLI should return the following message:

Project {{YOUR_PROJECT_HRN}} has been created.

Write down {{YOUR_PROJECT_HRN}}, as you need it in the next steps for the new project resources.

Notice that you can also create and manage your projects in the Workspace portal.

To verify that your app and user have access to the project, execute the olp access list command as follows:

olp project access list {{YOUR_PROJECT_HRN}}

If your command is successful, the CLI returns the following:

type           HRN                                                                         id                                           name
app            hrn:<partition>:account::<realm>:app/<AppId>                                <AppId>                                      <AppName>
user           hrn:<partition>:account::<realm>:user/<UserId>                              <UserId>                                     <UserName>

By default, the olp project create command grants project access both for the user who creates the project and for the app associated with the credentials. This behavior can be overridden with the --skip-sharing-with-user flag, which provides a way to not share the project with the user. Notice that without project access privileges, the user cannot view and manage the project in the HERE portal.

You can also verify that an app and user have access to the project in the portal. To do this, go to the Project Manager portal section and select {{YOUR_PROJECT_NAME}} in the list. Verify that your user name and app name are listed in the "Access and permissions" section of the project details page.

To grant access to a project for a particular user, app, or group, use the olp project access grant command as follows:

olp project access grant {{YOUR_PROJECT_HRN}} --app <appId> --user <uderId> --group <groupId>

Create a catalog in the scope of the project

Give your catalog a unique identifier and name, for example, {{YOUR_USERNAME}}-sdii-tutorial.

Create a configuration file for your catalog called sdii-versioned.json with the contents below and replace {{YOUR_CATALOG_ID}} with your own identifier:

{
  "id": "{{YOUR_CATALOG_ID}}",
  "name": "Simulated sensor data archive (From tutorial)",
  "summary": "Archive of simulated sensor data",
  "description": "Archive of simulated sensor data in form of SDII messages containing road sign observations.",
  "tags": ["Tutorial", "SDII", "SensorData", "RoadSign", "Simulated"],
  "layers": [
    {
      "id": "sdii-message-archive",
      "name": "sdii-message-archive",
      "summary": "Archive of simulated sensor data in form of SDII messages containing road sign observations",
      "description": "Archive of simulated sensor data in form of SDII messages containing road sign observations.\n\nThis data is meant to be used for demonstration purposes and \"playing\" with data.",
      "tags": ["Tutorial", "SDII", "SensorData", "RoadSign", "Simulated"],
      "contentType": "application/x-protobuf",
      "layerType": "versioned",
      "volume": {
        "volumeType": "durable"
      },
      "partitioning": {
        "scheme": "generic"
      },
      "schema": {
        "hrn": "hrn:here-cn:schema::olp-here:com.here.sdii:sdii_message_v3:3.3.8"
      }
    }
  ]
}

To create a catalog, use the olp catalog create command. Replace {{YOUR_CATALOG_ID}} with your own catalog ID and {{YOUR_PROJECT_HRN}} with the HRN of the project in the following command and execute it:

olp catalog create {{YOUR_CATALOG_ID}} \
    "Simulated sensor data archive from tutorial" \
    --config sdii-versioned.json \
    --scope {{YOUR_PROJECT_HRN}}

Observe the --scope param, as the value of this parameter is equal to the project HRN. We specify this parameter to create a catalog in the scope of the project. In the same way, you should use this parameter with other operations such as update, show, and delete for resources associated with the project. For more details about scopes, see the Scopes section of the CLI Developer Guide.

If your command is successful, the CLI returns the following:

Catalog {{YOUR_CATALOG_HRN}} has been created.

The HRN returned by this statement is what you use for all further CLI and HERE Data SDK for Java & Scala operations to identify this catalog.

You can verify the expected catalog configuration with the olp catalog show command as follows:

olp catalog show {{YOUR_CATALOG_HRN}} \
    --scope {{YOUR_PROJECT_HRN}}

The CLI returns the values matching those in your JSON file. In particular, the output that indicates that the versioned layer has been successfully created and configured looks as follows:

...
layers
type      id                                      name
Versioned sdii-message-archive                    sdii-message-archive
...

You can also verify that the new catalog is shared with your project in the platform portal. To do this, navigate to the portal and select {{YOUR_PROJECT_NAME}} in the list. Your new catalog should be listed in the "Resources" section of your project details page.

You can also inspect your new catalog configuration in the platform portal. To do this, navigate to the portal and select Simulated sensor data archive (From tutorial) from the list.

Inspect catalog configuration

Notice that the fields in your JSON configuration correspond to the values displayed in the portal.

The tags with which you have configured the catalog and layer appear in their respective catalog listings in the Data tab under the catalog name and the layer listings within a single catalog. You can also search for all catalogs with a given tag, such as "sensordata".

Click your catalog and select the "sdii-message-archive" layer in the "Layers" section below. For more information about layers and their types, see Data User Guide.

The JSON file above configures the layer to contain an archive of raw sensor data messages. Each partition in this layer contains a single protobuf-encoded sensor data message, which the JSON file indicates in the contentType field. The layerType is "versioned" and the volumeType is "durable", indicating that the partition data is intended to persist. If a client publishes new data to an existing partition, the old data remains available in previous versions of the layer. The partitioning scheme is "generic", meaning that the messages are not spatially partitioned in the catalog. Therefore, you can't bring up a visual representation of the partition by clicking on the Inspect tab. Instead, the tab reports "No spatial data."

Click the Schema tab to view the details of the schema that is bundled with the layer. The HRN of the schema is also specified in the layer configuration file. As you learned in Verify your credentials, an HRN identifies a resource such as a catalog or a schema. Client applications can use the schema HRN to discover the associated artifacts for processing and decoding data using the schema's specifications and language bindings. The platform portal itself uses the schema artifacts to decode the partition contents and display their values in the Inspect tab.

The newly created catalog is empty, so there are no partitions to decode.

Also, you can verify the catalog layer configuration with the olp catalog layer show command as follows:

olp catalog layer show {{YOUR_CATALOG_HRN}} sdii-message-archive \
    --scope {{YOUR_PROJECT_HRN}}

Link a public catalog to the project

We use public catalogs as the data source in further tutorials. Link the hrn:here-cn:data::olp-cn-here:here-map-content-china-2 public catalog to your project to be able to access this catalog in the scope of the project.

To do this, use the olp project resource link command.

Replace {{PUBLIC_CATALOG_HRN}} with hrn:here-cn:data::olp-cn-here:here-map-content-china-2 and {{YOUR_PROJECT_HRN}} with the HRN of your project in the following command, then execute it:

olp project resource link {{YOUR_PROJECT_HRN}} {{PUBLIC_CATALOG_HRN}}

If your command is successful, the CLI returns the following message:

Project resource hrn:here-cn:data::olp-cn-here:here-map-content-china-2 has been linked.

To verify that the public catalog has been successfully linked to the project, execute the olp project resource list command as follows:

olp project resource list {{YOUR_PROJECT_HRN}}

If your command is successful, the CLI returns the following message:

Available resources in the project:

HRN                                                                                                Type           Relation
hrn:here-cn:data::olp-cn-here:here-map-content-china-2                                                                catalog        reference
{{YOUR_CATALOG_HRN}}                                                                               catalog        home

Notice that there are other types of resources, like pipelines and schemas, which can be associated with the project, but in this tutorial we link only catalogs. Also, there are different types of relations between a project and the resources associated with it.

If a resource is created in the scope of the project, the relation type is "home". If you link a public catalog to the project, the relation type is "reference".

You can also verify that the public catalog is shared with your project in the platform portal. To do this, navigate to the "Projects" tab and find {{YOUR_PROJECT_NAME}} from the list.

Open the project and verify that the public catalog previously linked is present in the "Resources" section of your project details page.

You can also inspect the configuration of any catalog accessible to you on the platform portal. To do this, navigate to portal and find your public catalog by name.

Further information

You can use the catalog you created in this tutorial as the output catalog for the streaming application implemented in Run a Stream Application Locally.

For more details on projects, see Manage Projects.

For more details on OLP CLI project access management commands, see the relevant sections in Project Access Commands.

For more details on OLP CLI project management commands, see the relevant sections in Project Commands.

For more details on OLP CLI catalog commands, see the relevant Catalog Commands.

If you want to see schema decoding in action over a non-spatially partitioned layer (that is, whose partitioning scheme is "generic"), you can inspect the Administrative Locations layer of the HERE Map Content catalog.