By default, event and error log entries are sent to Splunk for review. How much information is reported in the logs depends on the logging level you select for each pipeline version when it is executed. Splunk is a data collection, indexing, and visualization engine for operational intelligence. For information on how to use Splunk, see the Splunk Enterprise User Documentation.
Pipeline Logging Basics
HERE platform pipelines use logging to provide more details during their operation. Different levels of logging are available for different purposes. HERE platform pipelines support the following levels of logging:
Debug — Includes fine-grained informational events that are most useful to troubleshoot a pipeline.
Info — Includes informational messages that highlight the progress of the pipeline at a coarse-grained level.
Warn — Includes information on potentially harmful situations; including other run-time situations that are undesirable or unexpected, but not necessarily "wrong".
Error — Includes other run-time errors or unexpected conditions such as error events that might still allow the pipeline to continue running.
Note
Pipelines have a default logging level of warn.
Change the Logging Level
The Logging Level is set for a Pipeline Version. All the Jobs use the logging level associated with the corresponding Pipeline Version.
Logging Levels can be set at the root level for the entire pipeline and/or at the individual logger level for a pipeline class. Due to operational latency, it takes a few minutes for the changes to take effect. This may delay the availability of the logs at the new level in Splunk.
Note
The displayed Pipeline Version details include either the last logging level modified by the user or the default logging level (that is, warn).
The logging level for a Pipeline Version can be changed only when it is in one of the following states:
Running — When the logging-level is changed for a Pipeline Version in a Running state, the system will change the logging-level of the currently running job.
Ready or Scheduled — When the logging-level is changed for a Pipeline Version in a Ready or Scheduled state, the system will run the future jobs using the new logging-level.
Paused — For a Pipeline Version in the Paused state, when the logging-level is changed and Pipeline Version is resumed, the system will run the future jobs using the new logging-level.
API Solution
To change a Pipeline Version's log-level to DEBUG at the root-level, use the following command from the REST API:
PUT /v2/pipelines/{pipelineId}/versions/{versionId}/logging-configuration
Each running Pipeline Version has a unique URL where the logs for that pipeline are stored. You can access the logs from the portal by clicking on the See Log link on the display for the running pipeline, as shown in Figure 1.
When using the CLI, the log URL information is supplied by the pipeline whenever a pipeline version is activated, upgraded, or when a status request is submitted. However, if you are running your pipeline by interfacing directly with the Pipeline REST API, you will need to make a status request to discover the logging URL. For more information, see the Logs, Monitoring and Alerts User Guide .
Example
The CLI show function is one way you can check status. In the case of a specific job, you can use the following CLI command.
olp pipeline version job show f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 d77f288e-2c89-4c94-b4ba-79fbd1e26e79 e8e31070-4f4b-4e4b-a8e0-40425844cb75
In this case, the loggingUrl parameter has a null value. This is what you would expect to find for a job that is not yet running. Once the job runs, the logging URL would be shown.
Note
Until a scheduled pipeline runs, the logging URL will remain null.
Custom Logging and Local Development
Your HERE platform pipelines may need to log additional custom events. You can add these in the pipeline application code, but there are certain limitations.
The HERE platform generally supports standard logging via SLF4J. If you require a specific logging back end that is SLF4J-compatible, you need to specify the configuration in the application.conf. Your runtime also requires an SLF4J backend. For more information on logging backend implementations for SLF4J, refer to the official SL4FJ Documentation.
When a developer wants to add logging to their pipeline code, they should use the SLF4J abstracted log API. The HERE platform will provide a binding on the platform, and an API to control log-levels on the fly, as described above.
No separate logging implementation JAR file should be included in the pipeline JAR file artifact; such as slf4j-api or slf4j-log4j12. For example, slf4j-api should be a provided JAR file defined in the BOM for the pipeline application's Fat JAR file.
When doing local development and testing of pipeline code, the developer is free to provide any SLF4J binding they choose, although we recommend logback.
Tools
Event Logging is handled by an embedded version of Splunk. In general, you will not need to be concerned because Splunk runs in the background. However, you can create a new Splunk dashboard or run reports on the platform portal.