PinotAdminHook

Pinot

This hook is a wrapper around the pinot-admin.sh script. For now, only small subset of its subcommands are implemented, which are required to ingest offline data into Apache Pinot (i.e., AddSchema, AddTable, CreateSegment, and UploadSegment). Their command options are based on Pinot v0.1.0.

View on GitHub

Last Updated: Apr. 27, 2021

Access Instructions

Install the Pinot provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

conn_idstrThe name of the connection to use.
cmd_pathstrThe filepath to the pinot-admin.sh executable
pinot_admin_system_exitboolIf true, the result is evaluated based on the status code. Otherwise, the result is evaluated as a failure if “Error” or “Exception” is in the output message.

Documentation

This hook is a wrapper around the pinot-admin.sh script. For now, only small subset of its subcommands are implemented, which are required to ingest offline data into Apache Pinot (i.e., AddSchema, AddTable, CreateSegment, and UploadSegment). Their command options are based on Pinot v0.1.0.

Unfortunately, as of v0.1.0, pinot-admin.sh always exits with status code 0. To address this behavior, users can use the pinot_admin_system_exit flag. If its value is set to false, this hook evaluates the result based on the output message instead of the status code. This Pinot’s behavior is supposed to be improved in the next release, which will include the following PR: https://github.com/apache/incubator-pinot/pull/4110

Was this page helpful?