KylinCubeOperator

Kylin

This operator is used to submit request about kylin build/refresh/merge, and can track job status . so users can easier to build kylin job

View Source

Last Updated: May. 7, 2021

Access Instructions

Install the Kylin provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

kylin_conn_idstrThe connection id as configured in Airflow administration.
projectstrkylin project name, this param will overwrite the project in kylin_conn_id:
cubestrkylin cube name
dsnstr(dsn , dsn url of kylin connection ,which will overwrite kylin_conn_id. for example: kylin://ADMIN:KYLIN@sandbox/learn_kylin?timeout=60&is_debug=1)
commandstr(kylin command include 'build', 'merge', 'refresh', 'delete', 'build_streaming', 'merge_streaming', 'refresh_streaming', 'disable', 'enable', 'purge', 'clone', 'drop'. build - use /kylin/api/cubes/{cubeName}/build rest api,and buildType is ‘BUILD’, and you should give start_time and end_time refresh - use build rest api,and buildType is ‘REFRESH’ merge - use build rest api,and buildType is ‘MERGE’ build_streaming - use /kylin/api/cubes/{cubeName}/build2 rest api,and buildType is ‘BUILD’ and you should give offset_start and offset_end refresh_streaming - use build2 rest api,and buildType is ‘REFRESH’ merge_streaming - use build2 rest api,and buildType is ‘MERGE’ delete - delete segment, and you should give segment_name value disable - disable cube enable - enable cube purge - purge cube clone - clone cube,new cube name is {cube_name}_clone drop - drop cube)
start_timeOptional[str]build segment start time
end_timeOptional[str]build segment end time
offset_startOptional[str]streaming build segment start time
offset_endOptional[str]streaming build segment end time
segment_namestrsegment name
is_track_jobbool(whether to track job status. if value is True,will track job until job status is in("FINISHED", "ERROR", "DISCARDED", "KILLED", "SUICIDAL", "STOPPED") or timeout)
intervalinttrack job status,default value is 60s
timeoutinttimeout value,default value is 1 day,60 * 60 * 24 s
eager_error_statustuple(jobs error status,if job status in this list ,this task will be error. default value is tuple(["ERROR", "DISCARDED", "KILLED", "SUICIDAL", "STOPPED"]))

Documentation

This operator is used to submit request about kylin build/refresh/merge, and can track job status . so users can easier to build kylin job

For more detail information in Apache Kylin

Was this page helpful?