Data Pipeline

If running on AWS you can use datapipeline() to access AWS Data Pipelines’ health easily.

datapipeline(region=None)

Initialize Data Pipeline wrapper.

Parameters:region (str) – AWS region for Data Pipeline queries. Eg. “eu-west-1”. Defaults to the region in which the check is being executed. Note that Data Pipeline is not availabe in “eu-central-1” at time of writing.

Methods of Data Pipeline

get_details(pipeline_ids)

Query AWS Data Pipeline IDs supplied as a String (single) or list of Strings (multiple). Return a dict of ID(s) and status dicts as described in describe_pipelines boto documentation.

Parameters:pipeline_ids (Union[str, list]) – Data Pipeline IDs. Example df-0123456789ABCDEFGHI Return type:dict

Example query with single Data Pipeline ID supplied in a list:

datapipeline().get_details(pipeline_ids=['df-exampleA'])
{
    "df-exampleA": {
        "@lastActivationTime": "2018-01-30T14:23:52",
        "pipelineCreator": "ABCDEF:auser",
        "@scheduledPeriod": "24 hours",
        "@accountId": "0123456789",
        "name": "exampleA",
        "@latestRunTime": "2018-01-04T03:00:00",
        "@id": "df-0441325MB6VYFI6MUU1",
        "@healthStatusUpdatedTime": "2018-01-01T10:00:00",
        "@creationTime": "2018-01-01T10:00:00",
        "@userId": "0123456789",
        "@sphere": "PIPELINE",
        "@nextRunTime": "2018-01-05T03:00:00",
        "@scheduledStartTime": "2018-01-02T03:00:00",
        "@healthStatus": "HEALTHY",
        "uniqueId": "exampleA",
        "*tags": "[{\"key\":\"DataPipelineName\",\"value\":\"exampleA\"},{\"key\":\"DataPipelineId\",\"value\":\"df-exampleA\"}]",
        "@version": "2",
        "@firstActivationTime": "2018-01-01T10:00:00",
        "@pipelineState": "SCHEDULED"
    }
}