2023-01-14

Kedro

What is Kedro

Kedro is an OSS pipeline management tool developed by Quantum Black Lab, a research arm of McKinsey & Company, that allows you to create highly reproducible, maintainable, and modular data science code.

Kedro features

Kedro has the following main features.

Pipeline visualization
- You can visualize the DAG of your pipeline.
Data catalog
- Lightweight data connectors for reading and writing data between many file formats (Pandas, Spark, Dask, NetworkX, Pickle, Plotly, Matplotlib, etc.) and file systems (S3, GCP, Azure, sFTP, DBFS, local) is provided.
Integration
- Excellent integration with Apache Spark, Pandas, Dask, Matplotlib, Plotly, fsspec, Apache Airflow, Jupyter Notebook, Docker, etc.
Project templates
- Project templates allow you to standardize how you organize your configuration, source code, tests, documentation, notebooks, etc.
Pipeline abstraction
- Kedro supports a dataset-driven workflow that automatically resolves dependencies between functions, eliminating the need to label the order of execution of tasks in the pipeline.
Flexibility in deployment
- Pipelines can be deployed to Argo, Prefect, Kubeflow, AWS Batch, AWS Sagemaker, Databricks, Dask, etc.

Kedro components

Kedro has the following four components

Node
Pipeline
DataCatalog
Runner

Node

A node is the unit of each process in the pipeline. You use node() to define the processing, input, and output.

In the following code, the return_greeting function is wrapped in a node called return_greeting_node, which has no input and names its output my_salutation.

from kedro.pipeline import node

# Prepare first node
def return_greeting():
    return "Hello"

return_greeting_node = node(func=return_greeting,
                            inputs=None,
                            outputs="my_salutation")

In the following code, the join_statements function is wrapped in a node called join_statements_node with my_salutation as input and the output named my_message.

# Prepare second node
def join_statements(greeting):
    return f"{greeting} Kedro!"

join_statements_node = node(
    join_statements,
    inputs="my_salutation",
    outputs="my_message"
)

It means that my_salutation is an output of return_greeting_node and an input of join_statements_node. Thus, inputs and outputs of node() can define the dependencies of the pipeline.

Pipeline

A pipeline is an execution pipeline. A pipeline is constructed by passing a list of Nodes to Pipeline(). The order of the list to be passed to the pipeline is arbitrary.

In the following pipeline, return_greeting_node is executed first, followed by join_statements_node.

from kedro.pipeline import Pipeline

# Assemble nodes into a pipeline
pipeline = Pipeline([return_greeting_node, join_statements_node])

DataCatalog

DataCatalog is a catalog that defines the data to be used in the pipeline; DataCatalog takes Node inputs and outputs as keys and reads and writes them into a Dataset. The Dataset supports various file formats and file systems. Please refer to the official documentation below for available Datasets.

The following code uses MemoryDataSet to store my_salutation in memory.

from kedro.io import DataCatalog, MemoryDataSet

# Prepare a data catalog
data_catalog = DataCatalog({"my_salutation": MemoryDataSet()})

Runner

There are three types of runners: SequentialRunner for executing pipelines in series, ParallelRunner for executing pipelines in parallel, and ThreadRunner for executing pipelines in threads.

File name (folder name)	Description
conf	Project configuration files
data	Local project data (not committed to version control)
docs	Project documentation
logs	Project output logs (not committed to version control)
notebooks	Project-related Jupyter notebooks (can be used for experimental code before moving the code to src)
pyproject.toml	Identifies the project root and contains configuration information
setup.cfg	Configuration options for `pytest` when doing `kedro test` and for the `isort` utility when doing `kedro lint`
src	Project source code

Starter	Description
`astro-airflow-iris`	The Kedro Iris dataset example project with a minimal setup for deploying the pipeline on Airflow with Astronomer.
`standalone-datacatalog`	A minimum setup to use the traditional Iris dataset with Kedro’s DataCatalog, which is a core component of Kedro. This starter is of use in the exploratory phase of a project. For more information, read the guide to standalone use of the DataCatalog. This starter was formerly known as mini-kedro.
`pandas-iris`	The Kedro Iris dataset example project
`pyspark-iris`	An alternative Kedro Iris dataset example, using PySpark
`pyspark`	The configuration and initialisation code for a Kedro pipeline using PySpark
`spaceflights`	The spaceflights tutorial example code

Kedro

What is Kedro

Kedro features

Kedro components

Node

Pipeline

DataCatalog

Runner

Hello Kedro

How to use Kedro

Installation

Project creation

kedro new

kedro new --config

kedro new --starter=pandas-iris

Other starters

References

MLflow Tracking

Kedro tutorial

Ryusei Kakujo