2023-01-14

Pipeline modular Kedro

Pipeline modular

Kedro memungkinkan Anda untuk memodularisasi pipeline Anda untuk membuat pipeline yang dapat digunakan kembali. Misalnya, Anda dapat menjalankan beberapa pipeline yang hanya berbeda dalam fitur. Pipeline modular lebih mudah dikembangkan, diuji, dan dipelihara serta lebih portabel.

Memperluas proyek dengan pipeline modular

Mari kita perluas pipeline spaceflights Starter yang disediakan oleh Kedro. Pertama, siapkan proyek spaceflights Anda.

$ kedro new --starter=spaceflights
$ cd spaceflights
$ pip install -r src/requirements.txt

Kita sekarang akan menambahkan namespaces dengan parameter berbeda berikut ini ke komponen pemodelan dari pipeline data science.

active_modelling_pipeline
candidate_modelling_pipeline

Edit conf/base/parameter/data_science.yml sebagai berikut.

conf/base/parameters/data_science.yml

-  model_options:
-    test_size: 0.2
-    random_state: 3
-    features:
-      - engines
-      - passenger_capacity
-      - crew
-      - d_check_complete
-      - moon_clearance_complete
-      - iata_approved
-      - company_rating
-      - review_scores_rating

+  active_modelling_pipeline:
+      model_options:
+        test_size: 0.2
+        random_state: 3
+        features:
+          - engines
+          - passenger_capacity
+          - crew
+          - d_check_complete
+          - moon_clearance_complete
+          - iata_approved
+          - company_rating
+          - review_scores_rating
+
+  candidate_modelling_pipeline:
+      model_options:
+        test_size: 0.2
+        random_state: 8
+        features:
+          - engines
+          - passenger_capacity
+          - crew
+          - review_scores_rating

Edit conf/base/catalog.yml sebagai berikut.

conf/base/catalog.yml

-  regressor:
-    type: pickle.PickleDataSet
-    filepath: data/06_models/regressor.pickle
-    versioned: true

+  active_modelling_pipeline.regressor:
+    type: pickle.PickleDataSet
+    filepath: data/06_models/regressor_active.pickle
+    versioned: true
+
+  candidate_modelling_pipeline.regressor:
+    type: pickle.PickleDataSet
+    filepath: data/06_models/regressor_candidate.pickle
+    versioned: true

Edit pipelines/data_science/pipeline.py sebagai berikut.

pipelines/data_science/pipeline.py

  from kedro.pipeline import Pipeline, node
- from kedro.pipeline import pipeline
+ from kedro.pipeline.modular_pipeline import pipeline

  from .nodes import evaluate_model, split_data, train_model


  def create_pipeline(**kwargs) -> Pipeline:
-      return pipeline(
-          [
-              node(
-                  func=split_data,
-                  inputs=["model_input_table", "params:model_options"],
-                  outputs=["X_train", "X_test", "y_train", "y_test"],
-                  name="split_data_node",
-              ),
-              node(
-                  func=train_model,
-                  inputs=["X_train", "y_train"],
-                  outputs="regressor",
-                  name="train_model_node",
-              ),
-              node(
-                  func=evaluate_model,
-                  inputs=["regressor", "X_test", "y_test"],
-                  outputs=None,
-                  name="evaluate_model_node",
-              ),
-          ]
-      )

+      pipeline_instance = pipeline(
+          [
+              node(
+                  func=split_data,
+                  inputs=["model_input_table", "params:model_options"],
+                  outputs=["X_train", "X_test", "y_train", "y_test"],
+                  name="split_data_node",
+              ),
+              node(
+                  func=train_model,
+                  inputs=["X_train", "y_train"],
+                  outputs="regressor",
+                  name="train_model_node",
+              ),
+              node(
+                  func=evaluate_model,
+                  inputs=["regressor", "X_test", "y_test"],
+                  outputs=None,
+                  name="evaluate_model_node",
+              ),
+          ]
+      )
+      ds_pipeline_1 = pipeline(
+          pipe=pipeline_instance,
+          inputs="model_input_table",
+          namespace="active_modelling_pipeline",
+      )
+      ds_pipeline_2 = pipeline(
+          pipe=pipeline_instance,
+          inputs="model_input_table",
+          namespace="candidate_modelling_pipeline",
+      )
+
+      return ds_pipeline_1 + ds_pipeline_2

Jika anda menjalankan kedro run, anda akan melihat bahwa kedua pipeline active_modelling_pipeline dan candidate_modelling_pipeline sedang berjalan.

Keyword argument	Description
`pipe`	The Pipeline object you want to wrap
`inputs`	Any overrides provided to this instance of the underlying wrapped Pipeline object
`outputs`	Any overrides provided to this instance of the underlying wrapped Pipeline object
`parameters`	Any overrides provided to this instance of the underlying wrapped Pipeline object
`namespace`	The namespace that will be encapsulated by this pipeline instance

Pipeline modular Kedro

Pipeline modular

Memperluas proyek dengan pipeline modular

pembungkus pipeline()

Referensi

Tutorial Kedro

Kedro Viz

Ryusei Kakujo