2022-06-10

Migration to Meltano Version 2.0

Introduction

In this article, I will introduce key points for migrating from Meltano Version 1.0 to Version 2.0.

Migrating to Adapter-Specific dbt Transformer

If you are using the dbt Transformer, it is recommended to install adapter-specific dependencies such as dbt-snowflake.

$ meltano add transformer dbt-snowflake

Updating dbt_project.yml

When installing a new Transformer, two important files will be generated in the transform/ directory:

  • transform/profiles/<adapter name>/profiles.yml
  • transform/dbt_project (<adapter name>).yml

By merging dbt_project.yml and dbt_project (<adapter name>).yml into a single file named dbt_project.yml, the migration from legacy dbt plugins will be completed.

Removing dbt Transformer Plugin and Associated Files

You can remove the legacy dbt Transformer plugin using the following commands:

# remove the transformer `dbt`
$ meltano remove transformer dbt

# remove the file bundle `dbt`
$ meltano remove files dbt

Support for transform in meltano elt Command

Meltano 2.0 continues to support EL processing with the meltano elt command, but for EL+T processing, use meltano run.

Using --state-id Instead of --job_id

The term "Job ID" used in Meltano's code and documentation will be replaced with the more accurate term "State ID".

References

https://docs.meltano.com/guide/v2-migration

Ryusei Kakujo

researchgatelinkedingithub

Focusing on data science for mobility

Bench Press 100kg!