2022-06-07

Handling Meltano Environments

Meltano Environments

In Meltano, you can manage and switch environment variables according to your development or production environments. The definitions for multiple environments are described in meltano.yml as follows:

meltano.yml
environments:
  - name: prod
    config:
      plugins:
        extractors:
          - name: tap-github
            config:
              organizations: [Meltano]
            select: ['*.*']
        loaders:
          - name: target-snowflake
            config:
              dbname: prod
              warehouse: prod_wh
              batch_size_rows: 100000
    env:
      SOME_PROD_ONLY_SETTING: abc
  - name: dev
    config:
      plugins:
        extractors:
          - name: tap-github
            config:
              organizations: [MeltanoLabs]
            select: ['repositories.*']
        loaders:
          - name: target-snowflake
            config:
              dbname: dev
              warehouse: dev_wh
              batch_size_rows: 1000

Environment Inheritance

If you want to use the same base plugin with different configurations, you can add a new plugin that inherits from an existing plugin. The new plugin inherits the configuration of the base plugin and can override it if necessary.

meltano.yml
.
.
.
plugins:
  extractors:
  - name: tap-google-analytics
    variant: meltano
    config:
      key_file_location: client_secrets.json
      start_date: '2020-10-01T00:00:00Z'
  - name: tap-ga--view-foo
    inherit_from: tap-google-analytics
    config:
      # key_file_location and start_date are inherited
      view_id: 123456
  - name: tap-ga--view-bar
    inherit_from: tap-google-analytics
    config:
      # key_file_location is inherited
      start_date: '2020-12-01T00:00:00Z' # start_date is overwritten
      view_id: 789012

Mapping Environment Variables

In Meltano, you can map environment variables as follows:

meltano.yml
environments:
  - name: dev
    env:
      MY_ENV_VAR: $MELTANO_PROJECT_ROOT/path/to/a/file.json

Specifying the Environment

You can specify the environment using the following methods:

  • Add the --environment option in the CLI command:
bash
$ meltano --environment=dev run tap-github target-csv
  • Set the MELTANO_ENVIRONMENT variable:
bash
$ export MELTANO_ENVIRONMENT=dev
$ meltano run tap-github target-csv

Additionally, you can specify the default environment by adding default_environment: staging in the meltano.yml file.

References

https://docs.meltano.com/concepts/environments

Ryusei Kakujo

researchgatelinkedingithub

Focusing on data science for mobility

Bench Press 100kg!