2022-06-09

How to Use Meltano Plugins

Introduction

Meltano project plugins are defined in the meltano.yml project file and installed within the .meltano directory. These plugins can be managed using various CLI commands and the user interface (UI).

Adding Plugins

You can add plugins by executing the meltano add command or by directly editing the meltano.yml file and then running the meltano install command.

Discoverable Plugins

The Extractors and Loaders supported by Meltano (which can be discovered using meltano add) are referred to as "Discoverable plugins" in the official documentation. Discoverable plugins can be added using the meltano add command.

bash

$ meltano add extractor tap-gitlab
$ meltano add loader target-postgres
$ meltano add transformer dbt
$ meltano add orchestrator airflow

When you execute the above command, the plugin section in the meltano.yml file will look as follows:

meltano.yml

plugins:
  extractors:
  - name: tap-gitlab
    variant: meltano
    pip_url: git+https://gitlab.com/meltano/tap-gitlab.git
  loaders:
  - name: target-postgres
    variant: datamill-co
    pip_url: singer-target-postgres
  transformer:
  - name: dbt
    pip_url: dbt
  orchestrators:
  - name: airflow
    pip_url: apache-airflow

For the same Extractor or Loader, there may be multiple variants, which are referred to as "variants" in the official documentation. When you execute meltano add without specifying a variant, Meltano automatically selects the recommended variant.

To specify a variant, use the following command:

bash

$ meltano add loader target-postgres --variant=transferwise

If you want to install multiple variants, execute commands like the following:

bash

$ meltano add loader target-snowflake --variant=transferwise --as target-snowflake--transferwise
$ meltano add loader target-snowflake --variant=meltano --as target-snowflake--meltano

Custom Plugins

If the desired plugin is not available as a Discoverable plugin, you can add any Singer tap or Singer target by adding the --custom option to the meltano add command. Follow the official documentation to add an Extractor that is not present in the Discoverable plugins.

bash

$ meltano add --custom extractor tap-covid-19

When you execute the above command, you will be prompted with interactive questions.

Adding new custom extractor with name 'tap-covid-19'...

Specify the plugin's namespace, which will serve as the:
- identifier to find related/compatible plugins
- default database schema (`load_schema` extra),
  for use by loaders that support a target schema

Hit Return to accept the default: plugin name with underscores instead of dashes

(namespace) [tap_covid_19]: tap_covid_19

Specify the plugin's `pip install` argument, for example:
- PyPI package name:
	tap-covid-19
- Git repository URL:
	git+https://gitlab.com/meltano/tap-covid-19.git
- local directory, in editable/development mode:
	-e extract/tap-covid-19
- 'n' if using a local executable (nothing to install)

Default: plugin name as PyPI package name

(pip_url) [tap-covid-19]: -e extract/tap-covid-19

Specify the plugin's executable name

Default: name derived from `pip_url`

(executable) [tap-covid-19]: tap-covid-19

Specify the tap's supported Singer features (executable flags), for example:
	`catalog`: supports the `--catalog` flag
	`discover`: supports the `--discover` flag
	`properties`: supports the `--properties` flag
	`state`: supports the `--state` flag

To find out what features a tap supports, reference its documentation or try one
of the tricks under [how to test a tap](/contribute/plugins#how-to-test-a-tap).

Multiple capabilities can be separated using commas.

Default: no capabilities

(capabilities) [[]]: catalog,discover,state

Specify the tap's supported settings (`config.json` keys)

Multiple setting names (keys) can be separated using commas.

A setting kind can be specified alongside the name (key) by using the `:` delimiter,
e.g. `port:integer` to set the kind `integer` for the name `port`

Supported setting kinds:
string | integer | boolean | date_iso8601 | email | password | oauth | options | file | array | object | hidden

- Credentials and other sensitive setting types should use the password kind.
- If not specified, setting kind defaults to string.
- Nested properties can be represented using the `.` separator, e.g. `auth.username` for `{ "auth": { "username": value } }`.
- To find out what settings a tap supports, reference its documentation.

Default: no settings

(settings) [[]]: api_token:password,user_agent:string,start_date:date_iso8601
Added extractor 'tap-covid-19' to your Meltano project

Installing extractor 'tap-covid-19'...
Installed extractor 'tap-covid-19'

The plugins section in the meltano.yml file will have the tap-covid-19 Extractor added.

meltano.yml

plugins:
  extractors:
  - name: tap-covid-19
    namespace: tap_covid_19
    pip_url: tap-covid-19
    executable: tap-covid-19
    capabilities:
    - catalog
    - discover
    - state
    settings:
    - name: api_token
    - name: user_agent
    - name: start_date

Plugin Inheritance

To inherit from a Discoverable plugin and install it with a different name, add the --inherit-from or --as option to the command.

bash

$ meltano add extractor tap-postgres--billing --inherit-from tap-postgres
$ meltano add extractor tap-postgres --as tap-postgres--billing

The plugins section in the meltano.yml file will look as follows:

meltano.yml

plugins:
  extractors:
  - name: tap-postgres--billing
    inherit_from: tap-postgres
    variant: transferwise
    pip_url: pipelinewise-tap-postgres

Specifying Specific Plugin Versions

If you want to use a specific plugin version, specify the pip_url in the meltano.yml file as follows:

meltano.yml

# Before:
pip_url: git+https://gitlab.com/meltano/tap-gitlab.git
pip_url: git+https://github.com/adswerve/target-bigquery.git

# After:
pip_url: git+https://gitlab.com/meltano/tap-gitlab.git@v0.9.11
pip_url: git+https://github.com/adswerve/target-bigquery.git@v0.10.2

# Alternatively:
pip_url: git+https://gitlab.com/meltano/tap-gitlab.git@2657b89e8896face4ce320a03b8413bbc196cec9
pip_url: git+https://github.com/adswerve/target-bigquery.git@3df97b951b7eebdfa331a1ff570f1fe3487d632f

Removing Plugins

You can remove plugins using the meltano remove command.

bash

$ meltano remove extractor tap-github
$ meltano remove loader target-csv target-bigquery

Using Custom Forked Plugins

If you want to use a forked plugin repository, modify the value of the pip_url in the plugins section of the meltano.yml file and then execute the meltano install command.

meltano.yml

.
.
.
plugins:
   extractors:
   - name: tap-gitlab
     variant: meltano
     pip_url: git+https://github.com/meltano/tap-gitlab.git@ref-name #この行を修正する

bash

$ meltano install extractor tap-github

If the plugin repository is private, you can handle it in two ways:

Placing the .netrc file in the home directory

.netrc

  machine <hostname> # e.g. gitlab.com or github.com
  login <username>
  password <personal-access-token-or-password>

Authenticating with SSH

  pip_url: git+ssh://git@gitlab.com/meltano/tap-gitlab.git

Changing Variants

If you have an existing variant and want to switch to a different variant, you can either add a new variant or override the existing one.

Here, we will demonstrate how to override an existing variant. Modify the plugins section in the meltano.yml file as follows:

meltano.yml

- plugins:
-   loaders:
-   - name: target-postgres
-     variant: datamill-co
-     pip_url: singer-target-postgres

+ plugins:
+   loaders:
+   - name: target-postgres
+     variant: meltano
+     pip_url: git+https://github.com/meltano/target-postgres.git # Optional

Next, run the meltano install command:

bash

$ meltano install loader target-postgres

Different variants often have different configuration values. Modify the meltano.yml file as follows:

meltano.yml

- config:
-   postgres_host: postgres.example.com
-   postgres_port: 5432
-   postgres_username: my_user
-   postgres_database: my_database

+ config:
+   host: postgres.example.com
+   port: 5432
+   user: my_user
+   dbname: my_database

If you have old configuration values, you can set new values using the following command:

bash

$ meltano config target-postgres unset postgres_password
$ meltano config target-postgres set password my_password

discovery.yml File

By default, Meltano adds plugins by referring to the discovery.yml file located at https://discovery.meltano.com/discovery.yml. If you want to create plugins that are not present in this discovery.yml file, you usually add the --custom option and execute the add command to input the configuration interactively. However, if you create your own discovery.yml file, you can avoid interactive configuration by running the command export MELTANO_DISCOVERY_URL=<YOUR discovery.yml PATH>.