Meltano Environments
In Meltano, you can manage and switch environment variables according to your development or production environments. The definitions for multiple environments are described in meltano.yml
as follows:
meltano.yml
environments:
- name: prod
config:
plugins:
extractors:
- name: tap-github
config:
organizations: [Meltano]
select: ['*.*']
loaders:
- name: target-snowflake
config:
dbname: prod
warehouse: prod_wh
batch_size_rows: 100000
env:
SOME_PROD_ONLY_SETTING: abc
- name: dev
config:
plugins:
extractors:
- name: tap-github
config:
organizations: [MeltanoLabs]
select: ['repositories.*']
loaders:
- name: target-snowflake
config:
dbname: dev
warehouse: dev_wh
batch_size_rows: 1000
Environment Inheritance
If you want to use the same base plugin with different configurations, you can add a new plugin that inherits from an existing plugin. The new plugin inherits the configuration of the base plugin and can override it if necessary.
meltano.yml
.
.
.
plugins:
extractors:
- name: tap-google-analytics
variant: meltano
config:
key_file_location: client_secrets.json
start_date: '2020-10-01T00:00:00Z'
- name: tap-ga--view-foo
inherit_from: tap-google-analytics
config:
# key_file_location and start_date are inherited
view_id: 123456
- name: tap-ga--view-bar
inherit_from: tap-google-analytics
config:
# key_file_location is inherited
start_date: '2020-12-01T00:00:00Z' # start_date is overwritten
view_id: 789012
Mapping Environment Variables
In Meltano, you can map environment variables as follows:
meltano.yml
environments:
- name: dev
env:
MY_ENV_VAR: $MELTANO_PROJECT_ROOT/path/to/a/file.json
Specifying the Environment
You can specify the environment using the following methods:
- Add the
--environment
option in the CLI command:
bash
$ meltano --environment=dev run tap-github target-csv
- Set the
MELTANO_ENVIRONMENT
variable:
bash
$ export MELTANO_ENVIRONMENT=dev
$ meltano run tap-github target-csv
Additionally, you can specify the default environment by adding default_environment
: staging in the meltano.yml
file.
References
AlloyDB
Amazon Cognito
Amazon EC2
Amazon ECS
Amazon QuickSight
Amazon RDS
Amazon Redshift
Amazon S3
API
Autonomous Vehicle
AWS
AWS API Gateway
AWS Chalice
AWS Control Tower
AWS IAM
AWS Lambda
AWS VPC
BERT
BigQuery
Causal Inference
ChatGPT
Chrome Extension
CircleCI
Classification
Cloud Functions
Cloud IAM
Cloud Run
Cloud Storage
Clustering
CSS
Data Engineering
Data Modeling
Database
dbt
Decision Tree
Deep Learning
Descriptive Statistics
Differential Equation
Dimensionality Reduction
Discrete Choice Model
Docker
Economics
FastAPI
Firebase
GIS
git
GitHub
GitHub Actions
Google
Google Cloud
Google Search Console
Hugging Face
Hypothesis Testing
Inferential Statistics
Interval Estimation
JavaScript
Jinja
Kedro
Kubernetes
LightGBM
Linux
LLM
Mac
Machine Learning
Macroeconomics
Marketing
Mathematical Model
Meltano
MLflow
MLOps
MySQL
NextJS
NLP
Nodejs
NoSQL
ONNX
OpenAI
Optimization Problem
Optuna
Pandas
Pinecone
PostGIS
PostgreSQL
Probability Distribution
Product
Project
Psychology
Python
PyTorch
QGIS
R
ReactJS
Regression
Rideshare
SEO
Singer
sklearn
Slack
Snowflake
Software Development
SQL
Statistical Model
Statistics
Streamlit
Tabular
Tailwind CSS
TensorFlow
Terraform
Transportation
TypeScript
Urban Planning
Vector Database
Vertex AI
VSCode
XGBoost