Skip to main content

How to configure credentials

This guide will explain how to configure your great_expectations.yml project config to populate credentials from either a YAML file or a secret manager.

If your Great Expectations deployment is in an environment without a file system, refer to How to instantiate a Data Context without a yml file for credential configuration examples.

Prerequisites: This how-to guide assumes you have:

Steps​

1. Save credentials and config​

Decide where you would like to save the desired credentials or config values - in a YAML file, environment variables, or a combination - then save the values.

In most cases, we suggest using a config variables YAML file. YAML files make variables more visible, easily editable, and allow for modularization (e.g. one file for dev, another for prod).

note
  • In the great_expectations.yml config file, environment variables take precedence over variables defined in a config variables YAML
  • Environment variable substitution is supported in both the great_expectations.yml and config variables config_variables.yml config file.

If using a YAML file, save desired credentials or config values to great_expectations/uncommitted/config_variables.yml or another YAML file of your choosing:

my_postgres_db_yaml_creds:
  drivername: postgresql
  host: localhost
  port: 5432
  username: postgres
  password: ${MY_DB_PW}
  database: postgres
note
  • If you wish to store values that include the dollar sign character $, please escape them using a backslash \ so substitution is not attempted. For example in the above example for Postgres credentials you could set password: pa\$sword if your password is pa$sword. Say that 5 times fast, and also please choose a more secure password!
  • When you save values via the CLICommand Line Interface, they are automatically escaped if they contain the $ character.
  • You can also have multiple substitutions for the same item, e.g. database_string: ${USER}:${PASSWORD}@${HOST}:${PORT}/${DATABASE}

If using environment variables, set values by entering export ENV_VAR_NAME=env_var_value in the terminal or adding the commands to your ~/.bashrc file:

export POSTGRES_DRIVERNAME=postgresql
export POSTGRES_HOST=localhost
export POSTGRES_PORT=5432
export POSTGRES_USERNAME=postgres
export POSTGRES_PW=
export POSTGRES_DB=postgres
export MY_DB_PW=password

2. Set config_variables_file_path​

If using a YAML file, set the config_variables_file_path key in your great_expectations.yml or leave the default.

config_variables_file_path: uncommitted/config_variables.yml

3. Replace credentials with placeholders​

Replace credentials or other values in your great_expectations.yml with ${}-wrapped variable names (i.e. ${ENVIRONMENT_VARIABLE} or ${YAML_KEY}).

datasources:
  my_postgres_db:
    class_name: Datasource
    module_name: great_expectations.datasource
    execution_engine:
      module_name: great_expectations.execution_engine
      class_name: SqlAlchemyExecutionEngine
      credentials: ${my_postgres_db_yaml_creds}
    data_connectors:
      default_inferred_data_connector_name:
        class_name: InferredAssetSqlDataConnector
  my_other_postgres_db:
    class_name: Datasource
    module_name: great_expectations.datasource
    execution_engine:
      module_name: great_expectations.execution_engine
      class_name: SqlAlchemyExecutionEngine
      credentials:
        drivername: ${POSTGRES_DRIVERNAME}
        host: ${POSTGRES_HOST}
        port: ${POSTGRES_PORT}
        username: ${POSTGRES_USERNAME}
        password: ${POSTGRES_PW}
        database: ${POSTGRES_DB}
    data_connectors:
      default_inferred_data_connector_name:
        class_name: InferredAssetSqlDataConnector

Additional Notes​

  • The default config_variables.yml file located at great_expectations/uncommitted/config_variables.yml applies to deployments created using great_expectations init.
  • To view the full script used in this page, see it on GitHub: how_to_configure_credentials.py