Skip to main content

Store

SetupArrowConnect to DataArrowCreate ExpectationsArrowValidate Data

Overview​

Definition​

A Store is a connector to store and retrieve information about metadata in Great Expectations.

Features and promises​

Great Expectations supports a variety of Stores for different purposes, but the most common Stores are Expectation Stores, Validations Stores, Checkpoint Stores, Metric Stores, and Evaluation Parameter Stores. Data Docs Stores can also be configured for Data Doc Sites. Each of these Stores is tailored to a specific type of information.

  • Expectation Store: a connector to store and retrieve information about collections of verifiable assertions about data. These are Stores for Expectation Suites.
  • Validation Result Store: a connector to store and retrieve information about objects generated when data is Validated against an Expectation Suite.
  • Checkpoint Store: a connector to store and retrieve information about means for validating data in a production deployment of Great Expectations.
  • Evaluation Parameter Store: a connector to store and retrieve information about parameters used during Validation of an Expectation which reference simple expressions or previously generated Metrics.
  • Data Docs Store: a connector to store and retrieve information pertaining to Human readable documentation generated from Great Expectations metadata detailing Expectations, Validation Results, etc.
  • Metric Stores: a connector to store and retrieve information about computed attributes of data, such as the mean of a column. These differ from the Evaluation Parameter Store in how they are formatted. The data in a Metrics Store is intended to be used for generating reports and analyzing trends, rather than as Evaluation Parameter values.

Relationship to other objects​

Each type of Store is designed to interact with a specific subset of information, and thus interacts with a specific subset of objects in Great Expectations. However, all Stores can be listed and modified through your Data Context. For further information on how a given type of Store relates to other objects, please see the corresponding Store type's technical term page:

Use cases​

Setup

Setup

All Stores are configured during the Setup step of working with Great Expectations. For the most part, the default configurations will permit you to work with Great Expectations in a local environment. You can, however, configure your Stores to be hosted elsewhere.

Setup

Create Expectations

When creating Expectations, you will use Expectation Stores to store your Expectation SuitesA collection of verifiable assertions about data.. You may also use Validation Result Stores or Evaluation Parameter Stores to configure some ExpectationsA verifiable assertion about data. that require Evaluation ParametersA dynamic value used during Validation of an Expectation which is populated by evaluating simple expressions or by referencing previously generated metrics. as part of their definition, provided you have previously stored relevant MetricsA computed attribute of data such as the mean of a column. in those Stores and then come back to create a new Expectation Suite that references them.

Setup

Validate Data

When Validating data, you may store new CheckpointsThe primary means for validating data in a production deployment of Great Expectations. (or retrieve existing ones) from your Checkpoint Store. Checkpoints may also use Expectation Suites retrieved by an Expectation Store, and Expectations that are run by a Checkpoint may retrieve Metrics as input (Evaluation Parameters) from Validation ResultsGenerated when data is Validated against an Expectation or Expectation Suite. in a Validation Results Store, or from values stored in an Evaluation Parameter Store. Checkpoints may also write information about Validation Results into a Validation Results Store, or information about Metrics into an Evaluation Parameter Store (which is why they are also known as Metric Stores).

Features​

Metadata storage and retrieval​

Stores permit the configuration of where certain types of data and metadata will be stored and retrieved from.

Centralized access and convenient API​

Stores provide an API for Great Expectations to perform the retrieval and storage of information. This API (e.g. the Stores themselves) is always available from your Data Context.

API Basics​

How to access​

You can find a list of almost all of your Stores in your great_expectations.yml file under the stores key. Alternatively, from the root folder of your Data Context you can retrieve a list of Stores with a CLI command:

Console command
great_expectations store list

Data Docs Stores are the one exception to the above. They are instead configured under the data_docs_sites key in the great_expectations.yml file. Each entry under data_docs_sites will have a store_backend key. The information under store_backend will correspond to the Data Doc Store used by that site. Data Doc Stores are not listed by the great_expectations store list command.

Configuration​

Configuration of your Stores is located in your great_expectations.yml file under the stores key. Three stores are required: an Expectation Store, a Validation Result Store, and an Evaluation Parameter Store must always have a valid entry in the great_expectations.yml file. Additional Stores can be configured for uses such as Data Docs and Checkpoints (and a Checkpoint Store configuration entry will automatically be added if one does not exist when you attempt to save a Checkpoint for the first time.)

Note that unlike other Stores, your Data Docs Stores are configured under each individual site under the data_docs_sites key in the great_expectations.yml file.

More details​

For more information about a specific type of Store, please see the Store type's technical term definition page: