Skip to main content

How to connect to data on GCS using Pandas

This guide will help you connect to your data stored on GCS using Pandas. This will allow you to ValidateThe act of applying an Expectation Suite to a Batch. and explore your data.

Prerequisites: This how-to guide assumes you have:

Steps​

1. Choose how to run the code in this guide​

Get an environment to run the code in this guide. Please choose an option below.

If you use the Great Expectations CLICommand Line Interface, run this command to automatically generate a pre-configured Jupyter Notebook. Then you can follow along in the YAML-based workflow below:

great_expectations datasource new

2. Instantiate your project's DataContext​

Import these necessary packages and modules.

from ruamel import yaml

import great_expectations as gx
from great_expectations.core.batch import Batch, BatchRequest, RuntimeBatchRequest

Load your DataContext into memory using the get_context() method.

context = gx.get_context()

3. Configure your Datasource​

Great Expectations provides two types of DataConnectors classes for connecting to GCS: InferredAssetGCSDataConnector and ConfiguredAssetGCSDataConnector

  • An InferredAssetGCSDataConnector utilizes regular expressions to infer data_asset_names by evaluating filename patterns that exist in your bucket. This DataConnector, along with a RuntimeDataConnector, is provided as a default when utilizing our Jupyter Notebooks.
  • A ConfiguredAssetGCSDataConnector requires an explicit listing of each DataAsset you want to connect to. This allows for more granularity and control than its Inferred counterpart but also requires a more complex setup.

As the InferredAssetDataConnectors have fewer options and are generally simpler to use, we recommend starting with them.

We've detailed example configurations for both options in the next section for your reference.

Authentication

It is also important to note that GCS DataConnectors support various methods of authentication. You should be aware of the following options when configuring your own environment:

  • gcloud command line tool / GOOGLE_APPLICATION_CREDENTIALS environment variable.
    • This is the default option and what is used throughout this guide.
  • Passing a filename argument to the optional gcs_options dictionary.
    • This argument should contain a specific filepath that leads to your credentials JSON.
    • This method utilizes google.oauth2.service_account.Credentials.from_service_account_file under the hood.
  • Passing an info argument to the optional gcs_options dictionary.
    • This argument should contain the actual JSON data from your credentials file in the form of a string.
    • This method utilizes google.oauth2.service_account.Credentials.from_service_account_info under the hood.

Please note that if you use the filename or info options, you must supply these options to any GX objects that interact with GCS (i.e. PandasExecutionEngine). The gcs_options dictionary is also responsible for storing any **kwargs you wish to pass to the GCS storage.Client() connection object (i.e. project)

For more details regarding storing credentials for use with Great Expectations see: How to configure credentials

For more details regarding authentication, please visit the following:

Using these example configurations, add in your GCS bucket and path to a directory that contains some of your data:

The below configuration is representative of the default setup you'll see when preparing your own environment.

datasource_yaml = rf"""
name: my_gcs_datasource
class_name: Datasource
execution_engine:
    class_name: PandasExecutionEngine
data_connectors:
    default_runtime_data_connector_name:
        class_name: RuntimeDataConnector
        batch_identifiers:
            - default_identifier_name
    default_inferred_data_connector_name:
        class_name: InferredAssetGCSDataConnector
        bucket_or_name: 
        prefix: 
        default_regex:
            pattern: (.*)\.csv
            group_names:
                - data_asset_name
"""

Run this code to test your configuration.

context.add_datasource(**yaml.load(datasource_yaml))

If you specified a GCS path containing CSV files you will see them listed as Available data_asset_names in the output of test_yaml_config().

Feel free to adjust your configuration and re-run test_yaml_config() as needed.

4. Save the Datasource configuration to your DataContext​

Save the configuration into your DataContext by using the add_datasource() function.

context.add_datasource(**yaml.load(datasource_yaml))

5. Test your new Datasource​

Verify your new DatasourceProvides a standard API for accessing and interacting with data from a wide variety of source systems. by loading data from it into a ValidatorUsed to run an Expectation Suite against data. using a Batch RequestProvided to a Datasource in order to create a Batch..

Add the GCS path to your CSV in the path key under runtime_parameters in your RuntimeBatchRequest.

Please note we support the following format for GCS URL's: gs://<BUCKET_OR_NAME>/<BLOB>.

batch_request = RuntimeBatchRequest(
    datasource_name="my_gcs_datasource",
    data_connector_name="default_runtime_data_connector_name",
    data_asset_name="",  # this can be anything that identifies this data_asset for you
    runtime_parameters={"path": ""},  # Add your GCS path here.
    batch_identifiers={"default_identifier_name": "default_identifier"},
)

Then load data into the Validator.

context.create_expectation_suite(
    expectation_suite_name="test_suite", overwrite_existing=True
)
validator = context.get_validator(
    batch_request=batch_request, expectation_suite_name="test_suite"
)
print(validator.head())

🚀🚀 Congratulations! 🚀🚀 You successfully connected Great Expectations with your data.

Additional Notes​

To view the full scripts used in this page, see them on GitHub:

To review the source code of these DataConnectors, also visit GitHub: