Great Expectations
Open-source data quality framework: declare expectations about your data, validate them in pipelines and CI.
About
Great Expectations (GX) is the standard open-source framework for data validation: you declare Expectations — verifiable assertions like 'order_id is unique' or 'amount is never negative' — bundle them into suites, and run them against real data batches in pipelines, notebooks, or CI. It ships 300+ built-in expectations, SQL pushdown execution for large tables, auto-generated Data Docs (browsable HTML reports), and Airflow-native operators. GX Core is Apache 2.0 and free to self-host; GX Cloud is the hosted SaaS adding a shared UI, role-based access, anomaly detection, and AI-assisted rule suggestions (ExpectAI). Limitations to know: the Python-first API has a real learning curve, large-table validation needs SQL pushdown to scale, and validation is batch-oriented — no streaming checks. Legacy 0.x APIs were sunset, so old tutorials using ge.from_pandas() no longer apply.
What it can do
- 300+ built-in expectations; custom Python/SQL expectations; SQL pushdown execution; Data Docs HTML reports; checkpoints with actions (Slack alerts, quarantine); Airflow operator; GX Cloud hosted UI with ExpectAI rule suggestions.
Setup
pip install great_expectations, initialize a GX project, connect a datasource, define expectation suites in Python or YAML, and wire checkpoints into your pipeline.
Prerequisites
Python 3.9+; access to the data source (SQLAlchemy-supported databases, Spark, or Pandas for small data).
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: GX Core Apache 2.0 open source, 300+ built-in expectations, Data Docs, Airflow operator corroborated by practitioner guide and engineering ADR.; GX Cloud features (ExpectAI, anomaly detection, hosted backend) from official docs mirrors.; Limitations disclosed: Python-first learning curve, batch-only validation, legacy 0.x API sunset..
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.