dlt
Open-source Python library that turns messy sources into clean datasets.
About
dlt (data load tool) is an open-source Python library for building data pipelines from REST APIs, SQL databases, files, and Python structures into warehouses and lakes. It infers and evolves schemas automatically, normalizes nested data, and automates pipeline maintenance with incremental loading, schema evolution, and data contracts.
With 200+ verified source connectors and destinations from Postgres and DuckDB to ClickHouse, S3, and Parquet, dlt is pure Python with no JVM and no heavy infrastructure: it runs in a notebook, a CI job, Airflow, serverless functions, or as a Dagster asset via the native dagster-dlt integration. The docs and workflows are built to be agent- and LLM-friendly, with an LLM-native workflow that takes pipeline code to data in a notebook across thousands of sources.
What it can do
- Declarative pipelines from APIs, databases, and files in pure Python
- Automatic schema inference, evolution, and nested data normalization
- Incremental loading with state management and data contracts
- LLM-native workflow plus native Dagster integration
Setup
pip install dlt, pick a verified source or define your own with the REST client, and run the pipeline to your destination of choice.
Prerequisites
Python 3.9+; credentials for your sources and a destination warehouse or lake.
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 98; Trending score 78/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.