Pathway

Pathway is a Python framework for building data pipelines that stay fresh: the same code runs on static files and live streams, with results updating incrementally as new data arrives. Python defines the transformations while a Rust engine built on differential dataflow executes them, which sidesteps the GIL and gives exactly-once consistency across batch and streaming. Connectors cover Kafka, SQL/NoSQL databases, REST APIs, and files, and templates exist for RAG and ETL pipelines that react to document changes in real time.

The project is strikingly popular, with over 62,000 GitHub stars and same-day commit activity, and it installs with a plain pip install on macOS and Linux. The license detail matters: it ships under the Business Source License 1.1, which is source-available rather than OSI-approved open source, so commercial users should read the license terms. For teams choosing between Flink/Spark complexity and ad-hoc scripts, Pathway is the pragmatic middle path for real-time Python data work.

1. Install: pip install -U pathway (macOS/Linux, Python 3.9+).
2. Write your pipeline in Python using Pathway tables (batch and streaming share code).
3. Connect sources: Kafka, databases, files, or REST.
4. Run locally, then deploy with Docker or Kubernetes.