Pyper

Concurrent Python made simple

Test Coverage Package version Supported Python versions

Introduction

Concurrency and parallelism are really hard to get right.

The Python package space has great support for achieving

  • concurrency in web applications (Django, Flask, FastAPI, etc.)
  • parallelism for distributed computing (Ray, Dask, etc.)

However, the solutions for general-purpose data processing are less established.

Pyper aims to offer a flexible framework for concurrent and parallel data-processing in Python

It is designed with the following goals in mind:

  • Unified API: Combine threads, processes and async code using one intuitive pattern
  • Functional Paradigm: Data pipelines compose together straightforwardly as functions
  • Lazy Execution: Built from the ground up to support generators, and provides mechanisms for fine-grained memory control
  • Error Handling: Data flows fail fast, even in long-running threads, and propagate their errors cleanly
  • Complex Data Flows: Data pipelines support branching/joining data flows, as well as sharing contexts/resources between tasks

In addition, Pyper enables developers to write code in an extensible way that can be integrated naturally with other frameworks like those aforementioned.

Installation

Install the latest version using pip:

$ pip install python-pyper

Where Next?

  • Check out the 📖 User Guide to get started with Pyper

  • See some 🎯 Examples of possible use cases