This repository contains common and extensible Prefect patterns to drive efficient workflows — we like to call these patterns our recipes!

Here you'll find starter code and more advanced example use cases.

• Getting Started

  • Subflows
  • Control Flow
  • Optimization
  • Notifications
  • Flow Run Observability
  • Parameters
  • Testing
  • Triggering Flow Runs
  • Deployments

• Diving Deeper

  • Data Engineering / DataOps
  • Azure Infrastructure
  • Legacy (Prefect 1.0)

• Contributing

• Issues & Bugs

• Recipe Requests

• Join the Discussion

• Thanks

• Getting Started With Subflows
• Run Multiple Subflows or Child Flows in Parallel
• Subflow with a Different Task Runner Than Parent Flow
• Create a Subflow and Block Until It's Completed

• Conditionally Stop a Task Run
• Ensure Tasks Immediately Fail If Upstream Task Fails
• Define State Dependencies Between Tasks

• Cache a Task Result To Prevent Recomputation

• Send Notifications with a Slack Webhook

• Interact with REST API
• Determine Whether a Flow Run Was Executed Ad Hoc or on a Schedule

• Use flow parameters

• Testing Flows, Subflows, and Tasks

• Triggering Flow Runs From a Deployment via API Call or From a Terminal Using Curl

• Deploy Flows to Run as a Local Process, Docker Container or a Kubernetes Job
• Deploy Flows to AWS
• Deploy Flows to GCP
• Deploy Flows to Azure

• Serverless Real-Time Data Pipelines on AWS with Prefect, ECS and GitHub Actions

• Setup Azure with Prefect
• Deploy Prefect Orion to an AKS Cluster with Azure Blob Storage
• Setup an Azure VM and Run the Prefect Agent

• Create Deployment with storage and infra Blocks on push to branch

• Register a Prefect Flow
• Run GraphQL Queries
• Airbyte Orchestration
• ETL with AWS S3 and Snowflake
• Use AWS Lambda for Event-Driven Flows
• Read Secrets into Prefect Cloud tenant
• Handle DBT Model Failures
• S3 Flow Storage on EKS

We're always looking for new contributions! You can add your Prefect 2.0 recipe and earn some swag in a few simple steps:

1. Look through the recipes to ensure your example is unique
2. Clone the prefect-recipes repo:

git clone [email protected]:PrefectHQ/prefect-recipes.git

3. Create and checkout a new branch:

git branch feat/recipe-name
git checkout feat/recipe-name

5. Add your code under the appropriate category, making sure it is reproducible and easy to understand.
6. Add your recipe to README.
7. Commit and push the code to your remote branch.
8. Create a PR 🤌

To report issues, typos, or link fixes, please open an issue. We appreciate it!

What are you interested in seeing examples of? Jot down your big idea here.

We use our Slack Community and Discourse to discuss all things Prefect-- such as FAQ, use cases and integrations. Join in the conversation 😄

Thank you for your contributions and efforts to improve prefect-recipes. We're glad to have you in our community!