prefecthq / prefect-recipes Goto Github PK

Create descriptive readme for secret import

💻

• Would you like to work on this feature?

What problem are you trying to solve?

Showcase benefits of prefect-email to your workflow

https://prefecthq.github.io/prefect-email/

Describe the solution you'd like

One or more recipes

Describe alternatives you've considered

n/a

Documentation, Adoption, Migration Strategy

No response

💻

• Would you like to work on this feature?

What problem are you trying to solve?

Showcase benefits of prefect-firebolt to your workflow

https://prefecthq.github.io/prefect-firebolt/

Describe the solution you'd like

One or more recipes

Describe alternatives you've considered

n/a

Documentation, Adoption, Migration Strategy

No response

💻

• Would you like to work on this feature?

What problem are you trying to solve?

Show off the benefits of the prefect-azure collection to your workflow.
https://prefecthq.github.io/prefect-azure/

Describe the solution you'd like

One or more recipes.

Describe alternatives you've considered

n/a

Documentation, Adoption, Migration Strategy

No response

https://docs.airbyte.io/deploying-airbyte/on-kubernetes

Optionally deploy as a stand alone cluster or with the prefect agent installed as well

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-instance.html

Create a single cloudformation template + bootstrap script to deploy the prefect agent onto an ec2 instance

💻

• Would you like to work on this feature?

What problem are you trying to solve?

Showcase benefits of prefect-metricflow

https://alessandrolollo.github.io/prefect-metricflow/

Describe the solution you'd like

One or more recipes

Describe alternatives you've considered

n/a

Documentation, Adoption, Migration Strategy

No response

💻

• Would you like to work on this feature?

What problem are you trying to solve?

Show off the benefits of the prefect-azure collection to your workflow.

https://alessandrolollo.github.io/prefect-cubejs/

Describe the solution you'd like

One or more recipes

Describe alternatives you've considered

n/a

Documentation, Adoption, Migration Strategy

No response

As an engineer, i want to ensure that my airbyte configuration data is not lost in the event of an instance failure.

💻

• Would you like to work on this feature?

What problem are you trying to solve?

Showcase the benefits of prefect-fugue in your workflow
https://fugue-project.github.io/prefect-fugue/

Describe the solution you'd like

One or more recipes

Describe alternatives you've considered

n/a

Documentation, Adoption, Migration Strategy

No response

As a solution engineer I want to have examples of popular CI tool gitlab CI registering prefect flows.

As a solution engineer I want to have examples of popular CI tool github actions registering prefect flows.

💻

• Would you like to work on this feature?

What problem are you trying to solve?

Showcase prefect-jupyter in your workflow

https://prefecthq.github.io/prefect-jupyter/

Describe the solution you'd like

One or more examples

Describe alternatives you've considered

n/a

Documentation, Adoption, Migration Strategy

No response

💻

• Would you like to work on this feature?

What problem are you trying to solve?

Showcase the benefits of using prefect-dbt.

https://prefecthq.github.io/prefect-dbt/

Describe the solution you'd like

One or more recipes

Describe alternatives you've considered

n/a

Documentation, Adoption, Migration Strategy

No response

💻

• Would you like to work on this feature?

What problem are you trying to solve?

Showcase the benefits of using prefect-databricks.
prefect-databricks

Describe the solution you'd like

One or more recipes

Describe alternatives you've considered

n/a

Documentation, Adoption, Migration Strategy

No response

As a solution engineer I want to have examples of popular CI tool travis registering prefect flows.

• template supports deployment from root of directory as well as sub directories within a gitlab repository
• template supports terraform validation, plan, apply and destroy stages

💻

• Would you like to work on this feature?

What problem are you trying to solve?

Showcase benefits of prefect-gcp in your workflow

https://prefecthq.github.io/prefect-gcp/

Describe the solution you'd like

One or more recipes

Describe alternatives you've considered

n/a

Documentation, Adoption, Migration Strategy

No response

user_data = templatefile("${path.module}/airbyte-install.sh",
{
region = "${data.aws_region.current.name}"
linux_type = "${var.linux_type}"
}
)

https://docs.airbyte.io/deploying-airbyte/on-aws-ec2

👋 Hey CS!

I would love to see/help with a terraform setup for deploying a "production-ready" Orion instance now that we have a basic k8s setup. As a consumer of this Orion instance, I would be particularly interested in:

• A setup that utilizes a managed Postgres setup to preserve run history
• A setup that supports horizontal scaling
• Goal Task Runs / second ~ 20

I would be very interested in paring on this 😄

💻

• Would you like to work on this feature?

What problem are you trying to solve?

Showcase the benefits of prefect-github in your workflow

https://prefecthq.github.io/prefect-github/

Describe the solution you'd like

One or more recipes

Describe alternatives you've considered

n/a

Documentation, Adoption, Migration Strategy

No response

💻

• Would you like to work on this feature?

What problem are you trying to solve?

Show off the benefits of using prefect-dask
https://prefecthq.github.io/prefect-dask/

Describe the solution you'd like

One or more recipes

Describe alternatives you've considered

n/a

Documentation, Adoption, Migration Strategy

No response

💻

• Would you like to work on this feature?

What problem are you trying to solve?

When implementing large workflows at scale we often only want failed tasks to be re-run in the event of a failure as this preserves resources for other jobs in the infrastructure where our workflows are running.

Describe the solution you'd like

Implementing cacheing on tasks and/or using results storage can help alleviate the need to re-run successful tasks in large workflows (caching docs). It would be helpful for users to have a template when implementing this for large workloads, in the same vein including reporting/logging for where the tasks may be failing and why.

Results (TBD)(currently uses local storage)

1.0 potential reference: https://discourse.prefect.io/t/how-to-resume-mapped-task-runs-from-failure-at-scale-or-limit-the-amount-of-allowed-runs-that-may-fail/715

Describe alternatives you've considered

None at the moment.

Documentation, Adoption, Migration Strategy

No response

https://docs.airbyte.io/deploying-airbyte/on-gcp-compute-engine

• template supports deployment from root of directory as well as sub directories within a gitlab repository
• template supports terraform validation, plan, apply and destroy stages

• autoscaling group monitors health of agent and responds to unhealthy events by building and registering a new agent

💻

• Would you like to work on this feature?

What problem are you trying to solve?

A recipe detailing use of ECSTask block as users would like to see it in action.

Describe the solution you'd like

Take a user from writing the flow through deployment!

Resources

Release Notes:
https://medium.com/the-prefect-blog/prefect-2-4-0-release-adds-support-for-aws-ecs-and-improves-deployment-cli-for-ci-cd-9582458e1e81

Docs/Prereqs:
https://docs.prefect.io/concepts/infrastructure/?h=ecstas#ecstask

Describe alternatives you've considered

n/a

Documentation, Adoption, Migration Strategy

No response

💻

• Would you like to work on this feature?

What problem are you trying to solve?

Showcase benefits of prefect-hex in your workflow

https://prefecthq.github.io/prefect-hex/

Describe the solution you'd like

One or more recipes

Describe alternatives you've considered

n/a

Documentation, Adoption, Migration Strategy

No response

Several issues I noticed when trying to deploy Orion server from the Helm chart.

• Mismatch between values.yaml and deployment.yaml postgresql configuration. Deployment.yaml line 9 references .Values.api.postgres.enabled, but values file contains a "postgresql" section and no "enabled" value.
• Mismatch between _helpers.tpl and values.yaml. Helpers omits the "api" from references to postgresql related values. For example, ".Values.postgresql" should be replaced with ".Values.api.postgresql"
• .Values.api.postgresql.existingSecret doesn't appear to be respected. Even when populated, the deployment is trying to overwrite the existing secret in the namespace and generating an error. Once I deleted the existing secret, the deployment was successful.

💻

• Would you like to work on this feature?

What problem are you trying to solve?

Showcase prefect-great-expectations in your workflow

https://prefecthq.github.io/prefect-great-expectations/

Describe the solution you'd like

One or more recipes

Describe alternatives you've considered

n/a

Documentation, Adoption, Migration Strategy

No response

💻

• Would you like to work on this feature?

What problem are you trying to solve?

As someone who is new to Pydantic, I am curious to see how using it can improve my workflows.

Describe the solution you'd like

not sure

Describe alternatives you've considered

not sure

Documentation, Adoption, Migration Strategy

No response

As a solution engineer I want consistent formatting for all code committed to recipe repos.

Create github action that runs black, flake8 and terraform checks to ensure that pre-commit formatting standards have been adhered to.

https://docs.airbyte.io/deploying-airbyte/on-kubernetes

Optionally deploy as a stand alone cluster or with the prefect agent installed as well

💻

• Would you like to work on this feature?

What problem are you trying to solve?

Event-Driven-Flows.
Dynamically created deployments.
Flows that use .context

Describe the solution you'd like

Recipes for this

Describe alternatives you've considered

Using the 1.0 guides

Documentation, Adoption, Migration Strategy

No response