Why we chose Discourse and how you can benefit from this new Prefect Community forum — We're excited to announce that we have launched Prefect Discourse! Discourse is a forum for online communities that provides an unmatched level of flexibility and customization. Going forward, we will use Discourse as a knowledge base that we can grow together as a community. It's a safe place to ask…

Introducing Monte Carlo data lineage tasks in Prefect — As recently announced, Prefect has a brand-new integration with Monte Carlo — a leading platform that adds observability features to your data warehouse. This hands-on post will dive into what Monte Carlo is and how to use it to add even more observability to your Prefect flows. You’ll learn the…

How to pass runtime-specific parameter values to your data pipelines — Parametrization is one of the most critical features of any modern workflow orchestration solution. It allows you to dynamically overwrite parameter values for a given run without having to redeploy your workflow. Most orchestration frameworks provide rather limited functionality in that regard, such as only allowing to override global variables…

How to use Prefect and dbt Cloud with a Snowflake data warehouse — This is the third post in a series of articles about orchestrating ELT data pipelines with Prefect. The first post dealt with organizing a project and orchestrating flows in a local environment. The second one discussed deploying the ELT project to Snowflake, AWS EKS, and building a CI/CD process. In…

A flow of flows: a guide on how to deploy large-scale data pipelines to production — This article is the second in a series of tutorials about orchestrating ELT data pipelines. The first post demonstrated how to organize and orchestrate a variety of flows written by different teams and how to trigger those in the correct order using Prefect. This post builds on that by capturing…

Can a more flexible workflow orchestration system handle both paradigms? — Many believe that streaming technologies are the only way to achieve real-time analytics. Since most data workloads these days are orchestrated through batch-processing platforms, the real-time requirement forces data teams to adopt a new set of tools. However, maintaining two separate systems for batch processing and real-time streaming introduces additional…

How to manage dependencies between data pipelines — Workflow orchestration platforms have historically allowed managing task dependencies within individual data pipelines. While this is a good start, what if you have dependencies between data pipelines? Say you have some flows or directed acyclic graphs (DAGs) that ingest operational data from various sources into the staging area of your…