Do you have to choose? — Many tools in the contemporary data landscape started to become increasingly polarized. On the one hand, there are products highly specialized in one specific area, such as data ingestion, transformation, scheduling, cataloging, experiment tracking, alerting, etc. On the opposite side of the spectrum, there are tools that attempt to re-bundle…

A guide to fully automated serverless real-time data pipelines — Most data platforms these days are still operated using batch processing. Even though streaming technology matured, building automated and reliable real-time data pipelines is still difficult and often requires a team of engineers to operate the underlying platform. But it doesn’t have to be that way. …

And why you may consider applying for this role at Prefect — There are many misconceptions about who a community engineer is. This blog post defines such a person as “someone who figures out how to help a community of people do what they want with what they have.” …

Let’s disambiguate the terms to understand workflow orchestration better — with a real-life analogy! — With the rise of the Modern Data Stack, many tools in the industry started positioning themselves as “data orchestrators” rather than “workflow orchestrators.” This article attempts to disambiguate the terms. …

Why we chose Discourse and how you can benefit from this new Prefect Community forum — We're excited to announce that we have launched Prefect Discourse! Discourse is a forum for online communities that provides an unmatched level of flexibility and customization. Going forward, we will use Discourse as a knowledge base that we can grow together as a community. It's a safe place to ask…

Introducing Monte Carlo data lineage tasks in Prefect — As recently announced, Prefect has a brand-new integration with Monte Carlo — a leading platform that adds observability features to your data warehouse. This hands-on post will dive into what Monte Carlo is and how to use it to add even more observability to your Prefect flows. …

How to pass runtime-specific parameter values to your data pipelines — Parametrization is one of the most critical features of any modern workflow orchestration solution. It allows you to dynamically overwrite parameter values for a given run without having to redeploy your workflow. Most orchestration frameworks provide rather limited functionality in that regard, such as only allowing to override global variables…

How to use Prefect and dbt Cloud with a Snowflake data warehouse — This is the third post in a series of articles about orchestrating ELT data pipelines with Prefect. The first post dealt with organizing a project and orchestrating flows in a local environment. The second one discussed deploying the ELT project to Snowflake, AWS EKS, and building a CI/CD process. In…