Modular Data Stack with dbt Cloud Prefect block — This short post will walk you through how to set up dbt Cloud jobs and orchestrate those with Prefect. It assumes that you have already signed up for dbt Cloud and know how to use dbt. dbt Cloud setup First, you need to retrieve the dbt Cloud account ID and create an API…

And many more enhancements! — Prefect 2.7 ships with robust flow run cancellation, pause and resume functionality allowing manual approval from the UI, logging of print statements, and agent concurrency limits. We’ll cover the enhancements in this post. As always, check the release notes and our documentation to learn more. Flow run cancellation We’re excited to announce a…

From Prefect 1 or another orchestration framework — At Prefect, we have an ambassador program called Club42. One of our club members recently asked: why should I migrate to Prefect 2? If you are using Prefect 1 or another orchestrator, here is a list of reasons you should consider migrating to Prefect 2. #1 Just Python Prefect 2 embraces the philosophy…

Coordinate dataflow across multiple domains and teams, and orchestrate your data platform with a parent flow in Prefect — This is a continuation of a series of articles about building a data platform with Prefect, dbt, and Snowflake. If you’re new to this series, check out the summary linking to previous posts. This demo will discuss how you can manage dependencies across multiple teams using a parent flow. First…

Coordinate ML, analytics & BI reporting data platform workflows with Prefect, dbt, Snowflake, and modular building blocks — Welcome to another article about building a data platform with Prefect, dbt, and Snowflake. If you’re new to this series, check out the summary linking to previous posts. This post will cover processes after data ingestion and transformation, including ML, analytics, and reporting flows. …

Failure handling with alert notifications and retries for data ingestion and dbt transformations in your Modern Data Stack — This is a continuation of a series of articles about building a data platform with Prefect, dbt, and Snowflake. If you’re new to this series, have a look at the summary linking to previous posts. …

Scheduling, data ingestion, and backfilling implemented with modular building blocks and simple deployment patterns — This is a continuation of a series of articles about building a data platform with Prefect, dbt, and Snowflake. If you’re new to this series, check out this summary linking to previous posts. …

Build a data platform with Snowflake & dbt, and use Prefect to observe and coordinate your data stack — Data platform engineers are increasingly crossing the boundaries of teams, business domains, systems, repositories, and data pipelines. Running things on a schedule alone often doesn’t cut it anymore. Some dataflows are real-time, event-driven, or triggered ad-hoc. …