How to Use Prefect and Monte Carlo to Achieve More Reliable Data Pipelines

Introducing Monte Carlo data lineage tasks in Prefect

Photo by Frederick Marschall on Unsplash

As recently announced, Prefect has a brand-new integration with Monte Carlo — a leading platform that adds observability features to your data warehouse. This hands-on post will dive into what Monte Carlo is and how to use it to add even more observability to your Prefect flows. You’ll learn the similarities and differences between the tools and why using both in tandem can be beneficial.

Table of contents:
· What is Monte Carlo?
· How is Monte Carlo different from Prefect?
· The problem that Prefect's integration for Monte Carlo can solve
· Prefect tasks for Monte Carlo
· Demo time!
  ∘ Adding nodes and edges to the lineage graph
  ∘ Adding standalone nodes and tags to the lineage graph
  ∘ Querying Monte Carlo resources
· Next steps

What is Monte Carlo?

Monte Carlo is an end-to-end, fully automated data observability platform that can assess how your data warehouse operates, while also giving users the ability to set custom rules and thresholds to meet the needs of their data SLAs.

By continuously processing the outputs of your data warehouse (such as query logs, table schemas, execution…