Prefect 2.5 Makes Modular and Serverless Dataflows Easier Than Ever

Prefect 2.5 adds new serverless infrastructure capabilities, batch deletion from the UI, and utilities to orchestrate flow runs from deployments

Prefect as a bot automating serverless infrastructure deployments while still observing what’s happening in the dataflow universe and carefully tracking dependencies, even dependencies between deployments.

Managing infrastructure and building deployment processes for dataflows can be challenging. Prefect helps solve that problem by providing flexible building blocks that allow you to easily provision and maintain the underlying runtime environment, including several serverless options. We are excited to share that Prefect now supports Google Cloud Run (and now also Azure Container Instances) for deploying dataflows as serverless containers.

As another serverless enhancement, we’ve recently added ECSTask customizations to allow setting (and overriding) custom AWS VPCs, subnets, security groups, and more.

Lastly, Prefect 2.5 ships with utilities to support a commonly requested dataflow design pattern that involves triggering flow runs from deployments, waiting for their completion, and tracking dependencies across deployments, thereby further improving collaboration between teams and code dependency management.

We’ll cover those new features in this post. To learn more about all improvements, check the release notes and…