Prefect 2.5 Makes Modular and Serverless Dataflows Easier Than Ever
Prefect 2.5 adds new serverless infrastructure capabilities, batch deletion from the UI, and utilities to orchestrate flow runs from deployments
--
Managing infrastructure and building deployment processes for dataflows can be challenging. Prefect helps solve that problem by providing flexible building blocks that allow you to easily provision and maintain the underlying runtime environment, including several serverless options. We are excited to share that Prefect now supports Google Cloud Run (and now also Azure Container Instances) for deploying dataflows as serverless containers.
As another serverless enhancement, we’ve recently added ECSTask customizations to allow setting (and overriding) custom AWS VPCs, subnets, security groups, and more.
Lastly, Prefect 2.5 ships with utilities to support a commonly requested dataflow design pattern that involves triggering flow runs from deployments, waiting for their completion, and tracking dependencies across deployments, thereby further improving collaboration between teams and code dependency management.
We’ll cover those new features in this post. To learn more about all improvements, check the release notes and our documentation.
Going with the (serverless) flow: Google Cloud Run
If you are not familiar with Prefect Deployments, they allow you to turn any flow into an API-managed entity. The key components of deployments are storage and infrastructure blocks that make it easy to configure (even from the UI) where your code lives and where it should be executed at runtime.
We’ve recently added one more infrastructure block to the mix: CloudRunJob. It allows you to deploy your flows as serverless containers to GCP, as well as to spin up arbitrary containerized tasks within that platform.
What are blocks? They combine stateful information (including secrets) with extra capabilities to provide you with flexible building blocks applicable to various scenarios. You can leverage the same block to configure where your flow runs should be deployed, and you can leverage…
