…and many enhancements based on user feedback! — We’ve just released Prefect 2.4.1. It includes enhancements that significantly improve the user experience around notifications and blocks, as well as deploying Prefect agents to Kubernetes. Apart from that, this release ships with some critical behind-the-scenes changes that we’ll briefly cover at the end. Improved notifications experience Sending alerts on failed or completed…

Orchestrate serverless on AWS with Python and Prefect — Amazon Elastic Container Service (ECS) is a platform for managing containers at any scale. The simple task definition API allows you to start containerized workloads both individually and as a continuously-running service. …

And how to define state and data dependencies in your Prefect data pipelines — Prefect is a coordination plane for the constantly evolving world of data. Using Prefect, you can orchestrate any dataflow and observe everything else happening around it. …

CI/CD, Scheduling, Parametrization and Serverless AWS ECS Deployments are now even easier — Prefect 2.4.0 ships with enhancements that further simplify your automated deployment patterns, parametrization, and scheduling. It adds the highly requested infrastructure block for AWS ECS, allowing you to easily deploy your flows as serverless containers running on AWS. This post will briefly demonstrate those new features. …

When and when not to schedule your workflows and why sensors and daemon processes are a waste of resources — One of the most common data engineering challenges is triggering workflows in response to events such as when a new file arrives in a certain directory. The approach taken by legacy orchestrators is to deploy continuously running background processes, such as sensors or daemons that poll for status. In this…

And several important fixes and enhancements! — Prefect 2.3.0 ships with some really cool and highly requested features. We’ll cover those in this post. To learn more about all improvements, check the release notes and our updated documentation. Table of contents: · Build and apply your deployments in one step · Skip upload if your code didn’t change ·…

Forget about sensors and DAGs. Run your flows based on events. — Prefect is a dataflow coordination plane designed for a highly dynamic and rapidly changing world. Your workflows don’t have to be scheduled unless they need to — you can run them from anywhere, at any time and scale, for any reason. This post will look at how to implement scheduled…

Command Line Interfaces and YAML are loved by DevOps engineers — we use the same for DataflowOps — In Prefect, Python is the API — we believe that your code is the best representation of your workflow. But when it comes to deploying your flows, an imperative Python API can become cumbersome. …

We hear more about tools than about using them the right way — There have been many debates lately about unbundling or rebundling, which led to some arguments over who’s gonna win the market for any given piece of the data stack. But those debates hide an important fact: tools are just tools. …