Event-driven Data Pipelines with AWS Lambda, Prefect and GitHub Actions
Forget about sensors and DAGs. Run your flows based on events.
6 min readAug 29, 2022
Prefect is a dataflow coordination plane designed for a highly dynamic and rapidly changing world. Your workflows don’t have to be scheduled unless they need to — you can run them from anywhere, at any time and scale, for any reason. This post will look at how to implement scheduled and event-driven data pipelines. We’ll demonstrate how you can deploy such workflows to AWS Lambda with the serverless framework and GitHub Actions and how you can orchestrate and observe their execution with Prefect.
Getting started
To get started, use the following repository template. It contains everything required to build and deploy event-driven serverless dataflows with Prefect and AWS Lambda, including flows, Dockerfiles, and Infrastructure as Code.
