Scheduled vs. Event-driven Data Pipelines — Orchestrate Anything with Prefect

When and when not to schedule your workflows and why sensors and daemon processes are a waste of resources

Marvin observes what’s currently happening in the galaxy

One of the most common data engineering challenges is triggering workflows in response to events such as when a new file arrives in a certain directory. The…