Prefect & Fivetran: integrate all the tools & orchestrate them in Python
What is Prefect?
It’s a flexible framework to build, reliably execute and observe your dataflow while supporting various execution and data access patterns. It lets you turn any Python script into a fully operationalized application.
What is Fivetran?
It’s a data integration platform that allows businesses to replicate data from various systems into a central data warehouse. It offers pre-built connectors for a wide range of data sources, including databases, SaaS applications, cloud storage, event logs, and more. Fivetran can help you keep your data consistent, complete, and up-to-date by automating the data integration process.
Why Fivetran + Prefect?
Prefect coordinates dataflow across your entire stack. With Fivetran, you can easily move data from sources to destinations. Prefect allows you to:
- schedule and observe your data replication syncs,
- add a sync to the part of your pipeline that needs to operate on the most up-to-date data from any given source system,
- ensure reliable dataflow through retries, alerting, and dependency management.
Getting started with Prefect
pip install prefect
prefect cloud login
This command will redirect you to your browser to automatically create an API key and CLI profile. This way, your local development machine gets authenticated with the Prefect Cloud workspace.
Then, create a script called
from prefect import flow
print("You're the Prefectionist now!🤗")
if __name__ == "__main__":
Finally, run your first flow using
python myflow.py. If you navigate to the Cloud UI, you’ll see a flow run that you triggered from your local terminal.