Prefect & Fivetran: integrate all the tools & orchestrate them in Python

Cloud data ingestion and orchestration made easy

Image by the author

What is Prefect?

It’s a flexible framework to build, reliably execute and observe your dataflow while supporting various execution and data access patterns. It lets you turn any Python script into a fully operationalized application.

What is Fivetran?

It’s a data integration platform that allows businesses to replicate data from various systems into a central data warehouse. It offers pre-built connectors for a wide range of data sources, including databases, SaaS applications, cloud storage, event logs, and more. Fivetran can help you keep your data consistent, complete, and up-to-date by automating the data integration process.

Why Fivetran + Prefect?

Prefect coordinates dataflow across your entire stack. With Fivetran, you can easily move data from sources to destinations. Prefect allows you to:

• schedule and observe your data replication syncs,
• add a sync to the part of your pipeline that needs to operate on the most up-to-date data from any given source system,
• ensure reliable dataflow through retries, alerting, and dependency management.

Getting started with Prefect

First, sign up for a free Prefect Cloud account and create a workspace. Then, install Prefect and log into your Cloud workspace from a terminal:

pip install prefect
prefect cloud login

This command will redirect you to your browser to automatically create an API key and CLI profile. This way, your local development machine gets authenticated with the Prefect Cloud workspace.

Then, create a script called myflow.py:

from prefect import flow

@flow(log_prints=True)
def hello():
    print("You're the Prefectionist now!🤗")

if __name__ == "__main__":
    hello()

Finally, run your first flow using python myflow.py. If you navigate to the Cloud UI, you’ll see a flow run that you triggered from your local terminal.