GCP and Prefect Cloud — from Docker Container to Cloud VM on Google Compute Engine

Repository template with GitHub Actions will deploy your first Python dataflows to Google Cloud in minutes

Little baby Marvin slowly getting acquainted with Prefect and Google Cloud

Google Compute Engine allows running virtual machines (VMs) on GCP. It’s a scalable platform for running a wide range of workloads, including custom (Python) applications, data processing, and machine learning — an ideal execution layer for Prefect flows. This post will help you get started with Prefect and Google Cloud by leveraging the prefect-cloud-gcp repository template.

Create a new repository

First, create a new repository from the prefect-cloud-gcp template. Then, inspect the repository structure. It consists of the following:

1. flows — example flows to get started with Prefect, both executing workflows locally from your machine, as well as remotely on GCP
2. prefect_utils module as a placeholder for any custom business logic and reusable code
3. setup.py and requirements.txt are used to package prefect_utils and other dependencies for both local and remote execution
4. Dockerfile combines your flow code, custom logic, and external dependencies into a portable…