What to measure and whether you should — Data teams are sometimes asked to prove their ROI to senior leadership to justify a budget for new hires, tools, projects, or process changes. But the work of data teams is inherently unmeasurable. Often the reason for this ROI question isn’t rooted in a lack of proper metrics but rather…

How to do it respectfully and in a way that improves your relationships rather than harming them — One of the most challenging aspects of giving and receiving negative feedback is understanding how people can respond to it. While some prefer more direct feedback, others can feel personally attacked by it. You need to know the person on the other side to assess that correctly. …

Cloud data ingestion and orchestration made easy — What is Prefect? It’s a flexible framework to build, reliably execute and observe your dataflow while supporting various execution and data access patterns. It lets you turn any Python script into a fully operationalized application. What is Fivetran? It’s a data integration platform that allows businesses to replicate data from various systems into a central data…

A quick but secure way to extract fields from JSON-based secrets — Several cloud vendors provide credentials as a JSON object. For instance, Google Cloud allows the creation of a credentials JSON file that contains a.o. project identifier, private key, and client email of a service account. …

Summary of selected talks and lessons learned — NormConf is an online tech conference about things that matter in data and ML but don’t get the spotlight. As something that started as a Twitter joke, NormConf 2022 exceeded anyone’s expectations. It encompassed many excellent presentations from smart people sharing stories from real-life experiences in the field. All talks…

Repository template with GitHub Actions will deploy your first Python dataflows to Google Cloud in minutes — Google Compute Engine allows running virtual machines (VMs) on GCP. It’s a scalable platform for running a wide range of workloads, including custom (Python) applications, data processing, and machine learning — an ideal execution layer for Prefect flows. …

Orchestration is just one part of the dataflow equation — What is Prefect? It’s a flexible framework to build, reliably execute and observe your dataflow while supporting a wide variety of execution and data access patterns. Why should you care? The Modern Data Stack encompasses a large array of highly specialized components. You can find tools for data ingestion, transformation, analysis, validation, cataloging — the list goes…

Modular Data Stack with dbt Cloud Prefect block — This short post will walk you through how to set up dbt Cloud jobs and orchestrate those with Prefect. It assumes that you have already signed up for dbt Cloud and know how to use dbt. dbt Cloud setup First, you need to retrieve the dbt Cloud account ID and create an API…