Event-driven open-source data orchestrator with a built-in code editor and a rich plugin ecosystem — This post shares the story behind Kestra and explains how you can use it to orchestrate data processing workflows in a declarative way. What is Kestra Kestra is an open-source, event-driven data orchestrator that strives to make data workflows accessible to a broader audience. The product offers a declarative YAML interface for workflow…

Short personal update — This article is a short personal update. I’m not a fan of talking in the first person in blog posts, so be warned — this post has an exceptional amount of I’s 👀. Before 2023 Over the last couple of years, I was lucky to carve a niche for myself, which lies…

Summary of recent trends for AI end-users (rather than AI engineers or researchers) — Many AI-driven applications and initiatives have recently surfaced and gained immense popularity in a remarkably short period of time. These tools utilize large language models (LLMs) to either assist in everyday tasks or help build AI-powered software. This post will provide an overview of some notable autonomous agents and tools…

From a prompt to a deployed application in 5 minutes — In this post, we’ll build a sentiment analysis application using ChatGPT and FastAPI, and deploy it to AWS using App Runner. We’ll use ChatGPT to generate and document the application code and write tests. Let’s get started! Building the FastAPI Application To create our sentiment analysis app, we’ll use ChatGPT, an AI language model…

Less is more — I wish more people and organizations would embrace concise writing and communication as a standard. It’s beneficial to everyone, regardless of whether you are: a content writer or reader, a student writing an essay or a teacher evaluating it, a programmer documenting your code (thanks if you do it) or…

Summary of the 2023 data infrastructure from the perspective of practitioner talking (a lot) to end-users from data communities — Matt Turck has recently published the 2023 MAD (Machine Learning, Artificial Intelligence & Data) Landscape overview. Similar graphics have been made for 2012, 2014, 2016, 2017, 2018, 2019 (Part I and Part II), 2020, and 2021, and now there is a PDF…

What to measure and whether you should — Data teams are sometimes asked to prove their ROI to senior leadership to justify a budget for new hires, tools, projects, or process changes. But the work of data teams is inherently unmeasurable. Often the reason for this ROI question isn’t rooted in a lack of proper metrics but rather…

How to do it respectfully and in a way that improves your relationships rather than harming them — One of the most challenging aspects of giving and receiving negative feedback is understanding how people can respond to it. While some prefer more direct feedback, others can feel personally attacked by it. You need to know the person on the other side to assess that correctly. …

Cloud data ingestion and orchestration made easy — ⚠️ Note: I no longer work at Prefect. This post may be completely outdated. Refer to Prefect docs and website to stay up-to-date. What is Prefect? It’s a flexible framework to build, reliably execute and observe your dataflow while supporting various execution and data access patterns. …