How Fivetran, Airbyte, Singer, dltHub and CloudQuery approached data ingestion and why it will never be fully solved — If you ask a data engineer what is the most frustrating and error-prone part of their job, chances are they’ll say data ingestion. Moving data from A to B is one of the most mundane and time-consuming tasks of any platform team. This post covers how Fivetran, Airbyte, Singer, dltHub…

Comparison of open-source dataframe and SQL frameworks for data engineering, machine learning, and analytics — Tabular format with rows and columns, popularized by relational databases and Microsoft Excel, is an intuitive way of organizing and manipulating data for analytics. There are two main ways of transforming and analyzing tabular data — SQL and dataframes (sorry, Excel!). SQL is a declarative language for querying datasets of…

How to turn AWS data lake into a data lakehouse using Iceberg, the open table format — This crash course will guide you on how to get started with Apache Iceberg on AWS. By the end of this tutorial, you’ll be able to create Iceberg tables, insert and modify data stored in S3 in Parquet format, query data and table metadata in plain SQL, and declaratively manage…

Why and when to use MotherDuck over local DuckDB — MotherDuck has recently launched the managed DuckDB product, which is currently in Beta. This post shares how to get started with both DuckDB and MotherDuck, what are key differences between them, and when to choose each of these options. Before diving into details, let’s clarify what DuckDB and MotherDuck are. DuckDB …

Everything as Code for Data Infrastructure — Everything as Code (EaC) is a development approach aiming to express not only software but also its infrastructure and configuration in code. Changes to resources are managed programmatically using a Git workflow and a code review process rather than deployed manually. …

How you can use open-source tools to ingest, transform and orchestrate data pipelines without vendor lock-in — The key benefit of the Modern Data Stack is that you can avoid vendor lock-in by selecting best-of-breed tools rather than paying expensive license fees for one inflexible solution. However, assembling your modular stack based on multiple SaaS solutions will only marginally improve that situation. It’s undoubtedly easier to swap…

Get rid of manual workflow deployments using Infrastructure as Code — You know the story. You write your data pipeline locally. You configure dependencies, and you validate that everything works as expected. If it does, it’s time to submit a pull request and ship it. Right? Well, that’s easier said than done. Moving to production without affecting existing applications and data…

Berlin Buzzwords is Europe’s leading conference about search technologies, modern data infrastructure, and ML. This article continues upon the post from Day 1 and highlights key takeaways from the second (and also last) conference day. Summary of selected talks At any time, there were several parallel talks. Each section below gives a brief summary…

Berlin Buzzwords is Europe’s leading conference for modern data infrastructure, search technologies, and ML. The participants are engineers, IT architects, data scientists, and analysts interested in data searchability, AI, and data processing at scale. Top experts present contemporary industry trends and demystify the technology buzzwords. Focusing on open-source software projects…