Parametrization is one of the most critical features of any modern workflow orchestration solution. It allows you to dynamically overwrite parameter values for a given run without having to redeploy your workflow. Most orchestration frameworks provide rather limited functionality in that regard, such as only allowing to override global variables…
This is the third post in a series of articles about orchestrating ELT data pipelines with Prefect. The first post dealt with organizing a project and orchestrating flows in a local environment. The second one discussed deploying the ELT project to Snowflake, AWS EKS, and building a CI/CD process. In…
This article is the second in a series of tutorials about orchestrating ELT data pipelines. The first post demonstrated how to organize and orchestrate a variety of flows written by different teams and how to trigger those in the correct order using Prefect. This post builds on that by capturing…
The last Timestream article demonstrated how to ingest data into Timestream using boto3, awswrangler, and CLI.
In this post, we’ll dive deeper into using time-series functions in Timestream’s query language, and we’ll visualize the data using Grafana. The end goal is to have a Grafana dashboard populated with new data…
Many believe that streaming technologies are the only way to achieve real-time analytics. Since most data workloads these days are orchestrated through batch-processing platforms, the real-time requirement forces data teams to adopt a new set of tools. However, maintaining two separate systems for batch processing and real-time streaming introduces additional…
Workflow orchestration platforms have historically allowed managing task dependencies within individual data pipelines. While this is a good start, what if you have dependencies between data pipelines?
Say you have some flows or directed acyclic graphs (DAGs) that ingest operational data from various sources into the staging area of your…
Timestream is a serverless time-series database service offered by AWS. It can be used for operational analytics, IoT device monitoring, financial forecasting, and many more use cases that deal with time-series data. For more background on this service, check out my previous article:
In this post, we will dive deeper…
Prefect is a flexible tool to orchestrate the modern data stack. In contrast to many other solutions on the market, it doesn't tie you to any specific execution framework or cloud provider — whether you want to use Kubernetes on GCP, AWS ECS, a bare-metal server, or an on-demand distributed…
In this article, we’ll discuss potential pitfalls that we came across when configuring ECS task definitions. While considering this AWS-specific container management platform, we’ll also examine some general best practices for working with containers in production.
Table of contents:
· #1 Wrong logging configuration
· #2 Failing to enable “Auto-assign public…
Engineering time is a scarce resource. We often have to balance many tasks and often conflicting priorities. However, there are some activities for which allocating more of that time can be beneficial. In this article, we’ll look at ten of them.
Have you ever deleted something prematurely only to figure…
Data Engineer, M.Sc. in BI, AWS Certified Solution Architect: www.annageller.com. Get my articles via email: https://annageller.medium.com/subscribe
