Build real-time analytics fast with Timestream and Grafana

The last Timestream article demonstrated how to ingest data into Timestream using boto3, awswrangler, and CLI.

In this post, we’ll dive deeper into using time-series functions in Timestream’s query language, and we’ll visualize the data using Grafana. The end goal is to have a Grafana dashboard populated with new data…

How to manage dependencies between data pipelines

Illustration of an ELT flow using Prefect and dbt

Workflow orchestration platforms have historically allowed managing task dependencies within individual data pipelines. While this is a good start, what if you have dependencies between data pipelines?

Say you have some flows or directed acyclic graphs (DAGs) that ingest operational data from various sources into the staging area of your…

How to ingest data into the AWS serverless time-series database

Night stars

Timestream is a serverless time-series database service offered by AWS. It can be used for operational analytics, IoT device monitoring, financial forecasting, and many more use cases that deal with time-series data. For more background on this service, check out my previous article:

In this post, we will dive deeper…

Serverless Data Engineering Pipelines in Python

Prefect is a flexible tool to orchestrate the modern data stack. In contrast to many other solutions on the market, it doesn't tie you to any specific execution framework or cloud provider — whether you want to use Kubernetes on GCP, AWS ECS, a bare-metal server, or an on-demand distributed…

Including AWS best practices to avoid them

In this article, we’ll discuss potential pitfalls that we came across when configuring ECS task definitions. While considering this AWS-specific container management platform, we’ll also examine some general best practices for working with containers in production.

Table of contents:

· #1 Wrong logging configuration
· #2 Failing to enable “Auto-assign public…

Some activities we don’t spend enough time on

People in a meeting

Engineering time is a scarce resource. We often have to balance many tasks and often conflicting priorities. However, there are some activities for which allocating more of that time can be beneficial. In this article, we’ll look at ten of them.

1. Backups and Preventing Accidental Deletion

Have you ever deleted something prematurely only to figure…

Make better design choices by avoiding those pitfalls

Female writing on a whiteboard

Designing a data model for analytics is not the same as doing it for transactional processing. You optimize for access patterns that are very different from row-level data retrieval used in OLTP systems. In this article, we’ll look at the most common pitfalls when designing schemas and tables for analytics.

1. Treating Schema Design as a One-Off Project

Anna Geller

Data Engineer, M.Sc. in BI, AWS Certified Solution Architect: www.annageller.com. Get my articles via email: https://annageller.medium.com/subscribe

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store