kafka

Use CDK to deploy a complete solution with Kafka, App Runner, EKS and DynamoDB

Posted on January 9, 2023 | 5 minutes | Abhishek Gupta

A previous blog post covered how to deploy a Go Lambda function and trigger it in response to events sent to a topic in a MSK Serverless cluster. [Read More]

Getting started with MSK Serverless and AWS Lambda using Go

Posted on January 3, 2023 | 4 minutes | Abhishek Gupta

In this blog post you will learn how to deploy a Go Lambda function and trigger it in response to events sent to a topic in a MSK Serverless cluster. [Read More]

kafka go serverless lambda aws

MySQL to DynamoDB: Build a streaming data pipeline on AWS using Kafka

Posted on June 28, 2022 | 9 minutes | Abhishek Gupta

Use change data capture with MSK Connect to sync data between Aurora MySQL and DynamoDB This is the second part of the blog series which provides a step-by-step walkthrough of data pipelines with Kafka and Kafka Connect. [Read More]

kafka dynamodb data engineering nosql database aws tutorial

Build a data pipeline on AWS with Kafka, Kafka connect and DynamoDB

Posted on June 21, 2022 | 10 minutes | Abhishek Gupta

Integrate DynamoDB with MSK and MSK Connect There are many ways to stitch data pipelines - open source components, managed services, ETL tools, etc. [Read More]

kafka dynamodb data-pipeline nosql aws

Getting started with Kafka Connector for Azure Cosmos DB using Docker

Posted on July 7, 2021 | 10 minutes | Abhishek Gupta

Having a local development environment is quite handy when trying out a new service or technology. Docker has emerged as the de-facto choice in such cases. [Read More]

kafka azure cosmos db NoSQL docker

Processing Time-Series Data with Redis and Apache Kafka

Posted on June 22, 2021 | 10 minutes | Abhishek Gupta

RedisTimeSeries is a Redis Module that brings native Time Series data structure to Redis. Time Series solutions which were earlier built on top of Sorted Sets (or Redis Streams) can benefit from RedisTimeSeries features such as high volume inserts, low latency reads, flexible query language, down-sampling and much more! [Read More]

Redis Grafana Azure NoSQL Spring Java Kafka

Getting started with Kafka and Rust: Part 2

Posted on April 14, 2021 | 10 minutes | Abhishek Gupta

This is a two-part series to help you get started with Rust and Kafka. We will be using the rust-rdkafka crate which itself is based on librdkafka (C library). [Read More]

rust kafka programming

Getting started with Kafka and Rust: Part 1

Posted on April 6, 2021 | 7 minutes | Abhishek Gupta

This is a two-part series to help you get started with Rust and Kafka. We will be using the rust-rdkafka crate which itself is based on librdkafka (C library). [Read More]

rust kafka programming

Real-Time Search and Analytics with Confluent, Azure, Redis, and Spring Cloud

Posted on March 25, 2021 | 12 minutes | Abhishek Gupta

Self-managing a distributed system like Apache Kafka ®, along with building and operating Kafka connectors, is complex and resource intensive. It requires significant Kafka skills and expertise in the development and operations teams of your organization. [Read More]

mysql redis kafka azure spring boot java azure spring cloud

PostgreSQL pgoutput plugin for change data capture

Posted on August 13, 2020 | 5 minutes | Abhishek Gupta

Set up a Change Data Capture architecture on Azure using Debezium, Postgres and Kafka was a tutorial on how to use Debezium for change data capture from Azure PostgreSQL and send them to Azure Event Hubs for Kafka - it used the wal2json output plugin. [Read More]

postgresql kafka azure