Aws

Its hard to operate stateful distributed systems at scale and Redis is no exception. Managed databases make life easier by taking on much of the heavy lifting. But you still need a sound architecture and apply best practices both on the server (Redis) as well as client (application).

This blog covers a range of Redis related best practices, tips and tricks including cluster scalability, client side configuration, integration, metrics etc. Although I will be citing Amazon MemoryDB and ElastiCache for Redis from time to time, most (if not all) will be applicable to Redis clusters in general.

Earlier, I wrote about a Serverless URL shortener application on AWS using DynamoDB, AWS Lambda and API Gateway.

In this blog post, we will deploy that as a REST API on AWS App Runner and continue to use DynamoDB as the database. AWS App Runner is a compute service that makes it easy to deploy applications from a container image (or source code), manage their scalability, deployment pipelines and more.

With the help of a practical example presented in this blog, you will:

Using AWS Lambda, DynamoDB and API Gateway

This blog post covers how to build a Serverless URL shortener application using Go. It leverages AWS Lambda for business logic, DynamoDB for persistence and API Gateway to provide the HTTP endpoints to access and use the application. The sample application presented in this blog is a trimmed down version of bit.ly or other solutions you may have used or encountered.

It’s structured as follows:

• I will start off with a quick introduction and dive into how to deploy try the solution.
• After that, I will focus on the code itself. This will cover:
  • The part which is used to write the infrastructure (using Go bindings for AWS CDK)
  • And also the core business logic which contains the Lambda function (using Lambda Go support) as well as the DynamoDB operations (using the DynamoDB Go SDK)

In this blog, you will learn:

Replicate DynamoDB data from one table to another

This blog post will help you get quickly started with DynamoDB Streams and AWS Lambda using Go. It will cover how to deploy the entire solution using AWS CDK.

The use case presented here is pretty simple. There are a couple of DynamoDB tables and the goal is to capture the data in one of those tables (also referred to as the source table) and replicate them to another table (also referred to as the target table) so that it can serve different queries. To demonstrate an end-to-end flow, there is also an Amazon API Gateway that front ends a Lambda function which persists data in the source DynamoDB table. Changes in this table will trigger another Lambda function (thanks to DynamoDB Streams) which will finally replicate the data into the target table.

Use change data capture with MSK Connect to sync data between Aurora MySQL and DynamoDB

This is the second part of the blog series which provides a step-by-step walkthrough of data pipelines with Kafka and Kafka Connect. I will be using AWS for demonstration purposes, but the concepts apply to any equivalent options (e.g. running these locally in Docker).

This part will show Change Data Capture in action that let’s you track row-level changes in database tables in response to create, update and delete operations. For example, in MySQL, these change data events are exposed via the MySQL binary log (binlog).

Integrate DynamoDB with MSK and MSK Connect

There are many ways to stitch data pipelines - open source components, managed services, ETL tools, etc. In the Kafka world, Kafka Connect is the tool of choice for “streaming data between Apache Kafka and other systems”. It has an extensive set of pre-built source and sink connectors as well as a common framework for Kafka connectors which standardises integration of other data systems with Kafka and making it simpler to develop your own connectors, should there be a need to do so.

This is a Go Lambda function that’s packaged and deployed using AWS SAM

Lambda Function URL is a relatively new feature (at the time of writing this blog) that provides dedicated HTTP(S) endpoint for your Lambda function. It is really useful when all you need is a single endpoint for your function (e.g. to serve as a webhook) and don’t want to setup and configure an API Gateway. Looks like I can’t seem to get enough of it! I have written a couple of blog posts on this topic that include a practical example of using it to build a serverless backend and then deploying that solution using AWS CDK.

Deploy a Serverless backend for Slack using Infrastructure-as-code (IaaC)

One of my previous blog post covered how to build a Serverless backend for Slack using by using Lambda Function URL as a webhook. Since I wanted to focus on the application itself, the infrastructure setup part was simplified - using AWS CLI, the function was packaged as a zip file, configured and finally a Function URL was created along with the required permissions.

A combination of AWS Lambda and Amazon API Gateway is a widely-used architecture for serverless microservices and API based solutions. They enable developers to focus on their applications, instead of spending time provisioning and managing servers.

API Gateway is a feature rich offering that includes with support for different API types (HTTP, REST, WebSocket), multiple authentication schemes, API versioning, canary deployments and much more! However, if your requirements are simpler and all you need is an HTTP(S) endpoint for your Lambda function (for example, to serve as a webhook), you can use Lambda Function URLs! When you create a function URL, Lambda automatically generates a unique HTTP(S) endpoint that is dedicated for your Lambda function.