Golang

A previous blog post covered how to deploy a Go Lambda function and trigger it in response to events sent to a topic in a MSK Serverless cluster.

This blog will take it a notch further.

• The solution consists of a MSK Serverless cluster, a producer application on AWS App Runner and a consumer application in Kubernetes (EKS) persisting data to DynamoDB.
• The core components (MSK cluster, EKS and DynamoDB) and the producer application will be provisioned using Infrastructure-as-code with AWS CDK.
• Since the consumer application on EKS will interact with both MSK and DynamoDB, you will also need to configure appropriate IAM roles.

All the components in this solution have been written in Go.

In this blog post you will learn how to deploy a Go Lambda function and trigger it in response to events sent to a topic in a MSK Serverless cluster.

• Prerequisites
• Infrastructure setup
• Send data to MSK Serverless using producer application
• Configure and deploy the Lambda function
• Verify the integration
• Conclusion

The following topics have been covered:

• How to use the franz-go Go Kafka client to connect to MSK Serverless using IAM authentication.
• Write a Go Lambda function to process data in MSK topic.
• Create the infrastructure - VPC, subnets, MSK cluster, Cloud9 etc.
• Configure Lambda and Cloud9 to access MSK using IAM roles and fine-grained permissions.

Prerequisites

You will need an AWS account, install AWS CLI as well a recent version of Go (1.18 or above).

The DynamoDB Scan API accesses every items in a table (or secondary index). It is the equivalent of a select * from query. One of the things I will cover in this blog is how to use Scan API with the DynamoDB Go SDK.

To scan a table, we need some data to begin with! So in the process, I will also go into how to use the Batch API to write bulk data in DynamoDB. You can use the BatchWriteItem API to create or delete items in batches (of twenty five) and it’s possible to you can combine these operations across multiple tables.

Learn with practical code samples

DynamoDB provides a rich set of data types including Strings, Numbers, Sets, Lists, Maps etc. In the Go SDK for DynamoDB, the types package contains Go representations of these data types and the attributevalue module provides functions to work with Go and DynamoDB types.

This blog post will demonstrate how to handle conversions between Go types in your application and DynamoDB. We will start off with simple code snippets to introduce some of the API constructs and wrap up with a example of how to use these Go SDK features in the context of a complete application (including a code walk though).

In an earlier blog post you saw how to use cdk8s with AWS Controllers for Kubernetes (also known as ACK), thanks to the fact that you can import existing Kubernetes Custom Resource Definitions using cdk8s! This made is possible to deploy DynamoDB along with a client application, from using cdk8s and Kubernetes.

But, what if you continue using AWS CDK for AWS infrastructure and harness the power cdk8s (and cdk8s-plus!) to define Kubernetes resources using regular code? Thanks to the native integration between the AWS EKS module and cdk8s, you can have the best of both worlds!

Deploy DynamoDB and a client app using cdk8s along with AWS Controller for Kubernetes

AWS Controllers for Kubernetes (also known as ACK) are built around the Kubernetes extension concepts of Custom Resource and Custom Resource Definitions. You can use ACK to define and use AWS services directly from Kubernetes. This helps you take advantage of managed AWS services for your Kubernetes applications without needing to define resources outside of the cluster.

Using AWS Controller for Kubernetes and CDK for Kubernetes

In this blog post, you will learn how to use ACK with Amazon EKS for creating a Redis cluster on AWS (with Amazon MemoryDB).

AWS Controllers for Kubernetes (also known as ACK) leverage Kubernetes Custom Resource and Custom Resource Definitions and give you the ability to manage and use AWS services directly from Kubernetes without needing to define resources outside of the cluster. It supports many AWS services including S3, DynamoDB, MemoryDB etc.

Earlier, I wrote about a Serverless URL shortener application on AWS using DynamoDB, AWS Lambda and API Gateway.

In this blog post, we will deploy that as a REST API on AWS App Runner and continue to use DynamoDB as the database. AWS App Runner is a compute service that makes it easy to deploy applications from a container image (or source code), manage their scalability, deployment pipelines and more.

With the help of a practical example presented in this blog, you will:

Build a Wordpress deployment as a cdk8s construct

Constructs are the fundamental building block of cdk8s (Cloud Development Kit for Kubernetes) - an open-source framework (part of CNCF) with which you can define your Kubernetes applications using regular programming languages (instead of yaml). In Getting started with cdk8s, you saw how to use the core cdk8s library.

You can also use the cdk8s-plus library (also covered this a previous blog) to reduce the amount of boilerplate code you need to write. With cdk8s-plus, creating a Kubernetes Deployment, specifying it’s container (and other properties) and exposing it via a Service is three function calls away.

Using AWS Lambda, DynamoDB and API Gateway

This blog post covers how to build a Serverless URL shortener application using Go. It leverages AWS Lambda for business logic, DynamoDB for persistence and API Gateway to provide the HTTP endpoints to access and use the application. The sample application presented in this blog is a trimmed down version of bit.ly or other solutions you may have used or encountered.

It’s structured as follows:

• I will start off with a quick introduction and dive into how to deploy try the solution.
• After that, I will focus on the code itself. This will cover:
  • The part which is used to write the infrastructure (using Go bindings for AWS CDK)
  • And also the core business logic which contains the Lambda function (using Lambda Go support) as well as the DynamoDB operations (using the DynamoDB Go SDK)

In this blog, you will learn: