Kubernetes

Say you have a web service running as a Kubernetes Deployment. There are a bunch of ways to access it over a public URL, but Kubexpose makes it easy to do so. It’s a Kubernetes Operator backed by a Custom Resource Definition and the corresponding controller implementation.

Kubexpose built using kubebuilder and available on GitHub

To try it out, jump into the next section or scroll down to How it works? to learn more

This blog post demonstrates how to auto-scale your Redis based applications on Kubernetes. Redis is a widely used (and loved!) database which supports a rich set of data structures (String, Hash, Streams, Geospatial), as well as other features such as pub/sub messaging, clustering (HA) etc. One such data structure is a List which supports operations such as inserts (LPUSH, RPUSH, LINSERT etc.), reads (LRANGE), deletes (LREM, LPOP etc.) etc. But that’s not all!

Welcome to part four of this blog series! So far, we have a Kafka single-node cluster with TLS encryption on top of which we configured different authentication modes (TLS and SASL SCRAM-SHA-512), defined users with the User Operator, connected to the cluster using CLI and Go clients and saw how easy it is to manage Kafka topics with the Topic Operator. So far, our cluster used ephemeral persistence, which in the case of a single-node cluster, means that we will lose data if the Kafka or Zookeeper nodes (Pods) are restarted due to any reason.

Azure Service Operator is an open source project to help you provision and manage Azure services using Kubernetes. Developers can use it to provision Azure services from any environment, be it Azure, any other cloud provider or on-premises - Kubernetes is the only common denominator!

It can also be included as a part of CI/CD pipelines to create, use and tear down Azure resources on-demand. Behind the scenes, all the heavy lifting is taken care of by a combination of Custom Resource Definitions which define Azure resources and the corresponding Kubernetes Operator(s) which ensure that the state defined by the Custom Resource Definition is reflected in Azure as well.

Over the course of the first two parts of this blog series, we setup a single-node Kafka cluster on Kubernetes, secured it using TLS encryption and accessed the broker using both internal and external clients. Let’s keep iterating! In this post, we will continue the Kafka on Kubernetes journey with Strimzi and cover:

• How to apply different authentication types: TLS and SASL SCRAM-SHA-512
• Use Strimzi Entity operator to manage Kafka users and topics
• How to configure Kafka CLI and Go client applications to securely connect to the Kafka cluster

The code is available on GitHub - https://github.com/abhirockzz/kafka-kubernetes-strimzi/

[Read More]

We kicked off the the first part of the series by setting up a single node Kafka cluster which was accessible to only internal clients within the same Kubernetes cluster, had no encryption, authentication or authorization and used temporary persistence. We will keep iterating/improving on this during the course of this blog series.

This part will cover these topics:

• Expose Kafka cluster to external applications
• Apply TLS encryption
• Explore Kubernetes resources behind the scenes
• Use Kafka CLI and Go client applications to test our cluster setup

The code is available on GitHub - https://github.com/abhirockzz/kafka-kubernetes-strimzi/

[Read More]

Some of my previous blog posts (such as Kafka Connect on Kubernetes, the easy way!), demonstrate how to use Kafka Connect in a Kubernetes-native way. This is the first in a series of blog posts which will cover Apache Kafka on Kubernetes using the Strimzi Operator. In this post, we will start off with the simplest possible setup i.e. a single node Kafka (and Zookeeper) cluster and learn:

• Strimzi overview and setup
• Kafka cluster installation
• Kubernetes resources used/created behind the scenes
• Test the Kafka setup using clients within the Kubernetes cluster

The code is available on GitHub - https://github.com/abhirockzz/kafka-kubernetes-strimzi

[Read More]

In Kafka Connect on Kubernetes, the easy way!, I had demonstrated Kafka Connect on Kubernetes using Strimzi along with the File source and sink connector. This blog will showcase how to build a simple data pipeline with MongoDB and Kafka with the MongoDB Kafka connectors which will be deployed on Kubernetes with Strimzi.

I will be using the following Azure services:

Please note that there are no hard dependencies on these components and the solution should work with alternatives as well

[Read More]

Very often, there is a need to expose Kubernetes apps running in minikube to the public internet. This post just provides a manual, yet quick and dirty hack for this using ngrok (it’s probably been done before, but here it goes anyway)

We’ll use a simple nginx app to test things out i.e. we’ll expose an nginx server (running as a single replica Kubernetes Deployment) as a publicly accessible URL.

Test app (nginx)

Start by creating the nginx Deployment