While doing some routine maintenance work on my EKS cluster recently, I ended up modernising part of my storage setup.

The cluster was already using the AWS EBS CSI driver, but it had originally been installed manually using eksctl rather than being managed through Terraform.

Because of that:

• The add-on wasn’t managed as infrastructure-as-code

• The driver version was behind the recommended version for my Kubernetes release

That led me to:

1. Bring the AWS EBS CSI driver under Terraform management

2. Migrate Grafana’s persistent storage from the legacy in-tree EBS provisioner to the modern CSI driver

But this work didn’t start as a storage migration task.

It started with a scheduling issue.

The Problem That Triggered This Work

While improving Grafana's scheduling in my EKS cluster, I ran into an issue regarding how stateful workloads interact with AWS EBS volumes.

I covered the scheduling side of this problem in more detail in a previous post about node affinity and zonal storage constraints in Kubernetes.

While investigating the issue further, I noticed that the Grafana volume had originally been provisioned using the legacy in-tree AWS EBS driver, rather than the modern AWS EBS CSI driver.

Since I was already reviewing the storage configuration, this became a good opportunity to migrate Grafana’s persistent storage to the CSI driver and bring the EBS CSI addon under Terraform management.

In-Tree vs CSI Storage Drivers

While investigating the storage configuration, I noticed that the Grafana volume had originally been created using the legacy in-tree AWS EBS provisioner.

Older Kubernetes clusters relied on built-in storage plugins provided by the cloud provider.

For AWS EBS this meant the provisioner looked like this:

kubernetes.io/aws-ebs

These drivers have since been deprecated in favour of CSI drivers.

You can easily identify which driver a cluster is using by inspecting the StorageClass.

In my case, the existing storage class looked like this:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: prometheus-stack-storageclass
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer

The important line here is the provisioner field, which shows the legacy in-tree driver:

provisioner: kubernetes.io/aws-ebs

Modern Kubernetes clusters instead use the AWS EBS CSI driver, which uses the following provisioner:

ebs.csi.aws.com

So the new storage class looks something like this:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-csi-gp3
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  fsType: ext4
allowVolumeExpansion: true
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer

Since the Grafana volume had been provisioned using the in-tree driver, this became a good opportunity to migrate it to a CSI-backed storage class.

Managing the CSI Driver with Terraform

During this work I also noticed that the AWS EBS CSI driver had originally been installed manually using eksctl.

While that works perfectly fine, the rest of my infrastructure is managed using Terraform, so leaving the add-on outside of Terraform would eventually lead to configuration drift.

To bring the driver under infrastructure-as-code management, I first defined the add-on in Terraform:

resource “aws_eks_addon” “aws_ebs_csi_driver” {
  cluster_name  = var.cluster_name
  addon_name    = “aws-ebs-csi-driver”
  addon_version = “v1.xx.x-eksbuild.x”

  service_account_role_arn = var.ebs_csi_driver_role_arn

  resolve_conflicts_on_create = “OVERWRITE”
  resolve_conflicts_on_update = “OVERWRITE”
}

Since the add-on already existed in the cluster, the next step was to import it into Terraform state rather than recreate it.

This can be done using:

terraform import aws_eks_addon.aws_ebs_csi_driver <cluster-name>:aws-ebs-csi-driver

After importing the resource, Terraform was able to recognise the existing driver and manage it moving forward.

This means future upgrades to the CSI driver can now be handled alongside the rest of the cluster infrastructure through Terraform.

Backing Up Grafana Before Migration

Grafana stores dashboards, plugin data and configuration on disk, so before migrating the volume, I created a snapshot of the existing EBS volume.

The snapshot was created at the AWS level first, which meant Kubernetes did not yet know about it.

To make the snapshot usable within Kubernetes, I needed to import it using Kubernetes snapshot resources.

Importing the AWS Snapshot into Kubernetes

Since the snapshot already existed in AWS, I created a VolumeSnapshotContent resource referencing the AWS snapshot ID.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotContent
metadata:
  name: grafana-snap-content
spec:
  deletionPolicy: Retain
  driver: ebs.csi.aws.com
  volumeSnapshotClassName: ebs-csi-snapclass
  source:
    snapshotHandle: snap-xxxxxxxx
  volumeSnapshotRef:
    name: grafana-snapshot
    namespace: prometheus

This effectively imports the AWS snapshot into Kubernetes.

Creating the VolumeSnapshot

Next I created the Kubernetes snapshot resource:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: grafana-snapshot
  namespace: prometheus
spec:
  volumeSnapshotClassName: ebs-csi-snapclass
  source:
    volumeSnapshotContentName: grafana-snap-content

Once created, Kubernetes recognised the snapshot as a restore source.

I confirmed it was ready:

kubectl get volumesnapshot -n prometheus

Output:

READYTOUSE   true

Restoring the Snapshot Into a CSI Volume

Once the snapshot was ready, I created a new PVC restoring the snapshot using the CSI storage class.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-pvc-csi
  namespace: prometheus
spec:
  storageClassName: ebs-csi-gp3
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 5Gi
  dataSource:
    name: grafana-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
    kmsKeyId: arn:aws:kms:eu-west-2:XXXXX:key/<XXXXXX>

Kubernetes then provisioned a new EBS volume using the CSI driver and restored the snapshot contents.

Updating Grafana to Use the New Volume

Grafana was deployed using the kube-prometheus-stack Helm chart.

I updated the Helm values so Grafana would use the restored claim:

grafana:
  persistence:
    enabled: true
    existingClaim: grafana-pvc-csi

After redeploying the chart, Grafana started successfully using the new CSI-backed volume.

All existing configuration was preserved, including dashboards and data sources.

Final Result

After completing the work:

• The AWS EBS CSI driver is now managed through Terraform

• Grafana storage has been migrated from the in-tree AWS EBS provisioner to the CSI driver

• The monitoring stack now runs on a modern storage architecture

• snapshots provide a safe rollback path for future changes

Closing Thoughts

This migration stemmed from a real operational issue.

A scheduling problem led me to review storage behaviour, which uncovered an outdated driver and an opportunity to modernise the cluster.

That kind of chain reaction is very common in platform engineering.

You start by fixing one issue, and if you take the opportunity, you leave the platform in a much better state than you found it.

While making some improvements to my EKS cluster recently, I revisited how Grafana was scheduled across nodes.

The cluster runs a mixture of on-demand and spot instances, which is a common cost optimisation strategy. However, workloads running on spot instances can disappear at any time when AWS reclaims the capacity.

Because of this, I wanted to improve how Grafana behaved when nodes were reclaimed and pods needed to be rescheduled.

During this work I ran into an issue that highlights an important design consideration when running stateful workloads in Kubernetes using AWS EBS volumes.

The Error That Exposed the Problem

After Grafana restarted due to spot instance being reclaimed, the pod failed to start and remained pending.

Inspecting the pod events revealed the following:

AttachVolume.Attach failed for volume “grafana” : rpc error: code = Internal desc =
Could not attach volume “vol-xxxxxxxxxxxx” to node “i-xxxxxxxxxxxx”:
could not attach volume “vol-xxxxxxxxxxxx” to node “i-xxxxxxxxxxxx”:
InvalidVolume.ZoneMismatch: The volume ‘vol-xxxxxxxxxxxx’ is not in the same
availability zone as instance ‘i-xxxxxxxxxxxx’

This can typically be seen with:

kubectl describe pod prometheus-grafana-xxxxx -n prometheus

or

kubectl get events -n prometheus

The key clue here is the message:

InvalidVolume.ZoneMismatch

This indicates that Kubernetes attempted to attach an EBS volume that exists in one availability zone to a node running in another availability zone.

Because my cluster spans multiple AZs, the scheduler had placed the replacement pod onto a node that could not attach the existing volume.

Understanding EBS Volume Attachments

Another important aspect of EBS is how it attaches to nodes.

Unlike many network-attached storage systems, EBS does not support multi-node read/write access.

A typical EBS volume supports:

ReadWriteOnce

This means the volume can only be attached to one node at a time.

In contrast, traditional network storage solutions often support:

ReadWriteMany

where multiple nodes can mount the same volume simultaneously.

Because EBS behaves more like a block device attached to a single machine, Kubernetes must schedule the pod onto the node where the volume can attach.

This is why the AZ mismatch caused the workload to fail.

Why Not Use EFS Instead?

One alternative would be Amazon EFS.

EFS behaves more like traditional network-attached storage and supports:

ReadWriteMany

meaning multiple pods across multiple nodes can mount the same filesystem simultaneously.

However there is a trade-off.

EFS is typically more expensive than EBS and has different performance characteristics.

Since Grafana only requires a single writer volume, using EBS remains the more cost-effective option for this workload.

So the architecture decision here was deliberate:

• EBS for cost efficiency

• single-attach storage

• careful scheduling considerations

Designing Scheduling Rules for Stateful Workloads

Given these constraints, we need to ensure Kubernetes schedules pods in a way that respects how the storage behaves.

Kubernetes provides several mechanisms for controlling where pods are placed:

• node selectors

• node affinity

• taints and tolerations

• topology-aware scheduling

For stateful workloads using EBS-backed volumes, it’s important to influence the scheduler so that pods are placed onto nodes that can successfully attach the volume.

Using Node Affinity

Node affinity allows us to define rules about which nodes a pod should run on, based on node labels.

A simplified example looks like this:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
                - eu-west-2a

Breaking this down:

requiredDuringScheduling

The rule must be satisfied when Kubernetes schedules the pod.

IgnoredDuringExecution

If the rule stops being true later, the pod will not be evicted.

This allows us to guide the scheduler without forcing unnecessary restarts of running workloads.

The Bigger Architectural Lesson

This issue highlights an important distinction in Kubernetes.

Stateless workloads can generally run anywhere in the cluster.

Stateful workloads, however, often have infrastructure constraints that influence where they can run.

When using EBS-backed storage, those constraints include:

• availability zones

• single-node attachment

• storage cost considerations

Understanding these constraints helps design scheduling policies that prevent runtime failures like the ZoneMismatch error.

Closing Thoughts

This issue surfaced while improving the resilience of Grafana in a cluster running spot instances.

What started as a simple scheduling tweak turned into a deeper look at how stateful workloads interact with cloud storage primitives.

These kinds of debugging exercises are valuable because they highlight how:

• Kubernetes scheduling

• cloud infrastructure

• and storage architecture

all interact with each other.

In the next post, I’ll walk through how I migrated Grafana’s storage from the legacy in-tree AWS EBS driver to the modern EBS CSI driver, while safely preserving the existing data using Kubernetes volume snapshots.

Recently I ran into an interesting issue while reviewing the upgrade readiness of one of my Amazon EKS clusters.

What initially looked like a routine upgrade warning turned into a small investigation into:

• Kubernetes version skew policies

• EKS managed vs self-managed add-ons

• Terraform cluster provisioning

• Cilium networking configuration

In the end, the fix was simple — but the journey revealed something that often happens in long-running Kubernetes clusters: silent infrastructure drift.

This post walks through how I discovered the issue and how I resolved it.

The Warning

While reviewing the EKS Upgrade Insights dashboard, I saw the following alert:

kube-proxy version skew
Checks version of kube-proxy in cluster to see if upgrade would cause non compliance
with supported Kubernetes kube-proxy version skew policy.

At first glance, this seemed minor. The cluster had been running perfectly fine.

But version skew warnings should always be investigated because they often indicate deeper compatibility issues.

Inspecting the kube-proxy Version

To check the running version of kube-proxy in the cluster, I inspected the daemonset:

kubectl -n kube-system get ds kube-proxy \
-o jsonpath=’{.spec.template.spec.containers[0].image}’

The result:

602401143452.dkr.ecr.eu-west-2.amazonaws.com/eks/kube-proxy:v1.29.0-minimal-eksbuild.1

The cluster control plane version was:

1.35

This immediately explained the warning.

Kubernetes enforces a version skew policy between control plane components and kube-proxy.

The rule is:

• kube-proxy must not be newer than the API server

• kube-proxy may be up to three minor versions older

In my case:

ComponentVersionControl Plane1.35kube-proxy1.29

That’s a six-version gap, well outside the supported window.

Even though the cluster was functioning, this configuration was technically unsupported.

Why Was kube-proxy So Old?

The next question was obvious:

Why hadn’t kube-proxy been upgraded automatically?

To answer that, I checked whether kube-proxy was installed as an EKS managed add-on.

aws eks describe-addon \
--cluster-name my-cluster \
--addon-name kube-proxy

The result:

ResourceNotFoundException: No addon: kube-proxy found in cluster

That meant kube-proxy was not managed by EKS.

Instead, it was running as a self-managed Kubernetes daemonset.

This made sense once I remembered how the cluster had been created.

The cluster was originally provisioned via Terraform, and at the time I had not explicitly configured EKS add-ons.

When clusters are created outside the AWS console, the default components — such as:

• kube-proxy

• CoreDNS

• AWS VPC CNI

may exist as self-managed components unless explicitly installed as managed add-ons.

A Complication: Cilium Networking

This cluster also uses Cilium as the CNI instead of the default AWS VPC CNI.

Before modifying kube-proxy, I needed to confirm whether Cilium was operating in kube-proxy replacement mode.

If it was, kube-proxy would not be required at all.

Checking the Cilium configuration:

kubectl -n kube-system get cm cilium-config -o yaml | grep kubeProxyReplacement

The command returned nothing.

I also confirmed the daemonset was actively running:

kubectl -n kube-system get ds kube-proxy

Result:

kube-proxy   6   6   6

This confirmed that kube-proxy was still responsible for Kubernetes Service routing.

Cilium was providing networking and policy features, but not replacing kube-proxy.

The Solution: Migrating kube-proxy to an EKS Managed Add-on

Since kube-proxy was self-managed and significantly outdated, the best solution was to migrate it to the EKS managed add-on system.

Managed add-ons provide several advantages:

• automatic compatibility with cluster versions

• simplified upgrades

• consistent lifecycle management

• integration with EKS upgrade workflows

Because the cluster itself is managed with Terraform, I chose to manage the add-on declaratively as well.

I added the following Terraform resource:

resource “aws_eks_addon” “kube_proxy” {
  cluster_name                = aws_eks_cluster.eks_cluster.name
  addon_name                  = “kube-proxy”
  addon_version               = “v1.35.0-eksbuild.2”
  resolve_conflicts_on_create = “OVERWRITE”
  resolve_conflicts_on_update = “OVERWRITE”
}

The OVERWRITE flag is important.

Because a self-managed daemonset already existed, this tells EKS to replace the existing configuration with the managed add-on.

Applying the Change

After adding the resource, I ran:

terraform plan

The output showed:

+ aws_eks_addon.kube_proxy will be created

Importantly:

The EKS cluster itself was not recreated.

Add-ons are independent resources and can be installed or updated without affecting the control plane.

After applying the change, I verified the add-on status:

aws eks describe-addon \
--cluster-name my-cluster \
--addon-name kube-proxy

Result:

ACTIVE

The kube-proxy daemonset was now running the correct version for Kubernetes 1.35.

Lessons Learned

Even small upgrade warnings can reveal interesting platform insights.

Here are a few takeaways from this experience.

Long-Running Clusters Accumulate Drift

Clusters that have been running for years often contain components installed using older practices.

Without explicit lifecycle management, these components may quietly fall out of compliance.

EKS Managed Add-ons Reduce Operational Burden

Moving critical components such as kube-proxy and CoreDNS to managed add-ons simplifies cluster maintenance and reduces upgrade risks.

Networking Choices Affect Upgrade Strategy

Using alternative CNIs like Cilium introduces additional considerations.

Before modifying cluster components, it’s important to understand which parts of the networking stack are actually responsible for service routing.

Infrastructure as Code Should Include Add-ons

If you are managing EKS with Terraform, it is worth managing add-ons explicitly as well.

This keeps the entire cluster lifecycle declarative and avoids surprises during upgrades.

Final Thoughts

Kubernetes platforms evolve quickly, and clusters that were created with one set of assumptions can drift away from best practices over time.

This issue was ultimately straightforward to resolve, but it highlights how important it is to periodically review the underlying components of your cluster — not just the workloads running on top of it.

Sometimes a simple upgrade warning is actually an opportunity to improve the overall platform architecture.

If you’re interested in platform engineering, Kubernetes infrastructure, and DevSecOps, I’ll be writing more posts about real-world problems like this as they arise.

The shoot was a mix of digital and film. I used the Pentax 67 medium format for softness and depth, alongside my Sony full-frame mirrorless for flexibility. Lighting was simple but intentional: a one-light setup using the Godox AD600 Pro, a parabolic softbox, and a 40x60" reflector to shape and balance the light.

You will find 16 images and a BTS video from the shoot below. Let me know your thoughts—and if you are a stylist or makeup artist looking to collaborate on future shoots, do feel free to get in touch out.

Look 1: Pinstripe Power

At first glance, I wanted to channel something grown-up and editorial. Jemilia wore a grey pinstripe oversized blazer and matching wide-leg trousers over a textured white blouse and a layered red top. The bold silhouette contrasted beautifully against the track setting—corporate meets playground rebellion.

Look 2: Utility Denim

The second look leaned more streetwear—oversized denim trousers with utility-style hardware, paired with a neutral shirt and a layered mesh top with pops of red. I liked the toughness of this look, softened by her natural poise. It grounded the set and gave it weight.

Look 3: Relaxed Structure

The final look was the simplest—an oversized cream shirt styled open over a crisp white shirt and wide black trousers. Paired with trainers, this outfit brought out her casual confidence. It felt like a quiet close to the day—light, breathable, self-assured.

This shoot reminded me of how much joy there is in building from scratch—thrifting, layering, experimenting—and watching a look come to life on someone who carries it with ease. Jemilia brought patience, movement, and presence to each frame.

Let me know your thoughts on the images. If you’re a stylist, makeup artist, or creative looking to collaborate, my inbox is open.

—

Achieving cost reduction while enhancing infrastructure is a testament to strategic decision-making and technical finesse. This accomplishment contributes to immediate cost savings and sets the stage for long-term success and sustainability in cloud operations.

Cost is always a hot topic when running Kubernetes in the cloud, especially for small businesses or teams with limited budgets. The challenge is to find the right balance between maintaining a reliable, scalable, and secure infrastructure and optimising costs.

This achievement demonstrated a keen understanding of needs by recognising the importance of cost efficiency without compromising reliability, scalability, and security. Cost savings were achieved through meticulous analysis and optimisation of a Kubernetes deployment on Amazon EKS. This showcases proficiency in managing cloud resources effectively, leveraging only what is necessary to meet personal and business objectives.

I recently reduced my EKS costs from $404 to $280 while maintaining the same infrastructure and adding two extra nodes. This demonstrates my ability to support scalability and flexibility, which are crucial for adapting to evolving business requirements and accommodating growth without incurring unnecessary expenses. While this is just a small win in the grand scheme of things, as the saying goes, a win is a win.

Understanding AWS Cost Structure

Before diving into how I reduced my EKS costs, it's essential to understand how AWS charges for its services and what factors impact the overall cost of running Kubernetes in the cloud. AWS pricing is based on multiple factors: compute, storage, and data transfer.

Here are some key cost-impacting elements in AWS:

• Cross Availability Zone (AZ) Communication: Data transfers between different AZs within the same region incur costs. While redundancy across AZs is essential for high availability, it can increase expenses if not appropriately managed.

• Cross-Region Data Transfer: Moving data between AWS regions has higher costs than intra-region transfers, so optimising data flow where possible is crucial.

• Requests Leaving the Internal AWS Network: Any data egressing from AWS to the public internet incurs additional charges. These costs can add up quickly if your workloads interact with external APIs or serve external traffic.

• Storage and Persistent Volumes: Unused or over-provisioned storage, especially persistent volumes allocated to workloads, can lead to unnecessary charges.

• Idle and Underutilised Compute Resources: Running EC2 instances or Kubernetes nodes with low utilisation leads to wasted spending.

By understanding these key cost components, I could identify optimisation areas that would most impact reducing my expenses.

How I Reduced My EKS Costs

The key to optimising any cloud infrastructure is visibility. You can’t fix what you don’t understand. I deployed an open-source cost-monitoring tool called Kubecost into my EKS cluster to better understand my Kubernetes-related expenses.

Deploying Kubecost for Cost Visibility

Kubecost is a tool that provides real-time cost monitoring for Kubernetes workloads. It breaks down expenses by namespace, pod, and service. Integrating Kubecost into my cluster allowed me to track my spending and identify savings opportunities.

• Once Kubecost was running, I started analysing my infrastructure costs over time. I discovered a few key areas where I was overspending:

• Underutilised Nodes: Some nodes ran with low CPU and memory usage, meaning I was paying for resources I wasn’t fully utilising.

• Inefficient Pod Scheduling: Certain workloads ran on more expensive instances when they could be distributed more efficiently.

• EBS Volume Costs: I allocated persistent volumes that weren’t fully utilised.

The Optimisation Process

Armed with the insights from Kubecost, I took the following steps to optimise my EKS setup:

1. Right-Sizing Nodes: I reassessed and optimised my instance types for my workloads instead of sticking with my previous node configurations. This allowed me to switch to a more cost-effective instance type without sacrificing performance.

2. Bin Packing Workloads More Efficiently: I improved pod scheduling to use my nodes more efficiently by fine-tuning Kubernetes resource requests and limits. This allowed me to add two extra nodes while reducing my overall costs.

3. Cleaning Up Unused Resources: Kubecost helped me identify idle volumes and resources accumulating unnecessary costs. By removing or resizing them, I reduced additional expenses.

4. Spot Instances and Savings Plans: While I haven’t fully transitioned to spot instances yet, I started experimenting with them for non-critical workloads, which has led to further cost reductions.

Looking Ahead

This achievement shows adeptness at navigating the complexities of Kubernetes and cloud infrastructure. Technical expertise has been instrumental in realising these improvements, from fine-tuning resource allocation to implementing cost-saving measures without compromising performance.

Treating this achievement as a win and celebrating it encourages continuous improvement. This mindset ensures that cost optimisation efforts remain prioritised, leading to further enhancements and efficiencies over time.

Moving forward, I plan to:

• Automate cost-monitoring alerts using Kubecost.

• Experiment with spot instances for more workloads.

• Explore additional cost-saving features like AWS Graviton-based instances.

• Implement cluster auto-scaling more aggressively to match workload demand.

Optimising costs while enhancing infrastructure showcases proficiency in balancing business objectives with technical knowledge. I enjoy tackling the challenge of maintaining a scalable and resilient Kubernetes setup while controlling costs.

This small win has already proven valuable, and I’m excited to see what further cost-saving measures I can implement in the coming months.

What strategies have worked for you if you also work on Kubernetes cost optimisation? Let's discuss them!

Although I was fortunate to come from a family that could provide me with the opportunities I needed, I also recognised the experience of living in an environment that wasn’t always advantageous. These are environments with limited resources, and opportunities are even scarcer, but we compensate for our material deficiencies with imagination, community, and the incredible energy of youthful joy. We transformed the streets into our playground, simple objects into toys, and every moment into a memory that would last a lifetime. Observing these children in the Dominican Republic, I recognised that same spirit—an unbreakable resilience that turns any place into a paradise, regardless of circumstances.

I spent days photographing them, watching as they raced one another down the dusty roads of a beautiful tropical country, played games, and shared the kind of laughter that can only come from a pure and unburdened heart. They reminded me of my childhood friends back in Lagos—the way we would chase after each other in the rain and find adventure in the most mundane corners of our world.

What struck me most was their ability to find joy in a world that shapes them in so many ways. The Dominican Republic, like many places, is a land of contrasts—stunning landscapes, vibrant communities, and a spirit that shines through in its people. These children, much like the ones I grew up with, are not defined by circumstances but by their resilience, their energy, and their endless creativity. They embrace what surrounds them—the warmth of friendship, the excitement of discovery, and the simple pleasures that make childhood so full of life, turning even the ordinary into something extraordinary through their boundless imagination and endless creativity.

Photography has always been more than just a craft for me. It’s a way to tell stories that might otherwise go unnoticed and to capture emotions that words sometimes fail to articulate. This series, The Youthful Spirit of Tropical Paradise (2024), honours the universal spirit of youth—the ability to find happiness even in adversity and to create light in spaces where darkness might otherwise prevail.

Through these images, viewers see more than just the faces of these children—they witness the stories they tell. From the young girl having her hair done under the scorching sun to friends catching up on the school bus after a long day, these moments capture the essence of childhood.

These stories speak of perseverance, the beauty of growing up in a paradise shaped by nature’s abundance, and the unbreakable joy that defines childhood, no matter where in the world you are. They reveal the quiet strength found in everyday moments—the laughter shared between friends, the determination in a child’s eyes as they chase their dreams, and the simple pleasures of playing beneath open skies. They remind us that childhood is a time of boundless wonder, where imagination flourishes, friendships deepen, and joy is discovered in life’s simplest moments.

Because, in the end, paradise isn’t just a place. It’s a feeling, a moment, a state of mind. And these children have mastered the art of living in it.

In an age of continually evolving cyber threats, anticipating and mitigating security risks is more crucial than ever. One of the most effective ways to achieve this is through threat modelling, a structured approach to identifying vulnerabilities, understanding potential attack vectors, and mitigating risks before they materialise into security incidents. Threat modelling is not just a technical exercise but a strategic process enabling organisations to secure their applications, systems, and infrastructure proactively.

At its core, threat modelling involves systematically evaluating how an application, system, or process might be attacked. This approach enables security professionals, developers, and architects to anticipate weaknesses and devise robust countermeasures before they can be exploited. Unlike traditional reactive security measures that address vulnerabilities after a breach, threat modelling is a proactive discipline that aims to strengthen defences from the outset.

The Foundation of Threat Modelling

To understand threat modelling, it is essential to distinguish between key security concepts. A weakness is an inherent flaw in an application’s design or implementation that can be exploited if left unaddressed. A vulnerability manifests a weakness an attacker can leverage to compromise a system. Together, these elements form the foundation of security risks that must be mitigated.

An attack or security incident typically consists of three components:

• The target, which represents a valuable asset, such as a database containing sensitive user information.

• The attack vector is the route through which an attacker gains access, such as an unprotected form on a website.

• The actor, the individual or entity executing the attack.

The concept of an attack surface further broadens the discussion. It encapsulates all the potential entry points that an attacker can exploit, including physical elements like hardware, software components such as operating systems and applications, and network elements like open ports and firewall configurations.

Assessing Risk: Likelihood and Impact

Threat modelling is deeply rooted in risk assessment. Security teams evaluate risk based on two primary factors: likelihood and impact. Likelihood refers to the probability of a given threat occurring, while impact measures the severity of consequences should an attack be successful. Together, these elements define the overall risk:

risk = impact * likelihood

Understanding this equation helps prioritise security efforts, ensuring the most critical vulnerabilities are addressed first. Threat modelling allows organisations to move beyond vague security concerns and focus on concrete, actionable threats.

The Value of Threat Modelling in Cybersecurity

The importance of threat modelling cannot be overstated. By incorporating it into the development lifecycle, organisations gain significant advantages:

• Proactive security: Anticipating threats before they materialise reduces the risk of security breaches.

• Efficiency in security responses: Preparedness leads to faster, more effective incident handling.

• Improved prioritisation: Resources can be allocated effectively by addressing the highest-impact threats first.

• Enhanced security awareness: Development teams gain a deeper understanding of security considerations, leading to more secure coding practices.

When and How to Conduct Threat Modelling

Threat modelling is most effective when performed early in the development process. Ideally, it should begin in the requirement gathering and design phases, allowing security concerns to be addressed before they become deeply embedded in the system. However, continuous threat modelling throughout the software development lifecycle—particularly in Agile and DevOps environments—ensures that security remains a persistent focus.

There are various methodologies for implementing threat modelling, each with unique strengths. Some organisations adopt an asset-centric approach, securing critical assets like databases and servers. Others take an attacker-centric approach, attempting to think like a malicious actor to identify potential entry points and weaknesses. Another common strategy is the application-centric approach, where an application's architecture and data flows are mapped to assess security risks at each stage.

Choosing the Right Methodology

Selecting the right threat modelling methodology depends on several factors, including organisational maturity, security expertise, and business goals. Popular methodologies include:

• PASTA (Process for Attack Simulation and Threat Analysis) is an iterative, business-driven approach emphasising dynamic threat landscapes.

• Microsoft Threat Modelling: A developer-friendly framework that integrates seamlessly into the software development lifecycle.

• OCTAVE (Operationally Critical Threat, Asset and Vulnerability Evaluation): A risk-analysis framework for evaluating organisational security risks.

• TRIKE: An automated, asset-centric threat modelling tool for risk assessment.

• VAST (Visual Agile Simple Threat Modelling): Tailored for Agile teams, enabling application and operational threat modelling in dynamic development environments.

Each methodology offers unique advantages, and organisations must choose an approach that aligns with their security needs and workflows.

The Threat Modelling Process in Action

Regardless of the methodology chosen, the general process of threat modelling follows a structured series of steps:

1. Define the scope: Identify what assets and components are within the scope for analysis.

2. Map the system architecture: Create data flow diagrams to visualise the system's architecture.

3. Identify security risks: Assess potential threats, vulnerabilities, and attack vectors.

4. Categorise threats: Use frameworks like STRIDE (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, and Elevation of Privilege) to classify threats.

5. Prioritise risks: Rank threats based on their impact and likelihood.

6. Implement countermeasures: Develop security controls to mitigate identified threats.

7. Document and iterate: Maintain a continuous feedback loop to refine the security posture.

Conclusion

Threat modelling is not a one-time task but an ongoing discipline that should be integrated into every software and system development stage. By identifying potential threats early, organisations can proactively strengthen their security defences, minimise risks, and build resilient applications.

Whether leveraging an asset, attacker, or application-centric approach, the key to effective threat modelling lies in systematic assessment, informed prioritisation, and continuous iteration. In today’s cyber landscape, organisations that invest in proactive security measures like threat modelling are best positioned to withstand evolving threats and safeguard their digital assets.

References:

Threat Modeling Manifesto

A few years ago, I wrote about logging best practices in traditional applications on my Medium account. However, logging has also evolved as software architectures have shifted towards microservices and containerised environments like Kubernetes. The old strategies still apply but must be extended to work effectively in distributed dynamic systems.

This revised post explores effective logging practices in microservices and Kubernetes applications. It clarifies each concept, details successful implementation strategies, and highlights key takeaways.

Structured Logging: Essential for Scalability and Security

What is Structured Logging?

Structured logging is a way of logging data in a predictable, machine-readable format, often JSON. In microservices, structured logging ensures logs are easily parsed, searchable, and consistently formatted across different services. However, logging should also be considered part of application security. For example, we audit infrastructure for security and compliance, and logging allows us to audit application events, trace back actions, and detect anomalies.

Why Logging is a Security Protocol

Logging isn’t just for debugging—it provides a historical record of system events, making it a crucial security component. By maintaining structured logs, organisations can:

• Detect tampering attempts or suspicious activities in real time.

• Identify unauthorised access attempts and security breaches.

• Ensure compliance with industry regulations like SOC 2, HIPAA, and GDPR.

• Investigate incidents through forensic analysis when issues arise.

• Mitigate repudiation attacks, where users deny performing specific actions by keeping immutable logs of all key events.

Log Privacy & Data Anonymisation

Logs should avoid storing sensitive user data to comply with regulations like GDPR and CCPA. Tokenisation and redaction can help prevent the exposure of personal data.

Example: Instead of logging:

{ "email": "user@example.com", "creditCard": "4242424242424242" }

Use:

{ "email": "u***@example.com", "creditCard": "**** **** **** 4242" }

How to Implement Structured Logging Securely

• Use JSON format for logs instead of unstructured text.

• Ensure all logs contain key security-relevant fields such as timestamps, request origins, and user activity.

• Implement role-based access control (RBAC) to restrict access to logs.

• Encrypt logs both in transit and at rest.

• Integrate logs with SIEM (Security Information and Event Management) solutions like Splunk or Datadog for proactive security monitoring.

What We Have

• Logs should be machine-readable (JSON format is preferred) for log aggregators.

• Easily searchable using tools like ELK, Loki, and Splunk.

• Ensures compliance and security auditing by logging critical system events.

• Allows efficient debugging in a distributed system.

Example using Serilog in C#:

Log.Information("User {UserId} performed an action at {Timestamp}", userId, DateTime.UtcNow);

We can build a more resilient and auditable system by treating logging as a security mechanism rather than just a debugging tool.

Providing Context in Logs

What is Context in Logging?

Context in logging refers to adding meaningful information to log entries to ensure that logs are easily understandable and traceable. It helps engineers and operators quickly determine what happened, where, and why when debugging or analysing system behaviour.

Context includes details such as:

• The application or service name.

• The request ID or correlation ID for tracking.

• Timestamps to track when the event occurred.

• The user or session ID (if applicable and security compliant).

• The method or function that generated the log.

However, while providing context, security must be a top priority—logs should never contain sensitive data such as passwords, or financial details.

How to Implement Context in Logs

• Include metadata such as serviceName, requestId, and timestamp in each log.

• Use structured logging formats like JSON to standardise context across logs.

• Leverage Correlation IDs to link logs across multiple services (see the section on Correlation IDs below).

• Avoid logging sensitive information to comply with security best practices.

Example structured log with context:

{
  "timestamp": "2024-01-29T12:34:56Z",
  "service": "user-service",
  "level": "INFO",
  "message": "User logged in",
  "userId": "12345",
  "requestId": "abcd-efgh-ijkl"
}

Providing structured and contextual logs ensures that logs remain useful for current developers and future teams who may need to debug issues years later.

What We Have

• Contextual logs that provide clarity and traceability.

• Use of structured formats like JSON to ensure uniformity.

• Emphasis on avoiding sensitive data exposure.

• Correlation ID integration for tracking distributed requests.

What is a Correlation ID?

A Correlation ID is a unique identifier assigned to a request or transaction as it traverses multiple services in a distributed system. It connects all logs associated with that request, simplifying the tracing of user actions, debugging of issues, and analysis of system performance.

Why Correlation IDs Are Important?

• Enhances Traceability: Connects logs from various microservices that manage the same request.

• Improves Debugging: Aids engineers in tracking a request's journey through the system.

• Supports Performance Monitoring: Identifies bottlenecks in request processing.

• Facilitates Business Intelligence: Aggregates logs to better understand user behaviour and transactions.

Example Use Case

A user starts a checkout request on an e-commerce site. This request activates calls to:

1. Authentication Service (to verify user identity)

2. Order Service (to create an order)

3. Payment Service (to process payment)

4. Inventory Service (to update stock)

By assigning a Correlation ID (e.g., correlationId: "xyz-123-abc"), logs from all these services can be connected, offering a comprehensive view of the request lifecycle.

Deciding Log Levels: Filtering Noise in a Microservices World

What are Log Levels?

Log levels categorise logs by severity, helping filter out unnecessary noise while surfacing critical issues. They define the importance of a log message and guide engineers in prioritising issues.

Common Log Levels:

• DEBUG: Provides detailed information for diagnosing issues during development. It should not be used in production.

• INFO: General operational messages that indicate normal application behaviour.

• WARN: Indicates a potential issue that does not affect functionality but may require attention.

• ERROR: Captures failures in the application that need investigation but do not cause an immediate crash.

• FATAL: Represents critical failures that may bring down the application.

Why It’s Important to Standardise Log Levels

Consistently using the correct log levels ensures that logs are meaningful and actionable. Teams should agree on how to classify logs to prevent overuse of ERROR or INFO messages where WARN would be more appropriate. This consistency helps:

• Enable effective log filtering to concentrate on genuine issues.

• Enhance log analysis and debugging by minimising unnecessary noise.

• Assist in prioritising fixes based on log severity.

• Assist junior engineers in understanding log categorisation via code reviews.

How to Implement Log Levels Effectively

• Define a standard log-level policy for the team.

• Review log levels during code reviews to maintain consistency and educate junior developers.

• Use dynamic log-level configurations to adjust verbosity in production without redeploying services.

• Ensure logging platforms allow filtering based on log levels for targeted analysis.

What We Have

• Clearly defined log levels for categorisation.

• Use of dynamic log-level settings in Kubernetes via environment variables.

• Encourage discussion on log levels during code reviews.

Performance Considerations in Logging

Logging and Its Impact on Performance

Although logging is essential, it can introduce performance overhead, particularly in high-throughput applications developed in C#, Java or any other language. Frequent and excessive logging can:

• Elevated CPU usage due to string formatting and file I/O operations.

• Cause disk bloat if logs are not rotated and archived correctly.

• Slow down the application when synchronous logging blocks execution.

How to Mitigate Performance Issues

• Use Asynchronous Logging: Libraries like Serilog and NLog in C# support asynchronous logging to avoid blocking execution.

• Batch Logging: Instead of writing logs immediately, accumulate and write them in batches to reduce I/O overhead.

• Stream Logs to Kafka: By passing logs through a distributed streaming system like Apache Kafka, logs can be processed asynchronously in a scalable manner. Kafka ensures:

  • Logs are distributed efficiently across consumers.

  • If an error occurs, processing can resume from the last offset.

  • Logs can be replayed if necessary.

• Implement Log Rotation: To retain logs efficiently, use log rotate or similar tools to archive and remove old logs, preventing disk space exhaustion.

Recommended Tools for Performance Optimisation

• Serilog & NLog: Support asynchronous logging and structured output.

• Apache Kafka: Distributed log streaming and processing.

• logrotate: Automates log rotation and cleanup to free disk space.

What We Have

• Use of asynchronous logging to prevent performance bottlenecks.

• Streaming logs to Kafka for distributed processing.

• Log rotation to manage disk space efficiently.

Rate-Limiting and Log Sampling

Rather than logging every request, only log a sample of non-critical messages, thereby reducing storage overhead.

Improvements to Logging Practices: Alerts, Multi-Environment Strategies, and Business Intelligence

Logging is not merely about capturing data; it’s about making it actionable. The improvements outlined below can enhance logging practices, rendering logs more valuable for monitoring, debugging, and business intelligence.

Alerts & Proactive Monitoring

What is Alert-Based Logging?

Alert-based logging triggers real-time notifications for anomalies, allowing teams to respond proactively. Instead of manually sifting through logs, alerts highlight key issues automatically.

How to Implement Alert-Based Logging

• Identify Essential Metrics: Pinpoint necessary logs, such as elevated error rates, unsuccessful transactions, or security violations.

• Establish Alert Notifications: Set up real-time notifications using log monitoring platforms like Grafana, Datadog, or Splunk.

• Connect with Communication Platforms: Set up alerts to inform teams through channels like Slack, PagerDuty, or Microsoft Teams.

• Optimise Alert Settings: Prevent alert fatigue by defining thresholds that guarantee only significant issues trigger notifications.

Example Alert Setup

• Configure Grafana to send an alert when the error log count exceeds a predefined threshold:

alert:
  conditions:
    - type: threshold
      threshold: 100
      operator: gt
      field: "log_error_count"
  actions:
    - notify: slack_channel
      message: "A high error rate has been detected in the production logs!"

Benefits of Alert-Based Logging

• Faster Incident Response: Detect and fix issues before they escalate.

• Automated Monitoring: Reduce manual log searches.

• Improved Reliability: Maintain system health by proactively addressing problems.

2. Multi-Environment Logging Strategies

What is Multi-Environment Logging?

Over logging ??

Different environments (development, staging, production) necessitate distinct logging strategies. Excessive logging in production may lead to performance issues, whereas insufficient logging in development can hinder debugging efforts.

We may not require the same logging level in the development environments as in the production environment. This will help us reduce noise, improve performance, or save costs.

How to Implement Multi-Environment Logging

• Set Environment-Specific Log Levels: Configure different log levels for each environment.

• Use Configuration Management: Store logging configurations in environment variables or configuration files.

• Filter Logs by Environment: Use log aggregation tools to separate logs by environment.

Example Multi-Environment Log Configuration in C#

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Development": "Debug",
      "Production": "Warning"
    }
  }
}

Benefits of Multi-Environment Logging

• Improved Debugging in Development: Enhanced logging allows developers to pinpoint problems more effectively.

• Enhanced Production Efficiency: Minimizing excessive logs conserves resources.

• Distinct Log Separation: Simplifies the identification of various environments in log management systems.

3. Logs as a Business Intelligence Tool

What is Business Intelligence Logging?

A business intelligence log entry is a log record that documents information pertinent to company operations, user engagement, system efficiency, and feature usage. These logs enable organisations to derive valuable insights for informed decision-making, analytics, and optimisation.

How to Implement Business Intelligence Logging

• Record Essential Business Events: Document occurrences like API response times, user interactions, and feature usage.

• Focus on operational and user behaviour trends rather than just debugging.

• Consolidate and Analyse Logs: Utilise platforms such as BigQuery, Snowflake, or AWS Athena to derive insights from structured logs.

• Design Dashboards: Develop live visualisations in Grafana or Kibana to monitor critical metrics.

Example Business Intelligence Log Entry

{
  "event": "APIResponseTime",
  "endpoint": "/checkout",
  "latency_ms": 250,
  "timestamp": "2024-01-29T12:35:00Z"
}

Benefits of Business Intelligence Logging

• Improve User Experience: Identify slow endpoints and optimise performance.

• Feature Adoption Tracking: Understand which features users engage with most.

• Data-Driven Decisions: Use logs to inform product development and operational improvements.

Wrapping Up: The Modern Approach to Logging

Logging in microservices and Kubernetes requires a structured, security-focused, and performance-optimised approach. Following these best practices can enhance debugging, security, and observability.

Your logs, with structured logging, correlation IDs, and observability tools, will help you debug and improve system reliability and security.

Organisations can extract more value from their logs by integrating alerts and monitoring, multi-environment logging, and business intelligence practices. Logging isn’t just about capturing data—it’s about using it to proactively manage applications, optimise performance, and drive business decisions.

Happy blogging!

Upgrading Kubernetes nodes is a critical task that ensures your cluster remains secure, performant, and compatible with the latest features. However, it can feel like walking a tightrope—especially in production environments where downtime isn’t an option. In this guide, I’ll walk you through the step-by-step process to upgrade a Kubernetes node while minimising risks and ensuring your workloads remain unaffected.

Why Upgrade Kubernetes Nodes?

Kubernetes updates bring essential improvements in security, bug fixes, and new features. Upgrading nodes ensures your cluster:

• Adheres to compatibility policies (version skew).

• Maintains security compliance.

• Performs optimally under new workloads.

That said, let’s get into the practical steps.

Step 1: SSH Into the Node

The first step is to log into the target node where the upgrade will take place. Use your preferred method:

ssh user@<node-ip>

Step 2: Validate Cluster Health Before Starting

Before upgrading, ensure that the cluster is in a healthy state to avoid exacerbating any existing issues.

kubectl get nodes kubectl get pods -A kubectl get events -A

Look for NotReady nodes, failing Pods, or unusual events before proceeding.

Step 3: Drain the Node

To safely upgrade the node, workloads running on it must be evicted and rescheduled to other nodes. This prevents disruptions during the upgrade.

1. Mark the Node as Unschedulable: Prevent new workloads from being scheduled:

kubectl cordon <node-name>

1. Evict Workloads: Drain all running Pods (except DaemonSets and static Pods):

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Note: Pods using emptyDir volumes will lose their data.

Step 4: Backup Cluster State

Take a backup of your cluster state to prevent data loss in case of failure. This is particularly important for clusters using etcd.

ETCDCTL_API=3 etcdctl snapshot save backup.db --endpoints=<etcd-endpoint> --cacert=<ca-file> --cert=<cert-file> --key=<key-file>

Adjust the command for your environment as needed.

Step 5: Check Current Versions

Before upgrading, note the current versions of kubeadm, kubelet, and kubectl on the node:

kubectl version kubelet --version kubeadm version

This helps ensure a seamless upgrade process by comparing against available updates.

Step 6: Validate the Upgrade Plan

Run the following command to validate the kubeadm upgrade plan and ensure compatibility:

kubeadm upgrade plan

This step provides details about the available versions and any potential conflicts.

Step 7: Upgrade Using kubeadm

Run the following command to upgrade the node’s Kubernetes components:

kubeadm upgrade node

If the version matches the control plane, Kubernetes skips the upgrade process for that node.

Step 8: Update Package Repositories

Ensure your package manager’s repositories are up-to-date so you can fetch the latest Kubernetes packages:

sudo apt update

Step 9: Check Latest Version Numbers

To verify the latest available versions of kubeadm, kubectl, and kubelet, use:

apt show kubeadm -a | grep Version 
apt show kubectl -a | grep Version 
apt show kubelet -a | grep Version

Step 10: Install the Latest Version

Install the updated versions of Kubernetes components:

sudo apt install -y kubelet=<version> kubectl=<version> kubeadm=<version>

Step 11: Verify the Upgrade

After installation, confirm that the components are running the correct versions:

kubectl version kubelet --version kubeadm version

Step 12: Restart Critical Pods (Optional)

After restarting kubelet, ensure critical system Pods like kube-proxy or CNI (network plugins) are running correctly.

kubectl get pods -n kube-system

Look for any Pods in CrashLoopBackOff or Pending states and restart them if necessary:

kubectl delete pod <pod-name> -n kube-system

Step 13: Restart the kubelet Service

Restart the kubelet service to apply the updates:

sudo systemctl restart kubelet

Step 14: Rejoin the Node (if Necessary)

If the node fails to join the cluster automatically, you can rejoin it manually:

1. On the Control Plane: Create a token and retrieve the join command:

kubeadm token create --print-join-command

1. On the Worker Node: Run the provided join command:

kubeadm join <control-plane-ip>:6443 --token <token> --discovery-token-ca-cert-hash <hash>

1. Restart the kubelet service:

sudo systemctl restart kubelet

Step 15: Uncordon the Node

Once the upgrade is complete, allow the node to schedule workloads again:

kubectl uncordon <node-name>

Step 16: Verify Node Health

Ensure the node has successfully rejoined the cluster and is ready to accept workloads:

kubectl get nodes

Check the status of the node—it should show as Ready.

Key Notes to Remember

• Always follow Kubernetes Version Skew Policies to ensure compatibility between the control plane and nodes.

• Draining nodes is critical to maintaining workload availability during upgrades.

• Keep an eye on resource usage after the upgrade to confirm stability.

Upgrading Kubernetes nodes can feel daunting, but by following these steps, you can execute the process confidently and safely. Keeping your cluster up-to-date ensures security, reliability, and the ability to leverage the latest Kubernetes features.

Did this guide help you? Share your thoughts or questions in the comments below!

Subscribe to this Substack for more Kubernetes and DevOps tips delivered straight to your inbox.

As Kubernetes continues to dominate the landscape of container orchestration, many developers and system administrators are opting for self-managed clusters to gain complete control over their infrastructure. In this guide, I’ll walk you through the steps required to set up a self-managed Kubernetes cluster using kubeadm.

By the end of this tutorial, you’ll have a functional Kubernetes cluster ready to handle workloads. Let’s dive in!

📝 Prerequisites

Before beginning, ensure that you have:

• An Odd Number of Servers: For failure tolerance, Kubernetes clusters work best with an odd number of servers. This setup allows for a simple quorum, following the formula (N-1)/2 to determine the number of tolerated failures. For a minimal setup, you can start with 1 master node (or control plane node) and 2 worker nodes.

• Ubuntu 20.04 or a similar Linux distribution on your nodes.

• At least 2 CPUs and 2 GB of RAM on each node (one control plane node and at least one worker node).

• The control plane node should have at least 20 GB of storage while the worker nodes should have at least 10 GB of storage.

• A reliable network connection

• Root access or a user with sudo privileges during set up.

Step 1: Prepare the Environment

Note: Repeat Step 1 on both the control plane (master) node and each worker node. This will ensure all nodes are prepared with the necessary dependencies.

Update System Packages

Before installing Kubernetes components, let’s make sure your system packages are up to date. Run:

sudo apt update && sudo apt upgrade -y

Add Kubernetes Package Repository

1. Download the Kubernetes GPG key:

sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

1. Add the Kubernetes apt repository:

echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

1. Update package lists:

sudo apt update

For more details, refer to the Kubernetes Installation Guide.

Step 2: Install Kubernetes Tools

Note: The first part of this step, where we install kubeadm, kubelet, and kubectl, should also be repeated on each worker node. These tools are required on both the control plane and worker nodes.

Install kubeadm, kubelet, and kubectl

These tools are essential for managing the Kubernetes cluster:

sudo apt install -y kubeadm kubelet kubectl

Install Container Runtime (containerd)

Kubernetes requires a container runtime like containerd to run containers. Install it with:

sudo apt update && sudo apt install -y containerd

Configure CGroups for containerd and kubelet

Control groups, or cgroups, are a Linux kernel feature that manage and allocate system resources such as CPU, memory, disk I/O, and network bandwidth for groups of processes. In the context of Kubernetes, cgroups play a crucial role in ensuring that containers are efficiently managed and isolated. Here’s what cgroups do:

1. Resource Allocation and Limiting:
   Cgroups allow Kubernetes to set limits and requests on the amount of CPU and memory a container can use. For example, you can restrict a container to use a maximum of 512 MB of memory or a limited percentage of CPU resources. This prevents any single container from consuming all resources on the node.

2. Resource Isolation:
   By isolating resources, cgroups ensure that containers and their processes run independently without interfering with each other. If one container crashes or spikes in resource usage, cgroups help prevent it from impacting other containers.

3. Resource Monitoring:
   Cgroups provide visibility into resource usage, allowing Kubernetes to monitor the CPU and memory consumption of containers and make decisions based on this data (like scaling or scheduling).

4. Resource Prioritization:
   Cgroups can be used to prioritize resources for certain containers over others, ensuring critical services receive the resources they need, even under high load.

Overall, cgroups help Kubernetes enforce resource limits, maintain stability, and optimize performance, which is essential for managing containerized applications in a multi-tenant environment.

Kubernetes components need to use consistent cgroup drivers. There are two main CGroup drivers, which are:

1. systemd cgroup driver

• Description: This driver uses systemd as the cgroup manager, which is the default init system on most modern Linux distributions (like Ubuntu and CentOS).

• Usage: Preferred for Kubernetes clusters as it aligns with the native Linux systemd init system, leading to better compatibility and stability in resource management.

1. cgroupfs cgroup driver

• Description: This driver uses cgroupfs as the cgroup manager, which directly manages cgroups within the filesystem.

• Usage: Often used with container runtimes like Docker but may lead to issues in Kubernetes because it can conflict with systemd. It’s generally recommended to switch to systemd for Kubernetes clusters, especially in production environments.

Note: Consistency between the cgroup driver used by kubelet and the container runtime (such as containerd or Docker) is essential for stable cluster operations. Using the systemd driver is typically recommended for production environments.

Let’s set them to use systemd . With this being the default, all we have to do is verify that the current init system is using systemd, if so then we do not need to do anything. 

1. Check the current init system by running:

ps -p 1

1. Create the configuration path for containerd:

sudo mkdir -p /etc/containerd

1. Generate the default containerd config:

sudo containerd config default | sudo tee /etc/containerd/config.toml

1. Edit the config to set SystemdCgroup to true:

sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

1. Verify the setting:

cat /etc/containerd/config.toml | grep -i SystemdCgroup -B 50

1. Restart containerd:

sudo systemctl restart containerd

Set the kubelet Cgroup Driver to cgroupfs (Optional)

This section can be skipped if you are using the default systemd driver. The whole point is to make sure we keep the container runtime and the kubelet using the same driver. 

1. Set the kubelet Cgroup Driver to cgroupfs

1. Edit the kubelet configuration file, typically located at /var/lib/kubelet/config.yaml:

sudo nano /var/lib/kubelet/config.yaml

1. Find the cgroupDriver setting and set it to cgroupfs:

cgroupDriver: cgroupfs

1. Save and close the file.

2. Restart kubelet to apply the changes:

sudo systemctl restart kubelet

If the kubelet configuration file doesn’t already exist, you can also specify the cgroup driver directly by adding the --cgroup-driver=cgroupfs flag in the kubelet service configuration file (typically at /etc/systemd/system/kubelet.service.d/10-kubeadm.conf), then restart the kubelet service.

2. Set the Container Runtime Cgroup Driver to cgroupfs

For containerd

1. Edit the containerd configuration file located at /etc/containerd/config.toml:

sudo nano /etc/containerd/config.toml

1. Locate the SystemdCgroup option under the [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] section, and set it to false:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]   SystemdCgroup = false

1. Save and close the file.

2. Restart containerd to apply the changes:

sudo systemctl restart containerd

• For Docker

1. Create or edit the Docker daemon configuration file located at /etc/docker/daemon.json:

sudo nano /etc/docker/daemon.json

1. Set the cgroup driver to cgroupfs in the JSON configuration:

{   "exec-opts": ["native.cgroupdriver=cgroupfs"] }

1. Save and close the file.

2. Restart Docker to apply the changes:

sudo systemctl restart docker

For more details, refer to the Container Runtime Configuration in Kubernetes documentation.

Step 3: Initialize the Control Plane Node

Disable Swap

Kubernetes requires swap to be turned off, the reason is Kubernetes manages memory resources for containers and relies on accurate memory allocation and usage data. If swap is enabled, it can interfere with Kubernetes’ ability to monitor and control resource usage accurately. This can lead to unexpected behavior, where containers exceed memory limits or become unstable, which affects overall cluster stability.

sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstab

Enable IP Forwarding

IP forwarding is a feature in the Linux kernel that allows the server to forward network packets from one network interface to another. By default, many Linux distributions disable IP forwarding, which means the system will not route packets between network interfaces.

To ensure network traffic flows correctly between different network interfaces, IP forwarding is essential in Kubernetes. This will enable pod-to-pod communication between nodes.

Why IP Forwarding is Needed in Kubernetes

1. Pod-to-Pod Communication Across Nodes:

• Kubernetes creates a virtual network where each pod can communicate with other pods, even if they’re on different nodes. For this to work, IP packets need to be routed between different interfaces (e.g., between the pod network interface and the node’s main network interface).

• Enabling IP forwarding allows each node to forward packets between these interfaces, facilitating cross-node communication in the cluster.

1. Service and Network Routing:

• Kubernetes uses various service types (like ClusterIP, NodePort, and LoadBalancer) to expose applications to internal and external traffic.

• With IP forwarding enabled, packets can be routed to their destination service or pod IP, even if that IP is on a different node within the cluster. This is crucial for Kubernetes networking components, such as kube-proxy, to correctly handle traffic routing.

1. Network Plugins and CNI (Container Network Interface):

• Many CNI plugins (like Flannel, Calico, and Weave) rely on IP forwarding to handle the complex routing required for networking in a multi-node cluster.

• Enabling IP forwarding ensures that the network plugins can set up and manage pod networking, allowing pods to communicate seamlessly across the cluster.

Enable IP forwarding to allow packets to be routed between different network interfaces:

#permanently change it
echo "net.ipv4.ip_forward = 1" | sudo tee -a /etc/sysctl.conf 

#to temporarily change it
sudo sysctl -w net.ipv4.ip_forward=1 

Or 

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
sudo sysctl --system

Initialize the Kubernetes Cluster

Let’s initialize the cluster on the control plane node. Replace <control-plane-ip> with the actual IP of your control plane node:

sudo kubeadm init --apiserver-advertise-address=<control-plane-ip> --pod-network-cidr="10.244.0.0/16" --upload-certs

Refer to the kubeadm Cluster Setup Guide for more details.

Configure kubectl for Cluster Access

Now, let’s set up kubectl so you can manage the cluster:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Step 4: Install Pod Network (CNI)

Install Flannel (Example CNI)

Kubernetes requires a container network interface (CNI) for pod-to-pod communication. Here, we’ll use Flannel. If you encounter issues with pods not running due to br_netfilter, load the module with:

sudo modprobe overlay | bridge #depending on the module being used.
sudo modprobe br_netfilter

Download and Configure Flannel

Kubernetes needs a Container Network Interface (CNI) for pod-to-pod communication. We’ll use Flannel as our CNI. Follow these steps to download and configure it:

1. Download the Flannel YAML configuration file and save it as kube-flannel.yml:

curl -LO https://raw.githubusercontent.com/flannel-io/flannel/v0.20.2/Documentation/kube-flannel.yml

1. Edit the Flannel Configuration:

• Open the kube-flannel.yml file using a text editor:

cd kube-flannel.yml

• Locate the args section within the kube-flannel container definition. It should look like this:

args:   
- --ip-masq   
- --kube-subnet-mgr

• Add an additional argument to specify the network interface:

args:   
- --ip-masq   
- --kube-subnet-mgr   
- --iface=eth0

• Adding --iface=eth0 ensures Flannel uses the correct network interface for pod communication.

1. Save and exit the file.

2. Deploy Flannel:

kubectl apply -f kube-flannel.yaml

For more about Flannel configuration, refer to the Flannel Documentation.

Step 5: Verify Control Plane Status

Run the following command to check if the control plane node is ready:

kubectl get nodes

You should see the control plane node in a “Ready” state if everything is set up correctly.

Step 6: Join Worker Nodes to the Cluster

Set Unique Hostnames for Worker Nodes

Each worker node should have a unique hostname. Set it with:

sudo hostnamectl set-hostname <worker-node-name>

Join Worker Nodes

After initializing the control plane, kubeadm generates a command to join worker nodes. It will look similar to this:

kubeadm join <control-plane-ip>:6443 --token <token> --discovery-token-ca-cert-hash <hash>

Run the command on each worker node.

Refer to the Adding Nodes with kubeadm for more information.

Troubleshoot Common Issues

If you encounter errors related to unsupported fields in the kubeadm config file, such as caCertificateValidityPeriod or certificateValidityPeriod, open the config file:

sudo nano <your-kubeadm-config-file>.yaml

Remove the unsupported fields if using v1beta3.

🧪 Step 7: Run Kubernetes End-to-End (E2E) Tests

Kubernetes provides a set of E2E tests to ensure that all cluster components are working together as expected. These tests are especially useful for validating your setup, troubleshooting issues, and ensuring the cluster meets expected performance and reliability standards.

Prerequisites for E2E Tests

1. Install kubectl and have access to your Kubernetes cluster.

2. Make sure the kubeconfig file is correctly configured, so kubectl can interact with your cluster:

export KUBECONFIG=$HOME/.kube/config

1. Install Go (if it’s not already installed):
   Kubernetes E2E tests require Go, as they are implemented in Go code.

2. Clone the Kubernetes Repository:
   The E2E test scripts are included in the Kubernetes GitHub repository.

git clone https://github.com/kubernetes/kubernetes.git cd kubernetes

Running Basic E2E Tests

Kubernetes includes a variety of E2E tests, from simple checks to complex scenarios. Follow these steps to run a basic test suite.

1. Build the E2E Test Binary:

make WHAT=test/e2e/e2e.test

1. Run the E2E Test Suite:

2. The following command will run the E2E test suite against your cluster. Replace <your-cluster-ip> with your actual API server IP.

./_output/local/bin/linux/amd64/e2e.test --host=https://<your-cluster-ip>:6443

3. Specify tests to run by using the --ginkgo.focus flag if you want to run specific tests. For example, to run only the tests focused on pod creation, use:

./_output/local/bin/linux/amd64/e2e.test --ginkgo.focus="\[sig-scheduling\] Pod should"

4. Viewing Test Results:

The E2E tests will output results to the console, indicating success or failure for each test.

• You can also export logs to a file:

./_output/local/bin/linux/amd64/e2e.test --host=https://<your-cluster-ip>:6443 > e2e_test_results.log

Running Tests with Sonobuoy (Alternative Method)

Sonobuoy is a diagnostic tool designed to run Kubernetes conformance tests and can simplify the process of running E2E tests across different Kubernetes clusters.

1. Download Sonobuoy: Download Sonobuoy:

curl -L https://github.com/vmware-tanzu/sonobuoy/releases/latest/download/sonobuoy_$(uname -s)_$(uname -m).tar.gz | tar xvz sudo mv sonobuoy /usr/local/bin

1. Run Sonobuoy Tests:

2. Run Sonobuoy to start the conformance test suite:

sonobuoy run

• Check the test status:

sonobuoy status

1. Retrieve and View Results:

2. When tests are complete, download the results:

sonobuoy retrieve .

• Extract the tarball:

tar -xf *.tar.gz

• Results will be stored in a set of JSON files, providing detailed insights into which tests passed or failed.

Additional Testing Tips

• Run Tests on a Dedicated Test Cluster: Running E2E tests can place a significant load on the cluster, so it’s advisable to run them in a testing or staging environment.

• Customize Test Runs: For large clusters, it might be useful to limit test scope by using tags like --ginkgo.focus or --ginkgo.skip to focus on specific test types.

• Regular Testing: For production-grade clusters, consider integrating E2E tests into a CI/CD pipeline to monitor cluster health continuously.

More on E2E testing is available in the [Kubernetes E2E Testing Guide](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests)

🎉 Wrapping Up

Congratulations! You’ve successfully set up a self-managed Kubernetes cluster. From here, you can deploy applications, set up monitoring, and explore Kubernetes’ powerful orchestration capabilities.

For more complex configurations, like high availability clusters, consider diving deeper into the Kubernetes documentation.

Happy Clustering!