Deploying microservices to a Kubernetes cluster with CI/CD: A Step-by-Step Guide

In this blog post, we’ll guide you through deploying a Kubernetes application using Jenkins. This guide will help you set up a CI/CD pipeline for deploying microservices to a Kubernetes cluster. We’ll cover creating roles, binding them to service accounts, and setting up a Jenkins pipeline for automation.

Prerequisites

Clone repository for terraform code
App Repo
Jenkins Installation: Ensure Jenkins is installed and running.
Kubernetes Cluster: Have a Kubernetes cluster set up and accessible.
Kubernetes CLI (kubectl): Install and configure kubectl on your system.
Jenkins Kubernetes Plugin: Install the Kubernetes plugin for Jenkins to enable Kubernetes integration.
Service Account: Create a Kubernetes service account with appropriate roles and permissions.
Token Generation: Generate a token for the service account to be used by Jenkins for authentication.

Key Points

Create a Kubernetes Role: Define permissions for resources.
Bind Role to Service Account: Assign permissions to the service account.
Generate and Use a Token: Authenticate the service account.
Set Up Jenkins Pipeline: Automate deployment and verification.

Setting Up the Environment

I have created a Terraform file to set up the entire environment, including the installation of required applications, tools, and the EKS cluster automatically created.

Environment Setup

HostName	OS
terrabox-svr	Ubuntu 24 LTS

Password for the root account on all these virtual machines is xxxxxxx

Perform all the commands as root user unless otherwise specified

Change the hostname: (optional)

sudo hostnamectl set-hostname Jenkins

Update the /etc/hosts file:
- Open the file with a text editor, for example:

sudo vi /etc/hosts

Replace the old hostname with the new one where it appears in the file.

Apply the new hostname without rebooting:

sudo systemctl restart systemd-logind.service

Verify the change:

hostnamectl

Update the package

sudo -i
apt update

Setup the Jenkins

Note down the public address of the VM and access it in the browser

<publicIP of VM :8080>

will run this command on VM sudo cat /var/lib/jenkins/secrets/initialAdminPassword to get the first time password.

Install the plugin in Jenkins

Docker
Docker Pipeline
Kubernetes
Kubernetes CLI
Multibranch Scan Webhook Trigger
Pipeline: Stage View

Configure tools in Jenkins

Configure the docker

Name- docker

[x] install automatically

docker version: latest

Set docker cred in Jenkins

Dashboard>Manage Jenkins > Credentials> System> Global credentials (unrestricted) ⇒ Click on "New credentials"

kind: "username with password"

username: your docker login ID

password: docker token

Id: docker-cred #it would be used in pipeline

Description:docker-cred

Creating a multipipeline in Jenkins:

name: microservice-ecommerce

item type: Multibranch pipeline

Syntax to configure the webhooks in github JENKINS_URL/multibranch-webhook-trigger/invoke?token=[Trigger token]

http://18.234.174.99:8080/multibranch-webhook-trigger/invoke?token=singh

go to github repo > setting> webhooks

Once you configure webhook then build the pipeline and you will see successfull build.

Images view from Docker Hub

Setup the EKS Cluster

On the virtual machine, Go to directory k8s_setup_file and open the file cat apply.log to verify the cluster is created or not.

ubuntu@ip-172-31-90-126:~/k8s_setup_file$ pwd
/home/ubuntu/k8s_setup_file
ubuntu@ip-172-31-90-126:~/k8s_setup_file$

Once EKS cluster is setup then need to run the following command to make it intract with EKS.

aws eks update-kubeconfig --name <cluster-name> --region 
<region>

The aws eks update-kubeconfig a command is used to configure your local kubectl tool to interact with an Amazon EKS (Elastic Kubernetes Service) cluster. It updates or creates a kubeconfig file that contains the necessary authentication information to allow kubectl to communicate with your specified EKS cluster.

What happens when you run this command:

The AWS CLI retrieves the required connection information for the EKS cluster (such as the API server endpoint and certificate) and updates the kubeconfig file located at ~/.kube/config (by default). It configures the authentication details needed to connect kubectl to your EKS cluster using IAM roles. After running this command, you will be able to interact with your EKS cluster using kubectl commands, such as kubectl get nodes or kubectl get pods.

Create Namespace:

kubectl create namespace webapps

Create Service Account:

# vi svc.yml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: webapps

Steps:

create a vi svc.yml and paste the above svc.yml content

run the following command

kubectl apply -f svc.yml

Create Role and Role Binding:

Creating a Kubernetes Role

To start, you'll need to define a role in Kubernetes that specifies the permissions for the resources you'll manage. Here's how to do it:

Create a YAML File: Define the role with necessary permissions (e.g., get, list, watch, create, update, patch, delete).

We start by defining a Kubernetes Role with specific permissions using a YAML file.

Create a role.yaml file to specify what resources the role can access and what actions it can perform (e.g., list, create, delete).
Apply this configuration with kubectl apply -f role.yaml.

# vi role.yml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: app-role
  namespace: webapps
rules:
  - apiGroups:
        - ""
        - apps
        - autoscaling
        - batch
        - extensions
        - policy
        - rbac.authorization.k8s.io
    resources:
      - pods
      - componentstatuses
      - configmaps
      - daemonsets
      - deployments
      - events
      - endpoints
      - horizontalpodautoscalers
      - ingress
      - jobs
      - limitranges
      - namespaces
      - nodes
      - pods
      - persistentvolumes
      - persistentvolumeclaims
      - resourcequotas
      - replicasets
      - replicationcontrollers
      - serviceaccounts
      - services
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

kubectl apply -f role.yaml

Assigning the Role to a Service Account:

We need to bind the created role to a service account using RoleBinding.
Create a bind.yaml file to link the role with the service account.
Apply this configuration with kubectl apply -f bind.yaml.

vi bind.yml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: app-rolebinding
  namespace: webapps 
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: app-role 
subjects:
- namespace: webapps 
  kind: ServiceAccount
  name: jenkins

alt text

Creating a Token for Authentication:

Generate a token for the service account to authenticate with Kubernetes.
Use a YAML file to create a Kubernetes Secret that stores the token.
Apply this configuration with kubectl apply -f secret.yaml.

# vi secret.yml
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  name: mysecretname
  annotations:
    kubernetes.io/service-account.name: jenkins

while applying, makesure, we will use namespace as below

kubectl apply -f secret.yml -n webapps

Retrieve the token using kubectl describe secret <secret-name> -n webapps.

ubuntu@ip-172-31-90-126:~$ kubectl get namespace
NAME              STATUS   AGE
default           Active   51m
kube-node-lease   Active   51m
kube-public       Active   51m
kube-system       Active   51m
webapps           Active   22m

ubuntu@ip-172-31-90-126:~$ kubectl get namespace webapps
NAME      STATUS   AGE
webapps   Active   22m
ubuntu@ip-172-31-90-126:~$

ubuntu@ip-172-31-90-126:~$ kubectl get secret -n webapps
NAME           TYPE                                  DATA   AGE
mysecretname   kubernetes.io/service-account-token   3      3m33s

kubectl describe secret mysecretname -n webapps

will save token somewhere, because we will be using the same token in CI/CD pipeline.

kubectl describe secret mysecretname -n webapps | grep token

Type:  kubernetes.io/service-account-token
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6InhpWmNCYi1ZaEFNaFJ0eWpmQVFvSFR0ZFlQbGJZSjNndXpEM3hCUDJhVkUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJ3ZWJhcHBzIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im15c2VjcmV0bmFtZSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJqZW5raW5zIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiZWJjZmM4OTUtOTk4My00ZTIxLThmMTMtN2VhZTgzZmJmZWFjIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OndlYmFwcHM6amVua2lucyJ9.Q4a0Er2viPrfPKZ7vW93FuC_P4S2uYXHkY9v37qvR69DLPXYgEJx9aXa2z2-WlKUt12WdRW-Gv53hAvF2hZjt8REwgqbe98Dohv1PmDLwxlycqj_WjCxTSFxobZqeDqHXo3VF6SawSTNPETx4WnXDqMjqyOKk0LHI-Sxi6CIOMVi4mlZUXWCEiyywE75RlK-E25yqTU9FB4M3hZ_v2cMNedyDOz2IITdLosr17L9HyvPo6-kmOk1qmrSryXwD9pX4cw4cRgiNZR3p5wy_9TF2WOxDsnKzuyjOOCBP1AKbdp673eJI20mGQS2EB8HFx13ql8f_pZn5-Bl82o0s83fBA

Setting Up Jenkins CD Pipeline:

Create a Jenkins pipeline to handle the deployment process.
Define the pipeline stages: deploy to Kubernetes and verify deployment.
Configure Jenkins to use the service account token for Kubernetes API interactions.
Use the pipeline syntax to apply Kubernetes configurations and monitor the deployment.

Configure the K8s token in Jenkins

Dashboard> Manage Jenkins> Credentials> System> Global credentials (unrestricted)

kind: "Secret Text"

Secret: Paste your token which you get from secert

Id: k8-token #it would be used in pipeline

Description:k8-token

Finally, set up a Jenkins pipeline to automate deployment:

Create a dummy Jenkins Pipeline: Define stages for deployment and verification.

add the following in pipeline

pipeline {
    agent any

    stages {
        stage('Deploy to Kubernets') {
            steps {
                withKubeCredentials(kubectlCredentials: [[caCertificate: '', clusterName: 'balraj-cluster', contextName: '', credentialsId: 'k8-token', namespace: 'webapps', serverUrl: 'https://7A88D591B76582F68E890F414CBE194C.gr7.us-east-1.eks.amazonaws.com']]) {
                     sh "kubectl apply -f deployment-service.yml"
                     sleep 60
               }
            }
        }    

        stage('Verify Deployment') {
            steps {
                withKubeCredentials(kubectlCredentials: [[caCertificate: '', clusterName: 'balraj-cluster', contextName: '', credentialsId: 'k8-token', namespace: 'webapps', serverUrl: 'https://7A88D591B76582F68E890F414CBE194C.gr7.us-east-1.eks.amazonaws.com']]) {
                     sh "kubectl get svc -n webapps"
                }
            }
        }
      }
    }

Same pipeline will be added into the git repo in main branch

Commit and Run: Commit the Jenkinsfile and let Jenkins pick it up. Monitor the deployment process and check the application URL once it’s up and running.

Deployment Verification

Once the pipeline is set up, Jenkins will deploy the microservices and provide a URL to access the application.
will browser the LB URL and website should be accessible.
Environment Cleanup:
- As we are using Terraform, we will use the following command to delete the EKS cluster first, then delete the virtual machine.

    terraform destroy --auto-approve

To clean up, delete the Kubernetes cluster with eksctl delete cluster --name <cluster-name> --region <region>.

I got below error message while deleting the EKS cluster; you may be experiencing the same, so the solution is to `manually delete the load balancer` in the AWS console and `delete the VPC`.

Now, time to delete the Virtual machine.

Go to folder "09.Real-Time-DevOps-Project/Terraform_Code/04.Code_IAC_Terraform_box" and run the terraform command.

    cd Terraform_Code/

    $ ls -l
    Mode                 LastWriteTime         Length Name
    ----                 -------------         ------ ----
    da---l          26/08/24   9:48 AM                04.Code_IAC_Terraform_box



    Terraform destroy --auto-approve

Key Takeaways

Automation: The pipeline automates the deployment process, making it easier to manage multiple microservices.
Flexibility: The use of Jenkins and Kubernetes allows for flexible and scalable deployment strategies.
Efficiency: Implementing automated CI/CD pipelines improves deployment speed and reliability.

What to Avoid

Hardcoding Secrets: Avoid hardcoding sensitive information like tokens in your pipeline scripts. Use Jenkins credentials and Kubernetes secrets.
Long Sleep Periods: Instead of using long sleep periods in your pipeline, consider using appropriate Kubernetes checks to confirm the status of deployments.

Key Benefits

Streamlined Deployment: Automates the deployment of multiple microservices with minimal manual intervention.
Improved Efficiency: Reduces deployment time and ensures consistency across environments.
Scalability: Easily scales to handle large numbers of microservices and complex deployment scenarios.

Conclusion

By following these steps and best practices, you can efficiently set up a CI/CD pipeline that enhances your deployment processes and streamlines your workflow.

Following these steps, you can successfully deploy and manage a Kubernetes application using Jenkins. Automating this process with Jenkins pipelines ensures consistent and reliable deployments. If you found this guide helpful, please like and subscribe to my blog for more content. Feel free to reach out if you have any questions or need further assistance!

Ref Link

Deploying microservices to a Kubernetes cluster with CI/CD: A Step-by-Step Guide

Prerequisites

Key Points

Setting Up the Environment

Environment Setup

Setup the Jenkins

Install the plugin in Jenkins

Configure tools in Jenkins

Set docker cred in Jenkins

Creating a multipipeline in Jenkins:

Setup the EKS Cluster

What happens when you run this command:

Create Namespace:

Create Service Account:

Create Role and Role Binding:

Assigning the Role to a Service Account:

Creating a Token for Authentication:

Setting Up Jenkins CD Pipeline:

Configure the K8s token in Jenkins

Deployment Verification

Environment Cleanup:

I got below error message while deleting the EKS cluster; you may be experiencing the same, so the solution is to `manually delete the load balancer` in the AWS console and `delete the VPC`.

Now, time to delete the Virtual machine.

Key Takeaways

What to Avoid

Key Benefits

Conclusion

Comments

More from this blog

Claude Code + NVIDIA NIM API + LiteLLM + Terraform MCP: A Full Stack AI Dev Setup

Building an End-to-End AI Agent: Claude Code+ MCP + AWS Bedrock + Confluence for Enterprise Knowledge Discovery

🚀Stop AI Hallucinations: Build a Real-World RAG System with Amazon Bedrock (Step-by-Step)

Mega Project Spotlight: Full-Stack DevSecOps Pipeline for EKS with GitHub Actions & Terraform

Automating EKS Deployment with Terraform and GitHub Actions – The DevOps Way

Command Palette

Prerequisites

Key Points

Setting Up the Environment

Environment Setup

Setup the Jenkins

Install the plugin in Jenkins

Configure tools in Jenkins

Set docker cred in Jenkins

Creating a multipipeline in Jenkins:

Setup the EKS Cluster

What happens when you run this command:

Create Namespace:

Create Service Account:

Create Role and Role Binding:

Assigning the Role to a Service Account:

Creating a Token for Authentication:

Setting Up Jenkins CD Pipeline:

Configure the K8s token in Jenkins

Deployment Verification

Environment Cleanup:

I got below error message while deleting the EKS cluster; you may be experiencing the same, so the solution is to manually delete the load balancer in the AWS console and delete the VPC.

Now, time to delete the Virtual machine.

Key Takeaways

What to Avoid

Key Benefits

Conclusion

Comments

More from this blog

I got below error message while deleting the EKS cluster; you may be experiencing the same, so the solution is to `manually delete the load balancer` in the AWS console and `delete the VPC`.