Balraj Singh's blog
Comprehensive CI/CD Pipeline for Private AKS: Seamless Integration with Azure DevOps, FrontDoor, Terraform, Private ACR, Azure SQL DB, Application Gat

Comprehensive CI/CD Pipeline for Private AKS: Seamless Integration with Azure DevOps, FrontDoor, Terraform, Private ACR, Azure SQL DB, Application Gat

Balraj Singh's photo
Balraj Singh
·

11 min read

Table of contents

Image

This document explains how to set up a CICD pipeline for deploying a web application on a private AKS cluster. It includes integration with Azure SQL Database, Private Azure Container Registry (ACR), and Application Gateway using Terraform. It also discusses using Azure Front Door for global load balancing and content delivery.

Prerequisites

Before diving into this project, here are some skills and tools you should be familiar with:

  • [x] Need to Create a Personal Access Token (PAT)

  • [x] Terraform code
    Note: Replace resource names and variables as per your requirement in terraform code

    • Update terraform.tfvars

  • [x] Application Repo:

Tools and Technologies: Used

  • Terraform: Used for infrastructure as code to create and manage Azure resources.

  • Azure DevOps: For CICD pipeline and repository management.

  • Docker: For containerizing the application.

  • Kubernetes: For orchestrating the deployment of the application on AKS.

  • Azure Front Door: For global load balancing and content delivery.

  • Azure Key Vault: For storing secrets and sensitive information.

  • Azure CLI: For managing Azure resources from the command line.

  • Azure Log Analytics Workspace: For collecting and analyzing logs.

  • Azure Private Endpoint: For secure access to Azure services within the VNet.

  • Azure Virtual Network (VNet): For isolating resources and enabling secure communication.

  • Azure Application Gateway: For load balancing and routing traffic to the AKS cluster.

  • Azure Container Registry (ACR): For storing Docker images.

  • Azure SQL Database: For storing application data.

  • Azure Kubernetes Service (AKS): For deploying the web application in a private cluster.

Key Points

Architecture Components:

  • Virtual Networks: Three virtual networks (AKS vnet, ACR vnet, Agent vnet) are used to isolate resources.

  • Private AKS Cluster: Deployed within the AKS vnet.

  • Application Gateway: Integrated with the AKS cluster for load balancing.

  • Private ACR: Hosted within the ACR vnet.

  • Self-hosted Agent: A virtual machine in the Agent vnet configured as a self-hosted agent for Azure DevOps.

  • Virtual Network Peering: Enables communication between the virtual networks.

  • Private Endpoints: Used for secure communication between services.

  • Private DNS Zones: For name resolution within the virtual networks.

  • Azure SQL Database: Backend database for the application.

  • Log Analytics Workspace: For collecting and analyzing logs.

  • Ingress Controller: Integrated with Application Gateway for routing traffic to the pods.

Setting Up the Infrastructure

First, we'll create the necessary virtual machines using terraform code.

Below is a terraform Code:

Once you clone repo and run the terraform command.

$ ls -l
-rw-r--r-- 1 bsingh 1049089  690 Jan 31 15:01 DevOps_UI.tf
-rw-r--r-- 1 bsingh 1049089 6115 Jan 31 15:01 group_lib.tf
-rw-r--r-- 1 bsingh 1049089 3011 Jan 29 09:47 KeyVault_Create_permission.tf
-rw-r--r-- 1 bsingh 1049089  921 Jan 24 13:03 KeyVault-Getdata.tf
-rw-r--r-- 1 bsingh 1049089  675 Jan 30 11:41 output.tf
-rw-r--r-- 1 bsingh 1049089  813 Jan 24 13:03 provider.tf
-rw-r--r-- 1 bsingh 1049089  326 Jan 24 13:03 ssh_key.tf
-rw-r--r-- 1 bsingh 1049089  924 Jan 30 11:29 Storage.tf
-rw-r--r-- 1 bsingh 1049089 4248 Jan 31 20:36 terraform.tfvars
-rw-r--r-- 1 bsingh 1049089 3727 Jan 30 11:58 variable.tf

You need to run the following terraform command.

terraform init
terraform fmt
terraform validate
terraform plan
terraform apply 
# Optional <terraform apply --auto-approve>

Image

Once you run the terraform command, then we will verify the following things to make sure everything is set up via a terraform.

Inspect the Cloud-Init logs:

Once connected to VM then you can check the status of the user_data script by inspecting the log files

# Primary log file for cloud-init
sudo tail -f /var/log/cloud-init-output.log
                    or 
sudo cat /var/log/cloud-init-output.log | more

  • If the user_data script runs successfully, you will see output logs and any errors encountered during execution.

  • If there’s an error, this log will provide clues about what failed.

Detailed Steps

  1. Setting Up the Architecture:

    • Azure Repos: Store the source code and Terraform configuration files.

    • Azure Storage Account: Store the Terraform state file securely.

    • Build Pipeline: Configure tasks to build and push Docker images.

    • Release Pipeline: Configure tasks to deploy resources using Terraform.

  2. Configuring the Build Pipeline:

    • Build and Push Docker Image: Use Docker to build the application image and push it to ACR.

    • Publish Terraform Files: Publish Terraform configuration files as pipeline artifacts.

  3. Configuring the Release Pipeline:

    • Initialize Terraform: Initialize the Terraform working directory and download necessary plugins.

    • Apply Terraform Configuration: Apply the Terraform configuration to create resources in Azure.

    • Deploy Web App: Deploy the Docker image to AKS.

    • Configure Auto-Scaling and Alerts: Set up auto-scaling rules and alert notifications.

  4. Terraform Configuration Files:

    • Provider Configuration: Define the Azure provider and authentication details.

    • Variable Definitions: Define variables for resource names, locations, and other configurations.

    • Main Configuration: Define the resources to be created, including VNets, AKS cluster, ACR, SQL Database, Application Gateway, and Private Endpoints.

  5. Creating Service Connections:

    • Azure Service Connection: Authenticate with Azure for deploying resources.

    • Docker Registry Service Connection: Authenticate with ACR for pushing Docker images.

  6. Pipeline Stages:

    • Infrastructure Creation: Using Terraform to create virtual networks, virtual machines, ACR, SQL Database, Application Gateway, and AKS cluster.

    • Application Deployment: Building and pushing Docker images to ACR, deploying the application to AKS, and configuring Ingress and Application Gateway.

    • Database Configuration: Setting up the SQL database and running migrations.

    • Monitoring and Logging: Configuring Log Analytics Workspace for monitoring.

  7. Azure Front Door:

    • Global Load Balancing: Distributes traffic across multiple backend endpoints.

    • Content Delivery Network (CDN): Caches and delivers static and dynamic content closer to users.

    • SSL/TLS Termination: Offloads encryption and decryption processes.

    • Application Layer Security: Provides protection against common web application threats.

    • Session Affinity: Ensures subsequent requests from the same client are directed to the same backend server.

    • Health Monitoring and Failover: Continuously monitors backend endpoints and reroutes traffic if necessary.

Step-by-Step Process:

Configure Service Connection

  • First, create a Service Connection in Azure DevOps.

  • Once you create a connection then note it down the connection Name, because that name will be used in the pipeline.

  • On the agent machine, ensure you are logged in with Azure, and the connection is active. If not, log in using the following steps.

      az login --use-device-code

  • Need to Create a service Connect in pipeline first.

    • Azure Service Connection

      Image

      Image

      Image

      Image

      Image

      Image

Update Secret variable value details.

  • Update secret variable value first

    • update servicePrincipalId

    • update servicePrincipalKey

    • update tenantid

  • Step-01. Go to key vault and update the value (Azure UI)

    • Update the other two values in the same way.

  • Step-02. Update the secret in Library at Azure DevOps

    Image

  • Step-03. Link secrets from an Azure key vault as variables..

    • Image

      Image

      Image

      Image

      Image

      Image

Update changes in the repo code as per project details.

  • Repo (Infra-as-code)

    • Step-01: script file need to be updated from agent-vm folder

      • update the Organization and PAT token

        Image

    • Step-02: Update Service Principle Name from private-acr Folder

      • To get the Service Principal name

        • List Service Principals:
        az ad sp list --query <query_string>

        Example:
        az ad sp list --query "[].{Name:displayName, AppId:appId}" --output table

        # filder with name for your service account
        az ad sp list --query "[?starts_with(displayName, 'Azure')].{Name:displayName, AppId:appId}" --output table

  • Update the Service Principal name:

    Image

Build a YAML pipeline.

  • Following is the sequence to create the setup using pthe ipeline.

    • VNet (Network/Subnet)

    • Agent VM

    • Azure Container Registry

    • SQL Database

    • Application Gateway

    • AKS Cluster

    • FrontDoor

  • Step-01: Build the YAML pipeline as below:

    Image

    Image

    Image

    Image

  • Adjust the parameters for vNet Job.

    Image

  • Adjust the parameters for vm Job.

    Image

  • Adjust the parameters for acr Job.

    Image

  • Save and run the pipeline.

    Image

  • Adjust the parameters for db Job.

    Image

  • Adjust the parameters for appgateway Job.

    Image

  • Adjust the parameters for AKS Cluster Job.

    Image

  • Adjust the parameters for FrontDoor Job.

    Image

  • Step-02: Adjust the ssh key in library for AKS cluster

    • SSH public key need to be updated

      Image

Run the pipeline.

Image

Install SQL package on Agent VM.

  • Open a PuTTY session for the virtual machine and install the required SQL package to check SQL connectivity.

    • Install mssql-tools:

      • sudo apt-get update

      • sudo apt-get install mssql-tools -y

  • 3. Locate the sqlcmd executable:

    • ls /opt/mssql-tools/bin/sqlcmd

  • Add the sqlcmd directory to your PATH:

    • echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc

  • Apply the changes:

    • source ~/.bashrc

  • Verify if the PATH is updated:

    • Run the following command to check if the path has been correctly added:

      • echo $PATH

Image

  • Verify the installation:

    • sqlcmd -S <server_address> -U -P -d <database_name>

      Image

  • Convert SQL cred to base 64
echo "Server=tcp:yourserver.database.windows.net,1433;Initial Catalog=yourdatabase;Persist Security Info=False;User ID=yourusername;Password=yourpassword;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;" | base64 | tr -d '\n'

Image

  • Decode base64 cred to Normal.
 echo "U2VydmVyPXRjcDpldGlja2V0ZGJzZXJ2ZXIzMDAxMjAyNS5kYXRhYmFzZS53aW5kb3dzLm5ldCwxNDMzO0luaXRpYWwgQ2F0YWxvZz1ldGlja2V0cy1kYjtQZXJzaXN0IFNlY3VyaXR5IEluZm89RmFsc2U7VXNlciBJRD1henVyZXVzZXI7UGFzc3dvcmQ9cGFzc3dvcmRAMTIzO011bHRpcGxlQWN0aXZlUmVzdWx0U2V0cz1GYWxzZTtFbmNyeXB0PVRydWU7VHJ1c3RTZXJ2ZXJDZXJ0aWZpY2F0ZT1G" | base64 -d

Image

Connect AKS cluster on Agent VM.

  • Run the following commands to connect AKS cluster

    • Login to your azure account
    az login
  • Set the cluster subscription
    az account set --subscription <ID>
  • Download cluster credentials
    az aks get-credentials --resource-group rg-devops --name aksdemo --overwrite-existing

Image

Verify Application Accessibility.

  • Try to access the application gateway IP in the browser, and you will see the error message below.

    Image

    Image

Build the second pipeline (Application).

  • Build application pipeline

  • Here is the updated file.

  • To get the all repositories in Azure Container Registry

    • You can use the Azure CLI to check the details of repositories and images stored in a private Azure Container Registry (ACR).

      1. List all repositories in Azure Container Registry
        az acr repository list --name <ACR_NAME> --output table
        🔹 Replace <ACR_NAME> with the name of your Azure Container Registry.
  1. Get details of a specific repository
        az acr repository show --name <ACR_NAME> --repository <REPOSITORY_NAME>
        🔹 Replace <REPOSITORY_NAME> with the name of the repository inside ACR.
  1. List all image tags in a repository
        az acr repository show-tags --name <ACR_NAME> --repository <REPOSITORY_NAME> --output table
        🔹 This command displays the available tags for a given repository.
  1. List all manifests of an image repository
        az acr repository show-manifests --name <ACR_NAME> --repository <REPOSITORY_NAME> --output table
        🔹 This command provides manifest details of all images in the repository.
  1. Show details of a specific image tag
        az acr repository show --name <ACR_NAME> --repository <REPOSITORY_NAME> --image <IMAGE_TAG>
        🔹 Replace <IMAGE_TAG> with the tag of the image you want details for.
  1. Check the last update timestamp of an image
        az acr repository metadata list --name <ACR_NAME> --repository <REPOSITORY_NAME>
        🔹 This provides metadata information, including the last update timestamp.

  • Update the deployment.yaml file as below for SQL Connection and image name in manifest file.

  • Run the pipeline.

    • It will ask for permission and approve it.

    • AKS deployment is waiting for approval

Troubleshooting.

  • I was getting an error imagepullbackoff and noticed that the SQL connection name was having a problem. To retrieve and update the string from Secret.

  • Retrieve and Update the Connection String from the Secret

    • Since we're using a Kubernetes secret for the database connection, check the secret value:
    kubectl get secret db-connection-secret -o jsonpath="{.data.connection-string}" | base64 --decode

  • Pipeline status

  • Image

  • Image status

    Image

  • Application is accessible now from Application Gateway.

    Image

  • Try to create account in website.

    Image

  • Following are the cred to login into webpage

    Image

Application accessibility from Front Door.

Image

Image

💡
Congratulations :-) the application is working and accessible.

Image

  • Pipeline Status

    Image

Pipeline for Cleanup Infra Setup.

  • Following is the sequence to delete the setup using the pipeline.

    • FrontDoor

    • AKS Cluster

    • Application Gateway

    • Azure Container Registry

    • SQL Database

    • Agent VM

    • VNet

  • Here is the 👉Updated pipeline for delete👈

Image

Environment Cleanup:

  • As we are using Terraform, we will use the following command to delete ssh_key, Vault and Storage account.

  • Run the terraform command.

      Terraform destroy --auto-approve

  • I was getting the below error message while deleting the setup. It is getting failed in deleting ResourceGroup

    • Rerun the destroy command and it will delete ResourceGroup as well.

      Image

Conclusion

This document outlines the comprehensive setup of a secure and efficient CICD pipeline for deploying a web application on a private AKS cluster, integrated with various Azure services. By leveraging Terraform for infrastructure as code, Azure DevOps for pipeline management, and Azure Front Door for global load balancing, the solution ensures high availability, security, and optimal performance for the application.

For detailed steps and configurations, refer to the provided Terraform scripts and Azure DevOps pipeline definitions.

Ref Link:

azure acrazure-devopsAzureTerraformazure_frontdoorAzure Application Gatewayvirtual machinevnetAKS,Azure kubernetes servicesaksSQL ServerSQLACRApplication Development #azure #devops #openshift #redhat #cloudnative #microservices #technology #jobs #freshers #careeropportunities #training #learning #kubernetes #cka #ckad #cks #discounts #jobs #cloud #aws #azure #devops #certification #training #learning #coupons