CI/CD Pipeline for Private AKS: Integrating Azure DevOps, Terraform, Private ACR, Azure SQL DB, Application Gateway, Azure Key Vault, and Storage

This document explains how to set up a CI/CD pipeline using Azure DevOps, Terraform, and different Azure services to deploy a web application on a private AKS cluster. The deployment is automated and ensures secure communication between services.

Prerequisites

Before diving into this project, here are some skills and tools you should be familiar with:

Need to create a PAT Access Token-

[x] Clone repository for terraform code
Note: Replace resource names and variables as per your requirement in terraform code
- Update terraform.tfvars
[x] Infra as Code
[x] Application Repo

Azure Subscription: An active Azure subscription is required to create and manage resources.
Azure DevOps Account: An Azure DevOps account to create and manage pipelines.
Terraform: Installed on the local machine or configured in the Azure DevOps pipeline.
Azure CLI: Installed on the local machine or configured in the Azure DevOps pipeline.
Service Principal: Created in Azure AD for authentication and authorization in the pipeline.

Technologies/Services Used

Azure Kubernetes Service (AKS): For deploying the web application in a private cluster.
Azure SQL Database: For storing application data.
Azure Container Registry (ACR): For storing Docker images.
Azure Application Gateway: For load balancing and routing traffic to the AKS cluster.
Azure Virtual Network (VNet): For isolating resources and enabling secure communication.
Azure Private Endpoint: For secure access to Azure services within the VNet.
Azure Key Vault: For storing secrets and sensitive information.
Azure Log Analytics Workspace: For collecting and analyzing logs.
Terraform: For infrastructure as code to automate resource creation.
Azure DevOps: For creating CI/CD pipelines.

Key Points

Architecture Overview:
- The architecture includes multiple VNets for AKS, ACR, and self-hosted agents.
- Virtual Network Peering is used to enable communication between VNets.
- Private Endpoints are used for secure access to Azure services.
Build Pipeline:
- The build pipeline includes tasks to build and push a Docker image to ACR.
- Terraform configuration files are published as pipeline artifacts for use in the release pipeline.
Release Pipeline:
- The release pipeline is linked to the artifacts published by the build pipeline.
- It includes stages for deploying resources to AKS using Terraform.
- Auto-scaling and alert configurations are set up for the web app.
Terraform Configuration:
- Separate files for providers, variables, and main configuration.
- Variables are stored securely and referenced in the configuration files.
- The configuration includes creating VNets, AKS cluster, ACR, SQL Database, Application Gateway, and Private Endpoints.
Service Connections:
- Azure DevOps service connections are created for authentication with Azure and Docker Registry.

Setting Up the Infrastructure

First, we'll create the necessary virtual machines using terraform code.

Below is a terraform Code:

Once you clone repo and run the terraform command.

$ ls -l
-rw-r--r-- 1 bsingh 1049089  690 Jan 31 15:01 DevOps_UI.tf
-rw-r--r-- 1 bsingh 1049089 6115 Jan 31 15:01 group_lib.tf
-rw-r--r-- 1 bsingh 1049089 3011 Jan 29 09:47 KeyVault_Create_permission.tf
-rw-r--r-- 1 bsingh 1049089  921 Jan 24 13:03 KeyVault-Getdata.tf
-rw-r--r-- 1 bsingh 1049089  675 Jan 30 11:41 output.tf
-rw-r--r-- 1 bsingh 1049089  813 Jan 24 13:03 provider.tf
-rw-r--r-- 1 bsingh 1049089  326 Jan 24 13:03 ssh_key.tf
-rw-r--r-- 1 bsingh 1049089  924 Jan 30 11:29 Storage.tf
-rw-r--r-- 1 bsingh 1049089 4248 Jan 31 20:36 terraform.tfvars
-rw-r--r-- 1 bsingh 1049089 3727 Jan 30 11:58 variable.tf

You need to run the following terraform command.

Now, run the following command.

terraform init
terraform fmt
terraform validate
terraform plan
terraform apply 
# Optional <terraform apply --auto-approve>

Once you run the terraform command, then we will verify the following things to make sure everything is setup via a terraform.

Inspect the `Cloud-Init` logs:

Once connected to VM then you can check the status of the user_data script by inspecting the log files

# Primary log file for cloud-init
sudo tail -f /var/log/cloud-init-output.log
                    or 
sudo cat /var/log/cloud-init-output.log | more

If the user_data script runs successfully, you will see output logs and any errors encountered during execution.
If there’s an error, this log will provide clues about what failed.

Detailed Steps

Setting Up the Architecture:
- Azure Repos: Store the source code and Terraform configuration files.
- Azure Storage Account: Store the Terraform state file securely.
- Build Pipeline: Configure tasks to build and push Docker images.
- Release Pipeline: Configure tasks to deploy resources using Terraform.
Configuring the Build Pipeline:
- Build and Push Docker Image: Use Docker to build the application image and push it to ACR.
- Publish Terraform Files: Publish Terraform configuration files as pipeline artifacts.
Configuring the Release Pipeline:
- Initialize Terraform: Initialize the Terraform working directory and download the necessary plugins.
- Apply Terraform Configuration: Apply the Terraform configuration to create resources in Azure.
- Deploy Web App: Deploy the Docker image to AKS.
- Configure Auto-Scaling and Alerts: Set up auto-scaling rules and alert notifications.
Terraform Configuration Files:
- Provider Configuration: Define the Azure provider and authentication details.
- Variable Definitions: Define variables for resource names, locations, and other configurations.
- Main Configuration: Define the resources to be created, including VNets, AKS cluster, ACR, SQL Database, Application Gateway, and Private Endpoints.
Creating Service Connections:
- Azure Service Connection: Authenticate with Azure for deploying resources.
- Docker Registry Service Connection: Authenticate with ACR for pushing Docker images.

Step-by-Step Process:

Configure Service Connection

First, create Service Connection in Azure Devops.
Once you create a connection, make sure to note down the connection name, as it will be used in the pipeline.
On the agent machine, ensure you are logged in with Azure and that the connection is active. If not, log in using the following steps.
```
  az login --use-device-code
```
Need to Create a service Connect in the pipeline first.
- Azure Service Connection

Update Secret variable value details.

Update secret variable value first
- update servicePrincipalId
- update servicePrincipalKey
- update tenantid
- Step-01. Go to the key vault and update the value (Azure UI)
Update the other two values in the same way.
- Step-02. Update the secret in the Library at Azure DevOps
- Step-03. Link secrets from an Azure key vault as variables..

Update changes in Repo code as per project details.

Repo (Infra-as-code)
- Step-01: script file need to be updated from agent-vm folder
  - update the Organization and PAT token
- Step-02: Update Service Principle Name from private-acr Folder
  - To get the Service Principal's name
    - List Service Principals:

        az ad sp list --query <query_string>

        Example:
        az ad sp list --query "[].{Name:displayName, AppId:appId}" --output table

        # filder with name for your service account
        az ad sp list --query "[?starts_with(displayName, 'Azure')].{Name:displayName, AppId:appId}" --output table

Update the Service Principal name:

Build a YAML pipeline.

Following is the sequence to create the setup using the pipeline.
- VNet (Network/Subnet)
- Agent VM
- Azure Container Registry
- SQL Database
- Application Gateway
- AKS Cluster
Step-01: Build YAML pipeline as below:
Adjust the parameters for vNet Job.
Adjust the parameters for vm Job.
Adjust the parameters for acr Job.
Save and run the pipeline.
Adjust the parameters for db Job.
Adjust the parameters for appgateway Job.
Adjust the ssh key in library for AKS cluster
- SSH public key need to be updated

Run the pipeline.

Install SQL package on Agent VM.

Open a PuTTY session for the virtual machine and install the required SQL package to check SQL connectivity.
- install mssql-tools
  - sudo apt-get update
  - sudo apt-get install mssql-tools -y
- Find the sqlcmd executable:
  - ls /opt/mssql-tools/bin/sqlcmd
- Add the sqlcmd directory to your PATH:
  - nano ~/.bashrc
  - export PATH="$PATH:/opt/mssql-tools/bin"
- Apply the changes:
  - source ~/.bashrc

Verify the installation:
- sqlcmd -S <server_address> -U -P -d <database_name>

Convert SQL cred to base 64

echo "Server=tcp:yourserver.database.windows.net,1433;Initial Catalog=yourdatabase;Persist Security Info=False;User ID=yourusername;Password=yourpassword;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;" | base64

Decode base64 cred to Normal.

 echo "U2VydmVyPXRjcDpldGlja2V0ZGJzZXJ2ZXIzMDAxMjAyNS5kYXRhYmFzZS53aW5kb3dzLm5ldCwxNDMzO0luaXRpYWwgQ2F0YWxvZz1ldGlja2V0cy1kYjtQZXJzaXN0IFNlY3VyaXR5IEluZm89RmFsc2U7VXNlciBJRD1henVyZXVzZXI7UGFzc3dvcmQ9cGFzc3dvcmRAMTIzO011bHRpcGxlQWN0aXZlUmVzdWx0U2V0cz1GYWxzZTtFbmNyeXB0PVRydWU7VHJ1c3RTZXJ2ZXJDZXJ0aWZpY2F0ZT1G" | base64 -d

Connect `AKS cluster` on Agent VM.

Run the following commands to connect AKS cluster
- Login to your azure account

    az login

Set the cluster subscription

    az account set --subscription <ID>

Download cluster credentials

    az aks get-credentials --resource-group rg-devops --name aksdemo --overwrite-existing

Verify Application Accessibility.

Try to access application gateway IP in the browser and you will below the error message

Build the second pipeline (Application).

Build application pipeline
run the pipeline.
It will ask for permission and approve it.
Update the deployment.yaml file as below (SQL Connection)
AKS deployment is waiting for approval

Troubleshooting.

I was getting an error imagepullbackoff and noticed that the SQL connection name was having a problem. To retrieve and update the string from Secret.
Retrieve and Update the Connection String from the Secret.
- Since you're using a Kubernetes secret for the database connection, check the secret value:

    kubectl get secret db-connection-secret -o jsonpath="{.data.connection-string}" | base64 --decode

Pipeline status
Image status

Update the image name in the manifest file.

Congratulations :-) the application is working and accessible.

Application is accessible now.
Try to create account in website.
Following are the cred to login into webpage

Pipeline for Cleanup Infra Setup.

Following are the sequence to delete the setup using pipeline.
- AKS Cluster
- Application Gateway
- Azure Container Registry
- SQL Database
- Agent VM
- VNet
Here is the 👉Updated pipeline👈

Environment Cleanup:

As we are using Terraform, we will use the following command to delete ssh_key, Vault and Storage account.
Run the terraform command.
```
  Terraform destroy --auto-approve
```
I was getting below error message while deleting the setup
Rerun the destroy command again and it will delete ResourceGroup as well.

Conclusion

This project showcases a strong CI/CD pipeline setup using Azure DevOps, Terraform, and various Azure services. By automating the deployment process, it ensures efficient, scalable, and secure deployment of a web application on a private AKS cluster. Using Terraform for infrastructure as code offers flexibility and easy management, making it a valuable approach for modern cloud-based applications.

Ref Link: