YouTube Clone: Automating CI/CD with Azure DevOps: Blue-Green Deployment and Self-Hosted Agents on Azure VM Scale Sets

App_YouTube Clone_Compress

Overview

This project provides a comprehensive guide to automating Continuous Integration and Continuous Deployment (CI/CD) using Azure DevOps. It covers two main topics: Blue-Green Deployment using Azure DevOps Release Pipelines and setting up Self-Hosted Agents using Azure Virtual Machine Scale Sets (VMSS). The project uses a YouTube clone application as a practical example to demonstrate the concepts and steps involved.

Prerequisites

Before diving into this project, here are some skills and tools you should be familiar with:

[x] Clone repository for terraform code
Note: Replace resource names and variables as per your requirement in terraform code
- Update terraform.tfvars.
[x] App Repo (YouTube Clone)
[x] Azure Account: You’ll need an Azure account to create resources like virtual Machines, Storage, and manage pipelines.
[x] Terraform Knowledge: Familiarity with Terraform is needed to set up, manage, and remove infrastructure.
[x] GitHub: Experience with GitHub for version control and managing repositories.
[x] Command-Line Tools: Basic comfort with using the command line for managing infrastructure and services.
[x] Basic CI/CD Knowledge: Some understanding of Continuous Integration and Deployment is recommended.
[x] Linux VM: Docker must be installed on a Linux virtual machine to run containers.

Azure DevOps Release Pipelines: Blue-Green Deployment

This section focuses on setting up Azure DevOps release pipelines to automate the deployment process. It explains how to create a release pipeline using the GUI-based editor, configure continuous deployment triggers, and implement deployment gates for additional control. The blue-green deployment strategy is highlighted, showing how to use deployment slots to minimize downtime during deployments.

Azure DevOps Self-Hosted Agents: Virtual Machine Scale Sets (VMSS)

This section delves into the setup and use of self-hosted agents on Azure VMSS. It compares Microsoft-hosted and self-hosted agents, explaining the benefits and use cases. Detailed steps are provided to set up self-hosted agents on VMSS, register the agents with Azure DevOps, and use them in pipelines. The section also covers the benefits of using VMSS, such as automatic scaling and high availability.

Key Points

Technologies Used: Azure DevOps, Azure Virtual Machine Scale Sets (VMSS)
Self-Hosted Agents: Dedicated agents for running CI/CD pipelines
Automatic Scaling: Provisioning and de-provisioning VMs based on load
Extensions and Applications: Installing necessary software on VMs
Service Connections: Secure authentication for Azure resources
Automated Deployment: Demonstrates how to automate the deployment process using Azure DevOps release pipelines.
Reduced Downtime: Ensures minimal downtime during deployments with the blue-green deployment strategy.
Enhanced Control: Provides additional control over the deployment process with deployment gates.
Scalability: Allows for easy scaling of the application using Azure App Service and deployment slots.
Custom Software Installation: Enables automatic installation of custom software on self-hosted agents.
High Performance: Provides high performance and specific resource requirements with self-hosted agents.
Network Security: Ensures secure handling of sensitive data by keeping it within a secure network.
Cost Efficiency: Optimizes resource management and costs with automatic scaling based on demand.

Setting Up the Infrastructure

I have created Terraform code to set up the entire infrastructure, including the automatic installation of necessary software, and tools, and the creation of the web app.

Below is a terraform Code:

Once you clone repo and run the terraform command.

$ ls -l
Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
dar--l          26/12/24   7:16 PM                pipeline
dar--l          23/12/24   3:38 PM                scripts
-a---l          25/12/24   2:31 PM            600 .gitignore
-a---l          26/12/24   9:29 PM           6571 VM_ACR.tf
-a---l          26/12/24   9:29 PM            892 main.tf
-a---l          26/12/24   9:29 PM            567 output.tf
-a---l          26/12/24   9:29 PM            269 provider.tf
-a---l          26/12/24   9:30 PM            223 terraform.tfvars
-a---l          26/12/24   9:30 PM            615 variable.tf

You need to run the following terraform command.

Now, run the following command.

terraform init
terraform fmt
terraform validate
terraform plan
terraform apply 
# Optional <terraform apply --auto-approve>

Step-by-Step Description

1. Project Setup

Clone the Repository: Clone the project repository from the source control.
Import into Azure DevOps: Import the project into Azure DevOps and set up the necessary service connections.

2. Create a Virtual Machine Scale Set

Azure Portal: Log in to the Azure portal and search for "Virtual Machine Scale Sets".
Create VMSS: Click on "Create" and fill in the required details such as subscription, resource group, and region.
Select VM Size: Choose an appropriate VM size (e.g., Standard B1s) and configure scaling options.
Networking and Management: Configure networking options and set the upgrade mode to manual.

3. Configure Extensions and Applications

Add Extensions: Navigate to the "Extensions and Applications" section in the VMSS settings.
Install Azure DevOps Pipeline Agent: Add the Azure DevOps pipeline agent extension to the VMSS.
Custom Scripts: Optionally, add custom scripts to install additional software (e.g., npm, Azure CLI).

4. Set Up Agent Pool in Azure DevOps

Project Settings: Go to the project settings in Azure DevOps and navigate to the "Agent Pools" section.
Add Pool: Create a new agent pool and select "Azure Virtual Machine Scale Set" as the pool type.
Configure Pool: Select the appropriate Azure subscription and VMSS, and configure the scaling options.

5. Use Self-Hosted Agent in Pipeline

Edit Pipeline: Edit the existing pipeline to use the newly created self-hosted agent pool.
Specify Pool: Update the pipeline configuration to specify the self-hosted agent pool.
Run Pipeline: Trigger the pipeline to run using the self-hosted agent.

6. Testing and Debugging

Monitor Agents: Check the status of the agents in the Azure DevOps agent pool section.
Debug Issues: Use Azure DevOps logs and VMSS monitoring tools to debug and fix any issues.

Detailed Steps

Step 1: Project Setup

Clone the Repository:
- Use Git to clone the repository containing the project files.
- Command: git clone https://github.com/piyushsachdeva/Youtube_Clone
Import into Azure DevOps:
- Navigate to Azure DevOps and create a new project.
- Import the cloned repository into the Azure DevOps project.

Step 2: Create a Virtual Machine Scale Set

Azure Portal:
- Log in to the Azure portal.
- Search for "Virtual Machine Scale Sets" in the search bar
- click on Azure CLI and run the following command
```
  az vmss show --resource-group <ResourceGroupName> --name <VMScaleSetName> --output table
```

After creating your scale set, navigate to your scale set in the Azure portal and verify the following settings:
- Upgrade policy - Manual
- Scaling - Manual scale

Important: Azure Pipelines does not support instance protection. Make sure you have the scale-in and scale-set actions instance protections disabled.

Step 3: Configure Extensions and Applications

Add Extensions:
- Navigate to the "Extensions and Applications" section in the VMSS settings.
- Click on "Add" to install extensions.

Note: Part of provisioning the server includes creating a storage account, setting up a container, and uploading the script to the "Script container."

Custom Scripts:
- Optionally, add custom scripts to install additional software (e.g., npm, Azure CLI).
- Example:
  - Search for "Custom Script for Linux" extension. Upload a script file to install the software:
    
    - if the above script failed then try with this one only
    #!/bin/bash apt-get update && apt-get install -y npm

Step 4: Set Up Agent Pool in Azure DevOps

Project Settings:
- Go to the project settings in Azure DevOps.
- Navigate to the "Agent Pools" section.
Add Pool:
- Create a new agent pool and select "Azure Virtual Machine Scale Set" as the pool type.
- Example:
  - Pool Name: devops-demo_vm
  - Azure Subscription: Select the appropriate subscription
  - VMSS: Select the created VMSS
  - Configure scaling options (e.g., maximum number of agents, idle instances)
Install Azure DevOps Pipeline Agent:
- Add the Azure DevOps pipeline agent extension to the VMSS.
- This extension allows the VMSS to act as a self-hosted agent for Azure DevOps pipelines.
- Navigate to your Azure DevOps Project settings, select Agent Pools under Pipelines, and select Add Pool to create a new agent pool.
- VMSS UI
- Click on Instances and notice that latest Model is showing No.
- Note: - "Azure Pipeline Agent for Linux"
  - It will take 5-10 minutes for the VMSS connection to become active.
  - Go to azure DevOps Portal and noticed that agent is online and accessible now.

Step 5: Build the Pipeline

Edit Pipeline:
- Click on pipeline and select Azure Repos Git (YAML)
- it will generate the basic structure.

Step 6: Use Self-Hosted Agent in Pipeline

Edit Pipeline:
- Edit the existing pipeline to use the newly created self-hosted agent pool.
- Example:
  - Update the pipeline YAML file to specify the agent pool:
```
  pool:
    name: devops-demo_vm
```
  - 👉Complete Pipeline👈
Run Pipeline:
- Trigger the pipeline to run using the self-hosted agent.
- Monitor the pipeline execution to ensure it uses the self-hosted agent.

Step 7: Testing and Debugging

Monitor Agents:
- Check the status of the agents in the Azure DevOps agent pool section.
- Ensure the agents are online and ready to execute pipeline jobs.
Debug Issues:
- Use Azure DevOps logs and VMSS monitoring tools to debug and fix any issues.
- Example: If a pipeline step fails due to missing software, use custom scripts to install the required software on the VMSS.

Step 8: Verify the Artifacts

Verify:
- Artficat will be produced once the pipeline is executed as below.

Step 9: Create WorkItem

Create WorkItem:
- In work Iteams, change the priority from P2 to P1.
Create Query:
- Will create a query and save it in Shared Queries
Create Chart in query:

Step 10: Create a slot in WebApp

Slot in Webapp:
- Create a slot:
- Azure UI > Web App > Deployment Slot

Step 11: Configure `Release Pipeline (CD)`

To Configure the Dev Stage in CD pipeline:

1.1. Add Artifact & GateCheck in CD pipeline:
- Click on Release under pipelines and search as follows.
- add artifact
- Enable auto-trigger CD pipeline
- Configure PreDeployment stage
- Configure the GateCheck in the pipeline

1.2. Configure the Job

Configure the Job in the pipeline

1.3 Configure the slot Option

Click Save

To Configure the Prod Stage in CD pipeline:

To add Production stage.
- clone the existing stage.
- To add pre stage for the Production stage.
- Change the slot from staging to prod.

Step 12: Update the query and workIteam in the pipeline

To Configure query and workIteam in CD pipeline:

run the pipeline and Cd job would be failed because we have set the workitem query set to doing while it should be in done state.
also, we need to update the query as below:
Add security in query:
change state from doing to done in workitem.

ReRun the pipeline again:
- CI passed
- CD- also passed

Website is accessible on staging.
It's asking for approval for Prod Environment.
Application is also accessible and working fine in Prod.

Swap the slot:
- swap the from UI
- Current:

Advantages of Using This Project

Automated Deployment: The project demonstrates how to automate the deployment process using Azure DevOps release pipelines.
Reduced Downtime: Blue-green deployment strategy ensures minimal downtime during deployments by using deployment slots.
Enhanced Control: Deployment gates provide additional control over the deployment process, ensuring only approved changes are deployed to production.
Scalability: Using Azure App Service and deployment slots allows for easy scaling of the application based on demand.

Impact of Using This Project

Improved Efficiency: Automating the deployment process saves time and reduces the risk of human error.
Increased Reliability: Deployment gates and blue-green deployment strategies ensure that only tested and approved changes are deployed to production, increasing the reliability of the application.
Faster Time-to-Market: Continuous deployment enables faster release cycles, allowing new features and updates to reach users more quickly.
Better User Experience: Reduced downtime during deployments ensures a better user experience, as the application remains available during updates.

Environment Cleanup:

As we are using Terraform, we will use the following command to delete
- run the terraform command.

Terraform destroy --auto-approve

Conclusion

By following the steps outlined in this project, users can automate their CI/CD processes, improve control over deployments, and ensure reliable and scalable application updates. The use of blue-green deployment and self-hosted agents further enhances the user experience by minimizing downtime and providing greater control and security. This project serves as a valuable resource for anyone looking to implement DevOps practices in their workflow.

Ref Link: