Simple Notes App for Community: End-to-End Implementation using CI/CD"
Tech enthusiast with 15 years of experience in IT, specializing in server management, VMware, AWS, Azure, and automation. Passionate about DevOps, cloud, and modern infrastructure tools like Terraform, Ansible, Packer, Jenkins, Docker, Kubernetes, and Azure DevOps. Passionate about technology and continuous learning, I enjoy sharing my knowledge and insights through blogging and real-world experiences to help the tech community grow!
Prerequisites
Before diving into this project, here are some skills and tools you should be familiar with:
[x] Clone repository for terraform code
Note: Replace resource names and variables as per your requirement in terraform code- from the Virtual machine main.tf (i.e key name-
MYLABKEY*)
- from the Virtual machine main.tf (i.e key name-
[x] App Repo
[x] Basic Knowledge of GitHub: You need to be familiar with version control, pull requests, and branching.
[x] Understanding of Docker: Docker is used for containerizing the application. Make sure you know how to create, manage, and run Docker containers.
[x] Familiarity with Jenkins: Jenkins is the CI tool that automates the testing and deployment process. A basic understanding of setting up Jenkins pipelines is essential.
[x] Experience with Python and Node.js: Since the backend of the app is likely in Python and the frontend in React (which relies on Node.js), make sure you're comfortable with both technologies.
[x] React Framework: This is used to build the user interface. Familiarity with React components, state management, and hooks is important.
Setting Up the Environment
I have created a Terraform code to set up the entire environment, including the installation of required applications, and tools.
⇒ Two EC2 machines will be created named "Jenkins Server & Agent"
⇒ Docker Install
Setting Up the Virtual Machines (EC2)
First, we'll create the necessary virtual machines using terraform.
Below is a terraform configuration:
Once you clone the repo then go to folder "14.Real-Time-DevOps-Project/Terraform_Code/Code_IAC_Terraform_box" and run the terraform command.
cd Terraform_Code/Code_IAC_Terraform_box
$ ls -l
da---l 29/09/24 12:02 PM k8s_setup_file
-a---l 29/09/24 10:44 AM 507 .gitignore
-a---l 01/10/24 10:50 AM 3771 agent_install.sh
-a---l 01/10/24 10:59 AM 8149 main.tf
-a---l 16/07/21 4:53 PM 1696 MYLABKEY.pem
-a---l 25/07/24 9:16 PM 239 provider.tf
-a---l 01/10/24 11:26 AM 10257 terrabox_install.sh
Note ⇒ Make sure to run main.tf from inside the folders.
13.Real-Time-DevOps-Project/Terraform_Code/Code_IAC_Terraform_box/
da---l 29/09/24 12:02 PM k8s_setup_file
-a---l 29/09/24 10:44 AM 507 .gitignore
-a---l 01/10/24 10:50 AM 3771 agent_install.sh
-a---l 01/10/24 10:59 AM 8149 main.tf
-a---l 16/07/21 4:53 PM 1696 MYLABKEY.pem
-a---l 25/07/24 9:16 PM 239 provider.tf
-a---l 01/10/24 11:26 AM 10257 terrabox_install.sh
You need to run main.tf file using the following terraform command.
Now, run the following command.
terraform init
terraform fmt
terraform validate
terraform plan
terraform apply
# Optional <terraform apply --auto-approve>
Once you run the terraform command, then we will verify the following things to make sure everything is set up via a terraform.
Inspect the Cloud-Init logs:
Once connected to the EC2 instance then you can check the status of the user_data script by inspecting the log files.
# Primary log file for cloud-init
sudo tail -f /var/log/cloud-init-output.log
If the user_data script runs successfully, you will see output logs and any errors encountered during execution.
If there’s an error, this log will provide clues about what failed.
Outcome of "cloud-init-output.log"
Verify the Installation
- [x] Docker version
ubuntu@ip-172-31-95-197:~$ docker --version
Docker version 24.0.7, build 24.0.7-0ubuntu4.1
docker ps -a
ubuntu@ip-172-31-94-25:~$ docker ps
- [x] Terraform version
ubuntu@ip-172-31-89-97:~$ terraform version
Terraform v1.9.6
on linux_amd64
- [x] AWS Cli version
ubuntu@ip-172-31-89-97:~$ aws version
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:
aws help
aws <command> help
aws <command> <subcommand> help
Change the hostname: (optional)
sudo terraform show
sudo hostnamectl set-hostname jenkins-svr
sudo hostnamectl set-hostname jenkins-agent- Update the /etc/hosts file:
- Open the file with a text editor, for example:
sudo vi /etc/hostsReplace the old hostname with the new one where it appears in the file.
Apply the new hostname without rebooting:
sudo systemctl restart systemd-logind.serviceVerify the change:
hostnamectlUpdate the package
sudo -i
apt updateSetup the Jenkins
Access Jenkins via http://:8080. Retrieve the initial admin password using:
sudo cat /var/lib/jenkins/secrets/initialAdminPassword
Setup the Jenkins agent
- Set the password for user "ubuntu" on both Jenkins Master and Agent machines.
sudo passwd ubuntu
- Need to do the password-less authentication between both servers.
sudo su
cat /etc/ssh/sshd_config | grep "PasswordAuthentication"
echo "PasswordAuthentication yes" >> /etc/ssh/sshd_config
cat /etc/ssh/sshd_config | grep "PasswordAuthentication"
cat /etc/ssh/sshd_config | grep "PermitRootLogin"
echo "PermitRootLogin yes" >> /etc/ssh/sshd_config
cat /etc/ssh/sshd_config | grep "PermitRootLogin"
- Restart the sshd reservices.
systemctl daemon-reload
or
sudo service ssh restart
- Generate the SSH key and share it with the agent.
ssh-keygen
Copy the public SSH key from Jenkins to Agent.
- The public key from Jenkins master.
ubuntu@ip-172-31-89-97:~$ cat ~/.ssh/id_ed25519.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIG4BFDIh47LkE6huSzi6ryMKcw+Rj1+6ErnplFbOK5Nz ubuntu@ip-172-31-89-97
From Agent.
Now, try to do the SSH to agent, and it should be connected without any credentials.
ssh ubuntu@<private IP address of agent VM>
Open Jenkins UI and configure the agent. Dashboard> Manage Jenkins> Nodes
Remote root directory: define the path. Launch method: Launch agents via ssh
Host: public IP address of agent VM
Credential of the agent. (will create the credential)
Kind: SSH Username with private key
private key from Jenkins Master server.
Host Key Verification Strategy: Non-Verifying Verification Strategy
Congratulations: the Agent is successfully configured and alive.
Install the plugin in Jenkins
Blue Ocean
Pipeline: Stage View
Run any job and verify that the job is executing on the agent node.
- create a below pipeline build it and verify the outcomes in the agent machine.
pipeline {
agent { label "balraj"}
stages {
stage('code') {
steps {
echo 'This is cloning the code'
git branch: 'main', url: 'https://github.com/mrbalraj007/django-notes-app.git'
echo "This is cloning the code"
}
}
}
}
- Build a Docker image. add the following state to main pipeline
stage('build') {
steps {
echo 'This is building the docker image'
sh "docker build -t notes-app:latest ."
echo "Image has been created successfully"
}
}
- Add the below the deploy the image.
stage('test') {
steps {
echo 'This is testing the code'
}
}
stage('Deploy') {
steps {
echo 'This is deploying the code'
sh 'docker run -d -p 8000:8000 notes-app:latest '
}
}
pipeline {
agent { label "balraj"}
stages {
stage('code') {
steps {
echo 'This is cloning the code'
git branch: 'main', url: 'https://github.com/mrbalraj007/django-notes-app.git'
echo "This is cloning the code"
}
}
stage('build') {
steps {
echo 'This is building the docker image'
sh "docker build -t notes-app:latest ."
echo "Image has been created successfully"
}
}
stage('test') {
steps {
echo 'This is testing the code'
}
}
stage('Deploy') {
steps {
echo 'This is deploying the code'
sh 'docker run -d -p 8000:8000 notes-app:latest '
}
}
}
}
Now, try to access it via the 8000 port
If you rerun the build, then you will get an error because port 8000 has already been used. So we will use here Docker Compose.
here is the updated pipeline
pipeline {
agent { label "balraj"}
stages {
stage('code') {
steps {
echo 'This is cloning the code'
git branch: 'main', url: 'https://github.com/mrbalraj007/django-notes-app.git'
echo "This is cloning the code"
}
}
stage('build') {
steps {
echo 'This is building the docker image'
sh "docker build -t notes-app:latest ."
echo "Image has been created successfully"
}
}
stage('test') {
steps {
echo 'This is testing the code'
}
}
stage('Deploy') {
steps {
echo 'This is deploying the code'
sh 'docker compose up -d'
}
}
}
}
Kill the existing image/deployment before build/executing it.
docker container ls
docker stop aa48f961a5de && docker rm aa48f961a5de
Push image to Docker hub.
create credential in Jenkins for Dockerhub
Bind credential in the pipeline
pipeline {
agent { label "balraj"}
stages {
stage('code') {
steps {
echo 'This is cloning the code'
git branch: 'main', url: 'https://github.com/mrbalraj007/django-notes-app.git'
echo "This is cloning the code"
}
}
stage('build') {
steps {
echo 'This is building the docker image'
sh "docker build -t notes-app:latest ."
echo "Image has been created successfully"
}
}
stage('test') {
steps {
echo 'This is testing the code'
}
}
stage('Push to DockerHub') {
steps {
echo "This is pushing image to Docker Hub"
withCredentials([usernamePassword(credentialsId:"dockerHubCred",passwordVariable:"dockerHubPass",usernameVariable:"dockerHubUser")]){
sh "docker login -u ${env.dockerHubUser} -p ${env.dockerHubPass}"
sh "docker image tag notes-app:latest ${env.dockerHubUser}/notes-app:latest"
sh "docker push ${env.dockerHubUser}/notes-app:latest"
}
}
}
stage('Deploy') {
steps {
echo 'This is deploying the code'
sh 'docker compose up -d'
}
}
}
}
Creating a webhook for Jenkins in Github
Go to repo setting and click on webhooks PayloadURL: http://54.144.163.163:8080/github-webhook/ (Jenkins URL with port)
Content type: Application/JSON
[!Note] I was getting a 302 error message, and when I followed the below procedure, it fixed itself.
I clicked on webhooks clicked on the recent deliveries and clicked on redeliver and the issue was fixed.
Now, we have to tick this option in Jenkins: "GitHub hook trigger for GITScm polling."
Try to modify anything in the Github repo and the build should be auto trigger.
it works :-)
Ref Link
