cicd_module-initial commit (#122)

* cicd_module-initial commit

* Fixed the broken links in CI/CD course

* Fixed the broken links and image links in CI/CD course

Co-authored-by: Krishna Nukala <krnukala@microsoft.com>
pull/123/head
Krishna 3 years ago committed by GitHub
parent c3235c6baf
commit a4319ee421
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

@ -0,0 +1,19 @@
## The Evolution of the CI/CD
Traditional development approaches have been around for a very long time. The [waterfall model](https://www.linkedin.com/pulse/waterfall-model-shobika-ramasubbarayalu) has been widely used in both large and small projects and has been successful. Despite the success, it has a lot of drawbacks like longer cycle times or delivery.
While multiple team members are working on the project, the code changes get accumulated and never integrated until the planned build date. The build usually happens on agreed cycles that range from a month to a quarter. This results in several integration issues and build failures as the developers were working on their features in silos.
It was a nightmare situation for the operations teams/for anyone to deploy the new builds/releases to the production environment because of lack of proper documentation on every change and the configuration requirements. So, to deploy successfully, often it required hot fixes and immediate patches.
Another big challenge was collaboration. It is rare that the developer meets the operation engineers and does not have a full understanding of the production environment. All these challenges have given rise to longer cycle times for the delivery of the code changes.
[Agile](https://www.linkedin.com/pulse/list-popular-agile-methodologies-used-organizations) methodology prescribes the delivery of incremental delivery of features in multiple iterations. So, the developers commit their code changes in smaller increments and roll out more frequently. Every code commit triggers a new build, and the integration issues are identified much early. This has improved the build process and thereby reduced the cycle time. This process is known as *continuous integration or CI*.
The big barrier between the developers and the operation teams has been shrunken with the emergence of the trend where organizations are adapting to the DevOps and SRE disciplines. The collaboration between the developers and the operation teams is improved. Moreover, the use of the same tools and processes by both the teams has improved coordination and avoided conflicting understanding of the process. One of the main drivers in this regard is the *continuous delivery (CD)* process that ensures the incremental deployment of smaller changes. There are multiple pre-production environments also called the staging environments before deploying to production environments.
## CI/CD and DevOps
The term **DevOps** represents the combination of Development (Dev) and Operations (Ops) teams. That is bringing developers and operations teams together for more collaboration. The development team often wants to introduce more features and more changes while the operation teams are more focused on the stability of the application in production. A change is always taken as a threat by the operations team as it can shake the stability of the environment. DevOps is termed as a culture that introduces the processes to reduce the barriers between developers and operations.
The collaboration between Dev and Ops allows better follow-up of end-to-end production deployments and more frequent deployments. So, thus CI/CD is a key element in the DevOps processes.

@ -0,0 +1,19 @@
## Applications in SRE Role
The Monitoring, Automation and Eliminating the toil are some of the core pillars of the SRE discipline. As an SRE, you may require spending about 50% of time on automating the repetitive tasks and to eliminate the toil. CI/CD pipelines are one of the crucial tools for the SRE. They help in delivering the quality application with the smaller and regular and more frequent builds. Additionally, the CI/CD metrics such as Deployment time, Success rate, Cycle time and Automated test success rate etc. are the key things to watch to improve the quality of the product thus improving the reliability of the applications.
* [Infrastructure-as-code](https://en.wikipedia.org/wiki/Infrastructure_as_code) is one of the standard practices followed in SRE for automating the repetitive configuration tasks. Every configuration is maintained as code, so it can be deployed using CI/CD pipelines. It is important to deliver the configuration changes to the production environments through CI/CD pipelines to maintain the versioning, consistency of the changes across environments and to avoid manual errors.
* Often, as an SRE, you are required to review the application CI/CD pipelines and recommend additional stages such as static code analysis and the security and privacy checks in the code to improve the security and reliability of the product.
## Conclusion
In this chapter, we have studied the CI/CD pipelines with brief history on the challenges with the traditional build practices. We have also looked at how the CI/CD pipelines augments the SRE discipline. Use of CI/CD pipelines in software development life cycle is a modern approach in the SRE realm that helps achieve greater efficiency.
We have also performed a hands-on lab activity on creating the CI/CD pipeline using Jenkins.
## References
1. [Continuous Integration(martinfowler.com)](https://martinfowler.com/articles/continuousIntegration.html)
2. [CI/CD for microservices - Azure Architecture Center | Microsoft Docs](https://docs.microsoft.com/en-us/azure/architecture/microservices/ci-cd)
3. [SREFoundationBlueprint_2 (devopsinstitute.com)](https://www.devopsinstitute.com/wp-content/uploads/2020/11/SREF-Blueprint.pdf)
4. [Jenkins User Documentation](https://www.jenkins.io/doc/)

@ -0,0 +1,12 @@
***Continuous Delivery*** means deploying the application builds more frequently in the non-production environments such as [SIT, UAT, INT](https://medium.com/@buttertechn/qa-testing-what-is-dev-sit-uat-prod-ac97965ce4f) and performing the integration tests and the acceptance tests automatically.
In the CD, the tests are performed on the integrated application instead of the single microservice in the cases of microservice based application. The tests must include all the functional tests and the acceptance tests that may contain the UI tests. The build must be immutable in nature, that is the same package must be deployed across all the environments including the Production.
The deployment to the Production is often manual after performing additional acceptance tests such as performance tests etc. So, the fully automated deployment to the Production environments is called the ***Continuous Deployment*** (whereas ***CD Continuous delivery*** doesnt automatically deploy to Production). The continuous deployment must have a [feature toggle](https://martinfowler.com/articles/feature-toggles.html) so that a feature can be toggled off without the need for redeploying the code.
Often, the deployment involves more than one production environment, for example in [blue-green environments](https://www.linkedin.com/pulse/using-blue-green-deployments-reduce-downtime-nessan-harpur) the application is first deployed to the blue environment and then to the green environment so that the downtime is not required.
![](./images/CD_Image1.JPG)
*Fig 3: Continuous Delivery Pipeline*

@ -0,0 +1,16 @@
CI is a software development practice where members of a team integrate their work frequently. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.
Continuous integration requires that all the code changes be maintained in a single code repository where all the members can push the changes to their feature branches regularly. The code changes must be quickly integrated with the rest of the code and automated builds should happen and feedback to the member to resolve them early.
There should be a CI server where it can trigger a build as soon as the code is pushed by a member. The build typically involves compiling the code and transforming it to an executable file such as JARs or DLLs etc. called packaging. It must also perform [unit tests](https://en.wikipedia.org/wiki/Unit_testing) with code coverage. Optionally, the build process can have additional stages such as static code analysis and vulnerability checks etc.
[Jenkins](https://www.jenkins.io/), [Bamboo](https://confluence.atlassian.com/bamboo/understanding-the-bamboo-ci-server-289277285.html), [Travis CI](https://travis-ci.org/), [GitLab](https://about.gitlab.com/), [Azure DevOps](https://azure.microsoft.com/en-in/services/devops/) etc. are the few popular CI tools. These tools provide various plugins and integration such as [ant](https://ant.apache.org/), [maven](https://maven.apache.org/) etc. for building and packaging, and Junit, selenium etc. are for performing the unit tests. [SonarQube](https://www.sonarqube.org/) can be used for static code analysis and code security.
![](./images/CI_Image1.JPG)
*Fig 1: Continuous Integration Pipeline*
![](./images/CI_Image2.JPG)
*Fig 2: Continuous Integration Process*

Binary file not shown.

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 22 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 35 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 117 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 40 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 51 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 70 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 55 KiB

@ -0,0 +1,19 @@
## Prerequisites
1. [Software Development and Maintenance](https://en.wikibooks.org/wiki/Introduction_to_Software_Engineering/Implementation/Documentation)
2. [Git](https://linkedin.github.io/school-of-sre/level101/git/git-basics/)
3. [Docker](https://linkedin.github.io/school-of-sre/level102/containerization_and_orchestration/containerization_with_docker/)
## What to expect from this course?
In this course, you will learn the basics of CI/CD and how it helps drive the SRE discipline in an organization. It also discusses the various DevOps tools in CI/CD practice and a hands-on lab session on [Jenkins](https://www.jenkins.io/) based pipeline. Finally, it will conclude by explaining the role in the growing SRE philosophy.
## What is not covered under this course?
The course does not cover DevOps elements such as Infrastructure as a code, continuous monitoring applications and infrastructure comprehensively.
## Table of Contents
* [What is CI/CD?](https://linkedin.github.io/school-of-sre/level102/continuous_integration_and_continuous_delivery/introduction_to_cicd)
* [Brief History to CI/CD and DevOps](https://linkedin.github.io/school-of-sre/level102/continuous_integration_and_continuous_delivery/cicd_brief_history)
* [Continuous Integration](https://linkedin.github.io/school-of-sre/level102/continuous_integration_and_continuous_delivery/continuous_integration_build_pipeline)
* [Continuous Delivery and Deployment](https://linkedin.github.io/school-of-sre/level102/continuous_integration_and_continuous_delivery/continuous_delivery_release_pipeline)
* [Jenkins based CI/CD pipeline - Hands-on](https://linkedin.github.io/school-of-sre/level102/continuous_integration_and_continuous_delivery/jenkins_cicd_pipeline_hands_on_lab)
* [Conclusion](https://linkedin.github.io/school-of-sre/level102/continuous_integration_and_continuous_delivery/conclusion)

@ -0,0 +1,15 @@
Continuous Integration and Continuous Delivery, also known as CI/CD, is a set of processes that helps in faster integration of software code changes and deployment to the end user in a reliable manner. The more frequent integrations and deployments helps reduce the software development lifecycle.
There are three practices in CI/CD:
* Continuous Integration
* Continuous Delivery
* Continuous Deployment
Lets look in detail at each of these in the coming sections.
## The Benefits of CI/CD
1. Significant reduction in integration problems.
2. Teams can develop cohesive software more rapidly.
3. Improved Collaboration between developers and operation teams can reduce the production integration issues.
4. Faster delivery of new features with less friction
5. Better debugging the production issues and fixing them in the next release/patch.

@ -0,0 +1,198 @@
## Jenkins based CI/CD Pipeline
Jenkins is an open-source continuous integration server for orchestrating the CI/CD pipelines. It supports integration with several components, infrastructure such as git, cloud etc. that helps in complete software development life cycle.
In this hands-on lab, let us:
* Create a build pipeline (CI) for a simple java application.
* Adding Test stage to build pipeline
This hands-on is based on the Jenkins running on docker on your local workstation, designed for Windows OS. For Linux OS, please follow the [demo](https://www.jenkins.io/doc/tutorials/build-a-java-app-with-maven/)
**Note:** The hands-on lab is designed with Jenkins on the docker. However, the steps are applicable for the direct docker installation on your windows workstation as well.
### Installing Git, Docker and Jenkins:
* Install git command line tool on your workstation. (Follow [this](https://suwebdev.github.io/wats-lab-faq/first-question.html#download-and-install-git) to install Git Locally·)
* Docker Desktop for windows is installed on the workstation. Follow the [instructions](https://docs.docker.com/docker-for-windows/install/#install-docker-desktop-on-windows) to install docker.
* Ensure that your Docker for Windows installation is configured to run Linux Containers rather than Windows Containers. See the Docker documentation for instructions to switch to Linux containers.
* [Refer](https://www.jenkins.io/doc/tutorials/build-a-java-app-with-maven/#on-windows) this to run and setup the Jenkins on docker.
* Configure Jenkins with initial steps such as create an admin user etc. Follow Setup wizard.
* If you have installed the Jenkins on your local workstation, make sure the maven tool is installed. Follow [this](https://maven.apache.org/install.html) to installl maven.
### Forking Sample java application:
For this hands-on, let us fork a simple java application from the GitHub [simple-java-maven-app](https://github.com/jenkins-docs/simple-java-maven-app).
1. Sign up for the GitHub account [Join GitHub · GitHub](https://github.com/signup?source=login). Once signed up, proceed to [login](https://github.com/login).
2. Open the simple-java-maven-app by clicking on this [link](https://github.com/jenkins-docs/simple-java-maven-app)
3. On the top right corner, click on the Fork to create a copy of the project to your GitHub account. (Refer [Fork A Repo](https://help.github.com/articles/fork-a-repo/))
4. Once forked, clone this repository to your local workstation.
### Create Jenkins Project:
1. Login to the Jenkins portal at [http://localhost:8080](http://localhost:8080) using the admin account created earlier during Jenkinss setup.
2. On your first login, the following screen will appear. Click on “**Create a Job**”.
![](./images/Jenkins1.png)
*Fig 4: Jenkins - Create a Job*
3. On the next screen, type *simple-java-pipeline* in the **Enter an Item Name** field. Select *Pipeline* from the list of items and click *OK*.
![](./images/Jenkins2.png)
*Fig 5: Jenkins - Create Pipeline*
4. Click the **Pipeline** tab at the top of the page to scroll down to the *Pipeline* section.
5. From the **Definition** field, choose the *Pipeline script from SCM* option. This option instructs Jenkins to obtain your Pipeline from Source Control Management (SCM), which will be your locally cloned Git repository.
6. From the **SCM** field, choose *Git*.
7. In the **Repository URL** field, specify the directory path of your locally cloned repository from the [Forking Sample Java application](#forking-sample-java-application:) section above.
Screen looks like below after entering the details.
![](./images/Jenkins3.png)
*Fig 6: Jenkins - Pipeline Configuration*
### Create Build pipeline using the Jenkinsfile:
**Jenkinsfile** is a script file containing the pipeline configuration and the stages and other instructions to Jenkins to create a pipeline from the file. This file will be saved at the root of the code repository.
1. Using your favorite text editor or IDE, create and save a new text file with the name *Jenkinsfile* at the root of your local *simple-java-maven-app* Git repository.
2. Copy the following declarative pipeline code and paste it into the empty *Jenkinsfile*.
```
pipeline {
agent {
docker {
image 'maven:3.8.1-adoptopenjdk-11'
args '-v /root/.m2:/root/.m2'
}
}
stages {
stage('Build') {
steps {
sh 'mvn -B -DskipTests clean package'
}
}
}
}
```
**Note:** If you are running Jenkins on your local workstation without the docker, please change the agent to **any** as shown below so that it runs on the localhost. Please ensure that the maven tool is installed on your local workstation.
```
pipeline {
agent any
stages {
stage('Build') {
steps {
sh 'mvn -B -DskipTests clean package'
}
}
}
}
```
In the above Jenkinsfile:
* We specified an agent where the pipeline should run. 'docker in the agent section indicates to run a new docker container with the specified image.
* In the stages section, we can define multiple steps as different stages. Here, we have a stage called Build, with the maven command for building the java application.
3. Save your Jenkinsfile and commit and push to your forked repository. Run the following commands from the command prompt.
```
cd <your local simple-java-maven-app repo cloned folder>
git add .
git commit -m "Add initial Jenkinsfile"
git push origin master
```
4. Go to Jenkins portal on your browser and click on the **Dashboard**. Open the [simple-java-pipeline](http://localhost:8080/job/simple-java-pipeline/) and from the left-menu, click on **Build Now**.
![](./images/Jenkins4.png)
*Fig 7: Jenkins - Building the Pipeline*
5. Notice the build running under the **Build History** menu. Click on the *build number* and it shows the stages.
![](./images/Jenkins5.png)
*Fig 8: Jenkins - View Running Builds*
6. We have successfully created a build pipeline with single stage and ran it. We can check the logs by clicking on the **Console Output** menu.
### Additional stages in the build pipeline:
In the previous section, we have created the pipeline with a single stage. Typically, your CI pipeline contains multiple stages such as Build, Test and other optional stages such Code scanning etc. In this section, let us add a Test stage to the build pipeline and run.
1. Go back to your text editor/IDE and open Jenkinsfile and the Test stage shown below.
```
stage('Test') {
steps {
sh 'mvn test'
}
post {
always {
junit 'target/surefire-reports/*.xml'
}
}
}
```
The Jenkinsfile looks like below after adding the Test stage.
```
pipeline {
agent {
docker {
image 'maven:3.8.1-adoptopenjdk-11'
args '-v /root/.m2:/root/.m2'
}
}
stages {
stage('Build') {
steps {
sh 'mvn -B -DskipTests clean package'
}
}
stage('Test') {
steps {
sh 'mvn test'
}
post {
always {
junit 'target/surefire-reports/*.xml'
}
}
}
}
}
```
* Here the stage Test is added which runs the maven command test.
* The **post -> always** section ensures that this step is executed always after the steps are completed. The test report is available through Jenkinss interface.
**Note:** If you are running Jenkins on your local workstation without the docker, please change the agent to **any** so that it runs on the localhost. Please ensure that the maven tool is installed on your local workstation.
```
pipeline {
agent any
stages {…
}
}
```
2. Save your Jenkinsfile and commit and push to your forked repository. Run the following commands from the command prompt.
```
cd <your local simple-java-maven-app repo cloned folder>
git add .
git commit -m "Test stage is added to Jenkinsfile"
git push origin master
```
3. Go to Jenkins portal on your browser and click on the **Dashboard**. Open the *simple-java-pipeline* and from the left-menu, click on Build Now.
4. Notice the Build and Test stages are showing in the Build screen.
![](./images/Jenkins6.png)
*Fig 9: Jenkins - Viewing the Running Builds with Test stage Included*
We have now successfully created CI pipeline with two stages: Build and Test stages.

@ -120,6 +120,14 @@ nav:
- Performance Improvements: level102/system_troubleshooting_and_performance/performance-improvements.md
- Troubleshooting Example: level102/system_troubleshooting_and_performance/troubleshooting-example.md
- Conclusion: level102/system_troubleshooting_and_performance/conclusion.md
- Continuous Integration and Continuous Delivery:
- Introduction: level102/continuous_integration_and_continuous_delivery/introduction.md
- What is CI/CD?: level102/continuous_integration_and_continuous_delivery/introduction_to_cicd.md
- Brief History: level102/continuous_integration_and_continuous_delivery/cicd_brief_history.md
- Continuous Integration: level102/continuous_integration_and_continuous_delivery/continuous_integration_build_pipeline.md
- Continuous Delivery and Deployment: level102/continuous_integration_and_continuous_delivery/continuous_delivery_release_pipeline.md
- CI/CD Pipeline - Hands-on: level102/continuous_integration_and_continuous_delivery/jenkins_cicd_pipeline_hands_on_lab.md
- Conclusion: level102/continuous_integration_and_continuous_delivery/conclusion.md
- Contribute: CONTRIBUTING.md
- Code of Conduct: CODE_OF_CONDUCT.md
- SRE Community: sre_community.md

Loading…
Cancel
Save