Azure pipeline as a core part of Azure DevOps, it allows for the creation of CI ( Continuous Integration) pipeline in a declarative way using YAML documents, it is also called build pipelines. Since last Build 2019, this capability is also extending to CD ( Continuous Delivery ) Pipelines which is also known as Release Pipelines. More than expected is now it is possible to define multi-stage pipelines-as-code for both Continuous Integration and Continuous Delivery with the same YAML definition file.
Since Github can be easily integrated with Azure DevOps nowadays, you can not only build your CI/CD pipeline based on your source code on Github, but also even Mapping your GitHub repository permissions with Azure DevOps.
I am writing this blog to explain how to use Azure CI/CD pipelines to provide an end-to-end automation experience to users when deploying a node.js application via Azure DevOps. Our solution will look like the following diagram :
End-to-End CI/CD architecture diagram
Before getting started, you have to prepare a couple of things :
Step 1: Get an Azure subscription
You need an Azure account: Create an Azure account by browsing to https://azure.microsoft.com/en-us/free/ or claim your MSDN benefits to get a visual studio subscription.
Step 2 : Get an Azure DevOps account and create organisation and project. You can check Quickstart: Create an organization or project collection to know more details.
You’ll need to activate the ‘ multi-stage pipeline’ from the settings of preview feature.
Activate the multi-stage pipelines preview feature
Step 3 : Get source from cloudmelon’s Github repository ( using the following URL ), fork it and use your own branch for our lab here : https://cloudmelon.github.io/oss-cicd-devops/
Provision infrastructure via ARM template
One of the most important part of end-to-end automation start from automating the infrastructure provision process, by implementing Infrastructure as code (IaC) mindsets, we can use either ARM or Terraform to provision our infrastructure running in Azure.
In this solution, we’re going to provision two App services Plan and host web app on each of them. In the enterprise environment, due to budget and performance concerns, usually, we deploy Development and Production environment separately. On the top of the solution, we can also use full stack monitoring solutions to gain more visibility of Azure resources which we’ll get to there later on.
Where DevOps magic happens: defining service connections
There are so many discussions around automation and DevOps going on over past a few years. From my point of view, one of the most charming features of DevOps is you can deploy and automate the whole process as long as you have the right permission. In our example, you only need to replace the service connections then you can get this solution up and running in your subscription. You only need to go to Project settings and add a new service connection as shown in the following :
Create service connections
Trigger and variable definition
Basically, Continuous integration (CI) triggers in Azure DevOps will set off a build to run when a push is made to the specified branches ( by setting path filters ) or a specified tag is pushed. Below is an example of trigger defined in Yaml:
trigger: - master
In Azure DevOps, you can use a variable in Yaml definition or using Variable group and then link up to your CI or CD definition. Below is an example of yaml-defined variables :
variables: # Agent VM image name vmImageName: 'ubuntu-latest' demorg: 'melon-cicd-rg' subscription : '(please replace it by your own service connection name)' webappname : 'melonsamplewebapp'
Define your CI pipeline
In your CI pipeline, you need to define as the following :
- stage: Build displayName: Build stage jobs: - job: Build displayName: Build pool: vmImage: $(vmImageName) steps: - task: AzureCLI@1 displayName: 'Azure CLI ' inputs: azureSubscription: $(subscription) scriptLocation: inlineScript inlineScript: 'az group create --location northeurope --name $(demorg)' - task: AzureResourceGroupDeployment@2 displayName: 'Azure Deployment:Create Or Update Resource Group action on $(demorg)' inputs: azureSubscription: $(subscription) resourceGroupName: '$(demorg)' location: 'North Europe' templateLocation: 'Linked artifact' csmFile: 'iac/webapp.json' csmParametersFile: 'iac/webapp.parameters.json' deploymentMode: 'Incremental' - task: NodeTool@0 inputs: versionSpec: '10.x' displayName: 'Install Node.js' - script: | npm install npm run build --if-present # npm run test --if-present displayName: 'npm install, build and test' - task: CopyFiles@2 displayName: 'Copy Files to: $(Build.ArtifactStagingDirectory)/$(webappname)' inputs: SourceFolder: '$(system.defaultworkingdirectory)' TargetFolder: '$(Build.ArtifactStagingDirectory)/$(webappname)' - task: ArchiveFiles@2 displayName: '$(webappname) Archive' inputs: rootFolderOrFile: '$(Build.ArtifactStagingDirectory)/$(webappname)' includeRootFolder: false archiveType: zip replaceExistingArchive: true archiveFile: '$(Build.ArtifactStagingDirectory)/$(webappname).zip' - task: PublishPipelineArtifact@0 displayName: 'PublishPipelineArtifact: drop' inputs: targetPath: '$(Build.ArtifactStagingDirectory)/$(webappname).zip'
Define your CD pipeline
Now yaml-defined multi-stage CD pipeline is also supported in Azure DevOps, if you’re not comfortable with yaml definition, you can also check Define your multi-stage continuous deployment (CD) pipeline to know more about multi-stage pipeline.
Please make sure you define different stages, dependencies and deployment condition :
Dev stage :
- stage: Dev displayName: Dev stage dependsOn: Build condition: succeeded('Build') jobs: - deployment: Dev displayName: Dev environment: 'development' pool: vmImage: $(vmImageName) strategy: runOnce: deploy: steps: - task: DownloadPipelineArtifact@1 displayName: 'Download Pipeline Artifact' inputs: buildType: 'current' - task: AzureWebApp@1 inputs: azureSubscription: '$(subscription)' appType: 'webApp' appName: '$(webappname)' package: '$(System.ArtifactsDirectory)/drop/$(webappname).zip' customWebConfig: '-Handler iisnode -NodeStartFile index.js -appType node' deploymentMethod: 'zipDeploy'
Prod stage :
- stage: Prod displayName: Prod stage dependsOn: Dev condition: succeeded('Dev') jobs: - deployment: Prod displayName: Prod environment: 'production' pool: vmImage: $(vmImageName) strategy: runOnce: deploy: steps: - task: DownloadPipelineArtifact@1 displayName: 'Download Pipeline Artifact' inputs: buildType: 'current' - task: AzureWebApp@1 inputs: azureSubscription: '$(subscription)' appType: 'webApp' appName: '$(prodwebappname)' package: '$(System.ArtifactsDirectory)/drop/$(webappname).zip' customWebConfig: '-Handler iisnode -NodeStartFile index.js -appType node' deploymentMethod: 'zipDeploy'
The environment is a new feature of Azure DevOps pipeline, it represents a collection of resources such as namespaces within Kubernetes clusters or Azure Web Apps, basically, anything is targeted by deployments from a pipeline. In our example environments include Development and Production environment.
Environments in Azure DevOps
Art of possible: Full stack, end-to-end visibility Azure unified monitoring solution
Since the end of 2018, Azure Log Analytics and Azure Application Insights were available as integrated features within Azure Monitor, it is a pretty cool path which shows the possibility to provide administrators and DevOps engineers full stack unified monitoring solution, you can also check End-to-end monitoring solutions in Azure for Apps and Infrastructure to have more details.
Up & Running
You can check your run history and different stages in Azure DevOps.
Deploying multi-stage pipelines in Azure DevOps
The value of end-to-end automation is to cut down time-to-market and to boot your business performance. The scenarios are not only on OSS or .Net application deployment but also make senses on API economy as well as microservices productivities. However, one of the biggest challenges of this approach is about security and compliance, where DevSecOps comes out, you can find more information about Secure DevOps here. Let’s stay tuned!
Originally published on 8th July 2019 on https://buildazure.com/end-end-ci-cd-automation-using-azure-devops-unified-yaml-defined-pipelines/ by Melony Qin.
Categories: Cloud Native Vision
aka CloudMelon. I am the book author for Microsoft Azure Infrastructure and the Kubernetes Workshop published by Packt Publishing, the technical reviewer for Azure for Architects, 3rd Edition. And more importantly, an educator and community advocate. Mainly working on OSS, DevOps, Kubernetes, Serverless, Big Data & AI and IoT on Microsoft Azure in the community. She can be reached out her blog website: cloud-melon.com, follow her YouTube Channel : CloudMelon Vision and Twitter @MelonyQ.