Part 1 in a multi-part series on Azure DevOps pipelines
Two of the pillars of a DevOps strategy are Continuous Integration and Continuous Deployment (CI/CD). At Mercury we have been utilizing Azure DevOps for our CI/CD process and have seen the implementation of Pipelines change and continuously improve. The trend has been towards a fully scripted pipeline that can be included in version control along with the code and infrastructure.
In this series I’ll be walking through setting up an end to end pipeline using multi-stage pipelines in YAML. It will assume some working knowledge of Azure DevOps, Azure and general DevOps concepts. Post in the comments if you would like to see some more detailed posts about these other topics.
Requirements to follow along
- Azure Subscription – Sign up for a free account https://azure.microsoft.com/en-us/free/
- Azure DevOps Account – Sign up for a free account https://azure.microsoft.com/en-us/services/devops/
- Repository – Any Git repository can be used and connected to Azure Pipelines but this walkthrough will utilize Azure Repos Git repository
- IDE – This walkthrough was created using Visual Studio Code which has extensions for Pipeline syntax highlighting
We will be deploying a .Net Core API project throughout this series. You can find the base project that these walkthroughs will be deploying here – https://github.com/cashewshideout/blog-azurepipeline. It is not necessary to have previous knowledge of .Net Core for this walkthrough; the concepts of creating the Pipeline are universal between all supported languages.
This is the tentative list of planned posts in the series. Links and list will be updated as posts are published.
- Intro and setting up the build steps (this post)
- Deployment steps, environment and approvals
- Pipeline templates
Getting Started – Basic structure and YAML
The Microsoft documentation for Azure Pipelines has a good breakdown of the pipeline hierarchy and the supported YAML syntax (see references for a link).
A pipeline is comprised of Stages, Jobs and Steps. A stage contains multiple jobs and jobs contain multiple steps. The YAML syntax following the outline above would be:
Keep an eye on the required indents and dashes when creating a pipeline. There are syntax checker add-ons in Visual Studio Code that can help prevent errors.
Planning the Build
As said previously, the goal of this post is to focus on the build steps for the application. For those familiar with the current setup of Azure Pipelines our end goal is to create the artifact that will be deployed.
This is the plan for the steps needed to create the final artifact:
- Install build requirements
- Restore dependencies (in this case, NuGet packages)
- Publish (create application packages)
- Create build artifact (to be used in future stages)
For this part of the pipeline we will go ahead and put all of these in a single stage and a single job. Multiple jobs will allow you to run those groups of steps in parallel which isn’t necessary here – all the steps are dependent on the
Time to start writing code – First Task
Create a file in your project with a .yml extension. There is not a required name or location for the file. In my projects I usually create a folder at the top level for it and name the file pipeline.yml.
We know there will be one stage, one job and up to six steps, but let’s start with just the first step.
The internal name of stages, jobs and tasks do not allow spaces and are not always
The build has to run on some kind of machine (agent) and we have two options – a Microsoft hosted agent or a private agent. This can be a separate topic in itself so we are going with the Microsoft hosted agent which we get one of for free. There are various images available to choose from. Here we are asking for whatever the latest version of Windows is available. For additional options
task: There are a wide variety of tasks available in Azure DevOps by default and available through the marketplace (https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/?view=azure-devops).
The first task is the Dot Net Core installer task. We want to make sure that a specific version of the .Net Core SDK is installed that is compatible with our application. The syntax ‘3.x’ is used to specify that it should install the latest of major version 3.
Pausing code writing – Running your new build
Let’s take a detour from building out the pipeline and go set up the pipeline in Azure so we can start testing as we go along (fail fast!).
In Azure DevOps, select ‘Pipelines’ in the navigation and then ‘New pipeline’.
Where is your code? For this walkthrough I am using Azure Repos Git. Setting up a connection to a different
repository is out of the scope of this series but once the connections are made the pipeline setup will be the same.
Configure your pipeline. There are options for some pre-made builds which can be useful starting points. For now select ‘Existing Azure Pipelines YAML file’. If you don’t see it in the default list then select the Show more button.
In the window that opens, select the branch and the path to the yaml file for your pipeline. The Path dropdown will pick up on appropriate files in your project.
After selecting the files it will show you the pipeline file.
Click the ‘Run’ blue button!
A screen is displayed with the build information and a drill down into the currently running job. On these screens you can see how the displayName property that was set is used.
And we have a passing build!
But, I don’t have a passing build!
If you don’t have a passing build then first double check that the syntax in yaml is correct. Here is an example where there was a missing space after the dash in ‘- job’ on line 5 in the script just used.
No drill down is available because the pipeline never executed with this error.
Finishing the Build
Only one task has been added so far to our script. Let’s add the additional tasks.
publish: You will notice that there are fewer steps in the script than what was outlined above. The .Net
Core publish command does the restore, build and publish all in one step.
Publish Artifact: This task is different from the .Net Core publish command. It will allow the artifact to
be available to other jobs in the pipeline (to be used in future posts in the series).
Check in the code and then in Azure DevOps watch the update pipeline run. It was set up previously and it will now automatically run the pipeline on any check in. There are ways to set specific triggers for when the pipeline will run but we’ll cover that later.
The success screen you see will be the same with a few new pieces of information
Related: Here you will see that there is 1 published item. This is the artifact that was created in the last step of the pipeline. Clicking on the link will allow you to see the full structure and download any files. This can be useful for debugging if all of the correct files were included.
Tests and coverage: The test project includes a single test (which hopefully passed).
You are on your way to making a full yaml pipeline. It is possible to stop here and only have the build in yaml then continue using the existing Release experience.
In future posts we will continue to expand this pipeline so the build and deployment steps are in the yaml pipeline. Keeping everything with the code and allowing changes to the pipeline also go through a full Pull Request review.
Please post comments below with questions! If you have a project that needs to be moved over to utilizing automated deployments, please contact us and we will be happy to assist.
Here are a few extra things that are helpful to have –
trigger & pr: Right now the pipeline will run on every branch on every check in. Specifying triggers will reduce how often the pipeline is run. Here it is specifying to only run the build on Pull Requests created for the master branch and on a merge to the
master branch. We have branch policies in place to require a passing build on
name: This creates a unique name for the build. By default it sets the date and the unique build ID. This has been overwritten to format the date differently and add the branch name. This can be modified to the format desired for your team.
variables: These are pipeline specific properties that can be reused throughout the file. A variable is referenced using $(variableName) syntax. For example, the AgentImage has been moved to a variable and referenced using $(AgentImage). While it is currently only used in one place, it will become useful in further posts as we extend the pipeline.
- Azure free account https://azure.microsoft.com/en-us/free/
- Azure DevOps free account https://azure.microsoft.com/en-us/services/devops/
- Base Project https://github.com/cashewshideout/blog-azurepipeline
- Microsoft documentation (Pipeline YAML) https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema%2Cparameter-schema
- Agent Image options https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops#use-a-microsoft-hosted-agent
- Azure Pipelines built in tasks https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/?view=azure-devops