Jekyll2021-02-04T16:50:40+00:00https://lgulliver.github.io/feed.xmlLiam’s BlogBlogging about Azure, Azure DevOps, DevOps and all that fun stuffLiam GulliverBuilding YAML CI/CD Pipelines in Azure DevOps [Part 1]2021-02-04T18:00:00+00:002021-02-04T18:00:00+00:00https://lgulliver.github.io/building-yaml-cicd-pipelines-in-azure-devops-part-1<p>Recently, <a href="https://twitch.tv/azureishlive">Azureish Live!</a> returned - our new project is to build a zero touch CI/CD pipeline in <a href="https://lgulliver.github.io/tags/#azure-devops">Azure DevOps</a> for the <a href="https://lgulliver.github.io/azureish-live!-building-a-git-commit-watcher-with-azure-functions-and-github-part1/">project we build last year for the show</a>.</p>
<p>You can watch the whole first part of the show above. Other than a minor technical hiccup, I think we did pretty well!</p>
<p>In this post, we’re going to take a quick look at what we’ve got to do and what we achieved in our first part.</p>
<p>What we created as part of our GitWatcher project was a complete serverless solution using Azure Functions, Azure SignalR and Azure Storage for hosting a static website build with Blazor WebAssembly.</p>
<p>The application currently looks like the below:</p>
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="319px" viewBox="-0.5 -0.5 319 187" content="<mxfile host="app.diagrams.net" modified="2021-02-04T15:03:31.699Z" agent="5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.96 Safari/537.36 Edg/88.0.705.56" etag="xrGKU2o0LIArBVjX5Qau" version="14.2.9"><diagram id="rnFcCY4i2SYY_jUFPXYP" name="Page-1">3Vddb5swFP01PG4KH6btY77aTVrVaJnUx8kFFzwZLjKXBPrrdykmhKJEtGu3LE/Y517j63POxcJy50l5o3kW30IolOVMwtJyF5bj2I5/RY8aqQwymTgNEmkZGqwD1vJJtIkGLWQo8l4iAiiUWR8MIE1FgD2Maw3bftojqP6uGY/EAFgHXA3Rexli3KCO6/pd4IuQUdxu7TOviSS8zTZHyWMewnYPcpeWO9cA2IySci5UTV9LTLPu+kB0V5kWKY5ZAKqa3iArsuwuxmka/7izw09Gnw1XhTmxKRarlgIREiNmChpjiCDlatmhMw1FGop6mwnNupxvABmBNoG/BGJl5OUFAkExJspEmz3rjQ6ezUA5FDoQRw7UmoTrSOCRPG+nAJlXQCJQV7ROC8VRbvp1cGOiaJfX0UwDw/QrWLcHrE9XXwfE8zxrLP0oy5rdfcYykCk+l8VmFlsQwpWMUgICokxoAmTy7O3ZI6RomLedDl/IJKLilXyoj5AHXNDzukgDlJDmn/NNtBNmIzSK8rg0QyrNAtcz/jefAIexZr7t+omZlHivkzz/g8i/PDfLOyMtf0Cnv2N5Z8D6SkNZnanpPXZipreH/P/nrvdGup79oevN0lXtvT2BXdYT2L1i/Vc0dZlVL8TblfF2Pb2BnGvqBK6+19QJvZHE3Qm0linq424T92IyqrFa7N0b6+Lc+oqN7CvnX94mbMD6TPEn0ITdT9e3luMrqn32QIAf1aM1UlUBhdcST6MzEDRFf06DgMTHd7x72OXLu8cfd/ewV4tF0+7/pfm0df+B7vI3</diagram></mxfile>" onclick="(function(svg){var src=window.event.target||window.event.srcElement;while (src!=null&&src.nodeName.toLowerCase()!='a'){src=src.parentNode;}if(src==null){if(svg.wnd!=null&&!svg.wnd.closed){svg.wnd.focus();}else{var r=function(evt){if(evt.data=='ready'&&evt.source==svg.wnd){svg.wnd.postMessage(decodeURIComponent(svg.getAttribute('content')),'*');window.removeEventListener('message',r);}};window.addEventListener('message',r);svg.wnd=window.open('https://viewer.diagrams.net/?client=1&page=0&edit=_blank');}}})(this);" style="cursor:pointer;max-width:100%;max-height:187px;"><defs /><g><path d="M 41 46 L 41 108.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke" /><path d="M 41 113.88 L 37.5 106.88 L 41 108.63 L 44.5 106.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all" /><image x="15.5" y="-0.5" width="50" height="46" xlink:href="https://app.diagrams.net/img/lib/mscae/Functions.svg" /><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe flex-start; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 53px; margin-left: 41px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; background-color: white; white-space: nowrap; ">API</div></div></div></foreignObject><text x="41" y="65" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">API</text></switch></g><path d="M 126 23 L 72.37 23" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke" /><path d="M 67.12 23 L 74.12 19.5 L 72.37 23 L 74.12 26.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all" /><image x="125.5" y="-0.5" width="50" height="46" xlink:href="https://app.diagrams.net/img/lib/mscae/Functions.svg" /><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe flex-start; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 53px; margin-left: 151px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; background-color: white; white-space: nowrap; ">Proxy</div></div></div></foreignObject><text x="151" y="65" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Proxy</text></switch></g><path d="M 66 140 L 281 140 L 281 52.37" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke" /><path d="M 281 47.12 L 284.5 54.12 L 281 52.37 L 277.5 54.12 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all" /><image x="15.5" y="114.5" width="50" height="50" xlink:href="https://app.diagrams.net/img/lib/mscae/SignalR.svg" /><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe flex-start; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 172px; margin-left: 41px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; background-color: white; white-space: nowrap; ">SignalR service</div></div></div></foreignObject><text x="41" y="184" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">SignalR...</text></switch></g><path d="M 256 23.5 L 216 23.5 L 182.37 23.08" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke" /><path d="M 177.12 23.01 L 184.16 19.6 L 182.37 23.08 L 184.07 26.6 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all" /><image x="255.5" y="0.5" width="50" height="45" xlink:href="https://app.diagrams.net/img/lib/mscae/Storage_Accounts.svg" /><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe flex-start; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 53px; margin-left: 281px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; background-color: white; white-space: nowrap; ">Blazor WASM<br />Static Site</div></div></div></foreignObject><text x="281" y="65" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Blazor W...</text></switch></g></g><switch><g requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility" /><a transform="translate(0,-5)" xlink:href="https://www.diagrams.net/doc/faq/svg-export-text-problems" target="_blank"><text text-anchor="middle" font-size="10px" x="50%" y="100%">Viewer does not support full SVG 1.1</text></a></switch></svg>
<p>The Blazor static site is ultimately something that needs to be compiled, as is the API we have. We chose to start with the API as the easiest part to build. Most importantly, for now, we have chosen only to build it and not deploy just yet to keep things simple.</p>
<p>The API is a simple Function App with a couple of methods to be called by the front-end. You can see the code we wrote for that over at the <a href="https://github.com/AzureishLive/gitwatcher/tree/main/backend">GitHub repo</a>.</p>
<p>It is also important to note that for this project we have a single repository that contains all components. Our approach also means that we will be having one YAML pipeline per component, all in the same repository.</p>
<p>A couple of constraints we set ourselves:</p>
<ol>
<li>We want a build to run for <em>every</em> commit.</li>
<li>We won’t be running tests because we haven’t written any (yet).</li>
</ol>
<p>Getting setup with a stage for <code class="language-plaintext highlighter-rouge">build</code> along with the relevant triggers for our GitHub Flow branching strategy is relatively straightforward. We need to define our triggers, agent pool and of course, the first stage.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">trigger</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">main</span>
<span class="pi">-</span> <span class="s">feature/*</span>
<span class="na">pool</span><span class="pi">:</span>
<span class="na">vmImage</span><span class="pi">:</span> <span class="s1">'</span><span class="s">ubuntu-latest'</span>
<span class="na">stages</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">stage</span><span class="pi">:</span> <span class="s">build</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">job</span><span class="pi">:</span> <span class="s">build</span>
<span class="na">steps</span><span class="pi">:</span>
</code></pre></div></div>
<p>The triggers set here means that a build will run any time a commit is pushed (or merged into) <code class="language-plaintext highlighter-rouge">main</code> along with any branches that appear under <code class="language-plaintext highlighter-rouge">feature/*</code> e.g. <code class="language-plaintext highlighter-rouge">feature/adding-pipelines</code>.</p>
<p>Our first task is to ensure we have the right SDK for dotnet core available to us - in our case, this is 3.1.x LTS as we are yet to move GitWatcher to .NET 5 (but this will happen in a future show).</p>
<p>To do this, we need to use the <code class="language-plaintext highlighter-rouge">UseDotNet</code> task. This task allows us to explicitly define the SDK (or runtime) version of dotnet we wish to use. We now have to include this step as the default version available on the Microsoft-hosted agents is now .NET 5.</p>
<p>It doesn’t have to be exact, it can be a latest minor version which is what we do here by using the following configuration:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">UseDotNet@2</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">packageType</span><span class="pi">:</span> <span class="s1">'</span><span class="s">sdk'</span>
<span class="na">version</span><span class="pi">:</span> <span class="s1">'</span><span class="s">3.1.x'</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Set</span><span class="nv"> </span><span class="s">SDK</span><span class="nv"> </span><span class="s">to</span><span class="nv"> </span><span class="s">3.1.x'</span>
</code></pre></div></div>
<p>Now that we have that, the next thing we needed to do was to actually build the project!</p>
<p>As I mentioned, we haven’t written tests for this, even though we probably should!</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">DotNetCoreCLI@2</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">command</span><span class="pi">:</span> <span class="s1">'</span><span class="s">build'</span>
<span class="na">projects</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(Build.SourcesDirectory)/backend/backend.csproj'</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Build</span><span class="nv"> </span><span class="s">backend</span><span class="nv"> </span><span class="s">service'</span>
</code></pre></div></div>
<p>What we’re doing here is being specific about the project we want to build. As I mentioned earlier, what we have is a single repo and we don’t want to build the entire solution all at once.</p>
<p>Our next step is to create a package to publish:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">DotNetCoreCLI@2</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">command</span><span class="pi">:</span> <span class="s1">'</span><span class="s">publish'</span>
<span class="na">publishWebProjects</span><span class="pi">:</span> <span class="no">false</span>
<span class="na">projects</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(Build.SourcesDirectory)/backend/backend.csproj'</span>
<span class="na">arguments</span><span class="pi">:</span> <span class="s1">'</span><span class="s">-o</span><span class="nv"> </span><span class="s">$(Build.ArtifactStagingDirectory)'</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">dotnet</span><span class="nv"> </span><span class="s">publish'</span>
</code></pre></div></div>
<p>We’ve set the output of the publish action to put in the <code class="language-plaintext highlighter-rouge">ArtifactStagingDirectory</code>. This is where we’ll publish our artifact from to our build for later use.</p>
<p>Ultimately, all this command is running is the <code class="language-plaintext highlighter-rouge">dotnet publish</code> command.</p>
<p>Then our final step is to publish the build artifact:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">PublishBuildArtifacts@1</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">PathtoPublish</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(Build.ArtifactStagingDirectory)'</span>
<span class="na">ArtifactName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">drop'</span>
<span class="na">publishLocation</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Container'</span>
</code></pre></div></div>
<p>But what if we don’t want to create a build artifact every time? In my opinion, we shouldn’t need to. At least not when we’re building from feature branches on every commit for validation. We also don’t need to take up the storage for it every time we do a build and it’s not form <code class="language-plaintext highlighter-rouge">main</code> (we can though, nothing stopping you doing that, it’s just not my preferred practice).</p>
<p>On all the steps we don’t want to run depending on the source branch, we added a <code class="language-plaintext highlighter-rouge">condition</code>. For this particular build definition, we added the <code class="language-plaintext highlighter-rouge">condition</code> to both the <code class="language-plaintext highlighter-rouge">dotnet publish</code> and <code class="language-plaintext highlighter-rouge">PublishBuildArtifact</code> tasks.</p>
<p>The condition itself, looks a little like this:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">condition</span><span class="pi">:</span> <span class="s">and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))</span>
</code></pre></div></div>
<p>This is saying that providing the previous step succeeded, and the branch that triggered the build is <code class="language-plaintext highlighter-rouge">main</code> then we’ll run this step.</p>
<p>The complete step with the condition looks like this:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">DotNetCoreCLI@2</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">command</span><span class="pi">:</span> <span class="s1">'</span><span class="s">publish'</span>
<span class="na">publishWebProjects</span><span class="pi">:</span> <span class="no">false</span>
<span class="na">projects</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(Build.SourcesDirectory)/backend/backend.csproj'</span>
<span class="na">arguments</span><span class="pi">:</span> <span class="s1">'</span><span class="s">-o</span><span class="nv"> </span><span class="s">$(Build.ArtifactStagingDirectory)'</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">dotnet</span><span class="nv"> </span><span class="s">publish'</span>
<span class="na">condition</span><span class="pi">:</span> <span class="s">and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))</span>
</code></pre></div></div>
<p>Putting the whole thing together gives us a YAML pipeline that at this stage allows us to do continuous integration builds on every commit.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">trigger</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">main</span>
<span class="pi">-</span> <span class="s">feature/*</span>
<span class="na">pool</span><span class="pi">:</span>
<span class="na">vmImage</span><span class="pi">:</span> <span class="s1">'</span><span class="s">ubuntu-latest'</span>
<span class="na">stages</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">stage</span><span class="pi">:</span> <span class="s">build</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">job</span><span class="pi">:</span> <span class="s">build</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">UseDotNet@2</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">packageType</span><span class="pi">:</span> <span class="s1">'</span><span class="s">sdk'</span>
<span class="na">version</span><span class="pi">:</span> <span class="s1">'</span><span class="s">3.1.x'</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Set</span><span class="nv"> </span><span class="s">SDK</span><span class="nv"> </span><span class="s">to</span><span class="nv"> </span><span class="s">3.1.x'</span>
<span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">DotNetCoreCLI@2</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">command</span><span class="pi">:</span> <span class="s1">'</span><span class="s">build'</span>
<span class="na">projects</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(Build.SourcesDirectory)/backend/backend.csproj'</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Build</span><span class="nv"> </span><span class="s">backend</span><span class="nv"> </span><span class="s">service'</span>
<span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">DotNetCoreCLI@2</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">command</span><span class="pi">:</span> <span class="s1">'</span><span class="s">publish'</span>
<span class="na">publishWebProjects</span><span class="pi">:</span> <span class="no">false</span>
<span class="na">projects</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(Build.SourcesDirectory)/backend/backend.csproj'</span>
<span class="na">arguments</span><span class="pi">:</span> <span class="s1">'</span><span class="s">-o</span><span class="nv"> </span><span class="s">$(Build.ArtifactStagingDirectory)'</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">dotnet</span><span class="nv"> </span><span class="s">publish'</span>
<span class="na">condition</span><span class="pi">:</span> <span class="s">and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))</span>
<span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">PublishBuildArtifacts@1</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">PathToPublish</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(Build.ArtifactStagingDirectory)'</span>
<span class="na">ArtifactName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">drop'</span>
<span class="na">publishLocation</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Container'</span>
<span class="na">condition</span><span class="pi">:</span> <span class="s">and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))</span>
</code></pre></div></div>
<p>You’ll perhaps notice though that currently, there’s one glaring omission that means that this build will run when <em>any</em> commit is pushed back to the repo, regardless of project.</p>
<p>On the next show, we’ll be demonstrating how to solve that and moving on to infrastructure as code!</p>Liam GulliverRecently, Azureish Live! returned - our new project is to build a zero touch CI/CD pipeline in Azure DevOps for the project we build last year for the show.DevOps Notts - January 2021 - Resilience Engineering with Tom Geraghty from Red Hat Open Innovation Labs2021-02-02T18:00:00+00:002021-02-02T18:00:00+00:00https://lgulliver.github.io/devops-notts-resilience-engineering-with-tom-geraghty<p>In January, Mica and I had the privilege of hosting <a href="https://twitter.com/tom_geraghty">Tom Geraghty</a> of <a href="https://www.redhat.com/en/services/consulting/open-innovation-labs">Red Hat Open Innovation Labs</a> at <a href="https://lgulliver.github.io/tags/#devops-notts">DevOps Notts</a>.</p>
<p>We ran this event slightly differently to our usual format by having Tom as the only speaker and running things as more of a workshop than anything else!</p>
<p>You can catch the whole session on the video above.</p>
<p>Tom also provided a huge list of resources along with his slides which you can get over on the <a href="https://github.com/devopsnotts/ResilienceEngineering">DevOps Notts GitHub</a>.</p>Liam GulliverIn January, Mica and I had the privilege of hosting Tom Geraghty of Red Hat Open Innovation Labs at DevOps Notts.Using Terrascan with Azure DevOps2021-01-21T18:00:00+00:002021-01-21T18:00:00+00:00https://lgulliver.github.io/terrascan-in-azure-devops<p><a href="https://lgulliver.github.io/scan-terraform-kubernetes-and-more-with-terrascan/">In my last post, I took a look at a new scanning tool called Terrascan.</a> It can be used to ensure your Kubernetes manifests, Terraform and more are compliant with a set of built-in, or customised rules.</p>
<p>So far, my initial impressions of Terrascan have been positive (albeit, the release notes could use a little work).</p>
<p>As you know, I’m a huge fan of <a href="https://lgulliver.github.io/tags/#azure-devops">Azure DevOps</a> and one of the things I wanted to do with Terrascan is get it working as part of a CI/CD pipeline with the results output to Azure DevOps.</p>
<p>So let’s take a look at that!</p>
<p>Since my last delve into Terrascan, it has in fact been updated to 1.3.1 too, so I’ll go ahead and use that. As an aside, it looks like they’ve now added another output format called “Human” too which is now the default. On the downside is that it still looks like the XML output isn’t in a format that Azure DevOps agrees with so for now, I’m going to be content with the fact the task will still fail when issues are found.</p>
<p>For this post, I’m going to test using my Terraform from my post on <a href="https://lgulliver.github.io/deploy-storage-account-static-site-terraform-azure-devops/">setting up an Azure Static Web Site with Terraform</a>.</p>
<p>Taking the pipeline YAML from my prior post, I’m going to add in a validation stage at the start of the pipeline. This is where I’m going to do all my compliance checks before I do anything else.</p>
<p>These are the changes I’ve made at this point to add in a stage for validation:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">stages</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">stage</span><span class="pi">:</span> <span class="s">validate</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">job</span><span class="pi">:</span> <span class="s">Compliance</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Run</span><span class="nv"> </span><span class="s">Terrascan</span><span class="nv"> </span><span class="s">to</span><span class="nv"> </span><span class="s">check</span><span class="nv"> </span><span class="s">for</span><span class="nv"> </span><span class="s">compliance'</span>
<span class="na">pool</span><span class="pi">:</span>
<span class="na">vmImage</span><span class="pi">:</span> <span class="s1">'</span><span class="s">ubuntu-latest'</span>
<span class="pi">-</span> <span class="na">stage</span><span class="pi">:</span> <span class="s">dev</span>
<span class="na">dependsOn</span><span class="pi">:</span> <span class="s">validate</span>
</code></pre></div></div>
<p>Unfortunately, Terrascan doesn’t currently have any marketplace extensions to add it to your CI/CD pipeline in Azure DevOps, but the great thing about Azure DevOps is you can practically install any tool you can think of to an agent. Even a Microsoft hosted one!</p>
<p>The way I’m going to do this here is by using the <code class="language-plaintext highlighter-rouge">script</code> task to pull down a specific version and install it.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">script</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">curl --location https://github.com/accurics/terrascan/releases/download/v1.3.1/terrascan_1.3.1_Linux_x86_64.tar.gz --output terrascan.tar.gz</span>
<span class="s">tar -xvf terrascan.tar.gz</span>
<span class="s">sudo install terrascan /usr/local/bin </span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Get</span><span class="nv"> </span><span class="s">tools'</span>
</code></pre></div></div>
<p>In the above snippet, I’m getting version 1.3.1 of Terrascan from GitHub, extracting it and installing it to the agent. As I’m using the Microsoft hosted agents, this step will need to be run every build.</p>
<p>I also want to run Terrascan as soon as it has installed. Doing that will require another <code class="language-plaintext highlighter-rouge">script</code> task and I’m going to make sure that it will be run from the directory my Terraform sits in.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">script</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">terrascan scan -t azure -i terraform</span>
<span class="na">workingDirectory</span><span class="pi">:</span> <span class="s">$(System.DefaultWorkingDirectory)/infrastructure/storage-account</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Run</span><span class="nv"> </span><span class="s">terrascan'</span>
</code></pre></div></div>
<p>In this snippet, I’m specifying that I’m going to use the Azure provider and the Terraform ruleset.</p>
<p>Bringing it all together looks a little something like this:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">trigger</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">master</span>
<span class="na">variables</span><span class="pi">:</span>
<span class="na">resource_group_tfstate</span><span class="pi">:</span> <span class="s1">'</span><span class="s">tfstate-uks-rg'</span>
<span class="na">product</span><span class="pi">:</span> <span class="s1">'</span><span class="s">staticsite'</span>
<span class="na">shortcode</span><span class="pi">:</span> <span class="s1">'</span><span class="s">lg'</span>
<span class="na">stages</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">stage</span><span class="pi">:</span> <span class="s">validate</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">job</span><span class="pi">:</span> <span class="s">Compliance</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Run</span><span class="nv"> </span><span class="s">Terrascan</span><span class="nv"> </span><span class="s">to</span><span class="nv"> </span><span class="s">check</span><span class="nv"> </span><span class="s">for</span><span class="nv"> </span><span class="s">compliance'</span>
<span class="na">pool</span><span class="pi">:</span>
<span class="na">vmImage</span><span class="pi">:</span> <span class="s1">'</span><span class="s">ubuntu-latest'</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">script</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">curl --location https://github.com/accurics/terrascan/releases/download/v1.3.1/terrascan_1.3.1_Linux_x86_64.tar.gz --output terrascan.tar.gz</span>
<span class="s">tar -xvf terrascan.tar.gz</span>
<span class="s">sudo install terrascan /usr/local/bin </span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Get</span><span class="nv"> </span><span class="s">tools'</span>
<span class="pi">-</span> <span class="na">script</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">terrascan scan -t azure -i terraform</span>
<span class="na">workingDirectory</span><span class="pi">:</span> <span class="s">$(System.DefaultWorkingDirectory)/infrastructure/storage-account</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Run</span><span class="nv"> </span><span class="s">terrascan'</span>
<span class="pi">-</span> <span class="na">stage</span><span class="pi">:</span> <span class="s">dev</span>
<span class="na">dependsOn</span><span class="pi">:</span> <span class="s">validate</span>
<span class="na">variables</span><span class="pi">:</span>
<span class="na">location</span><span class="pi">:</span> <span class="s1">'</span><span class="s">uksouth'</span>
<span class="na">environment_name</span><span class="pi">:</span> <span class="s1">'</span><span class="s">dev'</span>
<span class="na">location_short_code</span><span class="pi">:</span> <span class="s1">'</span><span class="s">uks'</span>
<span class="na">backendAzureRmContainerName</span><span class="pi">:</span> <span class="s">tfstate</span>
<span class="na">backendAzureRmKey</span><span class="pi">:</span> <span class="s">tfdev</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">job</span><span class="pi">:</span> <span class="s">Infrastructure</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Infrastructure'</span>
<span class="na">pool</span><span class="pi">:</span>
<span class="na">vmImage</span><span class="pi">:</span> <span class="s1">'</span><span class="s">ubuntu-latest'</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">AzureResourceManagerTemplateDeployment@3</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">ARM</span><span class="nv"> </span><span class="s">Template</span><span class="nv"> </span><span class="s">deployment:</span><span class="nv"> </span><span class="s">Resource</span><span class="nv"> </span><span class="s">Group</span><span class="nv"> </span><span class="s">scope'</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">deploymentScope</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Resource</span><span class="nv"> </span><span class="s">Group'</span>
<span class="na">azureResourceManagerConnection</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(armConnection)'</span>
<span class="na">subscriptionId</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(subscription_id)'</span>
<span class="na">action</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Create</span><span class="nv"> </span><span class="s">Or</span><span class="nv"> </span><span class="s">Update</span><span class="nv"> </span><span class="s">Resource</span><span class="nv"> </span><span class="s">Group'</span>
<span class="na">resourceGroupName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(resource_group_tfstate)'</span>
<span class="na">location</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(location)'</span>
<span class="na">templateLocation</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Linked</span><span class="nv"> </span><span class="s">artifact'</span>
<span class="na">csmFile</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(System.DefaultWorkingDirectory)/infrastructure/backend/tfbackend.deploy.json'</span>
<span class="na">deploymentMode</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Incremental'</span>
<span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">ARM Outputs@6</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">ConnectedServiceNameSelector</span><span class="pi">:</span> <span class="s1">'</span><span class="s">ConnectedServiceNameARM'</span>
<span class="na">ConnectedServiceNameARM</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(armConnection)'</span>
<span class="na">resourceGroupName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(resource_group_tfstate)'</span>
<span class="na">whenLastDeploymentIsFailed</span><span class="pi">:</span> <span class="s1">'</span><span class="s">fail'</span>
<span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">qetza.replacetokens.replacetokens-task.replacetokens@3</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Replace</span><span class="nv"> </span><span class="s">tokens</span><span class="nv"> </span><span class="s">in</span><span class="nv"> </span><span class="s">**/*.tfvars'</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">rootDirectory</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(System.DefaultWorkingDirectory)/infrastructure/storage-account'</span>
<span class="na">targetFiles</span><span class="pi">:</span> <span class="s1">'</span><span class="s">**/*.tfvars'</span>
<span class="na">escapeType</span><span class="pi">:</span> <span class="s">none</span>
<span class="na">tokenPrefix</span><span class="pi">:</span> <span class="s1">'</span><span class="s">__'</span>
<span class="na">tokenSuffix</span><span class="pi">:</span> <span class="s1">'</span><span class="s">__'</span>
<span class="na">enableTelemetry</span><span class="pi">:</span> <span class="no">false</span>
<span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">TerraformInstaller@0</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Install</span><span class="nv"> </span><span class="s">Terraform</span><span class="nv"> </span><span class="s">0.12.29'</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">terraformVersion</span><span class="pi">:</span> <span class="s">0.12.29</span>
<span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">TerraformTaskV1@0</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Terraform</span><span class="nv"> </span><span class="s">init'</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">provider</span><span class="pi">:</span> <span class="s1">'</span><span class="s">azurerm'</span>
<span class="na">command</span><span class="pi">:</span> <span class="s1">'</span><span class="s">init'</span>
<span class="na">workingDirectory</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(System.DefaultWorkingDirectory)/infrastructure/storage-account'</span>
<span class="na">commandOptions</span><span class="pi">:</span> <span class="s1">'</span><span class="s">-backend-config=$(System.DefaultWorkingDirectory)/infrastructure/storage-account/az-storage-account-variables.tfvars'</span>
<span class="na">backendServiceArm</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(armConnection)'</span>
<span class="na">backendAzureRmResourceGroupName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(resource_group_tfstate)'</span>
<span class="na">backendAzureRmStorageAccountName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(storageAccountName)'</span>
<span class="na">backendAzureRmContainerName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(backendAzureRmContainerName)'</span>
<span class="na">backendAzureRmKey</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(backendAzureRmKey)'</span>
<span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">TerraformTaskV1@0</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Terraform</span><span class="nv"> </span><span class="s">plan'</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">provider</span><span class="pi">:</span> <span class="s1">'</span><span class="s">azurerm'</span>
<span class="na">command</span><span class="pi">:</span> <span class="s1">'</span><span class="s">plan'</span>
<span class="na">workingDirectory</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(System.DefaultWorkingDirectory)/infrastructure/storage-account'</span>
<span class="na">commandOptions</span><span class="pi">:</span> <span class="s1">'</span><span class="s">-var-file="$(System.DefaultWorkingDirectory)/infrastructure/storage-account/az-storage-account-variables.tfvars"</span><span class="nv"> </span><span class="s">--out=planfile'</span>
<span class="na">environmentServiceNameAzureRM</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(armConnection)'</span>
<span class="na">backendServiceArm</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(armConnection)'</span>
<span class="na">backendAzureRmResourceGroupName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(resource_group_tfstate)'</span>
<span class="na">backendAzureRmStorageAccountName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(storageAccountName)'</span>
<span class="na">backendAzureRmContainerName</span><span class="pi">:</span> <span class="s">$(backendAzureRmContainerName)</span>
<span class="na">backendAzureRmKey</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(backendAzureRmKey)'</span>
<span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">TerraformTaskV1@0</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Terraform</span><span class="nv"> </span><span class="s">apply'</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">provider</span><span class="pi">:</span> <span class="s1">'</span><span class="s">azurerm'</span>
<span class="na">command</span><span class="pi">:</span> <span class="s1">'</span><span class="s">apply'</span>
<span class="na">workingDirectory</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(System.DefaultWorkingDirectory)/infrastructure/storage-account'</span>
<span class="na">commandOptions</span><span class="pi">:</span> <span class="s1">'</span><span class="s">-auto-approve</span><span class="nv"> </span><span class="s">planfile'</span>
<span class="na">environmentServiceNameAzureRM</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(armConnection)'</span>
<span class="na">backendServiceArm</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(armConnection)'</span>
<span class="na">backendAzureRmResourceGroupName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(resource_group_tfstate)'</span>
<span class="na">backendAzureRmStorageAccountName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(storageAccountName)'</span>
<span class="na">backendAzureRmContainerName</span><span class="pi">:</span> <span class="s">$(backendAzureRmContainerName)</span>
<span class="na">backendAzureRmKey</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(backendAzureRmKey)'</span>
<span class="pi">-</span> <span class="na">job</span><span class="pi">:</span> <span class="s">Deploy</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Deploy'</span>
<span class="na">pool</span><span class="pi">:</span>
<span class="na">vmImage</span><span class="pi">:</span> <span class="s1">'</span><span class="s">windows-latest'</span>
<span class="na">dependsOn</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Infrastructure'</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">AzureFileCopy@3</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">SourcePath</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(System.DefaultWorkingDirectory)/code'</span>
<span class="na">azureSubscription</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(armConnection)'</span>
<span class="na">Destination</span><span class="pi">:</span> <span class="s1">'</span><span class="s">AzureBlob'</span>
<span class="na">storage</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(shortcode)$(product)$(environment_name)$(location_short_code)stor'</span>
<span class="na">ContainerName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$web'</span>
<span class="pi">-</span> <span class="na">stage</span><span class="pi">:</span> <span class="s">prod</span>
<span class="na">dependsOn</span><span class="pi">:</span> <span class="s">dev</span>
<span class="na">variables</span><span class="pi">:</span>
<span class="na">location</span><span class="pi">:</span> <span class="s1">'</span><span class="s">uksouth'</span>
<span class="na">environment_name</span><span class="pi">:</span> <span class="s1">'</span><span class="s">prod'</span>
<span class="na">location_short_code</span><span class="pi">:</span> <span class="s1">'</span><span class="s">uks'</span>
<span class="na">backendAzureRmContainerName</span><span class="pi">:</span> <span class="s">tfstate</span>
<span class="na">backendAzureRmKey</span><span class="pi">:</span> <span class="s">tfprod</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">job</span><span class="pi">:</span> <span class="s">Infrastructure</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Infrastructure'</span>
<span class="na">pool</span><span class="pi">:</span>
<span class="na">vmImage</span><span class="pi">:</span> <span class="s1">'</span><span class="s">ubuntu-latest'</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">AzureResourceManagerTemplateDeployment@3</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">ARM</span><span class="nv"> </span><span class="s">Template</span><span class="nv"> </span><span class="s">deployment:</span><span class="nv"> </span><span class="s">Resource</span><span class="nv"> </span><span class="s">Group</span><span class="nv"> </span><span class="s">scope'</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">deploymentScope</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Resource</span><span class="nv"> </span><span class="s">Group'</span>
<span class="na">azureResourceManagerConnection</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(armConnection)'</span>
<span class="na">subscriptionId</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(subscription_id)'</span>
<span class="na">action</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Create</span><span class="nv"> </span><span class="s">Or</span><span class="nv"> </span><span class="s">Update</span><span class="nv"> </span><span class="s">Resource</span><span class="nv"> </span><span class="s">Group'</span>
<span class="na">resourceGroupName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(resource_group_tfstate)'</span>
<span class="na">location</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(location)'</span>
<span class="na">templateLocation</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Linked</span><span class="nv"> </span><span class="s">artifact'</span>
<span class="na">csmFile</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(System.DefaultWorkingDirectory)/infrastructure/backend/tfbackend.deploy.json'</span>
<span class="na">deploymentMode</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Incremental'</span>
<span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">ARM Outputs@6</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">ConnectedServiceNameSelector</span><span class="pi">:</span> <span class="s1">'</span><span class="s">ConnectedServiceNameARM'</span>
<span class="na">ConnectedServiceNameARM</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(armConnection)'</span>
<span class="na">resourceGroupName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(resource_group_tfstate)'</span>
<span class="na">whenLastDeploymentIsFailed</span><span class="pi">:</span> <span class="s1">'</span><span class="s">fail'</span>
<span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">qetza.replacetokens.replacetokens-task.replacetokens@3</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Replace</span><span class="nv"> </span><span class="s">tokens</span><span class="nv"> </span><span class="s">in</span><span class="nv"> </span><span class="s">**/*.tfvars'</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">rootDirectory</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(System.DefaultWorkingDirectory)/infrastructure/storage-account'</span>
<span class="na">targetFiles</span><span class="pi">:</span> <span class="s1">'</span><span class="s">**/*.tfvars'</span>
<span class="na">escapeType</span><span class="pi">:</span> <span class="s">none</span>
<span class="na">tokenPrefix</span><span class="pi">:</span> <span class="s1">'</span><span class="s">__'</span>
<span class="na">tokenSuffix</span><span class="pi">:</span> <span class="s1">'</span><span class="s">__'</span>
<span class="na">enableTelemetry</span><span class="pi">:</span> <span class="no">false</span>
<span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">TerraformInstaller@0</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Install</span><span class="nv"> </span><span class="s">Terraform</span><span class="nv"> </span><span class="s">0.12.29'</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">terraformVersion</span><span class="pi">:</span> <span class="s">0.12.29</span>
<span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">TerraformTaskV1@0</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Terraform</span><span class="nv"> </span><span class="s">init'</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">provider</span><span class="pi">:</span> <span class="s1">'</span><span class="s">azurerm'</span>
<span class="na">command</span><span class="pi">:</span> <span class="s1">'</span><span class="s">init'</span>
<span class="na">workingDirectory</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(System.DefaultWorkingDirectory)/infrastructure/storage-account'</span>
<span class="na">commandOptions</span><span class="pi">:</span> <span class="s1">'</span><span class="s">-backend-config=$(System.DefaultWorkingDirectory)/infrastructure/storage-account/az-storage-account-variables.tfvars'</span>
<span class="na">backendServiceArm</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(armConnection)'</span>
<span class="na">backendAzureRmResourceGroupName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(resource_group_tfstate)'</span>
<span class="na">backendAzureRmStorageAccountName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(storageAccountName)'</span>
<span class="na">backendAzureRmContainerName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(backendAzureRmContainerName)'</span>
<span class="na">backendAzureRmKey</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(backendAzureRmKey)'</span>
<span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">TerraformTaskV1@0</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Terraform</span><span class="nv"> </span><span class="s">plan'</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">provider</span><span class="pi">:</span> <span class="s1">'</span><span class="s">azurerm'</span>
<span class="na">command</span><span class="pi">:</span> <span class="s1">'</span><span class="s">plan'</span>
<span class="na">workingDirectory</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(System.DefaultWorkingDirectory)/infrastructure/storage-account'</span>
<span class="na">commandOptions</span><span class="pi">:</span> <span class="s1">'</span><span class="s">-var-file="$(System.DefaultWorkingDirectory)/infrastructure/storage-account/az-storage-account-variables.tfvars"</span><span class="nv"> </span><span class="s">--out=planfile'</span>
<span class="na">environmentServiceNameAzureRM</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(armConnection)'</span>
<span class="na">backendServiceArm</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(armConnection)'</span>
<span class="na">backendAzureRmResourceGroupName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(resource_group_tfstate)'</span>
<span class="na">backendAzureRmStorageAccountName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(storageAccountName)'</span>
<span class="na">backendAzureRmContainerName</span><span class="pi">:</span> <span class="s">$(backendAzureRmContainerName)</span>
<span class="na">backendAzureRmKey</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(backendAzureRmKey)'</span>
<span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">TerraformTaskV1@0</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Terraform</span><span class="nv"> </span><span class="s">apply'</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">provider</span><span class="pi">:</span> <span class="s1">'</span><span class="s">azurerm'</span>
<span class="na">command</span><span class="pi">:</span> <span class="s1">'</span><span class="s">apply'</span>
<span class="na">workingDirectory</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(System.DefaultWorkingDirectory)/infrastructure/storage-account'</span>
<span class="na">commandOptions</span><span class="pi">:</span> <span class="s1">'</span><span class="s">-auto-approve</span><span class="nv"> </span><span class="s">planfile'</span>
<span class="na">environmentServiceNameAzureRM</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(armConnection)'</span>
<span class="na">backendServiceArm</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(armConnection)'</span>
<span class="na">backendAzureRmResourceGroupName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(resource_group_tfstate)'</span>
<span class="na">backendAzureRmStorageAccountName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(storageAccountName)'</span>
<span class="na">backendAzureRmContainerName</span><span class="pi">:</span> <span class="s">$(backendAzureRmContainerName)</span>
<span class="na">backendAzureRmKey</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(backendAzureRmKey)'</span>
<span class="pi">-</span> <span class="na">job</span><span class="pi">:</span> <span class="s">Deploy</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Deploy'</span>
<span class="na">pool</span><span class="pi">:</span>
<span class="na">vmImage</span><span class="pi">:</span> <span class="s1">'</span><span class="s">windows-latest'</span>
<span class="na">dependsOn</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Infrastructure'</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">AzureFileCopy@3</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">SourcePath</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(System.DefaultWorkingDirectory)/code'</span>
<span class="na">azureSubscription</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(armConnection)'</span>
<span class="na">Destination</span><span class="pi">:</span> <span class="s1">'</span><span class="s">AzureBlob'</span>
<span class="na">storage</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$(shortcode)$(product)$(environment_name)$(location_short_code)stor'</span>
<span class="na">ContainerName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">$web'</span>
</code></pre></div></div>
<p>Now I have my validation step before any deployment begins. If it passes, it will deploy to my dev environment. If it fails, no deployment will happen.</p>
<p>I can see in the pipeline that currently, I have a failure with my scan.</p>
<figure class="">
<img src="/assets/images/posts/terrascan-pipeline.png" alt="Pipeline view" />
</figure>
<p>Digging into the failed task, as expected, I have a policy violation:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Violation Details -
Description : Ensure that Azure Resource Group has resource lock enabled
File : az-storage-account-main.tf
Line : 10
Severity : LOW
</code></pre></div></div>
<p>I could override this policy, but for the purpose of this post, I’m going to actually go ahead and fix the violation.</p>
<p>To fix this particular violation, all I need to do is add a resource lock to the resource group which I’m going to add to my <code class="language-plaintext highlighter-rouge">az-storage-account-main.tf</code>.</p>
<div class="language-hcl highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">resource</span> <span class="s2">"azurerm_management_lock"</span> <span class="s2">"rg"</span> <span class="p">{</span>
<span class="nx">name</span> <span class="p">=</span> <span class="s2">"rg-lock"</span>
<span class="nx">scope</span> <span class="p">=</span> <span class="nx">azurerm_resource_group</span><span class="err">.</span><span class="nx">rg</span><span class="err">.</span><span class="nx">id</span>
<span class="nx">lock_level</span> <span class="p">=</span> <span class="s2">"CanNotDelete"</span>
<span class="nx">notes</span> <span class="p">=</span> <span class="s2">"Locked for compliance"</span>
<span class="p">}</span>
</code></pre></div></div>
<p>A quick commit later and we’re away!</p>
<figure class="">
<img src="/assets/images/posts/terrascan-pipeline-success.png" alt="Pipeline view" />
</figure>Liam GulliverIn my last post, I took a look at a new scanning tool called Terrascan. It can be used to ensure your Kubernetes manifests, Terraform and more are compliant with a set of built-in, or customised rules.Scanning Terraform, Kubernetes and More for Policy Compliance with Terrascan2021-01-11T14:30:00+00:002021-01-11T14:30:00+00:00https://lgulliver.github.io/scan-terraform-kubernetes-and-more-with-terrascan<p>I was recently introduced a new security and compliance scanning tool called <a href="https://github.com/accurics/terrascan">Terrascan</a>. It’s another free and open source tool, just like another tool I’ve covered previously in this space called <a href="https://lgulliver.github.io/container-security-scanning-with-trivy-in-azure-devops/">Trivy</a>.</p>
<p>From the brief look I’ve had into Terrascan (a deeper dive to come!), it allows us to automate the compliance and security scans against a <a href="https://docs.accurics.com/projects/accurics-terrascan/en/latest/policies/">pre-defined set of policies</a> or custom policies as part of the CI process.</p>
<p>It has support for Terraform, Azure, GCP, AWS, Kubernetes (manifests, Helm, Kustomize), though as it doesn’t seem to have support for Dockerfiles, it’s a tool to be used alongside something like <a href="https://lgulliver.github.io/container-security-scanning-with-trivy-in-azure-devops/">Trivy</a>.</p>
<p>For my quick test, I installed Terrascan in my Ubuntu 20.04 WSL image on my machine (though you can also <a href="https://docs.accurics.com/projects/accurics-terrascan/en/latest/getting-started/usage/#using-docker">use it as a Docker image</a>) and pulled down a repo with a really basic Kubernetes application in, which I know also has some intentional mistakes/omissions in.</p>
<p>Installation:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>curl <span class="nt">--location</span> https://github.com/accurics/terrascan/releases/download/v1.2.0/terrascan_1.2.0_Linux_x86_64.tar.gz <span class="nt">--output</span> terrascan.tar.gz
<span class="nv">$ </span><span class="nb">tar</span> <span class="nt">-xvf</span> terrascan.tar.gz
x CHANGELOG.md
x LICENSE
x README.md
x terrascan
<span class="nv">$ </span><span class="nb">install </span>terrascan /usr/local/bin
</code></pre></div></div>
<p>To confirm Terrascan is installed, simply run the command <code class="language-plaintext highlighter-rouge">terrascan</code> in the terminal. You should receive something like the below back:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>terrascan
Terrascan
Detect compliance and security violations across Infrastructure as Code to mitigate risk before provisioning cloud native infrastructure.
For more information, please visit https://docs.accurics.com
Usage:
terrascan <span class="o">[</span><span class="nb">command</span><span class="o">]</span>
Available Commands:
<span class="nb">help </span>Help about any <span class="nb">command
</span>init Initialize Terrascan
scan Detect compliance and security violations across Infrastructure as Code.
server Run Terrascan as an API server
version Terrascan version
Flags:
<span class="nt">-c</span>, <span class="nt">--config-path</span> string config file path
<span class="nt">-h</span>, <span class="nt">--help</span> <span class="nb">help </span><span class="k">for </span>terrascan
<span class="nt">-l</span>, <span class="nt">--log-level</span> string log level <span class="o">(</span>debug, info, warn, error, panic, fatal<span class="o">)</span> <span class="o">(</span>default <span class="s2">"info"</span><span class="o">)</span>
<span class="nt">-x</span>, <span class="nt">--log-type</span> string log output <span class="nb">type</span> <span class="o">(</span>console, json<span class="o">)</span> <span class="o">(</span>default <span class="s2">"console"</span><span class="o">)</span>
<span class="nt">-o</span>, <span class="nt">--output</span> string output <span class="nb">type</span> <span class="o">(</span>json, yaml, xml<span class="o">)</span> <span class="o">(</span>default <span class="s2">"yaml"</span><span class="o">)</span>
</code></pre></div></div>
<p>Then to scan my Kubernetes manifests, all I need to do is navigate to the directory they’re sitting in and run the following command to tell Terrascan I’m scanning Kubernetes manifests (default is Terraform):</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>terrascan scan <span class="nt">-i</span> k8s
</code></pre></div></div>
<p>By default, the output is YAML and out to screen rather than to a file. In any case, the scan is incredibly quick (my initial test ran in less than a second) and produced the following:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">results</span><span class="pi">:</span>
<span class="na">violations</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">rule_name</span><span class="pi">:</span> <span class="s">containerResourcesNotDefined</span>
<span class="na">description</span><span class="pi">:</span> <span class="s">Container does not have resource limitations defined</span>
<span class="na">rule_id</span><span class="pi">:</span> <span class="s">accurics.kubernetes.IAM.107</span>
<span class="na">severity</span><span class="pi">:</span> <span class="s">MEDIUM</span>
<span class="na">category</span><span class="pi">:</span> <span class="s">Identity and Access Management</span>
<span class="na">resource_name</span><span class="pi">:</span> <span class="s">carsunlimited-cart-deployment</span>
<span class="na">resource_type</span><span class="pi">:</span> <span class="s">kubernetes_deployment</span>
<span class="na">file</span><span class="pi">:</span> <span class="s">/mnt/c/Checkout/Internal/CarsUnlimited-Kubernetes/02-carsunlimited/cart-deployment.yaml</span>
<span class="na">line</span><span class="pi">:</span> <span class="m">1</span>
<span class="pi">-</span> <span class="na">rule_name</span><span class="pi">:</span> <span class="s">containerResourcesNotDefined</span>
<span class="na">description</span><span class="pi">:</span> <span class="s">Container does not have resource limitations defined</span>
<span class="na">rule_id</span><span class="pi">:</span> <span class="s">accurics.kubernetes.IAM.107</span>
<span class="na">severity</span><span class="pi">:</span> <span class="s">MEDIUM</span>
<span class="na">category</span><span class="pi">:</span> <span class="s">Identity and Access Management</span>
<span class="na">resource_name</span><span class="pi">:</span> <span class="s">carsunlimited-inventory-deployment</span>
<span class="na">resource_type</span><span class="pi">:</span> <span class="s">kubernetes_deployment</span>
<span class="na">file</span><span class="pi">:</span> <span class="s">/mnt/c/Checkout/Internal/CarsUnlimited-Kubernetes/02-carsunlimited/inventory-deployment.yaml</span>
<span class="na">line</span><span class="pi">:</span> <span class="m">1</span>
<span class="pi">-</span> <span class="na">rule_name</span><span class="pi">:</span> <span class="s">containerResourcesNotDefined</span>
<span class="na">description</span><span class="pi">:</span> <span class="s">Container does not have resource limitations defined</span>
<span class="na">rule_id</span><span class="pi">:</span> <span class="s">accurics.kubernetes.IAM.107</span>
<span class="na">severity</span><span class="pi">:</span> <span class="s">MEDIUM</span>
<span class="na">category</span><span class="pi">:</span> <span class="s">Identity and Access Management</span>
<span class="na">resource_name</span><span class="pi">:</span> <span class="s">carsunlimited-purchase-deployment</span>
<span class="na">resource_type</span><span class="pi">:</span> <span class="s">kubernetes_deployment</span>
<span class="na">file</span><span class="pi">:</span> <span class="s">/mnt/c/Checkout/Internal/CarsUnlimited-Kubernetes/02-carsunlimited/purchase-deployment.yaml</span>
<span class="na">line</span><span class="pi">:</span> <span class="m">1</span>
<span class="pi">-</span> <span class="na">rule_name</span><span class="pi">:</span> <span class="s">containerResourcesNotDefined</span>
<span class="na">description</span><span class="pi">:</span> <span class="s">Container does not have resource limitations defined</span>
<span class="na">rule_id</span><span class="pi">:</span> <span class="s">accurics.kubernetes.IAM.107</span>
<span class="na">severity</span><span class="pi">:</span> <span class="s">MEDIUM</span>
<span class="na">category</span><span class="pi">:</span> <span class="s">Identity and Access Management</span>
<span class="na">resource_name</span><span class="pi">:</span> <span class="s">carsunlimited-web-deployment</span>
<span class="na">resource_type</span><span class="pi">:</span> <span class="s">kubernetes_deployment</span>
<span class="na">file</span><span class="pi">:</span> <span class="s">/mnt/c/Checkout/Internal/CarsUnlimited-Kubernetes/02-carsunlimited/web-deployment.yaml</span>
<span class="na">line</span><span class="pi">:</span> <span class="m">1</span>
<span class="pi">-</span> <span class="na">rule_name</span><span class="pi">:</span> <span class="s">defaultNamespaceUsed2</span>
<span class="na">description</span><span class="pi">:</span> <span class="s">The default namespace should not be used</span>
<span class="na">rule_id</span><span class="pi">:</span> <span class="s">accurics.kubernetes.OPS.461</span>
<span class="na">severity</span><span class="pi">:</span> <span class="s">LOW</span>
<span class="na">category</span><span class="pi">:</span> <span class="s">Operational Efficiency</span>
<span class="na">resource_name</span><span class="pi">:</span> <span class="s">carsunlimited-cart-deployment</span>
<span class="na">resource_type</span><span class="pi">:</span> <span class="s">kubernetes_deployment</span>
<span class="na">file</span><span class="pi">:</span> <span class="s">/mnt/c/Checkout/Internal/CarsUnlimited-Kubernetes/02-carsunlimited/cart-deployment.yaml</span>
<span class="na">line</span><span class="pi">:</span> <span class="m">1</span>
<span class="pi">-</span> <span class="na">rule_name</span><span class="pi">:</span> <span class="s">defaultNamespaceUsed2</span>
<span class="na">description</span><span class="pi">:</span> <span class="s">The default namespace should not be used</span>
<span class="na">rule_id</span><span class="pi">:</span> <span class="s">accurics.kubernetes.OPS.461</span>
<span class="na">severity</span><span class="pi">:</span> <span class="s">LOW</span>
<span class="na">category</span><span class="pi">:</span> <span class="s">Operational Efficiency</span>
<span class="na">resource_name</span><span class="pi">:</span> <span class="s">carsunlimited-inventory-deployment</span>
<span class="na">resource_type</span><span class="pi">:</span> <span class="s">kubernetes_deployment</span>
<span class="na">file</span><span class="pi">:</span> <span class="s">/mnt/c/Checkout/Internal/CarsUnlimited-Kubernetes/02-carsunlimited/inventory-deployment.yaml</span>
<span class="na">line</span><span class="pi">:</span> <span class="m">1</span>
<span class="pi">-</span> <span class="na">rule_name</span><span class="pi">:</span> <span class="s">defaultNamespaceUsed2</span>
<span class="na">description</span><span class="pi">:</span> <span class="s">The default namespace should not be used</span>
<span class="na">rule_id</span><span class="pi">:</span> <span class="s">accurics.kubernetes.OPS.461</span>
<span class="na">severity</span><span class="pi">:</span> <span class="s">LOW</span>
<span class="na">category</span><span class="pi">:</span> <span class="s">Operational Efficiency</span>
<span class="na">resource_name</span><span class="pi">:</span> <span class="s">carsunlimited-purchase-deployment</span>
<span class="na">resource_type</span><span class="pi">:</span> <span class="s">kubernetes_deployment</span>
<span class="na">file</span><span class="pi">:</span> <span class="s">/mnt/c/Checkout/Internal/CarsUnlimited-Kubernetes/02-carsunlimited/purchase-deployment.yaml</span>
<span class="na">line</span><span class="pi">:</span> <span class="m">1</span>
<span class="pi">-</span> <span class="na">rule_name</span><span class="pi">:</span> <span class="s">defaultNamespaceUsed2</span>
<span class="na">description</span><span class="pi">:</span> <span class="s">The default namespace should not be used</span>
<span class="na">rule_id</span><span class="pi">:</span> <span class="s">accurics.kubernetes.OPS.461</span>
<span class="na">severity</span><span class="pi">:</span> <span class="s">LOW</span>
<span class="na">category</span><span class="pi">:</span> <span class="s">Operational Efficiency</span>
<span class="na">resource_name</span><span class="pi">:</span> <span class="s">carsunlimited-web-deployment</span>
<span class="na">resource_type</span><span class="pi">:</span> <span class="s">kubernetes_deployment</span>
<span class="na">file</span><span class="pi">:</span> <span class="s">/mnt/c/Checkout/Internal/CarsUnlimited-Kubernetes/02-carsunlimited/web-deployment.yaml</span>
<span class="na">line</span><span class="pi">:</span> <span class="m">1</span>
<span class="na">count</span><span class="pi">:</span>
<span class="na">low</span><span class="pi">:</span> <span class="m">4</span>
<span class="na">medium</span><span class="pi">:</span> <span class="m">4</span>
<span class="na">high</span><span class="pi">:</span> <span class="m">0</span>
<span class="na">total</span><span class="pi">:</span> <span class="m">8</span>
</code></pre></div></div>
<p>The really cool thing here is that the policies seem to also be categorised against the <a href="https://docs.microsoft.com/en-us/azure/architecture/framework/">Well-Architected Framework</a>.</p>
<p>The output can also be provided as JSON or XML - you may recall if you read the <a href="https://lgulliver.github.io/trivy-scan-results-to-azure-devops/">Trivy posts I wrote last year that Azure DevOps needs the output in XML</a>.</p>
<p>To output it to XML, you need to append the <code class="language-plaintext highlighter-rouge">-o</code> or <code class="language-plaintext highlighter-rouge">--output</code> option with the value <code class="language-plaintext highlighter-rouge">xml</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>terrascan scan <span class="nt">-i</span> k8s <span class="nt">-o</span> xml
</code></pre></div></div>
<p>This will give you XML output <del>that should be compatible with the JUnit XML format</del>:</p>
<p><strong>UPDATE: No it isn’t compatible with JUnit/XUnit/NUnit or any other format supported by Azure DevOps</strong></p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><results></span>
<span class="nt"><violations></span>
<span class="nt"><violation</span> <span class="na">rule_name=</span><span class="s">"defaultNamespaceUsed2"</span> <span class="na">description=</span><span class="s">"The default namespace should not be used"</span> <span class="na">rule_id=</span><span class="s">"accurics.kubernetes.OPS.461"</span> <span class="na">severity=</span><span class="s">"LOW"</span> <span class="na">category=</span><span class="s">"Operational Efficiency"</span> <span class="na">resource_name=</span><span class="s">"carsunlimited-cart-deployment"</span> <span class="na">resource_type=</span><span class="s">"kubernetes_deployment"</span> <span class="na">file=</span><span class="s">"/mnt/c/Checkout/Internal/CarsUnlimited-Kubernetes/02-carsunlimited/cart-deployment.yaml"</span> <span class="na">line=</span><span class="s">"1"</span><span class="nt">></violation></span>
<span class="nt"><violation</span> <span class="na">rule_name=</span><span class="s">"defaultNamespaceUsed2"</span> <span class="na">description=</span><span class="s">"The default namespace should not be used"</span> <span class="na">rule_id=</span><span class="s">"accurics.kubernetes.OPS.461"</span> <span class="na">severity=</span><span class="s">"LOW"</span> <span class="na">category=</span><span class="s">"Operational Efficiency"</span> <span class="na">resource_name=</span><span class="s">"carsunlimited-inventory-deployment"</span> <span class="na">resource_type=</span><span class="s">"kubernetes_deployment"</span> <span class="na">file=</span><span class="s">"/mnt/c/Checkout/Internal/CarsUnlimited-Kubernetes/02-carsunlimited/inventory-deployment.yaml"</span> <span class="na">line=</span><span class="s">"1"</span><span class="nt">></violation></span>
<span class="nt"><violation</span> <span class="na">rule_name=</span><span class="s">"defaultNamespaceUsed2"</span> <span class="na">description=</span><span class="s">"The default namespace should not be used"</span> <span class="na">rule_id=</span><span class="s">"accurics.kubernetes.OPS.461"</span> <span class="na">severity=</span><span class="s">"LOW"</span> <span class="na">category=</span><span class="s">"Operational Efficiency"</span> <span class="na">resource_name=</span><span class="s">"carsunlimited-purchase-deployment"</span> <span class="na">resource_type=</span><span class="s">"kubernetes_deployment"</span> <span class="na">file=</span><span class="s">"/mnt/c/Checkout/Internal/CarsUnlimited-Kubernetes/02-carsunlimited/purchase-deployment.yaml"</span> <span class="na">line=</span><span class="s">"1"</span><span class="nt">></violation></span>
<span class="nt"><violation</span> <span class="na">rule_name=</span><span class="s">"defaultNamespaceUsed2"</span> <span class="na">description=</span><span class="s">"The default namespace should not be used"</span> <span class="na">rule_id=</span><span class="s">"accurics.kubernetes.OPS.461"</span> <span class="na">severity=</span><span class="s">"LOW"</span> <span class="na">category=</span><span class="s">"Operational Efficiency"</span> <span class="na">resource_name=</span><span class="s">"carsunlimited-web-deployment"</span> <span class="na">resource_type=</span><span class="s">"kubernetes_deployment"</span> <span class="na">file=</span><span class="s">"/mnt/c/Checkout/Internal/CarsUnlimited-Kubernetes/02-carsunlimited/web-deployment.yaml"</span> <span class="na">line=</span><span class="s">"1"</span><span class="nt">></violation></span>
<span class="nt"><violation</span> <span class="na">rule_name=</span><span class="s">"containerResourcesNotDefined"</span> <span class="na">description=</span><span class="s">"Container does not have resource limitations defined"</span> <span class="na">rule_id=</span><span class="s">"accurics.kubernetes.IAM.107"</span> <span class="na">severity=</span><span class="s">"MEDIUM"</span> <span class="na">category=</span><span class="s">"Identity and Access Management"</span> <span class="na">resource_name=</span><span class="s">"carsunlimited-cart-deployment"</span> <span class="na">resource_type=</span><span class="s">"kubernetes_deployment"</span> <span class="na">file=</span><span class="s">"/mnt/c/Checkout/Internal/CarsUnlimited-Kubernetes/02-carsunlimited/cart-deployment.yaml"</span> <span class="na">line=</span><span class="s">"1"</span><span class="nt">></violation></span>
<span class="nt"><violation</span> <span class="na">rule_name=</span><span class="s">"containerResourcesNotDefined"</span> <span class="na">description=</span><span class="s">"Container does not have resource limitations defined"</span> <span class="na">rule_id=</span><span class="s">"accurics.kubernetes.IAM.107"</span> <span class="na">severity=</span><span class="s">"MEDIUM"</span> <span class="na">category=</span><span class="s">"Identity and Access Management"</span> <span class="na">resource_name=</span><span class="s">"carsunlimited-inventory-deployment"</span> <span class="na">resource_type=</span><span class="s">"kubernetes_deployment"</span> <span class="na">file=</span><span class="s">"/mnt/c/Checkout/Internal/CarsUnlimited-Kubernetes/02-carsunlimited/inventory-deployment.yaml"</span> <span class="na">line=</span><span class="s">"1"</span><span class="nt">></violation></span>
<span class="nt"><violation</span> <span class="na">rule_name=</span><span class="s">"containerResourcesNotDefined"</span> <span class="na">description=</span><span class="s">"Container does not have resource limitations defined"</span> <span class="na">rule_id=</span><span class="s">"accurics.kubernetes.IAM.107"</span> <span class="na">severity=</span><span class="s">"MEDIUM"</span> <span class="na">category=</span><span class="s">"Identity and Access Management"</span> <span class="na">resource_name=</span><span class="s">"carsunlimited-purchase-deployment"</span> <span class="na">resource_type=</span><span class="s">"kubernetes_deployment"</span> <span class="na">file=</span><span class="s">"/mnt/c/Checkout/Internal/CarsUnlimited-Kubernetes/02-carsunlimited/purchase-deployment.yaml"</span> <span class="na">line=</span><span class="s">"1"</span><span class="nt">></violation></span>
<span class="nt"><violation</span> <span class="na">rule_name=</span><span class="s">"containerResourcesNotDefined"</span> <span class="na">description=</span><span class="s">"Container does not have resource limitations defined"</span> <span class="na">rule_id=</span><span class="s">"accurics.kubernetes.IAM.107"</span> <span class="na">severity=</span><span class="s">"MEDIUM"</span> <span class="na">category=</span><span class="s">"Identity and Access Management"</span> <span class="na">resource_name=</span><span class="s">"carsunlimited-web-deployment"</span> <span class="na">resource_type=</span><span class="s">"kubernetes_deployment"</span> <span class="na">file=</span><span class="s">"/mnt/c/Checkout/Internal/CarsUnlimited-Kubernetes/02-carsunlimited/web-deployment.yaml"</span> <span class="na">line=</span><span class="s">"1"</span><span class="nt">></violation></span>
<span class="nt"></violations></span>
<span class="nt"><count</span> <span class="na">low=</span><span class="s">"4"</span> <span class="na">medium=</span><span class="s">"4"</span> <span class="na">high=</span><span class="s">"0"</span> <span class="na">total=</span><span class="s">"8"</span><span class="nt">></count></span>
<span class="nt"></results></span>
</code></pre></div></div>
<p>To save it as an XML file, all you need to do is append <code class="language-plaintext highlighter-rouge">> result.xml</code>. The complete command looks as follows:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>terrascan scan <span class="nt">-i</span> k8s <span class="nt">-o</span> xml <span class="o">></span> result.xml
</code></pre></div></div>
<p>In a follow up post, I’ll cover integrating this into the CI/CD pipeline in Azure DevOps so that you can fail builds on Terrascan failures.</p>Liam GulliverI was recently introduced a new security and compliance scanning tool called Terrascan. It’s another free and open source tool, just like another tool I’ve covered previously in this space called Trivy.Azureish Live! Building a Git Commit Watcher with Azure Functions and GitHub [Part 2]2020-11-10T17:30:00+00:002020-11-10T17:30:00+00:00https://lgulliver.github.io/azureish-live!-building-a-git-commit-watcher-with-azure-functions-and-github-part2<p>Jonathan, Pete and I continue our GitWatcher project. We pivot towards the new Static Web App preview as we look to build our UI with Blazor.</p>
<p>We had a couple of issues with what we were building this time around so we didn’t get as far as we’d planned!</p>
<p>Check out the repository over on <a href="https://github.com/AzureishLive/gitwatcher">GitHub</a>!</p>
<p>Don’t forget, you can follow us over on our <a href="https://twitch.tv/azureishlive">Twitch</a> and <a href="https://www.youtube.com/channel/UCVQtNIXAgtJA-w9pd17WH5A">YouTube</a> channels.</p>Liam GulliverJonathan, Pete and I continue our GitWatcher project. We pivot towards the new Static Web App preview as we look to build our UI with Blazor.Agile Engineering Podcast - The Theory of Constraints with Tom Hoyland2020-11-06T11:30:00+00:002020-11-06T11:30:00+00:00https://lgulliver.github.io/agile-engineering-podcast-theory-of-constraints<p>I’m really excited for this episode of the podcast. Friend, DevOps Notts alum and all round great guy, Tom Hoyland of Sky Betting and Gaming joins us on the podcast to dig into The Theory of Constraints.</p>
<p>The theory was coined by Eliyahu M. Goldratt in his book <a href="https://amzn.to/3i1PpST">The Goal</a>. We take a look at what the theory is and what a practical application of the theory might look like. Give it a listen below!</p>
<iframe src="https://open.spotify.com/embed-podcast/episode/3LOvq3FzrfwletH3vJsFE8" width="100%" height="232" frameborder="0" allowtransparency="true" allow="encrypted-media"></iframe>
<p>We’ve also recently launched a Patreon page on <a href="https://www.patreon.com/agileengineering">patreon.com/agileengineering</a>. Our goal is to expand the type of content we provide to our audience and help to provide guidance and insights from people who do this stuff every single day direct to you; whether you are mildly interested in the topics through to if you’re the most passionate CTO! We’re looking to provide more video content and a live show and would appreciate any support you can give to help us level up! Take a look at my post on <a href="https://lgulliver.github.io/launching-patreon-for-agile-engineering-podcast/">Patreon</a> for more information.</p>Liam GulliverI’m really excited for this episode of the podcast. Friend, DevOps Notts alum and all round great guy, Tom Hoyland of Sky Betting and Gaming joins us on the podcast to dig into The Theory of Constraints.Azureish After Dark with Derek Campbell2020-11-03T22:00:00+00:002020-11-03T22:00:00+00:00https://lgulliver.github.io/azureish-after-dark-with-derek-campbell<p>Not one to sit still, this week <a href="https://twitter.com/pete_codes">Pete Gallagher</a>, <a href="https://twitter.com/jbjon">Jonathan Relf</a> and I launched companion show to <a href="https://lgulliver.github.io/2020-10-28-azureish-live!-building-a-git-commit-watcher-with-azure-functions-and-github-part1/">Azureish Live!</a> called Azureish After Dark.</p>
<p>Every Wednesday at 8PM UK Time on <a href="https://twitch.tv/azureishlive">Twitch</a>, I sit down with members from the global tech community to talk technology, cloud and more before we sit down to play some video games! At the moment, our game of choice is <a href="https://store.steampowered.com/app/945360/Among_Us/">Among Us</a>.</p>
<p>The reason we’ve opted for a show like this is that it’s different and fun! There’s plenty of <a href="https://www.meetup.com/DevOps-Notts/">meetups</a>, webinars, <a href="https://open.spotify.com/show/7r3FceDwIN1X47c4xfyzTG?si=8lefGt41TaaDUI46FxjDgA">podcasts</a> and <a href="https://www.youtube.com/channel/UCVQtNIXAgtJA-w9pd17WH5A">YouTube</a> videos out there but we wanted to take a different approach.</p>
<p>As with <a href="https://lgulliver.github.io/2020-10-28-azureish-live!-building-a-git-commit-watcher-with-azure-functions-and-github-part1/">Azureish Live!</a>, the shows will be available on our <a href="https://www.youtube.com/channel/UCVQtNIXAgtJA-w9pd17WH5A">YouTube channel</a> shortly after for on-demand.</p>
<p>The show is something we’ve been figuring out in the background for a little bit and we thought it’d be fun to do. We’ve had a blast on our first show so please consider subscribing to both our <a href="https://twitch.tv/azureishlive">Twitch</a> and <a href="https://www.youtube.com/channel/UCVQtNIXAgtJA-w9pd17WH5A">YouTube</a> channels.</p>
<p>Our first show is already up with special guest, <a href="https://twitter.com/devopsderek">Derek Campbell</a> where he and I talk about his journey to working in Developer Relations at <a href="https://octopus.com/">Octopus Deploy</a>.</p>Liam GulliverNot one to sit still, this week Pete Gallagher, Jonathan Relf and I launched companion show to Azureish Live! called Azureish After Dark.Azureish Live! Building a Git Commit Watcher with Azure Functions and GitHub [Part 1]2020-10-28T23:30:00+00:002020-10-28T23:30:00+00:00https://lgulliver.github.io/azureish-live!-building-a-git-commit-watcher-with-azure-functions-and-github-part1<p>This week I, along with my friends <a href="https://twitter.com/pete_codes">Pete Gallagher</a> and <a href="https://twitter.com/jbjon">Jonathan Relf</a> launched a new show dedicated to live coding with Azure called <a href="https://twitch.tv/azureishlive">Azureish Live!</a>.</p>
<p>The idea behind the show is that for all the projects we come up with and build, we show the whole process on the show. From conception to deployment, using Azure resources. Think of it a little like the show <a href="https://www.imdb.com/title/tt2007689/">Scrapheap Challenge/Junkyard Wars</a> of a few years ago, except we aren’t competing against anyone and working together to help teach ourselves something new and the audience.</p>
<p>We’ll also be pushing our code to public repositories on our <a href="https://github.com/AzureishLive">GitHub</a> as we go along too. Audience participation is encouraged so feel free to open issues and pull requests as we go too.</p>
<p>It’s live on <a href="https://twitch.tv/azureishlive">Twitch</a> every other Tuesday starting Octber 27th, 2020 at 12:30PM UK time for 1 hour and is available on our <a href="https://www.youtube.com/channel/UCVQtNIXAgtJA-w9pd17WH5A">YouTube channel</a> shortly after for on-demand.</p>
<p>The show is something we’ve been figuring out in the background for a little bit and we thought it’d be fun to do. We’ve had a blast on our first show so please consider subscribing to both our <a href="https://twitch.tv/azureishlive">Twitch</a> and <a href="https://www.youtube.com/channel/UCVQtNIXAgtJA-w9pd17WH5A">YouTube</a> channels.</p>
<p>In our first show, we start the journey of building our first project: a git commit watcher for the show itself! To do it, we’ve decided to use Azure Functions to consume webhooks from Github and eventually update a Blazor UI using SignalR to display the latest commit on our stream.</p>
<p>Check out the repository over on <a href="https://github.com/AzureishLive/gitwatcher">GitHub</a>!</p>
<p>We didn’t get all the way through it in the show on this occasion but we did manage to design our solution and start putting together some of the pieces of the puzzle get moving.</p>Liam GulliverThis week I, along with my friends Pete Gallagher and Jonathan Relf launched a new show dedicated to live coding with Azure called Azureish Live!.DevOps Notts - October 2020 - Lessons Learned from Building Microservices and Terraform with Azure DevOps with Thomas Thornton, Gregor Suttie and Mark Jones2020-10-27T22:30:00+00:002020-10-27T22:30:00+00:00https://lgulliver.github.io/devops-notts-october-2020-gregor-suttie-thomas-thornton-mark-jones<p>This month’s DevOps Notts has two great technical talks from some fantastic speakers.</p>
<p>Mark Jones joins us to talk about lessons learned from building choreographed microservices, along with a great guide on using Terraform with Azure DevOps from Gregor Suttie and Thomas Thornton.</p>Liam GulliverThis month’s DevOps Notts has two great technical talks from some fantastic speakers.Agile Engineering Podcast - Agile Principles - Part 2 and Part 3!2020-10-06T12:30:00+01:002020-10-06T12:30:00+01:00https://lgulliver.github.io/agile-engineering-podcast-agile-principles-part-2-and-3<p>I am unbelievably behind on my blog post by a good month at this stage. With that in mind, I wanted to point out that both part 2 and part 3 of the exploration of the agile principles on the podcast are available now!</p>
<p>In part 2 we explore principles 5 through to 8:</p>
<ol>
<li>Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.</li>
<li>The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.</li>
<li>Working software is the primary measure of progress.</li>
<li>Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.</li>
</ol>
<iframe src="https://open.spotify.com/embed-podcast/episode/03M568bjwKDQmYNHtSRcIT" width="100%" height="232" frameborder="0" allowtransparency="true" allow="encrypted-media"></iframe>
<p>In the final part of the series, we explore principles 9 through 12:</p>
<ol>
<li>Continuous attention to technical excellence and good design enhances agility.</li>
<li>Simplicity–the art of maximizing the amount of work not done–is essential.</li>
<li>The best architectures, requirements, and designs emerge from self-organizing teams.</li>
<li>At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.</li>
</ol>
<iframe src="https://open.spotify.com/embed-podcast/episode/4LyuqPshlNqRHgB7YH6Zgt" width="100%" height="232" frameborder="0" allowtransparency="true" allow="encrypted-media"></iframe>
<p>Both resulted in some fantastic discussions so please go ahead and take a listen and feel free to input into the discussion here, on <a href="https://twitter.com/AgileEngPodcast">Twitter</a> or on our website, <a href="https://www.agileengineeringpodcast.com">https://www.agileengineeringpodcast.com</a>.</p>
<p>We’ve also recently launched a Patreon page on <a href="https://www.patreon.com/agileengineering">patreon.com/agileengineering</a>. Our goal is to expand the type of content we provide to our audience and help to provide guidance and insights from people who do this stuff every single day direct to you; whether you are mildly interested in the topics through to if you’re the most passionate CTO! We’re looking to provide more video content and a live show and would appreciate any support you can give to help us level up! Take a look at my post on <a href="https://lgulliver.github.io/launching-patreon-for-agile-engineering-podcast/">Patreon</a> for more information.</p>Liam GulliverI am unbelievably behind on my blog post by a good month at this stage. With that in mind, I wanted to point out that both part 2 and part 3 of the exploration of the agile principles on the podcast are available now!