DevOps on Azure Readiness starts here...

Using GitHub Copilot Agent Mode to vibe code a Python shooting game

Fri, 28 Nov 2025 00:00:00 +0000

For about a year now, I’ve been teaching a lot on GitHub Copilot as part of my Microsoft role. Our program offers 2 different learning paths, one created by the Microsoft Content developers, AZ-2007, and the other one is managed by GitHub Content team, known as GH-300.

If you know my approach to teaching tech a bit, which a learner in my class lately called inspiring through technology, it means I’m trying to explain as much as possible through compelling, live demos. After walking learners through different GitHub Copilot features such as documenting/explaining code, generate application code (on different development frameworks), but also Azure CLI, CI/CD pipeline, Dockerfile, YAML, JSON and alike, I usually close with Agent Mode.

Having showed the first time in April when GitHub Copilot was still in preview, I usually show a demo where the Agent Mode builds me an ASP.NET webapp, modifies the Welcome home page, creates some sample employee data in a json-file, which then gets displayed in a table view in the webapp. If time allows, I also ask to then migrate everything into a SQLite setup, which brings in more complexity such as Entity Framework, SQL data migration steps and interaction with Azure KeyVault, since I specify I want to run this in Azure SQL, but not allowing connection strings in my appsettings.json.

(Now that I think about it, it might be another great blog post to write on in the near future…)

Earlier this week however, I came up with a new scenario, asking Agent Mode to develop a shooter game using Python code, bringing me back to my youth in the mid-80’s when I was playing such games on my first 486-PC.

Agent Mode Prompt

The prompt I used was this:

*"as a kid, I played arcade games a lot. I want to build a Python app, which tests me on my shooting reflexes. Help me developing a game which does the following:

1. aks for player name input
2. bottom middle of the game screen shows a shooter 
3. anywhere random on screen appears a target
4. player uses the space bar to simulate a shoot
5. calculate the time between the target appearing and player pressing the space bar to shoot
6. if that time is less than 0.3 sec, player wins, otherwise computer wins
7. show a "YOU WIN" or "YOU ARE TOO SLOW" dpeending on the outcome"

Agent Mode Processing

From here, the Agent started rolling…
Confirming with some sort of understanding what I asked for, followed by creating 3 todos:

Set Up Python Environment
Creating the Reflex Shooting Game
Testing the Game

This process took less than 1 minute, can you imagine? From there, it continued with providing detailed instructions on how to the game works.

Next, it also had a list of features included:

Time to start the game!!

Which allowed me to play exactly as I asked for. When I was too slow, it would tell me…

And several attempts later, I finally managed to win a game!!

Summary

I thought after using GitHub Copilot for training our customers and showing capabilities using live demos for about 5 hours per class, as well as using it for a lot of “coding” tasks as part of my role as trainer, mainly creating more demo scenarios (see Trainer-Demo-Deploy to get an idea what that means…), I thought I’d seen it all.

Yet, it keeps suprising me every single day when I try something new.

If I had access to this technology in the mid 80’s, I guess I would have spent more time learning about coding than playing games… although this game is actually pretty addicting already. Time to wrap up this post and go play a bit more! And feel 12 years old again.

If you want to see some similar version of this templategame in action, head over to my github repo.

Cheers!!

/Peter

Using GitHub Copilot Agent Mode to transform ARM templates to BICEP

Sat, 12 Jul 2025 00:00:00 +0000

Over the last few months, I’ve been working on an exciting project for our Microsoft Technical Trainer team, known as “Trainer-Demo-Deploy”, a catalog of Azure end-to-end demo scenarios, available as an Open-Source project.

While we managed to get about 50 templates live, there can never be enough scenarios to integrate into your Azure classes or POC activities if you ask me. One of the challenging tasks in the project is not only coming up with demo ideas, but also creating the actual artifacts, such as Azure templates with Bicep, sample apps and sample data.

I had an Azure Site Recovery Services scenario from a few years ago, written in modular ARM templates. With Bicep providing a great way to transform your ARM to Bicep, I could have gone through each template file and convert them. Have done several of those over the last few months.

But out of teaching AZ-2007 Accelerate app development by using GitHub Copilot, where I integrate a - what I think amazing demo on how to use Agent mode to deploy a sample web app - I started thinking about testing if Agent Mode could help me with this transformation project.

Fact I’m dedicating a blog post to it, is mainly to confirm it worked amazingly well, as well as sharing my excitement and some steps of what the process looked like. Hopefully this post inspires you to start embracing GitHub Copilot Agent Mode into your own tasks.

My starting templates

My original setup was pretty straightforward, having a folder “templates”, in which I have modular templates for each part of my Azure Site Recovery Vault deployment.

Each templates hold a snippet of ARM / JSON structured code to deploy one or more Azure Resources.

I opened up the folder structure in my Visual Studio Code, and opened GitHub Copilot, selecting Agent Mode. I clearly described in a prompt, what I wanted the Agent to perform as tasks. I didn’t provide much detail to be honest, as initially, I was merily experimenting to try and find out if Agent could actually help with this, or how far it would go in the process.

Agent Mode Prompt

The prompt I used was this: for each file in the templates folder, convert to azure bicep. create a new bicep file for each, keeping the same name as the original json file. the azuredeploy should be transformed to main.bicep. validate all pointers to all new bicep files to be correct"

From here, the Agent started rolling…
Informing me about the different steps it would take to handle this task, starting with exploring the templates folder to see all the files that need to be converted

Followed by going more in-depth into each and every template JSON file
Followed by starting the conversion process to Bicep files. Before doing that, it also highlighted it would check the Azure deployment best practices (although I didn’t explicitly asked to do that, nice one! )

It felt like it learned from the best practices, by starting with the azuredeploy.json conversion to main.bicep first. It could also be that it started with this, because I mentioned this in the prompt itself.

As the main.bicep conversion took a bit longer than normal - although it was only running over it for about a minute, it prompted, asking if it was ok to continue. Obviously, I confirmed to continue.

From here, it nicely continued looping through all smaller json-files, and transforming them into corresponding bicep-files. Since each file typically had only 1 or 2 resource references, the conversion went really smooth.

After a bit, it had finished the transformation of each ARM to Bicep, and started on updating the template links, as I asked for in my prompt, to also validate the references to all the deployment files.

With all references updated, it continued with its own error checking, and validating the different templates for any possible errors. Even more interesting, without me specifying, it detected an error in the azure.yaml, which I had in my project folder, from a baseline started AZD-template we use to create all our Trainer-Demo-Deploy scenarios.

Last, it also created a main.parameters.json, to capture any specific Parameters for the deployment.

From here, it went back to validating the Bicep templates again, where it detected a few different issues. (I didn’t check in detail what got identified as issue, as it didn’t prompt me to validate anything on my end…); based on the next informational message, it struggled with missing an output for myWorkspaceKey, in the deploy-infra.bicep file.

Chewing a bit on the myWorkspaceKey problem, it managed to find its own work-around to solve the problem. It even provided a clear explaining on why, identifying the dependency on the parent template.

Feeling we were close to the end of the process, it continued amazing me, as it now also created its own documentation in a BICEP_CONVERSION_SUMMARY.md Markdown file, in which it listed up what conversations it did.

With all that out of the way, it ran another final validation to conclude there were no more issues, closing the task with creating another README-BICEP.md file, describing how to run the actual deployment, using AZD.

Finally, the Agent provided a description within the Chat Agent window, clearly describing all the tasks accomplished with the necessary file references included:

As well as adding additional details on the task validation.

Finishing with describing different ways on how to run the actual deployment, using azd, Azure CLI and Azure PowerShell.

Last step was running the deployment, and this worked without any hiccups!

Summary

As mentioned earlier, I didn’t intend to go through this process as part of writing a blog post. Yet, the fact that the GitHub Copilot Agent Mode happily suprised me once more, I wanted to share my joy and excitement about this.

Starting from a somewhat complex JSON ARM template folder with about 10 modular arm-json files, it managed to nicely convert all of them into the new Bicep template language, with only a few minor issues throughout the process. Without asking assistance or halting the process, it ran its own troubleshooting and issue resolution, resulting in a 100% successful transformation.

Apart from the technical success of the task, what surprised me even more, is it only took the agent barely 5 minutes and only had to prompt me twice during the whole process!!

If you want to see this template in action, head over to my github repo and continue your Azure learning journey with more demo scenarios at Trainer-Demo-Deploy.

Cheers!!

/Peter

Azure Spring Clean - Application Insights - Inside Out

Thu, 06 Mar 2025 00:00:00 +0000

Hey folks,

Welcome to #AzureSpringClean, an initiative from Joe Carlyle and Thomas Thornton which celebrates its 4th edition this year. I’m thrilled to be part of this again for the 3nd time this year. My first article had security in mind, explaining the difference between Azure Service Principals and Managed Identity.

My 2nd article focused on understand DevSecOps, and how you can optimize security in your application deployment lifecycle, by “shifting left”. (https://www.007ffflearning.com/post/azure-spring-clean---devsecops-and-shifting-left-to-publish-secure-software/)

Where now for this 3rd article, where moving more towards the ’end’ of the traditional DevOps cycle, discussing Operations and Monitoring, by using Azure Application Insights.

Introduction

In today’s digital age, monitoring and maintaining the health of applications is crucial for ensuring optimal performance and user satisfaction. Azure Monitor, a comprehensive monitoring solution from Microsoft, offers a suite of tools to help developers and IT professionals keep their applications running smoothly. One of the key components of Azure Monitor is Application Insights, which provides deep insights into application performance and user behavior. In this article, we’ll explore Application Insights, its features, and how it integrates with Azure Monitor to deliver a robust monitoring solution.

Overview of Azure Monitor

Azure Monitor is a powerful platform that provides end-to-end monitoring for your applications and infrastructure. It collects and analyzes telemetry data from various sources, including Azure resources, applications, and on-premises environments. Azure Monitor helps you understand how your applications are performing and proactively identifies issues affecting them. It encompasses several services, including Log Analytics, Application Insights, and Azure Monitor for VMs, among others.

Application Insights Overview

Application Insights is an application performance management (APM) service within Azure Monitor. It is designed to monitor live applications, providing real-time insights into their performance and usage. By integrating with OpenTelemetry, Application Insights offers a vendor-neutral approach to collecting and analyzing telemetry data, enabling comprehensive observability of your applications. It supports various programming languages and frameworks, including .NET, Java, Node.js, and client-side JavaScript.

Key Features of Application Insights

End-to-End Transactions: Application Insights provides a detailed view of end-to-end transactions, allowing you to trace and diagnose issues across different components of your application4. This feature supports time scrubbing, enabling you to filter and analyze specific time periods in more detail.

Performance and Failures: The service offers tools to monitor performance and identify failures. It includes features like the Roles tab, which preserves role selection while navigating from the application map, and the availability tool, which helps you monitor the availability and responsiveness of your application endpoints.

Application Map: The application map provides a visual overview of your application’s architecture and the interactions between its components. It includes features like “Zoom to fit” and grouped nodes to make the map easier to read and navigate4.

Live Metrics: With Live Metrics Stream, you can monitor your application’s health metrics in real-time, even while deploying changes. This feature helps you quickly identify and address issues as they arise.

Smart Detection: Application Insights uses machine learning to detect anomalies in your application’s performance and sends alerts with embedded diagnostics. This proactive approach helps you address potential issues before they impact users.

Integration with Azure Services: Application Insights integrates seamlessly with other Azure services, such as Azure DevOps, Azure Kubernetes Service (AKS), and Azure Functions. This integration allows you to monitor and manage your applications using a unified platform.

Integration with Azure Monitor

Application Insights is a core component of Azure Monitor, and its integration with other Azure Monitor services enhances its capabilities. For example, you can use Log Analytics to query and analyze telemetry data collected by Application Insights. This integration provides a comprehensive view of your application’s performance and helps you identify trends and patterns.

Azure Monitor also offers out-of-the-box insights for various Azure resources, such as virtual machines, containers, and storage accounts. These insights are built on workbooks, which are interactive reports that you can customize to meet your specific needs. By leveraging these insights, you can gain a deeper understanding of your application’s performance and make data-driven decisions to optimize it.

Use Cases and Benefits

Proactive Monitoring: Application Insights enables proactive monitoring of your applications, helping you identify and address issues before they impact users. This proactive approach improves the overall user experience and reduces downtime.

Performance Optimization: By providing detailed insights into your application’s performance, Application Insights helps you identify bottlenecks and optimize your code. This optimization leads to faster and more efficient applications.

User Behavior Analysis: Application Insights offers tools to analyze user behavior, such as usage patterns, session durations, and user flows. This analysis helps you understand how users interact with your application and identify areas for improvement.

Cost Management: By monitoring resource usage and performance, Application Insights helps you manage costs more effectively. You can identify underutilized resources and optimize their usage to reduce costs.

Enhanced Security: Application Insights provides insights into potential security issues, such as failed login attempts and suspicious activities. By monitoring these activities, you can enhance the security of your applications and protect sensitive data.

Seeing it in action

Now that we covered the theoretical part, let’s have a look at what all this looks like from a sample application workload perspective.

I am using a sample app which I have been using in all my Azure Architecture (AZ-305) and Developing Azure Solutions (AZ-204) classes over the years, when talking about Application Insights. It recently got moved to a new ’trainer’ platform out of a project I’m leading within Microsoft, based on Azure Developer CLI deployments for trainer demo scenarios. (If you’re new to AZD, you should definitely check it out!)

Head over to Trainer-Demo-Deploy and search for tollbooth

![Trainer Demo Deploy - Tollbooth](../images/Screenshot 2025-03-07 060247.png)

Select the Tollbooth Serverless Architecture with Azure Functions card, and follow the Template Details instructions to get it deployed. Most important is having the Scenario-specific prereqs running on your local machine, as well as having Azure Developer CLI as well.

from azd up, it will ask you for your Azure subscription and the region where you want to deploy the scenario. Give it about 12-15min, and the fun can start…

As you can see from the architecture, it is using several different services in Azure, to replicate a Tollbooth / Automated Parking Lot management application. This will generate ’traffic’ to be monitored through Application Insights.

Using this demo scenario, you showcase a solution for processing vehicle photos as they are uploaded to a storage account, using serverless technologies on Azure. The license plate data gets extracted using Azure Cognitive Service, and stored in a highly available NoSQL data store on Azure CosmosDB for exporting. The data export process will be orchestrated by a serverless Azure Functions and EventGrid-based component architecture, that coordinates exporting new license plate data to file storage using the Blob Trigger Function. Each aspect of the architecture provides live dashboard views, and more detailed information can be viewed real-time from Azure Application Insights.

Azure App Service - Upload Images

The starting point of the demo scenario, is the imageupload web application. This simulates car traffic for 500 vehicles, which should be enough to see live data dashboard views across all architecture components. Now there is a 1-2 minute delay before the metrics actually show up in the dashboards.
Navigate to the imageupload website URL (https://%youralias%tbimageuploadapp.azurewebsites.net/)

Click the Upload Images button - this loops to 500

Azure Storage Account

Once the upload process is complete, navigate to the %youralias%datalake Azure Storage Account.
Navigate to the Images Container; notice the different image files, generated from the web application. Feel free to select a file and download it, to show it contains a car image with a license plate. You might open different images, to showcase there are different cars (Note: in reality, we used 10 different images, looping 50 times, to generate 500 images in total)

Azure Functions

Once the images are available in Azure Storage Account, an Azure Function ProcessImage, which sends image files to Azure Cognitive Service

Use this Azure Function to explain the concept of triggers (HTTP, Blob Trigger,…) and how the starting point is ‘something happens in Blob’, which kicks off the Function.
Once the data comes back from Cognitive Service, it triggers the next Azure Function SavePlateData, which stores text values in Azure CosmosDB.

Event Grid / Topics & Subscriptions

Notice that this Function is based on an Event Trigger, coming from Event Grid (%youralias%eventgridtopic). From the Azure Portal, navigate to Event Grid, and select Topics. Open the EventGridTopic resource**.

Highlight the Event Grid Topic is related to the Event Grid Subscription called SavePlate, which triggers the actual Azure Function SavePlateData. This also clarifies the use case, where Event Grid acts as the orchestrator, watching over certain events to occur, and based on the settings of the subscription, it triggers an Azure Function process.

Select the SavePlate Event Grid Subscription from the dashboard view. This opens a new dashboard, showing the hierarchy of the event:

Event Grid Topic : %youralias%eventgridtopic
Metrics - showing the 500 events
Azure Function - SavePlateData

While talking about Event Grid Subscriptions, there is actually a 2nd subscription in place, which watches over the Azure Blob Storage events. Navigate back to the Azure Storage Account %youralias%tbdatalake, and navigate to Events.

Notice the Event Grid Subscription blobtopicsubscription, which is a Web Hook, meaning, it gets triggered based on HTTP requests.
From within the Event Subscription detailed dashboard, showing Metrics initially, navigate to Filters. Highlight the subscription is based on filter Create Blob. This is what triggers the Azure Function, based on “a new blob is getting created”. All other events in the Storage Account are getting bypassed/neglected.

Cosmos DB

Open the %youralias%cosmosdb. Navigate to Data Explorer. Show the LicensePlates Database, which has 2 different Containers NeedsManualReview (not used in this demo scenario), and Processed. The Processed Container is where the actual text information returned from Azure Cognitive Service is getting stored.

Under Processed, open the Items view. This shows the different document items in the container, each document having the license plate, image file name and timestamp in a JSON document format.

Application Insights

Navigate to Application Insights, opening the %youralias%tbappinsights resource. Go to Live Metrics. This will show a lot of different views about the ongoing processing of Functions, Events, Storage activity and more.

Note: If you see the “Demo” page, it means you don’t have live metrics (anymore), and the processing of the car images is completed already. To generate (new) live data, go back to the imageupload web app, and generate new images by pressing the “Upload images” button.

From within the base charts, scroll down to Servers section. There should be anywhere between 2-10 visible. Explain that these “servers” reflect the different Azure Functions instances getting triggered, and handling the image processing from blob to CosmosDB.

Zoom in on the sample telemetry to the right hand side. Explain how the different API-streams of the application topology are visible here. Notice how it shows the Azure Function call “SavePlateData”, as well as interaction with “Azure Computer Vision”, etc…

Depending when you opened the Live Metrics view, the sample telemetry should have a red item Dependency, which simulates an issue from the Azure Function to Cognitive Service, showing you the details of the API POST Action call.

Next, select Application Map within Application Insights.

Explain the usage of Application Map, describing the 2 different views here. The first view %youralias%events, shows the number of running (Azure Functions) instances, with different metrics (performance details). The Events are representing communication with Azure CosmosDB. It shows the number of database calls, as well as the average performance between the Event Functions and CosmosDB.

Select the value metric in the middle between Events and CosmosDB, to open the more detailed view. This opens a blade to the right-hand side of the Azure Portal, exposing many more details about the processing of events. It shows details about the CosmosDB instance, as well as performance details of each CosmosDB action (GET, Create Document, Get Collection, etc…)
Click on Investigate Performance

Use this detailed dashboard to explain the different sections, reflecting chart representations of actual Log Analytics Queries. This can be demoed by selecting View Logs from the top menu, selecting a section, and opening it in Log Analytics.

Conclusion

Application Insights, as part of Azure Monitor, is a powerful tool for monitoring and optimizing the performance of your applications. Its comprehensive features, seamless integration with other Azure services, and real-time insights make it an essential component of any modern monitoring strategy. By leveraging Application Insights, you can ensure that your applications run smoothly, deliver a great user experience, and achieve your business goals.

Cheers!!

/Peter

Sending Emails from Azure DevOps using PowerShell and Azure LogicApps

Sun, 03 Mar 2024 00:00:00 +0000

Hi Readers,

This post is merely for myself, documenting all the steps I needed to go through to be able to send emails from a PowerShell script, as part of an Azure DevOps YAML pipeline flow. And since I was documenting it for our internal application, I thought it would hopefully help someone out there looking for a similar solution.

Why not using the built-in Email Notification system

Azure DevOps provides email notification features, but this comes with a few assumptions. The biggest one being the receiver(s) should be Project Members. Which was not the case in my scenario, since I want to send out deployment status emails, at the start of a pipeline and when completed successfully, to external recipients.

What’s wrong with Marketplace Extensions?

In my Classic Release Pipelines, I relied on Sendgrid, and it actually worked OK, and with the limited amount of emails (up to 100/daily), it was a free service on top.

Since I was migrating the full project away from Classic Release Pipelines to YAML, and wanted to customize the email flow a bit more, I started looking into other options. Keeping all functionality within my Azure subscription, was also a benefit.

The new setup architecture

In short, the new setup architecture is based on the following:

Azure LogicApp with HTTP Trigger, sending an Outlook 365 email based on parameters to an external recipient
ADO Release Pipeline using YAML tasks
One of the tasks is using Azure Powershell, to compose and send the email

Let’s detail each step a bit more.

Deploying and composing the Azure Logic App

The starting point is deploying the Azure Logic App, and composing the workflow steps.

Deploy an Azure Logic App resource using the Consumption plan (pay per trigger)
Once deployed, open the Logic App Designer, and choose **When an HTTP request is received" as trigger.
When it asks for a JSON Schema in the Request Body, you can use the ‘use sample payload to generate schema’, by entering the different parameters you need for the email details itself, in the JSON format. In my setup, I have the following fields:

Message, which is the actual body of the email, and gets send along as a PowerShell script object;
Subject, which contains the actual subject of the email, and also coming from the PowerShell script object;
To, which contains the recipient’s email address, and passed on from the PowerShell script object;

{
    "properties": {
        "Message": {
            "type": "string"
        },
        "Subject": {
            "type": "string"
        },
        "To": {
            "type": "string"
        }
    },
    "type": "object"
}

When you save the Logic App with this step, it will provide you with the unique LogicApp HTTP Trigger URL to connect to. Save this URL on the side, since you need it later in the PowerShell script.
Add a new step to the workflow, choosing Send an Email. Authenticate with your Office 365 credentials (although other security options such as Service Principal might be required in your production setup).
The required fields for this step are Body, Subject and To. Mapping with the 3 property fields from the HTTP Trigger. Note, I created the property “Message”, instead of calling it “Body”, since the full HTTP Trigger JSON response we get in through the PowerShell script, is known as “Body”. So I made some mistakes there initially in not getting the correct information mapped as expected.
The nice thing with Logic Apps, is that any next step in the workflow, can reuse input values from any previous step.
For the Body-parameter of the Send an email step, we want to use the “Message” content from the HTTP Trigger response. Therefore, click in the Body-field, which will open the Dynamic content window. From here, it will show all known properties from the “When an HTTP request is received” step in the workflow.

Select the necessary fields from the Dynamic content, and map them with the required fields of the Send an Email task:

1
2
3

Body -> Message
Subject -> Subject
To -> To

This is all for now to get the Logic App configured.

The PowerShell script for sending emails

The idea is that the PowerShell script gets triggered from the YAML pipeline task. I initially tried using the Inline option, but that didn’t recognize the here-strings (gets explained later if you don’t know what this means…) as well as some other limitations. I also didn’t like having the actual Message/Body details - for which I’m using HTML and tables - in my YAML pipeline. So using the ScriptPath option was much cleaner. And also allowing a more flexible scenario, where I could call different ‘sendemail’.ps1 scripts, depending on the specifics of the pipeline tasks.

Let me first explain the high-level setup of the script:

define param settings, capturing/transferring the parameters from the YAML task ScriptArguments
create a variable for the LogicApp HTTP Trigger Url
create a variable for the LogicApp HTTP Trigger Body data, holding the “To”, “Subject” and “Message” content
create a variable for the LogicApp HTTP Trigger Headers information
create a variable for the actual Invoke-RestMethod, sending all previous info along

Here are some of the snippets for each component of the script:

1. define param settings, capturing/transferring the parameters from the YAML task ScriptArguments

param (
    [string]$BuildDefinitionName,
    [string]$To

    )

2. create a variable for the LogicApp HTTP Trigger Url

$logicappUrl = "https://yourlogicappurl .centralus.logic.azure.com:443/workflows/1676b43cc2904b/triggers/manual/paths/invoke?api-version=2016-10-01&sp=%2Ftriggers%2Fmanual%2Frun&sv=1.0&sig=OK-Z_dgspUufMWMlBL531"

3. create a variable for the LogicApp HTTP Trigger Body data, holding the “To”, “Subject” and “Message” content

$body = @{
         To = $To
         Subject = @"
MTTDemoDeploy - $BuildDefinitionName - Deployment Kicked Off Successfully
"@
        Message = @"

        <actual email body content here - where I used a combination of text, HTML and CSS for the email layout>

"@                
}

Note the usage of the @" “@, which is known as here-strings. Here-strings allow you to define multiline strings without needing to escape special characters or use concatenation.

In this code snippet, I’m using a here-string to define the value of the Subject property in the $body hash table. The @” at the beginning indicates the start of the here-string. The text within the here-string (between the @" and the closing “@) is preserved exactly as written, including line breaks and any special characters. The variable $BuildDefinitionName is interpolated within the here-string, and corresponds to a param object as defined at the start of the script. This will hold the actual ScriptArguments object from the YAML pipeline steps later.

Also note the positioning of the “@ all the way at the start of the line, as the here-strings cannot recognize and spaces or tabs before it - this will throw an error when running the script. (Told you I really wanted to document all by observations and issues before I got this working smoothly…)

4. create a variable for the LogicApp HTTP Trigger Headers information When you configured the HTTP Trigger step in the LogicApp, it shows a little popup message, saying the trigger expect the Header to have the content type of application/json. this is how you specify this.

1
2
3

$headers = @{
    'Content-Type' = 'application/json'
}

5. create a variable for the actual Invoke-RestMethod, sending all previous info along

$sendemail = @{ Uri = $logicappUrl Method = ‘POST’ Headers = $headers Body = $body | ConvertTo-Json }

Invoke-RestMethod @sendemail

Eventually, save this file as start_email.ps1 or other filename that works for you, and make sure it is part of the ADO Repo where you have the YAML pipeline. To keep it a bit structured, I created a new subfolder Email with another subfolder .ado in which I stored the file.

REPO-ROOT /Email /.ado /start_email.ps1

I call this script at the start of the YAML pipeline, but also created a finish_email.ps1, which I’m calling at the end of the YAML pipeline successful completion.

The YAML pipeline task

With both LogicApp and PowerShell script created, the last step in the process is defining the YAML PowerShell task which sends the trigger and necessary parameters to the PowerShell script.

Like any YAML Pipeline, this is just another task, relying on some variables and task settings.

I created 2 new variables, one for the EmailDomain and one for the full EmailAddress:

variables:

name: pipelineName value: $(Build.DefinitionName)
name: EmailDomain value: ‘@company.com’
name: recipientEmail value: “${{parameters.User}}`$(EmailDomain)”

Critical here is the usage of the ` in the 2nd part of the value setting. Since the @ in the emaildomain is breaking the string, known as “splatting”, I needed to add that character to avoid this issue.

Next, within the stage / jobs / steps level of the YAML pipeline, I inserted the following task:

steps:

- task: AzurePowerShell@5
        displayName: 'Email - Deployment Kicked Off'
        inputs:
          azureSubscription:  '<yourADOServiceConnection>' 
          resourceGroupName: 'RG where you deployed the LogicApp' 
          azurePowerShellVersion: 'LatestVersion' #required to use the latest version of Azure PowerShell
          ScriptType: 'filePath'
          ScriptPath: $(System.DefaultWorkingDirectory)/Email/.ado/start_email.ps1  
          ScriptArguments: '-BuildDefinitionName:$(pipelineName) -To:$(recipientEmail)'

The ScriptArguments is the crucial step if you ask me, as it contains the different parameters you want to pass on to the PowerShell script; also, the parameters needed to map with the JSON properties in the HTTP Trigger step of the Logic App. (again, I needed multiple attempts to get all this working, hence my documentation on how I got this all glued together…)

It picks up a parameter called BuildDefinitionName, corresponding to the same param-name at the start of the PowerShell script, which contains the value of the variable pipelineName I defined earlier in the YAML pipeline. The 2nd parameter I’m passing on is the To-field, which corresponds with the composed recipientEmail variable in the YAML pipeline.

That’s pretty much it!!

FYI, it it also possible to send more YAML pipeline parameters or variables along with the ScriptArguments, such as deployment output or passwords or any other. For example, I have a pipeline where I’m creating an Azure Container Registry with a random name, as well as a unique created password for the admin.

`1`	`echo "##vso[task.setvariable variable=acrname]$acrname"`

which I then send on to the PowerShell script from the ScriptArguments like this:

ScriptArguments: '-BuildDefinitionName:$(pipelineName) -To:$(recipientEmail) -acrname:$(acrname) -location:${{ parameters.Location }} -adminPassword:"$(genPassword)"'  #needs quotes because split characters

Summary

Apart from the built-in ADO notification emails for the most ‘common’ scenarios when using pipelines, your DevOps project might need other emails to be sent, as part of your pipeline flow. Where you could use existing Marketplace tools such as SendGrid or other, I decided to come up with my own PowerShell-based script, interacting with Azure LogicApps.

While the setup involves jumping to several hoops, it is not all that difficult (easy to say once it all works, after spending half a day of troubleshooting at different levels - of which the main one was my rusty PowerShell skills…) once all pieces fall in place.

Let me know if you want to give this a try and share me your results!

Cheers!!

/Peter

Trigger a YAML Pipeline from a Classic Release Pipeline in Azure DevOps

Sat, 30 Sep 2023 00:00:00 +0000

Hi y’all!

Over the last 18 months, I’ve been developing a tool for our internal Microsoft Trainer team, allowing them to deploy trainer demo scenarios in Azure using a Blazor Front-End web app, connecting to Azure DevOps pipelines using REST API calls.

At the start of the project, Classic Release pipelines were still common, since YAML was too new, and rather unknown. However, over the last few months, more and more I was thinking of shifting from Classic to YAML Pipelines. But - although the app isn’t that big or complex - it means connecting to different API endpoints for YAML, reading out the status of ongoing pipelines would be different, and getting a listing of all deployed pipelines for a given trainer, would also be different.

So you hear me coming… I don’t want to rewrite the whole app, as it also means rewriting all Classic Pipelines to YAML syntax. While I know the export to YAML is available, I didn’t want to minimize the effort, nor like the time-pressure. As in the end, what is the added value to the end-user? (If that is not spoken like a true developer, I have no idea…)

Being more and more familiar with triggering DevOps Pipelines from REST API calls, I thought about the following… what if I could create a Classic Release Pipeline, which triggers a YAML Pipeline?

After searching around a little bit on how curl would help here, it seemed possible, since there is a Classic Pipeline Task for #BASH.

The YAML Pipeline components?

In order to trigger the YAML Pipeline from the Classic Release, you need to capture the YAML Pipeline Id, as well as capturing any parameter values you need to provide.

The YAML Pipeline Id; this is also known as the definitionId, and can be found by going to the YAML Pipeline under the ADO Pipelines section, and checking the URL in the address bar - it will be something like https://dev.azure.com///_build?definitionId=101
Take note of the definitionId number, as you need it in the Bash script later.
In my example, the YAML Pipeline has a section with parameters, where I’m using the trainer alias, the Service Connection/Azure Subscription info and the selected Azure Region for the deployment, like this:

parameters:
- name: MTTAlias
  type: string
  default: petender
- name: azureSubscription
  type: string
  default: MTTAliasServiceConnection
- name: Location
  type: string
  default: westus

Everything else in the YAML Pipeline is rather standard, triggering an Azure Bicep template deployment, using an Azure CLI Task and pointing to the Bicep file (/.azure/main.bicep), like this:

variables:
  RunName: "${{parameters.MTTAlias}}"
  RGName: "MTTDemoDeployRG${{parameters.MTTAlias}}TBOOTH"
  dotnetFunctionZipPath: $(Build.ArtifactStagingDirectory)/dotnet
  nodeFunctionZipPath: $(Build.ArtifactStagingDirectory)/node
  

stages :        
  - stage: Infra
    jobs:
    - job: Bicep
      steps:
      - checkout: self
      - task: AzureCLI@2
        name: deployBicep
        inputs:
          azureSubscription: ${{ parameters.azureSubscription }}
          scriptType: 'pscore'
          scriptLocation: 'inlineScript'
          inlineScript: |
            az group create --location ${{parameters.Location}} --name $(RGName)
            $out = $(az deployment group create -f ./FastCar-TollBooth/.azure/main.bicep -g $(RGName) --parameters namingConvention="${{parameters.MTTAlias}}" location=${{parameters.Location}} -o json --query properties.outputs) | ConvertFrom-Json
            $out.PSObject.Properties | ForEach-Object {
              $keyname = $_.Name
              $value = $_.Value.value
              echo "##vso[task.setvariable variable=$keyname;isOutput=true]$value"
            }

Composing The Classic Release Pipeline

Create a new Classic Release Pipeline, with a single Stage “Azure Infra”, and add the #Bash Script as Task

Select Inline as Type, and in the Script box, copy the following script, which does the following:

PIPELINE_ID = the number of the YAML PipelineId

`1`	`PIPELINE_ID="101"`

URL = the URL of the actual DevOps YAML Pipeline REST API to use

1
2

URL="$(SYSTEM.TEAMFOUNDATIONCOLLECTIONURI)$(SYSTEM.TEAMPROJECTID)/_apis/pipelines/$PIPELINE_ID/runs?api-version=6.0-preview.1"
echo $URL

Next, I specify 3 variables, for which the values (the $() notation) refer to Classic Release Pipeline Variables (These are passed on from the Blazor Web App to the Classic Pipeline).

1
2
3

MTTAliasValue=$(MTTAlias)
ServiceConnectionValue=$(ServiceConnection)
LocationVarValue=$(LocationVar)

Next, I specify the actual curl command to use, where I use the System.AccessToken Variable to allow OAuth authentication by the system account, followed by specifying we’re sending a JSON header, and the actual JSON snippet which holds the data to pass along to the YAML pipeline - finding the correct syntax to capture the variable values from earlier, was the biggest challenge here, taking a few failed attempts when triggering the pipeline :)…

curl -s --request POST \
 -u ":$(System.AccessToken)" \
 --header "Content-Type: application/json" \
 --data '{
   "resources": {
       "repositories": {
           "self": {
               "refName": "refs/heads/main"
           }
       }
   },
   "templateParameters": {
       "MTTAlias": "'"${MTTAliasValue}"'",
       "azureSubscription": "'"${ServiceConnectionValue}"'",
       "Location": "'"${LocationVarValue}"'"
   }
}' \
 $URL

The full #Bash script looks as follows:

PIPELINE_ID="101"
URL="$(SYSTEM.TEAMFOUNDATIONCOLLECTIONURI)$(SYSTEM.TEAMPROJECTID)/_apis/pipelines/$PIPELINE_ID/runs?api-version=6.0-preview.1"
echo $URL

MTTAliasValue=$(MTTAlias)
ServiceConnectionValue=$(ServiceConnection)
LocationVarValue=$(LocationVar)

curl -s --request POST \
 -u ":$(System.AccessToken)" \
 --header "Content-Type: application/json" \
 --data '{
   "resources": {
       "repositories": {
           "self": {
               "refName": "refs/heads/main"
           }
       }
   },
   "templateParameters": {
       "MTTAlias": "'"${MTTAliasValue}"'",
       "azureSubscription": "'"${ServiceConnectionValue}"'",
       "Location": "'"${LocationVarValue}"'"
   }
}' \
 $URL

Specify the Correct Permissions on the YAML Pipeline

Remember we used the System.AccessToken variable, which refers to the Build-In DevOps System Process. In order to make this Classic Release Pipeline trigger work, the Build-In account must have Queue Build permissions on the YAML Pipeline.

Navigate to the YAML Pipeline
Click the elipsis (the 3 dots) at the end of the Pipeline line, and from the context menu, select Manage Security

Select the Build Service Group, and set Allow for the Queue Builds permission

Running the Pipeline

Trigger the Classic Release Pipeline, wait for it to complete

With the Classic Release completed, navigate to the YAML Pipeline, and see this one is getting triggered / already running

Summary

In this post, I walked you through a use case within Azure DevOps, where it might be useful to build an integration between both Pipeline worlds, Classic Releases and YAML Pipeline Releases. By using a #Bash script with Curl to call the YAML Pipeline REST API endpoint, as well as passing some parameters in a JSON structure, it is possible to trigger YAML Pipelines from a Classic Release Pipeline.

I am using this scenario to give myself some time to continue updating the development work on the Blazor Front-end, pointing to YAML Pipelines only at some point, but for now, it gives me the flexibility to keep the same Classic URL Endpoints for my REST APIs, why gradually setting up new YAML Pipelines, migrating Classic to YAML etc…

Cheers!!

/Peter

You do not have permissions to view this directory or page after publishing to Azure App Service

Sun, 30 Apr 2023 00:00:00 +0000

Earlier this week, I got an interesting phenomenon when publishing a code update to an existing Azure App Service. As this was a small project, I’ve always been deploying updates in a manual way from right-click / publish in Visual Studio. But than I said to myself, Peter, as a passionate DevOps engineer and trainer, just go out and create a pipeline for this.

And that’s what happened.

Running the release pipeline came back successful, but the site threw an error:

I was like well, OK, no worries, let’s go back to the manual deployment from Visual Studio for now. Only to find out that one didn’t succeed either anymore, just giving me a spinning publishing task.

So this was not really helping me further. Time to open up the App Service Logs settings on the App Service to dig in.

After which I could check the live logs using App Service Log Stream (notice the verbose option to get immediate and full feedback…)

2 things here got my attention:

HTTP Error 403.13 Forbidden
A default document is not configured for the requested URL

So it seems like the index.html I use in my app, couldn’t be found on the Web Server. Let’s validate with App Service Editor

Interesting… so I have my drop folder with the application zip package and some other deployment artifacts, but not the actual application files in an extracted format. The drop folder was also something that made me curious… As that is what Azure DevOps is using to publish the package… Let’s go back to my Azure DevOps Release pipeline and check something…

EUREKA!!! Looks like Peter made a mistake here, by not setting the package file to use. So what the ADO Pipeline does here, is just copying the /drop folder with the artifact into the Azure App Service, but not extracting it.

This is what this setting should look like:

With these new changes, let’s run the release deployment again and see what happens…

and the website is running as expected!

Last check, validating if a new deployment from Visual Studio is running as expected again… and that runs successful again too!

Summary

In this post, I wanted to help you sharing some troubleshooting steps for Azure App Services. And also admitting I made a minor mistake in my ADO pipeline setup. So additional lesson learned: always doublecheck your setting when something doesn’t work anymore as it should be :)

Cheers!!

/Peter

Collecting Feedback in ADO work items from Office Forms

Sun, 26 Mar 2023 00:00:00 +0000

Hey folks,

I’m a fond user of Azure DevOps for testing application builds, running CI/CD pipelines to publish Azure demo scenarios and train our Microsoft global customers on it every few weeks out of AZ-400 training deliveries.

One of the lesser-known, yet AWESOMELY POWERFUL features besides ‘running pipelines’ is Azure Boards, providing an end-to-end project methodology platform using Scrum, Agile, CMMI or custom approach.

In this post, I mainly wanted to zoom in on the custom capabilities, capturing feedback from employees - which got entered through an Office Forms form, picked up by Azure Logic Apps, and stored in a customized Azure Boards Work Item.

In short, the following steps are needed:

Create custom Office Forms with questions and fields to complete
Create custom Azure DevOps Process Methodology containing the custom Work Item fields and form layout
Create new Azure DevOps Project, linked to the custom Project Methodology
Create Azure Logic Apps flow, mapping each custom field from Office Forms to the custom Work Item fields
See it in action :)

Here we go…

Create custom Office Forms with questions and fields to complete

The first step involves creating a custom Office Form, which is probably one of the easiest parts in the process. This is a free service within Microsoft Office Online, typically used for collecting user input, such as surveys, quizzes and polls, and all you need is a Microsoft Account such as Outlook.com, Hotmail.com or an organizational Office 365 account.

Browse to https://forms.office.com, and select New Form
Next, specify the different questions, together with the answer type (e.g. multiple choice, text field,…) I won’t cover the details on how to do this, as I think it’s self-explanatory.
A sample form I’ll be using in this post looks like this:

With the Office Form ready, we can move on to the next step, creating a new custom ADO Process Methodology.

Create custom Azure DevOps Process Methodology

Azure DevOps provides several Process Methodologies, such as Scrum, Agile, Basic, but also allows you to create customized versions of those. (More info on each process and how to choose is documented on Microsoft Learn).

Log on to Azure DevOps with an Organizational admin account
Select Azure DevOps (the logo in the upper left corner), and select Organization Settings
Within the Settings menu, select Process under the Boards section
This shows a list of default Azure Boards processes (Basic, Agile, Scrum, CMMI)
In this example, we will build a new custom deviation from the Scrum process, but you can choose any you want. Hover the mouse over the Scrum process, and select the ellipsis (the 3 dots). From the context menu, select Create inherited process
Provide a name and (optional) description for the process. I called mine Forms Test Process
Once created, select the new process. This opens a list of Work Item types such as Bug, Epic, Task and other. As the new Work Item we create is so custom, it doesn’t really matter which one to choose. If you have about 50% or more that’s identical to an existing Work Item, you can use that as a baseline.
Click New Work Item Type
Provide a Name, Description, Icon, Icon Color of choice. Confirm by pressing the Create button.
Once created, select the new work item type. This opens the Layout editor, where we will add custom fields, reflecting the different questions/items from the Office Form earlier.
A Work Item is based on Tabs. In my example, I only use a single tab, called Details. Know you can as many Tabs as needed. Within each Tab, the Work Item layout is built-up of 3 panes, a left, holding a Description field, the middle pane, holding Custom fields, and the right pane, which has Deployment, Development and Related Work as default items - at least in my setup.
Add the custom fields you want to have on the Work Item, specifying the field type (e.g. I added an open question, set as Text Multiple Lines, as well as adding the Geography and Category as Text Single Line items). These fields somewhat correspond with the different items on the Office Form.

Although there are a lot more customizations possible, I hope these basic steps are helping you building the baseline for what I want to guide you through in this post.

We now have the Azure DevOps Process created, as well as the customized version of the Work Item we want to use. Let’s hook this up to a new Azure DevOps Project.

Create new Azure DevOps Project, linked to the custom Process

Click the Azure DevOps logo (upper left corner), and press the + New Project button.
Provide a Project Name, (optional) Description, Visibility. Next, click Advanced to specify the Work Item Process. Click the Work Item Process field and select the custom process created earlier.
Once the project got created, navigate to Boards. Here, click New Work Item, notice the new Work Item type is available.
Select the new Work Item, which shows the detailed view. Notice the custom fields we added earlier are nicely showing up here.

The flexibility we now have, is that the Work Item can be created from both the Office Form, as well as still being available from within Azure DevOps. (Note: while we added custom fields, I didn’t add any field content to choose from, such as EMEA,APAC,USA in the Geography field - which would a viable option). In my use case, the only way to create a new Work Item is through the Office Form, as no typical users got access to Azure DevOps to do that (Permissions :)).

Awesome, we are now about 3/4 through the process, with the remaining part being the creation of the flow, using Azure Logic Apps.

Create an Azure Logic App flow to capture Office Forms data to ADO Work Item

Log on to Azure with administrative permissions to create an Azure Logic App resource.
When creating the Logic App, specify a unique name, Resource Group, Location and Plan (consumption would be OK).
Once the resource got created, it automatically opens Logic App Designer, which allows for the setup of the actual flow. From the list of sample scenarios, select Blank Logic App.
In the Search connectors and triggers field, search for Microsoft Forms.
Next, select When a new response is submitted as trigger.
In the Form Id field, select the Office Form name you used earlier.
Click + New Step to add the next step in the Logic App flow.
In the Search connectors and triggers field, search for Azure DevOps.
From the list of Actions, select Create a new work item.
Complete the fields, selecting your DevOps Organization, the DevOps Project and the Work Item Type as created earlier.
Now, we map the custom fields, by selecting Add new parameter, and selecting Other Fields.
Within the little table of Other Fields (the key/value), select the Key object; this opens the Logic Apps Dynamic Content. Here, click the See more option
This is where Logic Apps is awesome. It allows you to select (all) previous fields from all previous steps in the flow process.
Notice how the Forms information is returned as Body, which is not what we need. We want to reach out each answer to each Form’s question, instead of the full body. To make this possible, we have to add another step in-between the Forms step and the Azure DevOps step.
Click on the + sign in-between both steps, and select Add New Action. Search for Microsoft Forms again. This time, it will show an action called Get Response Details.
In the Forms Id field, select the name of the Office Forms; In the Response Id field, click See More and select List of Response notifcations Response id from the Dynamic Content list of options.
With that step added, return to the Create a work item step in the Logic App Flow, and navigate to the Other Fields section in the parameters. Select the Enter Key field, which opens the Dynamic Content blade again. This time, notice how the different response details (the Form’s questions) are visible.
From here, the idea is that you ‘map’ each custom field object from the ADO Work Item, with a corresponding value from the Office Forms. For example, the work item “geography” created earlier, maps with what is your geography question I have on the form.
Once done with all field mappings, Save the Logic App.

This completes the configuration of the Azure Logic Apps (Note: there is more work needed if you have more fields…)

Which brings us to the last step… seeing it in action…

Testing the Azure DevOps Work Item creation

Return to the Office Form, and click Collect Responses.
Complete the different questions and fields on the Form.
Wait for about a minute, and return to the Azure Logic Apps flow created earlier. From the Overview blade, navigate to Runs History.
The Forms completion resulted in a successful workflow trigger. Select the line, which opens a more detailed view. .
Last, return to the Azure DevOps Project, navigate to Boards and open Work Items. Notice the newly created Work Item.
Open the item, to see how the custom fields got completed.

That’s pretty much it!! Nice isn’t it…

Summary

In this post, I wanted to share more details on how you can allow end-users (or customers) to create Azure DevOps Work Items (of pretty much any type with any custom fields), using an integration of Microsoft Forms and Azure Logic Apps.

Don’t hesitate reaching out if you have any additional questions on this, or if you want to share how you used this in your own scenarios.

Cheers!!

/Peter

DevOps Workflow Generator

Sun, 03 Apr 2022 00:00:00 +0000

Hey awesome people,

For the ones who know me, it shouldn’t be a surprise I’m interested in DevOps, mainly using Azure DevOps and GitHub as core technologies, as well as several side-solutions that integrate with them. So when I heard about the DevOps Workflow Generator, a new free tool from the Microsoft Research Lab division, I wanted to give it a spin.

The Concepts of DevOps

DevOps according to Microsoft’s Definition: The Union of People, Processes and Products, to enable continuous delivery of value to the business

The tricky part with DevOps is that it’s not just about 1 team using a single tool, but potentially a complex group of people (the DevOps engineering team), using a multitude of tools and solutions to perform their role. Obviously the main DevOps pipeline engine (Azure Pipelines, GitHub Actions, Octopus Deploy, Jenkins, GitLab, etc…), but most probably, it also involves Infrastructure as Code tools such as Terraform, Azure Bicep or ARM Templates, Configuration as Code tools like PowerShell DSC, Ansible, Chef or Puppet, as well as DevSecOps tools where Snyk, Aqua, SonarQube, WhiteSource bolt and Veracode are just some of the popular ones. (Btw, if you missed my recent post on integrating DevSecOps by Shifting Left, which I wrote for Azure Spring Clean, you can find it here)

When to use what

If you went to the Azure Spring Clean post I referred to earlier, you now know that DevOps is relying on a lot of tools. I’m not saying it’s the most important aspect, but without tools, there is no DevOps. Period.

So coming back to the DevOps Workflow Generator, it literally helps organizations, and their DevOps teams, to get a clearer view on what the DevOps process is about, as well as what different tools are being used in each and every phase. You might ask, what’s the point in that.

Well, let me tell you. Since DevOps is more about the culture than the tools, the better view your team has on what tools are being used, the better the team will operate. Apart from bringing - what’s traditionally 2 separate worlds - Developers and Operations teams together, there might actually be a sub-project as part of the DevOps adoption, to try and unify the tooling that is being used. Let’s say Developers are OK with using Visual Studio, or maybe Visual Studio Code. Where Operations folks are probably also using Visual Studio Code, in favor of Visual Studio. This might lead to an agreement that from now on, Visual Studio Code is a standard tool across the DevOps team. Maybe even sharing extensions (ARM Templates, Docker, Kubernetes, Bicep, Azure,…) are just some of my favorites.

How to use the DevOps Workflow Generator

You most probably don’t need my help from this blog post to find out how the Workflow Generator is working, it’s really that easy to use. However, after a first quick look, I went back (in preparation to write this article) and actually discovered some new things. So hopefully there is still something useful for you to discover:

Browse to https://devopsworkflowgenerator.research.microsoft.com/

From the top menu, select Map Workflow

This presents you with a rather generic/standard DevOps workflow process; which at first, was what I used. Until I discovered you can actually make customizations to it. Following my own best practices - Shifting Left - I added some of the process steps as outlined in the referred article earlier. The updated workflow looks like this now:

Next, move over the Select Tools step in the top menu. This allows you to select DevOps solutions and tools (single or multiple) for each cycle in your DevOps Workflow. And the list is extensive…!

Once all tools have been mapped with each phase, it’s time to compile a report, by navigating to the Download Report menu option.

The outcome presents a nice-looking PDF document, which looks like this:

Pretty cool, right?

Summary

In this article, I wanted to introduce you to DevOps Workflow Generator , a free tool by Microsoft Research Labs, allowing DevOps teams to get a better view on their DevOps process(es), as well as highlighting the different solutions and tools used for each phase of the DevOps process.

Have a look at it, and let me know your thoughts!

If you liked this article, consider giving back a small token of appreciation:

Peter

Azure Spring Clean - DevSecOps and Shifting Left

Thu, 17 Mar 2022 00:00:00 +0000

Hey folks,

Welcome to #AzureSpringClean, an initiative from Joe Carlyle and Thomas Thornton which celebrates its 3rd edition this year. I’m thrilled to be part of this again for the 2nd time this year. My first article had security in mind, explaining the difference between Azure Service Principals and Managed Identity.

For this second article, I’m staying in the security focus, helping you understand DevSecOps, and how you can optimize security in your application deployment lifecycle, by “shifting left”.

I hope you learn from it, enjoy reading through and got inspired to check back the whole week here at Azure Spring Clean, as there are A TON of great topics that will be covered.

The Concepts of DevOps

Microsoft’s Definition: The Union of People, Processes and Products, to enable continuous delivery of value to our end users

What DevOps teams are doing is providing an automated deployment process, starting from:

a) checking in their application source code into a source control system, such as Azure DevOps Repos, GitHub Repositories or similar; b) Once the code has been checked in, it loops through a functional testing process; c) This forms the starting point of Continuous Integration (CI), where the code gets compiled into an artifact or deployable package; d) From here, the package typically gets deployed into a running state, known as Continuous Deployment (CD); this could be to a dev/test, staging or production environment; e) Once the application workload is published, there’s a handover to the Operations team, who integrate monitoring, watch over incidents and fix the problem.

This (somewhat simplified) process gets repeated over and over (CI/CD pipeline automation), and should lead to faster deployments, less bug fixes needed and reliable workloads.

If we look at this process from a linear perspective, it would look like this:

So now you know what DevOps stands for, let’s zoom in more on the DevSecOps extension

Shifting Left

When you look at this application lifecycle overview, most of the security handling, but also vulnerabiltiies, are getting detected and handled during the running phase. That’s where we have DDOS attacks, identity or credential theft, networking attacks, crypto and similar malware etc…

While there’s nothing wrong in handling security all the way at the end, it should not be ONLY all the way at the end, but rather be moved all the way to the beginning; and becoming part of each and every stage in our DevOps process

That’s what the industry calls “shifting left”

So what are some of the security best practices DevOps teams can (easily) integrate into their DevSecOps process you ask? Actually there’s a lot of different options and possibilities. Good news is, you can apply different security features in the different DevOps stages:

I’ll drill down a bit more on these obviously, but let’s look at some examples:

DEV

Threat Modeling - Microsoft provides a free Threat Modeling Tool, which helps you outlining the potential threats and vulnerabilities for a generic application architecture;
Credentials & Secrets Management - NEVER store your secrets and credentials hard-coded into your source control system. That’s just a NO GO!! Rather, look into secret variables, variable groups or even better, a secret store such as Azure Key Vault or GitHub Secrets
Peer Review

VALIDATE

Code Analysis - this is where you integrate source control code scanning tools such as Snyk, Sonar, WhiteSource Bolt and many others. The easiest way is going through them in your own DevOps pipelines. If you are using Azure DevOps, have a look at the Azure DevOps Marketplace, and search for security At present, there are 91 different extensions available to integrate security into your DevOps Organization and Projects.

PACKAGE

Secured Containers - Containers are becoming more and more popular to speed up the development process, simplifying the dependency on a platform and overall allowing for standardization thanks to Docker images. While containers are bringing in a lot of good things into a developer’s scenario, they also might bring in vulnerabiities and security risks, as you don’t always know where the image comes from, what’s running inside the container or who built it. That’s where I can definitely recommend a vulnerability scanner for containers. Tools such as Aqua or Twistlock are popular reference here. If you are using Azure as your container runtime environment, know that you can enable Microsoft Defender for COntainers, which provides a built-in security vulnerabilty for your containers stored in Azure Container Registry.
Quality Gates
Cloud Configuration
Security & Pen-testing

RUN

Cloud Platform Security - Any cloud vendor provides robust security features as part of the platform. Azure comes with Azure Security Center for years, which recently got rebranded to [Microsoft Defender for Cloud](https://docs.microsoft.com/en-us/azure/defender-for-cloud/defender-for-cloud-introduction). Core characteristics are detecting your Secure Score, a value for your security posture, together with an extensive list of recommendations, on how to optimize your security.
RBAC permissions model - This corresponds to the concept of “least set of privileges”, which means your DevOps teams should only get the administrative permissions they really need to do there job, but no more. Or even better, only give them administrative permissions when they need to perform admin tasks. Services such as Azure Identity Protection and Privileged Identity Management are no luxury in any organization. Keep in mind these require an Azure AD Premium P2 license. Which - if you ask me - is more than worth the extra cost!
Credentials & Secret Management

OPERATE

Security Monitoring
Threat Detection - This brings us back to the “original” approach, having the necessary security guardrails in place to protect our runtime environments. SIEM solutions such as Azure Sentinel
Mitigation

Summary

In this post, I gave you an overview of the typical DevOps process, and what challenges exist around security. Often coming in all the way at the end (during the operations cycle), security should be part of each and every step of the DevOps concept, preferably as early in the process as possible. This is what the industry calls shifting left. I tried to share some “easy to implement solutions” to optimize security, by sharing several tools and services available today in Azure, Azure DevOps and GitHub. If you should have any questions on this, or you want to see a demo on what the tools can do, I’m only a nudge away ;)

Once more, I very appreciated thank you for reading, and for Joe Carlyle and Thomas Thornton for having accepted my submission for this 2022 #AzureSpringClean edition. Enjoy your Spring Clean week, stay safe and healthy!

Peter

AzCopy failing in Azure Devops with error ServiceCode=AuthorizationPermissionMismatch

Mon, 14 Dec 2020 00:00:00 +0000

Hi all,

This is one of the nail biting challenges of our industry, failing in running a straight-forward task. Especially when it is failing…

What happened?

I was creating a pipeline in Azure DevOps to deploy an ARM template for VM setups with VM Extensions. To prep this deployment, the artifacts (DSC scripts) should be copied to Azure Blob Storage, in order for the Azure DevOps build agent to “find” it. Easy-peasy, there is a predefined task for that, called Azure Blob File Copy

I defined source (my Azure Repos folder) and target (Blob Container in Azure Storage Account), but saw the copy failing with an interesting error message:

INFO: Authentication failed, it is either not correct, or expired, or does not have the correct permission
RESPONSE Status: 403 This request is not authorized to perform this operation using this permission.

Knowing I’m running this deployment with a valid Service Principal linked to my Azure DevOps Service Connection which deployed Azure Resources successfully before (including a new Azure Storage Account from within the same Job earlier in the process), I had no immediate idea what was wrong. Also, I have used AzCopy for a while already, without issues.

What to check?

Next challenge is, where do I start troubleshooting?

My Service Principal only got created last week, so definitely not expired (and working for other Azure deployments);
My Service Principal got created with all default settings and scoped to my subscription as Contributor (az ad sp create)(https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli); While you should definitely limit the scope in a production environment, not that important for my demos. And thus also not the blocking factor;
Check the differences between a working AzCopy pipeline and the non-working version to try and identify any differences;

BINGO!

Not yet, but at least I found the clue… in the AzCopy documentation which specifies how to enable Azure Blob storage access using Azure Active Directory instead of the traditional SAS token (A Service Principal is an Azure Active Directory object, therefore I was not looking into SAS tokens anymore…). Following the link under Option 1 (Azure AD), pointed me to the following updates in AzCOpy:

- The level of authorization that you need is based on whether you plan to upload files or just download them.

- *If you just want to download files, then verify that the Storage Blob Data Reader role has been assigned to your user identity, managed identity, or service principal.

- If you want to upload files, then verify that one of these roles has been assigned to your security principal:

    - Storage Blob Data Contributor
    - Storage Blob Data Owner

Let’s give this a try:

From your Azure DevOps Project, select Project Settings / Service Connections
Select the Service Connection you use for the given Pipeline deployment and choose Manage Service Principal Roles
This opens Azure Active Directory - Access Control from where you can add a role assignment by clicking Add Role Assignment
From the list of roles, select Storage Blob Data Contributor, and search for your Service Principal name in the Select field (if you don’t know the exact name of your Service Principal anymore, from Azure DevOps / Service Connections, select “Manage Service Principal”, which will open your Service Principal blade in Azure Active Directory for this specific object - the name will be visible from there)
Save your changes. (Note - I’m allowing this permission for this Service Principal across the full subscription; in a real-life scenario, it would be enough to allocate this Azure Role scoped to the specific storage account)

The result should look about similar to below screenshot:

![Add Role Assignment](../images/2020-12-14_6.jpg)

Run the Pipeline again from Azure DevOps, and behold… a successful run this time :)!

I hope this helps anyone bumping into the same issue as I did. For me lesson learned is reading the Azure Docs a bit more every now and then, especially when something isn’t working right away…

thanks, Peter

Azure Back To School (with Azure DevOps)

Sun, 13 Sep 2020 00:00:00 +0000

Hey everyone,

Wow, it’s September already! Where has summer been? And honestly, where has the (nice) year been? Most of you, just like myself, are probably still working from home, and I guess the same are your kids.

Where September used to feel like a fresh start (remember that smell of your new school outfit, the new school equipment, fresh-sharpened pencils, the excitement of learning new things, meeting new class mates,… almost seems like I want to go back to school myself.

Trust me, I live this moment almost every single week as an Azure Technical Trainer, feeling the excitement from the attendees to learn knew things about Azure, getting their solutions validated or hearing how there problems can be solved by using Azure cloud services. I even feel how nervous they are about taking a Microsoft Certification and become Azure certified.

Azure Back To School

So to stay within the “spirit of continuous learning”, I was more than happy to contribute to Dwayne Natwick’s event Azure Back to School,

and present a session on “Azure DevOps is not for IT Pro (says no one ever again)”. Why was I so stupid enough to try and cram this amazing tool in a 30 minute session? I still don’t know why :)

Anyway, during this session, I’m scratching the surface of Azure DevOps capabilities:

Azure Devops Repos - GIT-compatible source/version control
Azure DevOps Boards - Following on Scrum, Agile and alike development project management, this component provides work items, bug tracking,… including graphical boards
Azure Pipelines - Probably the core of the solution, allowing you to enable a complete CI/CD (continuous integration/continuous deployment) scenario using YAML or Classic Editor, and deploy our templates/workloads to Azure (or other platforms)

The goal in my 30 min session was to just show you the core capabilities, to at least get you looking into it. If I can get you creating your Azure DevOps Organization and creating your first Azure DevOps project… I have reached my goal :).

Everything else will follow in future blog posts here at 007FFFLearning.com, no worries.

That said, if you made it all the way through this post, and you cannot wait to get started with Azure DevOps, send me a DM on Twitter or leave me an email with the exact subject “@zure DevOps - Back To Sch00l” , and I’ll return you a little gift, allowing you to jump into Azure DevOps right away…!

One last thing, there is a lot more cool and interesting Azure stuff to be found on Azure Back to School** for the rest of the month, so grab that opportunity to keep your internal learning daemon happy ;).

See you around! Take care,

thanks, Peter

DevOps for Dummies - Emily Freeman - Review

Sat, 22 Aug 2020 00:00:00 +0000

Hey,

As some of you know me for years already, you also know I’m rather bad in coding (although I’m learning Blazor since a few weeks, but more on that journey in future posts…), and always “got scared” about DevOps. Especially the Dev part in the word 😊. I still remember being in Bangalore, India early 2016 to deliver an Azure workshop to Microsoft GSI Partners (Wipro, Tech Mahindra, Accenture,…) as a freelance trainer together with Microsoft AzureCAT engineers. While my sessions were totally in my Azure comfort zone (Networking, Storage, Azure Active Directory), I also learned a lot from the Azure App sessions my colleagues delivered. But then all of a sudden, I had to jump in on Wednesday afternoon, taking over the “DevOps practices” session due to some urgent facts. Oh man, down went my comfort level. Me, the guy they saw earlier in the week, delivering Azure Infra sessions with ease, now needing to talk about DevOps, something I didn’t know at all? I hardly used Visual Studio at that time, let alone I could talk about this process.

So instead of delivering the foreseen session (DevOps processes, Visual Studio integration with source control, CI/CD Pipelines,…), I just talked about my personal perspective of DevOps, how ARM templates and Azure Automation was “the DEV part” in my world, helped me combining my +15 years background in building Microsoft datacenters at customers “the OPS guy”, and do amazing things with it. All in all, the talk got very much appreciated, as it was personal.

I already know Azure DevOps, so why bother?

Jumping 5 years further in time, I have been using Azure DevOps as a tooling myself for about 2 years now, and also delivered several successful Azure AZ-400 (https://docs.microsoft.com/en-us/learn/certifications/exams/az-400) workshops out of my Microsoft role as Azure Technical Trainer. And then I stumbled into Emily Freeman’s (https://twitter.com/editingemily) Devops for Dummies book (https://www.amazon.com/DevOps-Dummies-Computer-Tech/dp/1119552222).

At first, I wasn’t really interested in getting me a copy, given it was “for Dummies”. And remembering what other “for Dummies” I read in the past, it was always on something I didn’t know at all, yet learned a lot from it. So I gave it a chance – also because the Kindle edition is only $15,99 USD – that’s like 3 Starbucks coffees 😊. And wow, I was hooked to it from the first chapter.

What got me hooked?

First of all, Emily is a subject matter expert, period. I’ve seen her presenting both in-person and at virtual conferences, and I enjoy learning from her stories. She manages to combine technical expertise with understandable explanations, calm presentation style, funny twists,… everything you look for in a presenter. And next to that, passion for the topic. It is clear Emily can look back to a huge amount of in-the-field experience on helping customers implementing (or transforming for that matter) DevOps best practices into their IT lifecycle management.

It was this natural flow of words, the logic of chapters, how the whole book is getting build up from basics to more complex scenarios and everything in between, that makes it worthwhile for anyone in IT to read it. This book is not about Azure DevOps just to be clear. It focuses on the processes, the challenges, how to get from where you are today to where you can be in the (near) future as an organization, and also nicely describes the real-life challenges that comes with it.

Next, every couple of pages, she makes a nice annotation to an interesting quote, additional side-reference material, a funny story around DevOps,… that makes it still “light to digest”, yet still hugely informational.

About the book structure

The book is about 300 pages long, split up in 6 chapters:

Demystifying Devops – this is one of the best 50 pages I know describing what DevOps REALLY is. What’s good about, where are the challenges, why/how around colleagues not liking it and how to convince them,…
Establishing a Pipeline – From planning to software lifecycle management, and how to get there, also explaining how your development code itself should take DevOps into account
Connecting the circuit – this section covers how to handle and get feedback, manage iterations, success and failure of the process
Kaizen – Not about a Japanese Hero, but how to continuously improve your processes, learn from mistakes and optimize the business overall
DevOps Tools – A nice overview of different devops tools, useful for private and public cloud platforms
Top 20 – 10 reasons where DevOps is crucial, and another 10 reasons where it can fail

What I remember from it

“DevOps is approachable and real”, is one of the best quotes

Aside from that, I took a lot of notes around processes, how to inspire people to start embracing DevOps practices, the challenges that comes with it, but mainly how it can help any organization to transform their current mode of operations and software lifecycle into an optimized, well-oiled flow, where IT is becoming a business driver, no longer a cost center or cost generator.

I honestly wish I had this book 5 years ago, showing me there is nothing scary or anything to be afraid of when considering DevOps. It’s about people, processes and products. It’s not about being a full-stack developer or being the expert datacenter specialist. It’s how both worlds can nicely work together. (Which in these difficult times of COVID-19 and racial disturbance could actually be baby-steps in the positive direction)

Ping me if you got any questions on the book, DevOps or Azure DevOps in general.

Cheers, Peter

Azure DevOps Pipelines - YAML or Classic Designer

Sun, 21 Jun 2020 00:00:00 +0000

During several of my AZ-400 Designing and Implementing Microsoft DevOps Solutions training deliveries, one recurring point of conversation is Should we use YAML or the Classic Designer for our Release pipelines?

So I thought sharing my view in another blog post could be helpful.

Before answering the question more accurately, let’s go over each scenario a bit more in detail:

Classic Designer

Classic Designer has been the long-standing approach on how Azure DevOps Pipelines have been created. Using a user-friendly graphical User Interface, one can add tasks to create a pipeline just by searching for them from a list of tasks, and complete necessary parameters.

Below example is what I use for building Docker Containers:

As you can see, this looks quite straight forward to anyone, even if you are totally new to Azure DevOps.

If I want to update my pipeline with another task, for example Docker CLI Installer, I just click on add task and search for all “Docker” related tasks from the list,

and select the related task I want:

Once you are familiar with the actual steps on how to build and compile containers from a command line, moving the manual steps to an Azure Pipeline are almost 100% the same. In the end, you are literally automating your manual approach.

YAML

Now, let’s take a look at the YAML (Yet Another MarkUp Language) approach. There is no graphical designer here, but rather a text config file you need to build up, describing the different steps you want to run as part of your Azure Release Pipeline.

YAML got introduced into Azure DevOps mid 2018 already, but I still see a lot of customers not using it that often yet.

Using a similar example as before, the YAML file looks like this:

pool:
  name: Azure Pipelines
steps:
- task: qetza.replacetokens.replacetokens-task.replacetokens@3
  displayName: 'Replace tokens in appsettings.json'
  inputs:
    rootDirectory: '$(build.sourcesdirectory)/src/MyHealth.Web'
    targetFiles: appsettings.json
    escapeType: none
    tokenPrefix: '__'
    tokenSuffix: '__'

- task: qetza.replacetokens.replacetokens-task.replacetokens@3
  displayName: 'Replace tokens in mhc-aks.yaml'
  inputs:
    targetFiles: 'mhc-aks.yaml'
    escapeType: none
    tokenPrefix: '__'
    tokenSuffix: '__'

- task: DockerInstaller@0
  displayName: 'Install Docker 17.09.0-ce'

- task: DockerCompose@0
  displayName: 'Run services'
  inputs:
    dockerComposeFile: 'docker-compose.ci.build.yml'
    action: 'Run services'
    detached: false

- task: DockerCompose@0
  displayName: 'Build services'
  inputs:
    dockerComposeFile: 'docker-compose.yml'
    dockerComposeFileArgs: 'DOCKER_BUILD_SOURCE='
    action: 'Build services'
    additionalImageTags: '$(Build.BuildId)'

- task: DockerCompose@0
  displayName: 'Push services'
  inputs:
    dockerComposeFile: 'docker-compose.yml'
    dockerComposeFileArgs: 'DOCKER_BUILD_SOURCE='
    action: 'Push services'
    additionalImageTags: '$(Build.BuildId)'

- task: DockerCompose@0
  displayName: 'Lock services'
  inputs:
    dockerComposeFile: 'docker-compose.yml'
    dockerComposeFileArgs: 'DOCKER_BUILD_SOURCE='
    action: 'Lock services'

- task: CopyFiles@2
  displayName: 'Copy Files'
  inputs:
    Contents: |
     **/mhc-aks.yaml
     **/*.dacpac
     
    TargetFolder: '$(Build.ArtifactStagingDirectory)'

- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact'
  inputs:
    ArtifactName: deploy

That’s rather different, isn’t?

Each individual task I just had to select before, now requires a set of instructions in a config file. Luckily, Azure DevOps still provides a graphical interface to pick the tasks, which could be helpful in the beginning, when YAML is too new to you.

Good news is, from a Build Pipeline perspective, both methods provide the same result. So the key question is, which one to go for?

Classic Designer or YAML

After discussing on this topic with several students during my deliveries, I came up with a good and bad list for each. Know this is far from complete, not trying to push you in a certain direction at all, but merely providing an overview.

Advantages of Classic Editor

Ease of Use
Clear overview of what tasks the pipelines is based on
Lots of preconfigured task snippets available
No “development” language to learn

Disadvantages of Classic Editor

Less obvious Source Control/Version Control
Specific to Azure DevOps
Slow to create or update your pipelines
Microsoft-native
While not immediately, it will phase out at some point

Advantages of YAML

100% code-based, which means you can manage it like your application source code in source/version control
Easy to make changes (once you know how the language works)
Easier to compare changes (e.g. Azure Repos “file compare feature”)
Code snippets can be shared easily with colleagues, much easier than screenshots
Same YAML concept is used by Docker, Kubernetes,… and several other “configuration as code” tools
View YAML option in Classic Editor to see the snippet of GUI task translated to YAML

Disadvantages of YAML

Scary at first, especially if you are not a developer
Harder to learn the “language”, when you are used to using the Graphical UI

Summary

Again, this list is probably far from complete, and mostly depends on your personal preferences. For me, I still see myself going often to the Classic Editor rather than using YAML, but I also try to change my behavior :). Knowing YAML is somehow becoming a standard in other tools and platforms (think of Docker, Kubernetes,…), it makes total sense to also adopt this into Azure DevOps. Next, there is a tendency to move to a anything as code (Infrastructure as Code, Configuration as Code, now Pipelines as Code,…) which allows for easier creation, change, version control and collaboration across teams. And isn’t that the ultimate idea about DevOps after all?

Ping me on Twitter or send me an Email if you want to share your feedback on this.

Stay safe and healthy you all!

/Peter