Testing Infrastructure with Pester

This post is about using Pester for infrastructure validation (Skype for Business is used for the example infrastructure).

So what is Pester? They describe it very well on their wiki:  “Pester provides a framework for running Unit Tests to execute and validate PowerShell commands. Pester follows a file naming convention for naming tests to be discovered by pester at test time and a simple set of functions that expose a Testing DSL for isolating, running, evaluating and reporting the results of PowerShell commands.”

Really, Pester is meant to provide unit testing for Powershell scripts in the spirit of Test-Driven Development (TDD).

However, it can also be used for generic validation of object properties. In the case of infrastructure validation this means getting a setting, property, or the status of something and comparing the actual value to the expected value.

This can be a useful step after deploying or modifying infrastructure.

(if you don’t care about any of this and just want to see an example, check out the example I go through in my Github repo)

Getting Started

Fortunately it’s very easy to start using Pester. It’s included by default in WMF 5, meaning Windows 10 and Windows Server 2016. If you do not already have Pester you can grab it from Github here.

Whether you are using Pester for TDD or infrastructure validation determines how you actually use pester. In the case of TDD, you have a separate test file and execute it against another file which contains the functions you want to test.

In the case of infrastructure validation, what I’ll be doing in this post, you only have the test file itself.

Within this test file you will have a series of organized tests.

Describe, Context, and It blocks

Describe is used for describing and grouping your sets of tests. For example, it might be used to describe something like server health. Under describe, there might be multiple context blocks which are then used to group together a handful of it blocks (which contain the actual tests).

As an example, let’s say that I want to check the general health of my server. I want to know the disk health of each disk and check that the required ports are open:

Describe "Check Server Health" {
    Context "Disk Health" {
        it "Disk C has sufficient space" {
            (Get-DiskSpace -Disk "A").SpaceByPercentage | should BeGreaterThan '10'
        }
        it "Disk D has sufficient space" {
            (Get-DiskSpace -Disk "D").SpaceByPercentage | should BeGreaterThan '10'
        }
    }
    Context "Port Status" {
        foreach ($port in $required_ports) {
            it "$port is open" {
                Get-Port $port | should be "Open"
            }
        }
    }
}

The describe block is used to keep this group of tests all under the “Check Web Server Health”, the context block is used to group together the types of health checks and then the it blocks are used to run individual tests.

If I wanted to run the actual test I would simply run the following Powershell command

Invoke-Pester .\web_server_tests.ps1

Real Example

I’ll be going through an example of some basic tests you could do with Skype for Business. I’m only using SfB as the example because it’s a product that I already have deployed in my home lab and I know some things I could test.

Some of the tests done in this example:

• Checking DNS entries
• Checking open ports
• Checking service states
• Checking Database versions
• Checking database failover state
• Runs synthetic transactions to perform functional tests

I also want to make a quick disclaimer: This post is focused on using Pester, not writing good Powershell. This was all written pretty awful and I cut some corners. The point of this post is to get a feel for how to use Pester.

The example I’ll be going over can be found on Github here.

The first portion can be mostly ignored. It’s mostly just information used for the tests, such as credentials, sessions (required for some of the sfb commands), server information, services to check, etc.

If I was going to make a script actually used for production purposes, I would likely want to pull that information from some place and definitely not keep in hard-coded in the test.

So let’s look at the first test: the front end services tests

# check that services are running
Describe "Skype for Business Front End Health" {
    Context "Front End Services" {
        foreach ($server in $fe_servers) {
            foreach ($service in $fe_services) {
                it "$service on $server is running" {
                    (Get-CsWindowsService -ComputerName $server -Name $service -ErrorAction SilentlyContinue).Status | should be "Running"
                }
            }
        }
    }
}

This test starts off by explaining what the group of tests will be trying to achieve. Similar to the earlier example, there are likely plenty of sub-categories which could be added which cumulatively make the “front end health”.

With that in mind, I created a ‘context’ which represents of these health aspects — services. I want to make sure that all of the SfB-related services are in the appropriate state. If I wanted to add another aspect of front end health, all I need to do is add another context block and add the additional tests.

There are a couple of standard Powershell loops here as well, one for looping through each server and another for lopping through each of these SfB services. Then in each loop iteration I will run a specific test within the ‘it’ block.

The test is very straight-forward: get the service and check its status property. Then compare that value to what it should be. In this case, I want all of the services to be running. (‘should be’ is not the only operator in Pester, you can see the other potential assertions here)

I can run the test by doing

Invoke-Pester .\Validate-SkypeForBusiness.ps1

This is what it will look like when I run it

I am using VS Code’s integrated terminal to run the tests which is the reason for the odd background color, but the point is that successful tests will show as green with plus signs next to them.

Also, notice that the output includes the describe, context, and it block’s test descriptions as well as slight indentation.

What’s more interesting is what happens when tests fail. Further down I have some database tests. One of the tests checks the database version and compares it to what it should be. Additionally, it checks that some of the important SQL ports are open, including the default port used for a SQL witness.

# SQL, db version, ports, database mirroring primary on primary
Describe "SQL Databases" {
    Context "Check Database Versions" {
        $db_results = Test-CsDatabase -ConfiguredDatabases -SqlServerFqdn $be_servers
        foreach ($db in $db_results) {
            It "$($db.DatabaseName) database version is correct" {
                [string]$installed_db_version = "$($db.InstalledVersion.SchemaVersion).$($db.InstalledVersion.SprocVersion).$($db.InstalledVersion.UpgradeVersion)"
                [string]$expected_db_version = "$($db.ExpectedVersion.SchemaVersion).$($db.ExpectedVersion.SprocVersion).$($db.ExpectedVersion.UpgradeVersion)"
                $installed_db_version | should be $expected_db_version
            }
        }
    }
    Context "Check SQL Ports" {
        $sql_ports = @('1433', '5022', '7022')
        foreach ($port in $sql_ports) {
            it "Server is listening on port $port" {
                $port_state = Invoke-Command -Session $be_session -ScriptBlock {param($port)(Get-NetTCPConnection -LocalPort $port -ErrorAction SilentlyContinue).State} -ArgumentList $port
                $port_state.Value -contains "Listen" | should be $true
            }
        }
    }
    Context "Check Mirror Status" {
        $db_state = Get-CsDatabaseMirrorState -PoolFqdn $fe_pools
        foreach ($db in $db_state) {
            it "$($db.DatabaseName) database is on primary" {
                $db.StateOnPrimary -eq "Principal" | should be true
            }
        }
    }
}

And when I run the test:

Pester makes it very obvious when there are issues. In addition to the minus sign and the red text color, it actually tells you what the value is versus what was expected.

Running this test just showed me that one of databases failed to update and that my SQL server is not listening on the default witness port. This helped me determine that I needed to perform an upgrade and it reminded me that I forgot to deploy a witness server.

Other Tests

Towards the end of these tests, I threw in some SfB Synthetic transactions as a way to really tests the functionality of the stack rather than just checking the properties of the environment

I thought this would be a good way to validate things are working after applying patches or a cumulative update.

You could really get creative with these tests such as

• Invoke a web request against the load balancer and make sure you are getting a 200 OK back or even some specific data
• Check Event Viewer for a certain type of entry (fail the test if a log matches)
• Validate the appropriate software and its dependencies are installed
• Verify that the correct databases are running for a given instance
• Check disk space (free and total)

Additionally, outputting the results to the console is not the only option. The results of these tests can be structured as something called NUnit XML which many CI tools can understand.

For example, you could kick off the tests and display the results with something like Jenkins, Azure Automation, Visual Studio Team Services, AppVeyor, or TeamCity .

In my case, I tested out the setup of a hybrid worker and kicked of the test as an Azure Runbook to the on-premise worker agent which ran the tests. The results of the test were then sent to Azure and OMS for viewing (this was for fun, but didn’t prove to be a great way to do it).

Why not just perform these checks without Pester?

At this point you are probably asking yourself “why do I need to use Pester, why don’t I just run these checks with normal Powershell commands?”

First of all, as I mentioned above, the output for these tests will easily tie into a number of tools. That is huge and plays a critical role in the automation that you should be trying to achieve.

Second, this is a way to standardize your testing practices. It can be significantly easier/quicker to troubleshoot if everyone is using the same framework and same methodology for performing their tests.

And lastly, the most obvious reason is that it’s just faster. The framework already exists and is well supported. There is no reason not to use it for your testing.

Ok….but why do I really need Pester?

Well you don’t. But you do need an automated testing framework for your infrastructure and Pester happens to be the best way to do that in Powershell.

There is far more to this than just running a script which outputs some results to a screen. The big picture is that in whatever you execute you need to be sure that it actually happened — especially at scale.

Services are becoming more distributed, more modular, and overall just more complicated. In addition to this complexity, the scale at which services need to grow is incredible. In order to keep up with the pace, tools have been created to help automate and keep this new world we live in.

Automating deployment, monitoring, patching, updating, and all of the other operational tasks is pointless (and potentially dangerous) if you don’t know that the automation is actually working the way that it should.

If you want to deploy 50 services to Azure at once you need a way to validate that all of them are actually working how you would expect. Testing frameworks are the way that you ensure that things are the way they should be in a scalable fashion.

Automated testing should be inserted into your environment so that any time a piece of the infrastructure is modified, the tests are performed. This is especially important when it comes to provisioning cloud services but just as applicable when taking the ‘infrastructure as code’ approach to on-prem infrastructure.

Final Thoughts

In the words of Jeffrey Snover: “Pester is an important skill that every PowerShell user should master.”

Whether for testing code or validating infrastructure, an automated testing framework is a key component of continuous integration. Microsoft actually outlines the role of automated testing (using Pester as their example) within this fantastic whitepaper from 2016.

There is much more to Pester than what I show here. If you would like to learn more you can check out the documentation on Github and/or check out the Pester book (which receives continual updates).

Advertisement