Web performance and load testing with Visual Studio1st July 2019
Any time you want to release a new product or application to your customers, it's important to ensure that your software & hardware infrastructure is able to handle the expected user load smoothly, meaning that at the maximum expected request rate, your application behaves normally and serve responses back to the users. The best way to test this and be prepared for unexpected behaviors is through stress tests. Stress tests can reveal bugs in all levels of your infrastructure, such as concurrency issues on application level or dead locks in databases. Luckily you can use Visual Studio to create and load test your applications. This post will show you the basic steps to create a load test and stress test an API endpoint. More specifically we will see:
- Installation requirements
- Create and configure a web test
- Create and configure a load test
- Run stress tests with parameters
Load tests are only available in Enterprise Versions of Visual Studio, 2017 or 2019 will be fine. After installing an Enterprise version of Visual Studio open the installer and click Individual components. Under the Debugging and testing section make sure that Web performance and load testing tools is checked and if not, check it and click Modify to install it.
Open Visual Studio as Administrator and create a new project by selecting the Web Performance and Load Test Project template. The solution should look like this.
Let's assume that we have a simple API that returns a list of Product items as follow:
Create and configure a web test
A web test defines what you want to stress test. In the test project right click WebTest and select Add Request
The default request being created points to http://localhost/ so right click it, select properties and set the URL property to the URL you want to stress test
At this point you can test that your web test request is valid by clicking the Run test option on the upper left menu.
Make sure you the API under stress test is running and fire up the web test request. Confirm that the request run successfully.
Create and configure a load test
A load test defines how you wish to run a web test, meaning the configuration to be used through the stress test. Right click the Project (not the solution) and select Add New Item. Select the Test option on the left and click the Load Test option. Clicking the Add button will open the New Load Test Wizard where you will configure your new load test. All the settings are pretty much explanatory but let's pass and configure the most important.
- Welcome: Select On-premises Load Test
- Run Settings: Here you can set how you wish to run the test, whether you want to run it for a period of time or run exact number of test iterations. Select Test iterations and leave the default 100 value
- Load Pattern: The load pattern defines the number of concurrent users used during the test. This option won't affect your test if you have selected the Test iterations option on the previous step
- Test Mix: Here you define the web test(s) you wish the load test to use which in our case is the web test created before. Click the Add.. button, add it to the selected tests list and click OK
As you have probably understood, you can add more that one tests to your load test. Click Finish and Visual Studio will create the Load Test for you. You can re-configure all the settings applied before by selecting the appropriate option on the load test.
Start the load test in the same way you did with the web test, by selecting Run Load Test on the upper left menu. When the test finishes you see the result summary as follow.
This is a simple result summary that displays important information about how the API behaved such as Average Page Time or Request per second. Most of the times you will need to run a load test for a specific time period by incrementally increasing the load to your app. During the load test you will be able to see the total request per second, failed tests, total concurrent users etc.. To simulate this scenario you need to re-configure your load test. First let's configure the load test to start with 10 users and incrementally adding 10 every 10 seconds till 200 users reached. Select Step Load Pattern on the Load Test options and then set the Pattern to Step as follow:
Select Run Settings on the left list and change the Use Test Iterations to false. Also the Storage Type should be equal to None.
Next we set the duration to 5 minutes in order to allow the test to reach the maximum 200 users. I have changed a little bit the Products API code to produce random latencies and also randomly throw exceptions.
Let's run and see the results.
As you can see there are many requests that failed and the average response time was ~2.5 seconds which makes sense since we were randomly sleeping between 0-5 seconds.
Run stress tests with parameters
Some times your API calls you wish to stress test will require some parameters, e.g. query string parameters. This is totally fine and something that you can easily configure on your web test level. Let's assume we have the following API call.
Right click the test project and add a new Web Performance Test. Add a new request pointing to the new API action.
For the product name query string parameter we will use an CSV file containing a list of product names. I created a Google Spread Sheet and in a column named ProductName added 180 records of product names (the first 100 are valid products). Then saved the sheet as a Comma-separated values file (CSV).
Back in the project, find and click the Add Source button on the upper menu. It's the one with the plus sign.
Name the data source, select CSV File and click Next.
Select the CSV file you created before and confirm that the Wizard can preview the data contained.
When asked if you wish to add the file as a project item select Yes. After completing the source wizard you will see it on the webtest as well.
While still on the webtest, right click the request and select the Add URL QueryString Parameter option.
Set the parameter name to name or whatever your API action requires and on the value field, find and select the right column from the data source added.
The webtest should look like this now.
Confirm that the webtest has been setup properly by running a test run. In the following screenshot you can see that the test used the first row from the CSV file.
Back to the load test, you can remove the previous WebTest from the Test Mix and add the new one. You can right click on the Text Mix item and select Edit Test Mix. Then you can either remove or specify the distribution for each web test contained on the load test. In our case we will just remove the previous one.
Now we are ready to run the load test which on every request to the GetProduct action will use a row value from the products CSV data source and add it as a query parameter to the request. The CSV contains more that 100 product names so we expect that some of the requests will fail. Let's run it!
We didn't have any latency this time as expected but we did get errors since the CSV contained 180 products where only 100 were valid.
That's it we finished! We have seen the basics on how to get started with Visual Studio Web Performance tools and stress testing APIs. Stress tests are important whether you know the expected load on your systems and you need to ensure that you can handle the load or you don't know the expected load but you want to be proactive and know the limits of your system, so that you can take action when required.