In the previous post, I had explained what Artillery is, what are the USP of the tool and we ran a simple test successfully. Now let’s do a deep dive on the test and see how we can structure the tests.
As Artillery runs a test, it will track and record performance metrics, such as API response times, throughput, and errors. As the test runs, Artillery will report those metrics so that performance of the system can be observed in real-time. A summary report is also provided at the end of the test run.
Test Structuring
Let’s see our tests from the previous post
A test definition is written as a YAML file with different customisations. The artillery CLI is then used to run that test definition.
As you can see from the yaml above, the tests can be structured into sections – test config (in the config section and then the test scenario can be defined in one or multiple scenarios in the test – as you can see a single scenario in this test.
A single scenario can have multiple api calls using http verbs like GET, POST etc – or there can be multiple scenarios, each with different http calls. We’ll look at the various sections in detail in below sections.
Virtual Users (VU)
Artillery , like it’s competitor tools like k6 is centered around the concept of virtual users.
A “virtual user” is exactly what the name suggests – a simulation of a real user (or a client) using a service or an API. Each virtual user is completely independent of other virtual users – just like in the real world. For example, when testing an HTTP-based service, each virtual user will open & maintain its own TCP connection(s), and maintain its own cookies and any other data.
For example, if you’re testing a RESTful ecommerce backend, a scenario may be composed of:
- Making a request to the search API endpoint to search for a product.
- Making a request to add one of the products returned in the results to basket.
- Loading the basket.
- Initiating the checkout flow.
config
Let’s take a look at the code above. The first section that comes up is the config section. config section lets you define the target for the tests – something on the lines of baseurl and then the workload that you want to push it in the tests.
So config contains two configuration settings:
target: This allows you to specify the base URL for all of the requests defined later in the test script.phases: Here, you’ll specify the duration of the test, how many virtual users you want to send during the test, and how often.
As mentioned above, the target key lets you define the base url for the target web service that is to be used for the tests – in a single test I have generally seen one config , which defines the target webservice.
phases
The most crucial configuration for this test is the phases setting, which grants you control over the load amount you wish to send. A load phase tells Artillery how many virtual users to create over a period of time.
The duration key enables you to specify the duration in seconds for which you want Artillery to execute the test.
The arrivalRate key informs Artillery about the number of virtual users to send to your system every second. Different settings can be applied to this configuration, as demonstrated in other examples later on.
The phases section is very important section since it let’s you define the workload for the tests – which is the main thing in the performance tests. It lets you define whether you want to created a sustained load, or you want to ramp up the users after certain interval. A config can have multiple phases.
The config section also lets you define different environments – so let’s say you have two different environments – QA and Staging, then you can run tests define this in the config section
scenarios
The scenarios setting allows you to specify one or more requests you want. A scenario in an artillery test lets you define the test scenario -in a general sense of speaking. If you have a target url defined in the config section, then inside your scenarios you can define the path that you want to use for the tests.
All tests should be written in the scenarios section and should contain:
- GET, POST, PUT, DELETE, and some other commands;
- a URL for every endpoint;
- the body text in JSON format;
- all checks you want to run.
Each scenario is a sequence of steps that run in order, as defined by the flow key. The flow key contains each step in the scenario to run. Since we’re load-testing an HTTP API, we’ll use HTTP request methods such as GET and POST to tell Artillery what to do throughout the process.
A scenario can have multiple flows and there can be multiple scenarios in a single config section.
hooks
You can also define hooks that you might want to run before each scenario. This can be done using the hooks key in the artillery yaml file. We can write the custom logic in a separate hooks.js file and then call that file in our tests in the hooks section.
Note
Each virtual user will pick and run one of the scenarios in the test definition and run it to completion. A test run ends when all virtual users finish running their scenarios. This is an important distinction to understand: the duration of a test is not the combined duration of all of its load phases, but the amount of time it takes for all virtual users created in those load phases to finish their scenarios
TESTEROPS 