In the previous post, I had explained what
Artillery is, what are the USP of the tool and we ran a simple test successfully. Now let’s do a deep dive on the test and see how we can structure the tests.
As Artillery runs a test, it will track and record performance metrics, such as API response times, throughput, and errors. As the test runs, Artillery will report those metrics so that performance of the system can be observed in real-time. A summary report is also provided at the end of the test run.
Let’s see our tests from the previous post
A test definition is written as a YAML file with different customisations. The
artillery CLI is then used to run that test definition.
As you can see from the
yaml above, the tests can be structured into sections – test config (in the
config section and then the test scenario can be defined in one or multiple scenarios in the test – as you can see a single scenario in this test.
A single scenario can have multiple api calls using
http verbs like
POST etc – or there can be multiple scenarios, each with different
http calls. We’ll look at the various sections in detail in below sections.
Virtual Users (VU)
Artillery , like it’s competitor tools like
k6 is centered around the concept of
A “virtual user” is exactly what the name suggests – a simulation of a real user (or a client) using a service or an API. Each virtual user is completely independent of other virtual users – just like in the real world. For example, when testing an HTTP-based service, each virtual user will open & maintain its own TCP connection(s), and maintain its own cookies and any other data.
For example, if you’re testing a RESTful ecommerce backend, a scenario may be composed of:
- Making a request to the search API endpoint to search for a product.
- Making a request to add one of the products returned in the results to basket.
- Loading the basket.
- Initiating the checkout flow.
Let’s take a look at the code above. The first section that comes up is the
config section lets you define the target for the tests – something on the lines of
baseurl and then the workload that you want to push it in the tests.
config contains two configuration settings:
target: This allows you to specify the base URL for all of the requests defined later in the test script.
phases: Here, you’ll specify the duration of the test, how many virtual users you want to send during the test, and how often.
As mentioned above, the
target key lets you define the base url for the target web service that is to be used for the tests – in a single test I have generally seen one
config , which defines the
The most crucial configuration for this test is the
phases setting, which grants you control over the load amount you wish to send. A load phase tells
Artillery how many virtual users to create over a period of time.
duration key enables you to specify the duration in seconds for which you want
Artillery to execute the test.
arrivalRate key informs
Artillery about the number of virtual users to send to your system every second. Different settings can be applied to this configuration, as demonstrated in other examples later on.
phases section is very important section since it let’s you define the workload for the tests – which is the main thing in the performance tests. It lets you define whether you want to created a sustained load, or you want to ramp up the users after certain interval. A
config can have multiple phases.
config section also lets you define different environments – so let’s say you have two different environments –
Staging, then you can run tests define this in the
scenarios setting allows you to specify one or more requests you want. A
scenario in an artillery test lets you define the test scenario -in a general sense of speaking. If you have a
target url defined in the
config section, then inside your
scenarios you can define the
path that you want to use for the tests.
All tests should be written in the
scenarios section and should contain:
- GET, POST, PUT, DELETE, and some other commands;
- a URL for every endpoint;
- the body text in JSON format;
- all checks you want to run.
Each scenario is a sequence of steps that run in order, as defined by the
flow key. The
flow key contains each step in the scenario to run. Since we’re load-testing an HTTP API, we’ll use HTTP request methods such as
POST to tell Artillery what to do throughout the process.
scenario can have multiple flows and there can be multiple
scenarios in a single
You can also define
hooks that you might want to run before each scenario. This can be done using the
hooks key in the artillery
yaml file. We can write the custom logic in a separate
hooks.js file and then call that file in our tests in the
Each virtual user will pick and run one of the scenarios in the test definition and run it to completion. A test run ends when all virtual users finish running their scenarios. This is an important distinction to understand: the duration of a test is not the combined duration of all of its load phases, but the amount of time it takes for all virtual users created in those load phases to finish their scenarios