Performance Tests with Artillery

When you hear the word Artillery what does that brings to mind? Guns, ammunitions, cannons, tanks etc. right. Since the basic dictionary definition of artillery is

very large guns that are moved on wheels or metal tracks, or the part of the army that uses these

And these guns are powerful or at least used to be powerful in the earlier battles.

And so when I heard about a performance testing tool called Artillery, I was intrigued . I am one who is fascinated by the history – particularly the wars, battle stuff. And so the concept of artillery was not alien to me. But it was very intriguing to have a testing tool, named after artillery. So this weekend, I thought why not give it a try.

Artillery.IO

Artillery.io is a modern performance testing software that is powerful yet quite simple to navigate. It suits different types of performance testing, on local machines and as part of CI. What’s even better? – Artillery provides a free ‘Dev’ option if you want to test the waters before diving in. So, let’s make some exploration!

Artillery is a Node.js package designed for testing backend systems, such as APIs. Out of the box, it supports your standard HTTP APIs, along with protocols like WebSockets. It also integrates with the Socket.io library for Node.js applications. Artillery also sports a plugin interface to extend its functionality.

An important distinction to make with Artillery is that it’s not designed for testing web applications through the browser or client-side JavaScript. If you need to do load testing on a web application through the browser, you’ll need a tool focused on UI testing or built to do web performance testing. For example, you can do performance testing using TestCafe or use a more robust load testing tool like Apache JMeter.

Flavors of Artillery.IO

Artillery comes in two flavors – and by flavors I mean versions. There is a normal open source version which you can use as it is – since it is a Node.JS tool, you can use it with any node project.

The team behind Artillery also provides a paid version called Artillery Pro, which contains added functionality like better integration with Amazon AWS services and distributed load testing.

For the purpose of our testing, we’ll using the free version of Artillery since it gives a host of good functions and integrations that can be used for our testing purposes.

Why Artillery?

Now most of the folks will have this question. The performance testing market is already pretty populated right now – there are obvious heavyweights like Jmeter, Loadrunner, Neoload and then there are more newcomers who are making waves in the market like K6, Gatling, locust.

So why Artillery?

Let’s see why I thought Artillery is good choice for performance tests.

• Node JS tool

Artillery is a node js package that allows us to define our performance tests. It has a clean CLI interface. Also since most of the tools now a days that I am using rely heavily on JS – Front end in React, Back end is Node – so it is much easier for me to use a node js package that can be installed along with the project dependencies.

• Support for various Protocols

Out of the box, Artillery.io will support:

• HTTP
• Socket.IO
• WebSocket

However, you can use Artillery.io to test any stack, thanks to the plugin interface. Here are a few examples:

• Apache Kafka
• Amazon Kinesis
• HLS (HTTP Live Streaming)

• YAML format

Unlike most (if not all) it’s competitors , which rely on the use of a programming language, Artillery uses the YAML syntax for defining your tests. The use of YAML makes it much easier for a lay man to understand the basic architecture of the performance tests defined. If you have a basic idea of YAML, you can then write the tests.

• Re-usable Scenarios

With Artillery’s scenarios and phases, you can build more flexible tests and cover multiple use cases in parallel. Also, it takes you just a few steps and conditions to set up tests for complex user behaviours. Finally, it’s easy to reuse scenarios for comparison and future testing.

• Plugins and Integrations

Artillery.io seamlessly integrates with CI (for example, check the integration guide for GitHub Actions), making the process very organized. It also works with Docker, so you can run your Artillery tests inside a Docker container.

Also Artillery comes with a rich-set of integrations called the plugins. These plugins enhance the existing functionality of the basic artillery tool.

• In built reporting

Artillery has in built reporting modules that help in the basic reporting of the tests. It’s not very fancy, but it does the job, in most cases, and you wouldn’t need to install any other reporting module for the reports.

Installation

Since Artillery is a Node.js package, it’s relatively simple to install as long as your system runs Node.js (version 12.0 or above). You can download Node.js to install it in your system if you want to follow along, or install it using my preferred method or setting up nvm (Node Version Manager).

Once you have Node.js set up, installing Artillery is a simple one-line command:

npm install -g artillery

Note : During the course of my testing, I found that if I use artillery inside the project module , after installing it in the project dependencies and do not install it as a global module – as done above, I get command not found when running artillery. I was not able to debug this and installing artillery as a global module fixed the issue.

First Test

Now let’s run our first test – open any code editor of your choice – my choice being VS Code, and then copy paste this code

config:
  target: "http://lab.artillery.io"
  phases:
    - name: "First Test"
      duration: 5
      arrivalRate: 2
      rampTo: 2

scenarios:
  - flow:
    - get:
        url: "/movies"

Save this as first-test.yml and then run using this command

artillery run first-test.yml

Now see the results

And voila. You have successfully run your first test with artillery. It’s ok if it doesn’t makes much sense right now. We’ll dissect all the information from this file in the next portion. For now, this is enough. We’ve seen what artillery is and how it is different from the existing tools. We’ve run the first test successfully.

Now let’s move on to the second part.