When you hear the word
Artillery what does that brings to mind? Guns, ammunitions, cannons, tanks etc. right. Since the basic dictionary definition of
very large guns that are moved on wheels or metal tracks, or the part of the army that uses these
And these guns are powerful or at least used to be powerful in the earlier battles.
And so when I heard about a performance testing tool called
Artillery, I was intrigued . I am one who is fascinated by the history – particularly the wars, battle stuff. And so the concept of artillery was not alien to me. But it was very intriguing to have a testing tool, named after
artillery. So this weekend, I thought why not give it a try.
Artillery.io is a modern performance testing software that is powerful yet quite simple to navigate. It suits different types of performance testing, on local machines and as part of CI. What’s even better? – Artillery provides a free ‘Dev’ option if you want to test the waters before diving in. So, let’s make some exploration!
Artillery is a Node.js package designed for testing backend systems, such as APIs. Out of the box, it supports your standard HTTP APIs, along with protocols like WebSockets. It also integrates with the Socket.io library for Node.js applications. Artillery also sports a plugin interface to extend its functionality.
Flavors of Artillery.IO
Artillery comes in two flavors – and by flavors I mean versions. There is a normal open source version which you can use as it is – since it is a Node.JS tool, you can use it with any node project.
The team behind Artillery also provides a paid version called Artillery Pro, which contains added functionality like better integration with Amazon AWS services and distributed load testing.
For the purpose of our testing, we’ll using the free version of Artillery since it gives a host of good functions and integrations that can be used for our testing purposes.
Now most of the folks will have this question. The performance testing market is already pretty populated right now – there are obvious heavyweights like
Neoload and then there are more newcomers who are making waves in the market like
Let’s see why I thought
Artillery is good choice for performance tests.
- Node JS tool
Artillery is a node js package that allows us to define our performance tests. It has a clean CLI interface. Also since most of the tools now a days that I am using rely heavily on JS – Front end in React, Back end is Node – so it is much easier for me to use a node js package that can be installed along with the project dependencies.
- Support for various Protocols
Out of the box, Artillery.io will support:
However, you can use Artillery.io to test any stack, thanks to the plugin interface. Here are a few examples:
- Apache Kafka
- Amazon Kinesis
- HLS (HTTP Live Streaming)
- YAML format
Unlike most (if not all) it’s competitors , which rely on the use of a programming language,
Artillery uses the
YAML syntax for defining your tests. The use of
YAML makes it much easier for a lay man to understand the basic architecture of the performance tests defined. If you have a basic idea of
YAML, you can then write the tests.
- Re-usable Scenarios
Artillery’s scenarios and phases, you can build more flexible tests and cover multiple use cases in parallel. Also, it takes you just a few steps and conditions to set up tests for complex user behaviours. Finally, it’s easy to reuse scenarios for comparison and future testing.
- Plugins and Integrations
Artillery.io seamlessly integrates with CI (for example, check the integration guide for GitHub Actions), making the process very organized. It also works with Docker, so you can run your Artillery tests inside a Docker container.
Artillery comes with a rich-set of integrations called the plugins. These plugins enhance the existing functionality of the basic
- In built reporting
Artillery has in built reporting modules that help in the basic reporting of the tests. It’s not very fancy, but it does the job, in most cases, and you wouldn’t need to install any other reporting module for the reports.
Artillery is a Node.js package, it’s relatively simple to install as long as your system runs Node.js (version 12.0 or above). You can download Node.js to install it in your system if you want to follow along, or install it using my preferred method or setting up nvm (Node Version Manager).
Once you have Node.js set up, installing Artillery is a simple one-line command:
npm install -g artillery
Note : During the course of my testing, I found that if I use
artilleryinside the project module , after installing it in the project dependencies and do not install it as a global module – as done above, I get
command not foundwhen running
artillery. I was not able to debug this and installing
artilleryas a global module fixed the issue.
Now let’s run our first test – open any code editor of your choice – my choice being
VS Code, and then copy paste this code
config: target: "http://lab.artillery.io" phases: - name: "First Test" duration: 5 arrivalRate: 2 rampTo: 2 scenarios: - flow: - get: url: "/movies"
Save this as
first-test.yml and then run using this command
artillery run first-test.yml
Now see the results
And voila. You have successfully run your first test with
artillery. It’s ok if it doesn’t makes much sense right now. We’ll dissect all the information from this file in the next portion. For now, this is enough. We’ve seen what artillery is and how it is different from the existing tools. We’ve run the first test successfully.
Now let’s move on to the second part.