As a Gerrit administrator, making sure there is no performance impact while upgrading from one version to another can be difficult.

It is essential to:

• have a smooth and maintainable way to reproduce traffic profiles to stress your server
• easily interpret the results of your tests

Tools like wrk and ab are simple and good to run simple benchmarking tests, but when it comes to more complex scenarios and collection of client-side metrics, they are not the best tools to use.

Furthermore, they only support Close Workload Models, which might not always fit the behaviour of your system.

For those reasons in GerritForge we started to look at more sophisticated tools, and we started adopting Gatling, an open-source load testing framework.

The tool

The Gatling homepage describes it this way:

“Gatling is a highly capable load testing tool. It is designed for ease of use, maintainability and high performance…

Out of the box, Gatling comes with excellent support of the HTTP protocol…..

As the core engine is actually protocol-agnostic, it is perfectly possible to implement support for other protocols…

Based on an expressive DSL, the scenarios are self-explanatory. They are easy to maintain and can be kept in a version control system…”
In this article, we focus on the maintainability and protocol agnosticism of the tool.

What about Git?

Gatling natively supports HTTP protocol, but since the core engine is protocol-agnostic, it was easy to write an extension to implement the Git protocol. I started working on the Gatling git extension in August during a Gerrit hackathon in Sweden, and I am happy to see that is starting to get traction in the community.

This way we ended up among the official Gatling extension on the official Gatling homepage:

The code of the git extension if opensource and free to use. It can be found here, and the library can be downloaded from Maven central. In case you want to raise a bug, you can do it here.

Maintainability

Gatling is written in Scala, and it expects the load tests scenarios to be written in Scala. Don’t be scared; there is no need to learn crazy functional programming paradigms, the Gatling DSL does a good job in abstracting the underneath framework. To write a scenario you just have to learn the building blocks made available by the DSL.
Here a couple of snippets extracted from a scenario to understand the DSL is:

class ReplayRecordsFromFeederScenario extends Simulation {
  // Boireplate to select the protocol and import the configuration
  val gitProtocol = GitProtocol()
  implicit val conf = GatlingGitConfiguration()

  // Feeder definition: the data used for the scenario will be loaded from "data/requests.json"
  val feeder = jsonFile("data/requests.json").circular

  // Scenario definition: 
  val replayCallsScenario: ScenarioBuilder =
    scenario("Git commands") // What's the scenario's name?
      .forever {  // How many time do I need to run though the feed?
        feed(feeder) // Where shall I get my data?
          .exec(new GitRequestBuilder(GitRequestSession("${cmd}", "${url}"))) // Build a Git request
      }

  setUp(
    replayCallsScenario.inject(
      // Traffic shape definition....pretty self explanatory
      nothingFor(4 seconds),
      atOnceUsers(10),
      rampUsers(10) during (5 seconds),
      constantUsersPerSec(20) during (15 seconds),
      constantUsersPerSec(20) during (15 seconds) randomized
    ))
    .protocols(gitProtocol) // Which protocol should I use?
    .maxDuration(60 seconds) // How long should I run the scenario for?
}

That is how the feeder looks like:

[
  {
    "url": "ssh://admin@localhost:29418/loadtest-repo.git",
    "cmd": "clone"
  },
  {
    "url": "http://localhost:8080/loadtest-repo.git",
    "cmd": "fetch"
  },
  {
    "url": "http://localhost:8080/loadtest-repo.git",
    "cmd": "push",
    "ref-spec": "HEAD:refs/for/master"
  }
]

Here another example reproducing the creation of a WIP change using the REST API:

class WIPWorkflow extends Simulation {

  // Configuration bolierplate
  implicit val conf: GatlingGitConfiguration = GatlingGitConfiguration()
  val baseUrl = "https://review.gerrithub.io"
  val username: String = conf.httpConfiguration.userName
  val password: String = conf.httpConfiguration.password
  val httpProtocol: HttpProtocolBuilder = http
    .baseUrl(baseUrl)
    .userAgentHeader("Gatling test")
  val request_headers: Map[String, String] = Map(
    "Content-Type" -> "application/json"
  )
  
  val scn = scenario("WIP Workflow") // What's the name of my scenario?
    .exec(
      http("Create WIP change")
        .post("/a/changes/") // Which url and which HTTP verb should I use? 
        .headers(request_headers) 
        .basicAuth(username, password) // How do I authenticate?
        .body( // What's the body of my request?
          StringBody("""{
            "project" : "GerritForge/sandbox/e2e-tests",
            "subject" : "Let's test this Gerrit! Create WIP changes!",
            "branch" : "master",
            "work_in_progress": "true",
            "status" : "NEW"
          }""")
        )
        .check(status.is(201)) // What's the response code I expect?
    )

  setUp(scn.inject(
    atOnceUsers(1) // Traffic profile
  )
  ).protocols(httpProtocol)
}

Jenkins integration

Running load tests can be tedious and time-consuming, but yet essential to spot any possible performance regression in your application.
Providing the least possible friction is essential to incentivize people in running them. If you are already using Jenkins in your company, you can leverage the Gatling plugin to scale your load quickly and provide easy access to metrics.

A real use case: Gerrit v3.0 Vs Gerrit v3.1 load test…in production!

Let’s go through a real case scenario to show how useful and easy to read are the metrics provided by Gatling.

The closest environment to your production one is…production!

gerrithub.io runs Gerrit in a multi-site configuration, and this gives us the luxury of doing canary releases, only upgrading a subset of the master nodes running Gerrit. One of the significant advantages is that we can run A/B tests in production.

That allows us also to run meaningful load tests against the production environment. See below a simplified picture of our Gerrit setup where it is possible to see the canary server with a higher Gerrit version.

We ran against 2 servers in Germany the same load tests which:

• Create 100 chained Change Sets via REST API
• Submit all the changes together via REST API

We then compared the server-side and client-side metrics to see if a good job has been done with the latest Gerrit version.

Server-side metrics

Server-side metrics come from the Prometheus exporter plugin. The image is showing the HTTP requests mean time:

We can see the improvement in the latest Gerrit version. The mean requests time is almost halved and, of course, the overall duration is decreased.

Client-side metrics

Let’s see what is going on on the client-side using the metrics provided by Gatling.

Among all the metrics we are going to focus on one step of the test, since we have more data points about it, the creation of the change:

We can already see from the overall report the reduction of the response time in the latest Gerrit version:

If we look in-depth to all the response times, we can see that the distribution of the response times is pretty much the same, but the scale is different….again we confirmed the result we previously encountered.

What can we say…Good job Gerrit community!

Wrapping up

(You can see my presentation about Gatling in the last Gerrit User summit in Sunnyvale here <- add this when the talk will be sharable)

I have touched superficially several topics in this blog posts:

• simplicity and maintainability provided by the Gatling DSL
• Integration with Jenkins
• Gatling extensions and reuse of the statistic engine
• Example scenarios

I would like to write more in-depth about all these topics in some follow-up blog posts. Feel free to vote for the topic you are more interested in or suggest new ones in the comments section of this post.
If you need any help in setting up your scenario or understand how to run load tests against your Gerrit installation effectively, GerritForge can help you.

Fabio Ponciroli (aka Ponch) – GerritForge
Gerrit Code Review and Gatling Contributor