Gerrit v3.4.0-rc2: weekly update

Merged Changes

From Milutin Kristofic:

    Fix underlining on hover for subject column in dashboard
    Change-Id: Ie0c55bbe550f1714f1f9f630ac81f6c686184221

    Revert "A11y - show outline when focused on tab title in change view"    
    Change-Id: I6d8f74c4d01004a88b0dbf207d9d9741a6965755

From Mike Frysinger:

    Fix typo in hashtags docs
    Change-Id: Idabd8994651451183472eb3b777064e891e8a765

From Dmitrii Filippov:

    Do not reload page after clicking Cancel in apply-fix dialog    
    Change-Id: I5830caac655df9d3dc7058dc9674ba727685012b

From Paladox:

    Add undefined check for access[repo] in getRepoAccess
    Change-Id: I16f3344c9f11cb230adafa7fdc8457ac37b4ae70

    Add web links to project:<project>
    Change-Id: Iefda4bb5571d917d5fa9f24c02c8e4a3be2c6c52

From Dhruv Srivastava:

    Turn off syntax highlighting in comment context if disabled
    Change-Id: I89625a9b15678c1a63543d36f9f7d835bb39b359

    Allow comment context to take up remaining width 
    Change-Id: I1dab57df26b0fed98efead1a18083d9a504ed40f

    Make the line length indicator color clearer
    Issue: Bug 13648
    Change-Id: I523d971f39db477d67e212d1175ec6f5be8be24d

    Remove Add/No Patchset description label for merged changes
    Change-Id: I18643bcf01e10177a28411089ac1b312222d3e23

From David Ostrovsky:

    Bazel: Disable worker multiplexer to avoid sporadic build failures
    Change-Id: I2265c4ed7128a4b6ed259f44c5594ab717de58b0

From Ben Rohlfs:

    Allow line wrapping in check result messages
    Change-Id: Icedc05bbc28fbe513c60a5eb9853dadde636381b

Issues Fixed

  • Issue 13648: Make the line width guide in commit description to be darker

Issues Raised

Gatling E2E tests results

Git Protocol Simulation:

Gatling full results:
https://gerrit-ci.gerritforge.com/job/gatling-gerrit-test/305/gatling/report/gerritgitsimulation-20210419190719877/source/index.html

Gatling simulation class:
https://github.com/GerritForge/gatling-sbt-gerrit-test/blob/master/src/test/scala/gerritforge/GerritGitSimulation.scala

Gerrit UI REST Simulation

Gatling full results:
https://gerrit-ci.gerritforge.com/job/gatling-gerrit-test/305/gatling/report/gerritrestsimulation-20210419190957647/source/index.html

Gatling simulation class:
https://github.com/GerritForge/gatling-sbt-gerrit-test/blob/master/src/test/scala/gerritforge/GerritRestSimulation.scala

Gatling 10-days trend

Gerrit v3.4.0-rc1: weekly update

Starting from this week, GerritForge provides a weekly status update of how the Gerrit v3.4.0 release plan is progressing:

  1. List of merged changes since the previous RC
  2. List of issues fixed
  3. Result of the Gatling E2E tests on AWS

We hope that having these regular updates would help you focus on what is changing during the release plan execution and do more research, learning or specific investigation on the areas that are more pertinent to your use case.

Also, because the Gerrit release candidates do not come with an associated set of release notes, the list below would help people understand the new functionalities or fixes coming through every week.

Both Gatling-Git and the AWS-Gerrit project with a complete production-ready setup are projects started by GerritForge and contributed as OpenSource to the Gerrit community.

Merged changes

From Matthias Sohn:

    Log memory allocated per command in httpd_log 
    Change-Id: Ie0ce1382a8515e6dfb7d0d3fe10b3e64c0cf9aee

    Log cpu usage per http request   
    Change-Id: I9e78bed5219f9baf57a2b76f0f947efff334ffe5

    Log memory allocated per command in sshd_log
    Change-Id: Ifc1d274bf42eb3cb9b2cf46271b6be0117aa8b18

    Add metrics for monitoring Java memory pools
    Change-Id: I60e5960899c0cff8c05983d299b414d7a646bb07

    Log cpu usage in sshd_log
    Change-Id: I1c53f64caf982c2f85195e6bda4c6d790f79a810

    Encapsulate fields of SshScope.Context
    Change-Id: If989630425ad40922aaf8958c4335aab0bb5c2c9

    Log "-" for missing log fields in sshd_log
    Change-Id: I90adc7618864f702b42029ab596c6014bd4c6cfe

From Ben Rohlfs:

    Remove backend support for HTML UI plugins
    Change-Id: I44cc0d15910937de7e1f9b9780a799d4b85b0673

    Stop producing html version of plugins
    Change-Id: I1036f06e385f2997f7bea849755729df2789acaa

From Milutin Kristofic:

    A11y - show outline when focused on tab title in change view
    Change-Id: Ie9456a5d886e70a77dae8f055a54a3a1a0045daf

    A11y - show outline when focused on change subject in dashboard
    Change-Id: I9c73e49de661a17928c6f96a290c2069503bdfb4

    A11y - fix and improve label when navigating dashboard
    Change-Id: Id01302aea38c783687443401290995bdd0764126

From Hermann Loose:

    Allow setting image viewer max-width and max-weight externally
    Change-Id: Ie151a85d8ede5cb7fa0899d9367ca0dccd887538

From Han-Wen Nienhuys:

    Fix meta_diff documentation
    Change-Id: I9c59a4857724cdfd59b995a6dc255f77d29b017e

From Paladox:

    Update plugins/codemirror-editor and plugins/delete-project
    Change-Id: Ieca7d2e36b9eefffe5c830962109e1fa62134b5c

    gr-confirm-move-dialog: Fix _getProjectBranchesSuggestions
    Change-Id: I8d4ead0bf3a8c30d0ecefdc190e4ba4ea7ede29d

    Remove unused _handleDropdownTap
    Change-Id: Ie4a8effd6d814985cafa6dea9239903f503dae33

    Remove unused _handleDropdownTap
    Change-Id: Ie4a8effd6d814985cafa6dea9239903f503dae33

From Dhruv Srivastava:

    Clean up upload change help dialog
    Change-Id: I0c72450c37326ec2c2922b74928e0b059df0043e

From Saša Živkov:

    Fix binding of DELETE REST calls from plugins
    Change-Id: I9b9632e8f719937e5f7c61466996be79e6f29c14

Issues fixed

  • Issue 14335: CodeMirror plugin broken
  • Issue 14127: REST API DELETE query for delete-project plugin doesn’t work

Gatling E2E tests results

Git Protocol Simulation

Gatling full results:
https://gerrit-ci.gerritforge.com/job/gatling-gerrit-test/287/gatling/report/gerritgitsimulation-20210412213548212/source/index.html

Gatling simulation class:
https://github.com/GerritForge/gatling-sbt-gerrit-test/blob/master/src/test/scala/gerritforge/GerritGitSimulation.scala

Gerrit UI REST Simulation

Gatling full results:
https://gerrit-ci.gerritforge.com/job/gatling-gerrit-test/287/gatling/report/gerritrestsimulation-20210412213912089/source/index.html

Gatling simulation class:
https://github.com/GerritForge/gatling-sbt-gerrit-test/blob/master/src/test/scala/gerritforge/GerritRestSimulation.scala

Stress your Gerrit with Gatling

As a Gerrit administrator, making sure there is no performance impact while upgrading from one version to another can be difficult.

It is essential to:

  • have a smooth and maintainable way to reproduce traffic profiles to stress your server
  • easily interpret the results of your tests

Tools like wrk and ab are simple and good to run simple benchmarking tests, but when it comes to more complex scenarios and collection of client-side metrics, they are not the best tools to use.

Furthermore, they only support Close Workload Models, which might not always fit the behaviour of your system.

For those reasons in GerritForge we started to look at more sophisticated tools, and we started adopting Gatling, an open-source load testing framework.

The tool

The Gatling homepage describes it this way:

“Gatling is a highly capable load testing tool. It is designed for ease of use, maintainability and high performance…

Out of the box, Gatling comes with excellent support of the HTTP protocol…..

As the core engine is actually protocol-agnostic, it is perfectly possible to implement support for other protocols…

Based on an expressive DSL, the scenarios are self-explanatory. They are easy to maintain and can be kept in a version control system…”
In this article, we focus on the maintainability and protocol agnosticism of the tool.

What about Git?

Gatling natively supports HTTP protocol, but since the core engine is protocol-agnostic, it was easy to write an extension to implement the Git protocol. I started working on the Gatling git extension in August during a Gerrit hackathon in Sweden, and I am happy to see that is starting to get traction in the community.

This way we ended up among the official Gatling extension on the official Gatling homepage:

Screenshot 2019-12-16 at 15.03.32.png

The code of the git extension if opensource and free to use. It can be found here, and the library can be downloaded from Maven central. In case you want to raise a bug, you can do it here.

Maintainability

Gatling is written in Scala, and it expects the load tests scenarios to be written in Scala. Don’t be scared; there is no need to learn crazy functional programming paradigms, the Gatling DSL does a good job in abstracting the underneath framework. To write a scenario you just have to learn the building blocks made available by the DSL.
Here a couple of snippets extracted from a scenario to understand the DSL is:

class ReplayRecordsFromFeederScenario extends Simulation {
  // Boireplate to select the protocol and import the configuration
  val gitProtocol = GitProtocol()
  implicit val conf = GatlingGitConfiguration()

  // Feeder definition: the data used for the scenario will be loaded from "data/requests.json"
  val feeder = jsonFile("data/requests.json").circular

  // Scenario definition: 
  val replayCallsScenario: ScenarioBuilder =
    scenario("Git commands") // What's the scenario's name?
      .forever {  // How many time do I need to run though the feed?
        feed(feeder) // Where shall I get my data?
          .exec(new GitRequestBuilder(GitRequestSession("${cmd}", "${url}"))) // Build a Git request
      }

  setUp(
    replayCallsScenario.inject(
      // Traffic shape definition....pretty self explanatory
      nothingFor(4 seconds),
      atOnceUsers(10),
      rampUsers(10) during (5 seconds),
      constantUsersPerSec(20) during (15 seconds),
      constantUsersPerSec(20) during (15 seconds) randomized
    ))
    .protocols(gitProtocol) // Which protocol should I use?
    .maxDuration(60 seconds) // How long should I run the scenario for?
}

That is how the feeder looks like:

[
  {
    "url": "ssh://admin@localhost:29418/loadtest-repo.git",
    "cmd": "clone"
  },
  {
    "url": "http://localhost:8080/loadtest-repo.git",
    "cmd": "fetch"
  },
  {
    "url": "http://localhost:8080/loadtest-repo.git",
    "cmd": "push",
    "ref-spec": "HEAD:refs/for/master"
  }
]

Here another example reproducing the creation of a WIP change using the REST API:

class WIPWorkflow extends Simulation {

  // Configuration bolierplate
  implicit val conf: GatlingGitConfiguration = GatlingGitConfiguration()
  val baseUrl = "https://review.gerrithub.io"
  val username: String = conf.httpConfiguration.userName
  val password: String = conf.httpConfiguration.password
  val httpProtocol: HttpProtocolBuilder = http
    .baseUrl(baseUrl)
    .userAgentHeader("Gatling test")
  val request_headers: Map[String, String] = Map(
    "Content-Type" -> "application/json"
  )
  
  val scn = scenario("WIP Workflow") // What's the name of my scenario?
    .exec(
      http("Create WIP change")
        .post("/a/changes/") // Which url and which HTTP verb should I use? 
        .headers(request_headers) 
        .basicAuth(username, password) // How do I authenticate?
        .body( // What's the body of my request?
          StringBody("""{
            "project" : "GerritForge/sandbox/e2e-tests",
            "subject" : "Let's test this Gerrit! Create WIP changes!",
            "branch" : "master",
            "work_in_progress": "true",
            "status" : "NEW"
          }""")
        )
        .check(status.is(201)) // What's the response code I expect?
    )

  setUp(scn.inject(
    atOnceUsers(1) // Traffic profile
  )
  ).protocols(httpProtocol)
}

Jenkins integration

Running load tests can be tedious and time-consuming, but yet essential to spot any possible performance regression in your application.
Providing the least possible friction is essential to incentivize people in running them. If you are already using Jenkins in your company, you can leverage the Gatling plugin to scale your load quickly and provide easy access to metrics.

A real use case: Gerrit v3.0 Vs Gerrit v3.1 load test…in production!

Let’s go through a real case scenario to show how useful and easy to read are the metrics provided by Gatling.

The closest environment to your production one is…production!

gerrithub.io runs Gerrit in a multi-site configuration, and this gives us the luxury of doing canary releases, only upgrading a subset of the master nodes running Gerrit. One of the significant advantages is that we can run A/B tests in production.

That allows us also to run meaningful load tests against the production environment. See below a simplified picture of our Gerrit setup where it is possible to see the canary server with a higher Gerrit version.

Screenshot 2019-12-14 at 16.26.52.png

We ran against 2 servers in Germany the same load tests which:

  • Create 100 chained Change Sets via REST API
  • Submit all the changes together via REST API

We then compared the server-side and client-side metrics to see if a good job has been done with the latest Gerrit version.

Server-side metrics

Server-side metrics come from the Prometheus exporter plugin. The image is showing the HTTP requests mean time:

Screenshot 2019-12-14 at 16.32.12.png

We can see the improvement in the latest Gerrit version. The mean requests time is almost halved and, of course, the overall duration is decreased.

Client-side metrics

Let’s see what is going on on the client-side using the metrics provided by Gatling.

Among all the metrics we are going to focus on one step of the test, since we have more data points about it, the creation of the change:

Screenshot 2019-12-14 at 16.36.45.png

We can already see from the overall report the reduction of the response time in the latest Gerrit version:

Screenshot 2019-12-14 at 16.38.05.png

If we look in-depth to all the response times, we can see that the distribution of the response times is pretty much the same, but the scale is different….again we confirmed the result we previously encountered.

Screenshot 2019-12-14 at 16.39.17.png

What can we say…Good job Gerrit community!

Wrapping up

(You can see my presentation about Gatling in the last Gerrit User summit in Sunnyvale here <- add this when the talk will be sharable)

I have touched superficially several topics in this blog posts:

  • simplicity and maintainability provided by the Gatling DSL
  • Integration with Jenkins
  • Gatling extensions and reuse of the statistic engine
  • Example scenarios

I would like to write more in-depth about all these topics in some follow-up blog posts. Feel free to vote for the topic you are more interested in or suggest new ones in the comments section of this post.
If you need any help in setting up your scenario or understand how to run load tests against your Gerrit installation effectively, GerritForge can help you.

Fabio Ponciroli (aka Ponch) – GerritForge
Gerrit Code Review and Gatling Contributor