2023: New Year and opportunities for GerritForge and Gerrit Code Review

TL;DR: GerritForge has been dedicating its efforts to organising and managing the Gerrit User Summit in London back in November 2022, in conjunction with the release of Gerrit v3.7. The event has been a great success, with a significant presence on-site and record-breaking attendees on the GerritForge TV youtube channel. It has also committed to its promises to research and improve the JGit and Gerrit scalability to large mono-repos, with tens of millions of objects and refs. 2023 will see the finalisation of these efforts with an increase in development efforts and a new JGit Committer for pushing the platform to a new level of performance and scalability and a new innovating system for collecting and optimising the repository metrics automatically. Stay tuned.

Read the full story here below (9 mins read).


2022 has been a critical year for turning the Gerrit Code Review community and development back on track after the COVID-19 pandemic. At GerritForge, we’ve been working hard to make sure that the development, support, and innovation of Gerrit Code Review continue on its main objectives.

Gerrit Code Review v3.6 and v3.7

We have continued to deliver on the development and release of Gerrit Code Review and its plugins, helping the testing and releasing of versions v3.6.0 (May) and v3.7.0 (November).

Some numbers of the past 12 months’ development contributions by individual committers and companies:

  • 3,627 Changes have been merged on 76 projects related to the Gerrit Code Review platform, including JGit
  • 113 committers from 42 different organisations

A special mention to the top #10 contributors: Google (Ben Rohlfs, Edwin Kempin, Chris Pouchet, Dhruv Srivastava, Frank Borden, Milutin Kristofic), GerritForge (Luca Milanesio), Wikimedia (Paladox) and SAP (Matthias Sohn and Thomas Dräbing).

In comparison with 2021, we had 25% fewer changes merged but with more contributors coming from more companies, which is a symptom to a very healthy and thriving ecosystem of maintainers.

GerritForge has committed to resuming the face-to-face user summits, which were suspended since 2020.

The Gerrit User Summit 2022 took place in London, UK the 10-11 of November in a hybrid format, with people having the opportunity to participate either on-site or remotely on GerritForge’s YouTube TV channel.

It was a glorious success, with record-breaking attendance from all around the globe:

  • 50 people registered to attend on-site, 26 of them managed to arrive despite the London tube strike, whilst the others attended remotely
  • 235 people viewed the summit on YouTube with an average view time of 40 mins (one talk)

The summit survey had an outstanding report showing a huge acceptance and appreciation of the event:

  • 82% rated the remote video streaming as “good” or “outstanding”
  • 96% rated the quality of the summit as “good” or “outstanding.”
  • 100% would recommend the summit to a colleague, with 83% strongly recommending it

GerritHub.io SLA gets closer to five-nines.

We have been working hard to make Gerrit more stable and resilient throughout 2022, discovering and fixing many issues in the code base and on the multi-site software architecture.
In 2022, GerritHub.io had only six small hiccups for a total of 19 mins of downtime (SLA = 99.997%) over a 12-month period, a 75% reliability improvement compared to 2021.

We have run extensive RCAs on the causes of the downtime and identified two leading issues, which are explained in the details below.

The “anonymous unlimited query” hole in Gerrit
GerritHub.io has been subject to a 15 mins outage because of anonymous users being able to bring offline all the sites before the system could auto-recover.
Gerrit allows bypassing of all limits set in the ACLs for running queries by simply adding the “no-limit” parameter.
Returning an arbitrary payload without limits could allow a single user to generate a server-side workload for collecting and building a GBytes-sized JSON payload; unfortunately, that option was available to everyone, including anonymous users making any publicly faced Gerrit Code Review installation subject to deny-of-service attacks.
We have identified the issue, reported and fixed it in Gerrit with Change 333304, which has been included in Gerrit v3.3.10, v3.4.4, v3.5.1, and all v3.6.0 or later releases.

More granular monitoring and alerting
We have lowered the threshold of uptime checks on GerritHub.io to 1 minute, giving us the ability to detect and react immediately to 4 smaller hiccups. We have detected a lack of scalability for some specific higher-load projects. Those hiccups have been responsible for 2 mins of downtime over the 2nd part of 2022. Many more projects are also planning to be onboarded on GerritHub.io; hence we do need to address this project-specific capacity needs.

Scaling Gerrit Code Review and JGit beyond its limits

We have been investing a massive effort in building a test environment designed to stress Gerrit and JGit to its limits and identify all the limitations and bottlenecks that prevented us from scaling further.

Scaling the test repository
We have created over the months some test repositories that increased in every dimension:

  • Tens of millions of refs as both refs/changes and refs/heads
  • Millions of delta-chains
  • Tens of millions of Git objects
  • Packfiles of tens of Giga-bytes and packed refs of hundreds of megabytes

For generating a significant load on both client and server side, we have invested more into the aws-gerrit cloud setups and gatling-git performance loading tool.

There were some “well-known” issues and additional surprising ones.

SHA1 complexity and CPU utilization for large entities
JGit has been used SHA1 for identifying uniqueness not just for Git objects but also for other large entities. However, computing SHA1 has become increasingly CPU intensive because of the relatively recent findings about collisions on shattered.io.
We have highlighted two major potential improvements in cooperation with Matthias Sohn (SAP) on the raw SHA1 performance and its application for detecting packed-refs changes on the filesystem.

Commit priority queues
JGit has a custom implementation of priority queues which are intensively used in RevWalk, which has almost quadratic complexity. That isn’t a problem for small to medium chains of commits; however, when the number of commits reaches millions, the performance degradation becomes unbearable.
We have replaced the JGit’s custom implementation with the one provided by the Java JVM library, which has a logarithmic complexity that massively improves its performance with large commit chains.

Unwanted reachability checks
JGit needs to perform a full reachability check whenever a remote unknown client is advertising refs, which makes sense when serving a remote client. However, the cost of full reachability of millions of advertised refs can be a daunting task that may be alleviated if the remote end can be considered trusted.

Fixing JGit bitmaps
Since the introduction of Git bitmap, the whole community has learned how key they are in speeding up the counting and selection during the clone phase.
However, large and unoptimized bitmaps could be so unhelpful for Git that instead of speeding up, they could represent a massive overhead for the system, causing CPU spikes and, eventually, lowering the throughput of the server.
Git bitmaps are compressed using the JavaEWAH library, which is good for memory consumption but evil for CPU utilization: that is the reason why the smaller is best for performance.
We have discovered and fixed a critical issue with the JGit bitmap generation that was causing the inclusion of all commits and BLOBs pointed by annotated tags. Also, we have introduced the ability to inform JGit about the heads that can be excluded from the bitmap, allowing to shorten the creation tens of thousands times (5h generation time for a 2k refs to as little as 60s) and increase its effectiveness by 200%.

Millions of unneeded ref logs
When performing a clone of a repository with millions of heads, JGit created one local reflog file for every remote ref, including the ones there were not actually cloned but just fetched as remote references. This was creating a significant performance gap between JGit and Git, which would instead lazily create the reflog files once they are effectively checked out the first time. Cloning a single branch of a repository with millions of remote refs took around 1h, compared to a few minutes of Git.

All of the findings were included in multiple updates on the following components:

  • JGit changes: all fixes were also provided to stable-5.13, the last supported branch for Java 8, which allows benefiting from these improvements for older versions of Gerrit from v2.16 onwards.
  • pull-replication went through major performance improvements, achieving a 1000x times faster execution time compared to the traditional replication plugin
  • aws-gerrit is going through upgrades for making use of pull-replication plugin, including the support for the bearer token which allows to replicate virtually any repository, including All-Users.git
  • gatling-git: we have upgraded the Gatling version and JGit to the latest stable-5.13 to include the latest performance improvements.
  • git-repo-metrics: we have introduced a brand-new plugin that allows us to keep under control the major dimensions of a repository and therefore graph their increase over time.

GerritForge goals for 2023

We are definitely not done yet with the performance improvements on Gerrit and JGit: there are still significant improvements to be made, and JGit changes to get merged into the mainstream branches.
We believe we are on track to finalize the job and allow a stable and scalable platform for large Git repositories in 2023.

Finalise what we cooked in 2022 for JGit
JGit has a new maintainer, David Ostrovsky, awarded in 2022 as Git committer of the project. GerritForge’s devs are focused to get more reviews and attention to the JGit performance improvements. We are committed to finalising all the open changes related to large repositories.

JGit multi-pack indexes support
There is still a major gap between JGit and Git when dealing with very active repositories: multi-pack indexes. The proliferation of packfiles would eventually lead to a long and painful search-for-reuse phase for BLOBs which could be cut down 100s of times with a multi-pack index.

Git repository optimiser for Gerrit
We have been working on tracking the live information on the Git repository, thanks to the git-repo-metrics plugin. Wouldn’t it be nice to have a tool that can do something with it and automatically?
We would be doing R&D on how to correlate the repository metrics, the Git audit trail, and the performance data for making AI-based decisions on what needs to be improved on the repository.
This work stream is going to be useful for any Git repository, not just the ones powered by Gerrit Code Review. The ‘git-repo-metrics’ and the repository optimiser would also apply to other products, including GitHub and GitLab.

Gerrit v3.8 and projects-specific change numbers
We will finalise the design document for the transition to project-specific change numbers in Gerrit v3.8. That would allow the seamless migration of projects across Gerrit setups without having to worry about changes renumbering anymore.

Gerrit Code Review testing and GerritForge-certified binaries
GerritForge is spending a tremendous amount of time developing test environments and tools for serving the Gerrit community with more stable releases and improving the quality of its code. We want to intensify the effort and also offer our platinum support customers a unique service that includes the GerritForge digital signature and rubber stamp on the binaries of Gerrit Code Review and its plugins that have been successfully tested and validated for being production-ready.
Stay tuned; more details are coming soon …

GerritForge company forecast in 2023

GerritForge Inc. will finalise its roll-out to the USA, and all contracts and services will be run from Sunnyvale, CA and Europe. Over 2022, 60% of the customers and businesses have already been moved, and the operation will be completed over the course of 2023.

We are looking forward to doubling our revenue figures in 2023 and also our contributions to the open-source community, with a main focus on JGit as the driver of performance growth for Gerrit Code Review.


2023 is going to be an incredible year for GerritForge, Gerrit Code Review, and the JGit community altogether.

Happy New start of the Year 2023!

Luca Milanesio (GerritForge)
Gerrit Code Review Maintainer and Release Manager
Member of the Gerrit Engineering Steering Committee

Gerrit v3.4.0-rc5: weekly update

Merged Changes

From Luca Milanesio:

    Allow GerritAccount Cookie authentication for Git/HTTP
    Bug: Issue 14508
    Change-Id: I2a56197ee0dad479f0973192157e5970d9deac25

    Avoid multiple auth requests for Git/HTTP access
    Bug: Issue 14497
    Change-Id: Ibe41df0357b6be10bcdf0bd1f5a1b6160c34d4a4

    Introduce LDAP metrics
    Bug: Issue 14490
    Change-Id: I18e5d5b797b272ca11a6745bc39dcd73cab68c34

    Add unit-tests for ProjectBasicAuthFilter
    Change-Id: Ib9abf133d2128b6a29751ecbeda26b0b43115bb3

From Han-Wen Nienhuys:

    Documentation/user-review-ui: remove intraline diff mention
    Change-Id: I15faa971d5cfe80ab82295f1af4e307da1d17f10

    Documentation/user-review-ui: show just a single shortcut help screenshot
    Change-Id: I2ad396a20767221f293d2ec59e6ac61a85e12629

    Documentation/user-review-ui: remove mention of old UI
    Change-Id: I88adb24c5883410bc62b9e290e4077e6b845474a

    Documentation: rename all images for user-review.txt to gwt-*.png
    Change-Id: I42735800679736ef277a053e042908677c4db5ef

    Documentation: remove unused GWT image files
    Change-Id: Ie470c983509310b4a6eb35eedbc03ffa5e8028cd

From Matthias Sohn:

    Update jgit to a9579ba60cd2fd72179dfd8c2c37d389db5ec402
    [1] https://bugs.eclipse.org/bugs/show_bug.cgi?id=552173
    [2] https://bugs.eclipse.org/bugs/show_bug.cgi?id=573328
    Change-Id: Id3a4adc0faf27e7bcd2017ab439aa2230cf92b33

    Add new command convert-ref-storage to index
    Change-Id: If3a93e65333d1a5f299273711162d81e1b653e1b

From Antoine Musso:

    download_file: download to GERRIT_CACHE_HOME when set
    Change-Id: Ie4fac83928527e0e71b159b9500983234c2261ac

From Prudhvi Akhil Alahari:

    Fix EqualsLabelPredicate to not fail when calling match() from a plugin
    Change-Id: Icd2541fe26c18a8e61ce855862e0c9814a91f5ef

From Thomas Dräbing:

    Respect auth.userNameToLowerCase when creating accounts via REST or SSH
    Bug: Issue 14246
    Change-Id: If0f120f188e9f5bdf8008c4e66a55568180e7351

Issues Fixed

  • Issue 14246: Creation of internal account does not respect auth.userNameToLowerCase
  • Issue 14490: Missing LDAP metrics for authentication
  • Issue 14497: Git/HTTP traffic overloads LDAP with duplicate authentication requests
  • Issue 14508: Issue 14508: Gerrit authenticates more than once for a Git/HTTP high-level operation

Issues Raised

  • Issue 14388: Mobile diff context buttons do not work properly

Gatling E2E results

Git protocol simulation:

Gatling full results:
https://gerrit-ci.gerritforge.com/job/gatling-gerrit-test/360/gatling/report/gerritgitsimulation-20210511183716601/source/index.html

Gatling simulation class:
https://github.com/GerritForge/gatling-sbt-gerrit-test/blob/master/src/test/scala/gerritforge/GerritGitSimulation.scala

Gerrit UI REST simulation:

Gatling full results:
https://gerrit-ci.gerritforge.com/job/gatling-gerrit-test/360/gatling/report/gerritrestsimulation-20210511184140917/source/index.html

Gatling simulation class:
https://github.com/GerritForge/gatling-sbt-gerrit-test/blob/master/src/test/scala/gerritforge/GerritRestSimulation.scala

Gatling 10-days trend

Gerrit v3.4.0-rc4: weekly update

Merged Changes

From David Ostrovsky:

    Tidy up dev-plugins.txt documentation
    Change-Id: Ia3b09d5febcfe9ab1805bd2e55963aa6c6624f6a

    Reftable: Add convert-ref-storage ssh command
    Bug: Issue 14138
    Change-Id: I0ddd2370619de287ec757597ba57e8b757611808

    Bazel: Tidy up gerrit_js_bundle rule
    Change-Id: I9b3e757717a64e8538d5d05bdb86b3b26c9f363d

    Make srcs in gerrit_js_bundle optional
    Change-Id: Ifaca618836098798dfbad10c2f5c2aee10f6097f

    Fix link for change.enableAssignee configuration option
    Change-Id: I2faef25ea62119184e1a837b647b9cccdc246354

    ErrorJsonLogEntry: Use IP address as host value in logging event
    Change-Id: Ie3a246ff277a0b8646dbe11acdb2b53aa659800e

    Bazel: Add gerrit_js_bundle rule
    Change-Id: Ia8421a9ef6f1ad91808bb698769da84022f63973

From Edwin Kempin:

    Allow tests to check how much Counter0 metrics are increased by a call
    Change-Id: I79da799c86325eb64922a92caec8a476e4e6a4ae

From Ben Rohlfs:

    Improved and more sophisticated handling of check action results
    Change-Id: I4f139bbe913c60fbdcea27dd9ea1e3f605bb19cb

    Add documentation for the new Checks JavaScript Plugin API
    Change-Id: I82cd3c9af9c918b1c26ea1cb8a70e3283138ac31

    Add links to frontend plugin examples
    Bug: Issue 14421
    Change-Id: I123f1717ac691eda20709c20764db457c58b42e2

    Remove draft warning from Checks API
    Change-Id: I267b2792efee3baa27ebbe6915b0c7813efab566

    Update all the plugin examples
    Change-Id: Idc6473377bb5966f0bc199bfc68977297df4f802

    Tiny documentation fix
    Change-Id: I505419cdf03bfe5dd3a1dd28eeb84def36a3669a

    Update the JavaScript Plugin API documentation
    Bug: Issue 14421
    Change-Id: I5869c8168fe27bea81226b0b8f1dc40b271c2454

From Youssef Elghareeb:

    Rename SubmitRequirement to LegacySubmitRequirement
    Change-Id: Ib55038627c95ece82a7eddd64e317183f1752487

From Patrick Hiesel:

    SubmitRecord.Status: Reserve RULE_ERROR for user-caused errors
    Change-Id: I018ce9e13fa40f2f24546df49ecf7fc3fbb72b1b

Issues Fixed

  • Issue 14138: expose JGit reftable conversion as Gerrit SSH command
  • Issue 14421: Update the Frontend Plugin Documentation

Issues Raised

None

Gatling E2E results

Git protocol simulation:

Gatling full results:
https://gerrit-ci.gerritforge.com/job/gatling-gerrit-test/348/gatling/report/gerritgitsimulation-20210504002334245/source/index.html

Gatling simulation class:
https://github.com/GerritForge/gatling-sbt-gerrit-test/blob/master/src/test/scala/gerritforge/GerritGitSimulation.scala

Gerrit UI REST simulation:

Gatling full results:
https://gerrit-ci.gerritforge.com/job/gatling-gerrit-test/315/gatling/report/gerritrestsimulation-20210426212355447/source/index.html

Gatling simulation class:
https://github.com/GerritForge/gatling-sbt-gerrit-test/blob/master/src/test/scala/gerritforge/GerritRestSimulation.scala

Gatling 10-days trend

Gerrit v3.4.0-rc3: weekly update

Merged Changes

From David Ostrovsky:

    Bazel: Remove assets parameter in polygerrit_plugin rule
    Change-Id: I86b996a4e98c7ea8ea35bfda097ff60092078028

    EventGsonProvider: Register generic event serializer adapter
    Change-Id: I7dc6e6a859152ee3e30a90075ddd34814720eba1

    EventJsonTest: Add test for custom event serialization
    Change-Id: I7e45f77aa980c03896e3891c2d48680e31d2e0a6

    Disable peer IP in reflog record per default
    Bug: Issue 10810
    Change-Id: Ia7a04cdce0425989059b92131afd8c2f6030d10b

    Allow to suppress peer IP in reflog records
    Bug: Issue 10810
    Change-Id: I5645a4e68fb48ad155609573aea2518480074697

    Remove enableReverseDnsLookup configuration option
    Change-Id: If33c6e1901b70ca64d9b32b0ebb310f3a9daf017

    Revert "Respect enableReverseDnsLookup option in ErrorLogJsonLayout"
    Change-Id: I8f74ca039246753f775b7bfc96596e3f5159af01

    Clean up polygerrit_plugin rule
    Change-Id: Ie46a8f02a2b485a856916fe14002dc6d0e710c20

From Paladox:

    Remove KnownExperimentId from gr-comment-thread
    Change-Id: Ib13d07e6dc71e3133fdc249c72809082c0fa915b

    Cleanup undefined check for access in getRepoAccess
    Change-Id: Ieefab7e709a4124409ced7cb86ecdde37748d906

From Frank Borden:

    Upgrade package.json dependencies
    Change-Id: Ifb400606b6c420f732bb36e87bf43018253757ee

From Ben Rohlfs:

    Add a SHOW ALL button to check result sections    
    Change-Id: I6be8585c285db6255e25060f61008cb57232f154

    Add support for top-level links in the Checks Tab
    Change-Id: Icca6e93b44b28d33607032e2a4d74327a684c2a8

    Remove experiments for comment context and checks
    Change-Id: I130b7846414e281062906def588151da71d72d96

    Implement a detailed check run hovercard
    Screenshot https://imgur.com/a/9grF5sm
    Change-Id: I185264d0ce02726487facda19c116b9351fc9a7f

    Fix target area for expanding/collapsing check result rows
    Change-Id: Ibdd08e8732e047c62a453b02324408466817305a

    Fix counting of selected and filtered check results
    Change-Id: Iea4c1816dd23272678bcd5aa0f3ff549257606ed

    Do not always show the checks error message
    Change-Id: I61656729770322bb1795c44fd297fda4d1a0256d

    Refined UI for checks error responses
    https://imgur.com/a/baILcJl
    Change-Id: I63f1cc5f128681f46cae3ab875744946b8537ea4

    Add handling of not-logged-in and error states
    Change-Id: I701a2fbdebca8d5b4adc437554b49597c8cc166a

    Fix checks utility for link icons
    https://imgur.com/a/ZMFEuGr
    Change-Id: Ie242e578ff91def52c559a10124d82b836077b90

    Change check tag/label font color from deemphasized to primary
    Change-Id: Ie2de29d019d7ccab8075d212600a3da0267fc8e1

    Add actions and links to check result rows    
    Change-Id: I2ff5a594bcc927910855d89c52484a5aef4581e0

    Fix the color of the attention and status icons
    Change-Id: I2a32e1f22848e122c0fd3016d8b91fbc16d029a6

    Only render expanded checks row once the user has clicked the row
    Change-Id: I4039953a63a84995577f309dcb2fbfee5680be86

From Milutin Kristofic:

    Remove new-change-summary feature flag from gr-change-view
    Change-Id: I971bc62bacc61b1985adb5de8d4fd36da68f2f46

    Remove old implementation of gr-related-changes-list
    Change-Id: Ic42f9af760042cd0e080a29b09c3d21d20bd161c

    Move tests from old to new gr-related-changes-list
    Change-Id: I631cde95d00a1e54b1820232733424b28f74bff4

    Remove new-change-summary feature flag from gr-change-summary
    Change-Id: Id27fb05cd42186cea5de068da791a73c7f0064cb

    Remove new-change-summary feature flag from gr-change-metadata    
    Change-Id: Ie2116e83e258b46c546056b6a9b6867e1dadcd35

    Remove new-change-summary feature flag from gr-thread-list   
    Change-Id: I8eaf790199c80956cf3c32e70d7e6fde133cabbd

    Remove new-change-summary feature flag from gr-thread-list    
    Change-Id: I8eaf790199c80956cf3c32e70d7e6fde133cabbd

    Remove new-change-summary feature flag from gr-editable-content    
    Change-Id: Ic324b6f210fc37ee5e52253dc0658c89e395c0ef

    Remove new-change-summary feature flag from gr-reviewer-list
    Change-Id: I6675f9e57244fd70b940d6a659e0dcd8d1533388

    Remove 'shared-styles' from most frequent gr-elements
    Change-Id: I7ab764397fb0bda92e7c784e5c0aac2d17c2e63d

    Remove new-change-summary feature flag from gr-change-requirements
    Change-Id: I77e6f174a73979b8c6e52f67953909e97964ae9e

    New change summary UI - fix small A11y issues
    Change-Id: I29310115583b7c5ac349923d6014dd49f66aaa1a

    Always show add topic row in metadata
    Change-Id: I794c028c8f9c81b43918511fac193b702c20f147

    Pressing enter on focused button supress keyboard shortcuts
    Change-Id: I6d28ebd022041a617be645fcd7f89ecb68d9cc54

From Dhruv Srivastava:

    Only show Merged As info if change has been merged
    Change-Id: I2405bf98dbb6542cb36cc0767dc79d61061bf3c8

    Remove patchset level comments from patchset dropdown count
    Change-Id: Ib4fc43e0c3797834b72a765ae2afce6f70871e9d

    Show user first in list of reviewers in change view
    Screenshot: https://imgur.com/a/FW60KC2
    Issue: Bug 14366
    Change-Id: I5ae9fc2e1b90f2dbbb7040d81be0e69f44ce9e3a

    Clean up Add Patchset Description from File List header
    Change-Id: Ia1ce16f776f81f734c0b03acce60a02e46190465

    Move Merged as info from change header to change summary
    Screenshot: https://imgur.com/a/MgZNe7G
    Change-Id: Ie2a5715dc65a20447f296f5246962b6445836490

    Show Code Review quick approve button with missing labels
    Change-Id: I3f9179177d92a7b7932353da1a36c5decad599ec
    (cherry picked from commit 25ca8ea4e9b9abb65ec6cc05af767a7e03b479da)

From Dmitrii Filippov:
    Fixes for Typescript 4.2
    Change-Id: I2387d5664c6748a1194ca52a0560d1d961ca64e4

    Fix and update license generator
    Bug: Issue 14175
    Change-Id: Ifd38a8e3ae7f06354e00aee420f282773b64583a

From Edwin Kempin:

    Export PluginPushOption for use in plugins
    Bug: Issue 14362
    Change-Id: Ie64c22bc4fed65950844dd4cd9a3e5328fe4eb8f

From Matthias Sohn:

    Welcome administrator of freshly installed Gerrit site
    Bug: Issue 12209
    Change-Id: Ib26372a610ab2e9c8266a71c9b2c41cfda9a051a

From Luca Milanesio:

    Avoid NPE when logging incoming HTTP request
    Bug: Issue 14410
    Change-Id: I4bdbe7418ccc4b223fa57b7fb49c2aa46f5d3982

    Add initial stream-events acceptance tests
    Bug: Issue 13799
    Change-Id: I6987d413ba911039e56950425b501573b4555204

Issues Fixed

  • Issue 10810 : Gerrit leaks user IP address to project owners
  • Issue 12209: Remind users that install Gerrit newly to join the Gerrit community and register on repo-discuss
  • Issue 13799: No tests for stream-events
  • Issue 14175: //Documentation:check_licenses target fails with python 3 on stable-3.3 branch
  • Issue 14362: uploadvalidator check should be bypassable
  • Issue 14410: NPE when tracking Gerrit HTTP logs
  • Issue 14366: Always show self first in list of reviewers on change page

Issues Raised

  • Issue 14336: PolyGerrit: no way to configure checkers using UI provided by checks plugin

Gatling E2E tests results

Git Protocol Simulation:

Gatling full results:
https://gerrit-ci.gerritforge.com/job/gatling-gerrit-test/315/gatling/report/gerritgitsimulation-20210426212117440/source/index.html

Gatling simulation class:
https://github.com/GerritForge/gatling-sbt-gerrit-test/blob/master/src/test/scala/gerritforge/GerritGitSimulation.scala

Gerrit UI REST Simulation:

Gatling full results:
https://gerrit-ci.gerritforge.com/job/gatling-gerrit-test/315/gatling/report/gerritrestsimulation-20210426212355447/source/index.html

Gatling simulation class:
https://github.com/GerritForge/gatling-sbt-gerrit-test/blob/master/src/test/scala/gerritforge/GerritRestSimulation.scala

Gatling 10-days trend

Gerrit v3.4.0-rc2: weekly update

Merged Changes

From Milutin Kristofic:

    Fix underlining on hover for subject column in dashboard
    Change-Id: Ie0c55bbe550f1714f1f9f630ac81f6c686184221

    Revert "A11y - show outline when focused on tab title in change view"    
    Change-Id: I6d8f74c4d01004a88b0dbf207d9d9741a6965755

From Mike Frysinger:

    Fix typo in hashtags docs
    Change-Id: Idabd8994651451183472eb3b777064e891e8a765

From Dmitrii Filippov:

    Do not reload page after clicking Cancel in apply-fix dialog    
    Change-Id: I5830caac655df9d3dc7058dc9674ba727685012b

From Paladox:

    Add undefined check for access[repo] in getRepoAccess
    Change-Id: I16f3344c9f11cb230adafa7fdc8457ac37b4ae70

    Add web links to project:<project>
    Change-Id: Iefda4bb5571d917d5fa9f24c02c8e4a3be2c6c52

From Dhruv Srivastava:

    Turn off syntax highlighting in comment context if disabled
    Change-Id: I89625a9b15678c1a63543d36f9f7d835bb39b359

    Allow comment context to take up remaining width 
    Change-Id: I1dab57df26b0fed98efead1a18083d9a504ed40f

    Make the line length indicator color clearer
    Issue: Bug 13648
    Change-Id: I523d971f39db477d67e212d1175ec6f5be8be24d

    Remove Add/No Patchset description label for merged changes
    Change-Id: I18643bcf01e10177a28411089ac1b312222d3e23

From David Ostrovsky:

    Bazel: Disable worker multiplexer to avoid sporadic build failures
    Change-Id: I2265c4ed7128a4b6ed259f44c5594ab717de58b0

From Ben Rohlfs:

    Allow line wrapping in check result messages
    Change-Id: Icedc05bbc28fbe513c60a5eb9853dadde636381b

Issues Fixed

  • Issue 13648: Make the line width guide in commit description to be darker

Issues Raised

Gatling E2E tests results

Git Protocol Simulation:

Gatling full results:
https://gerrit-ci.gerritforge.com/job/gatling-gerrit-test/305/gatling/report/gerritgitsimulation-20210419190719877/source/index.html

Gatling simulation class:
https://github.com/GerritForge/gatling-sbt-gerrit-test/blob/master/src/test/scala/gerritforge/GerritGitSimulation.scala

Gerrit UI REST Simulation

Gatling full results:
https://gerrit-ci.gerritforge.com/job/gatling-gerrit-test/305/gatling/report/gerritrestsimulation-20210419190957647/source/index.html

Gatling simulation class:
https://github.com/GerritForge/gatling-sbt-gerrit-test/blob/master/src/test/scala/gerritforge/GerritRestSimulation.scala

Gatling 10-days trend

Gerrit v3.4.0-rc1: weekly update

Starting from this week, GerritForge provides a weekly status update of how the Gerrit v3.4.0 release plan is progressing:

  1. List of merged changes since the previous RC
  2. List of issues fixed
  3. Result of the Gatling E2E tests on AWS

We hope that having these regular updates would help you focus on what is changing during the release plan execution and do more research, learning or specific investigation on the areas that are more pertinent to your use case.

Also, because the Gerrit release candidates do not come with an associated set of release notes, the list below would help people understand the new functionalities or fixes coming through every week.

Both Gatling-Git and the AWS-Gerrit project with a complete production-ready setup are projects started by GerritForge and contributed as OpenSource to the Gerrit community.

Merged changes

From Matthias Sohn:

    Log memory allocated per command in httpd_log 
    Change-Id: Ie0ce1382a8515e6dfb7d0d3fe10b3e64c0cf9aee

    Log cpu usage per http request   
    Change-Id: I9e78bed5219f9baf57a2b76f0f947efff334ffe5

    Log memory allocated per command in sshd_log
    Change-Id: Ifc1d274bf42eb3cb9b2cf46271b6be0117aa8b18

    Add metrics for monitoring Java memory pools
    Change-Id: I60e5960899c0cff8c05983d299b414d7a646bb07

    Log cpu usage in sshd_log
    Change-Id: I1c53f64caf982c2f85195e6bda4c6d790f79a810

    Encapsulate fields of SshScope.Context
    Change-Id: If989630425ad40922aaf8958c4335aab0bb5c2c9

    Log "-" for missing log fields in sshd_log
    Change-Id: I90adc7618864f702b42029ab596c6014bd4c6cfe

From Ben Rohlfs:

    Remove backend support for HTML UI plugins
    Change-Id: I44cc0d15910937de7e1f9b9780a799d4b85b0673

    Stop producing html version of plugins
    Change-Id: I1036f06e385f2997f7bea849755729df2789acaa

From Milutin Kristofic:

    A11y - show outline when focused on tab title in change view
    Change-Id: Ie9456a5d886e70a77dae8f055a54a3a1a0045daf

    A11y - show outline when focused on change subject in dashboard
    Change-Id: I9c73e49de661a17928c6f96a290c2069503bdfb4

    A11y - fix and improve label when navigating dashboard
    Change-Id: Id01302aea38c783687443401290995bdd0764126

From Hermann Loose:

    Allow setting image viewer max-width and max-weight externally
    Change-Id: Ie151a85d8ede5cb7fa0899d9367ca0dccd887538

From Han-Wen Nienhuys:

    Fix meta_diff documentation
    Change-Id: I9c59a4857724cdfd59b995a6dc255f77d29b017e

From Paladox:

    Update plugins/codemirror-editor and plugins/delete-project
    Change-Id: Ieca7d2e36b9eefffe5c830962109e1fa62134b5c

    gr-confirm-move-dialog: Fix _getProjectBranchesSuggestions
    Change-Id: I8d4ead0bf3a8c30d0ecefdc190e4ba4ea7ede29d

    Remove unused _handleDropdownTap
    Change-Id: Ie4a8effd6d814985cafa6dea9239903f503dae33

    Remove unused _handleDropdownTap
    Change-Id: Ie4a8effd6d814985cafa6dea9239903f503dae33

From Dhruv Srivastava:

    Clean up upload change help dialog
    Change-Id: I0c72450c37326ec2c2922b74928e0b059df0043e

From Saša Živkov:

    Fix binding of DELETE REST calls from plugins
    Change-Id: I9b9632e8f719937e5f7c61466996be79e6f29c14

Issues fixed

  • Issue 14335: CodeMirror plugin broken
  • Issue 14127: REST API DELETE query for delete-project plugin doesn’t work

Gatling E2E tests results

Git Protocol Simulation

Gatling full results:
https://gerrit-ci.gerritforge.com/job/gatling-gerrit-test/287/gatling/report/gerritgitsimulation-20210412213548212/source/index.html

Gatling simulation class:
https://github.com/GerritForge/gatling-sbt-gerrit-test/blob/master/src/test/scala/gerritforge/GerritGitSimulation.scala

Gerrit UI REST Simulation

Gatling full results:
https://gerrit-ci.gerritforge.com/job/gatling-gerrit-test/287/gatling/report/gerritrestsimulation-20210412213912089/source/index.html

Gatling simulation class:
https://github.com/GerritForge/gatling-sbt-gerrit-test/blob/master/src/test/scala/gerritforge/GerritRestSimulation.scala

Stress your Gerrit with Gatling

As a Gerrit administrator, making sure there is no performance impact while upgrading from one version to another can be difficult.

It is essential to:

  • have a smooth and maintainable way to reproduce traffic profiles to stress your server
  • easily interpret the results of your tests

Tools like wrk and ab are simple and good to run simple benchmarking tests, but when it comes to more complex scenarios and collection of client-side metrics, they are not the best tools to use.

Furthermore, they only support Close Workload Models, which might not always fit the behaviour of your system.

For those reasons in GerritForge we started to look at more sophisticated tools, and we started adopting Gatling, an open-source load testing framework.

The tool

The Gatling homepage describes it this way:

“Gatling is a highly capable load testing tool. It is designed for ease of use, maintainability and high performance…

Out of the box, Gatling comes with excellent support of the HTTP protocol…..

As the core engine is actually protocol-agnostic, it is perfectly possible to implement support for other protocols…

Based on an expressive DSL, the scenarios are self-explanatory. They are easy to maintain and can be kept in a version control system…”
In this article, we focus on the maintainability and protocol agnosticism of the tool.

What about Git?

Gatling natively supports HTTP protocol, but since the core engine is protocol-agnostic, it was easy to write an extension to implement the Git protocol. I started working on the Gatling git extension in August during a Gerrit hackathon in Sweden, and I am happy to see that is starting to get traction in the community.

This way we ended up among the official Gatling extension on the official Gatling homepage:

Screenshot 2019-12-16 at 15.03.32.png

The code of the git extension if opensource and free to use. It can be found here, and the library can be downloaded from Maven central. In case you want to raise a bug, you can do it here.

Maintainability

Gatling is written in Scala, and it expects the load tests scenarios to be written in Scala. Don’t be scared; there is no need to learn crazy functional programming paradigms, the Gatling DSL does a good job in abstracting the underneath framework. To write a scenario you just have to learn the building blocks made available by the DSL.
Here a couple of snippets extracted from a scenario to understand the DSL is:

class ReplayRecordsFromFeederScenario extends Simulation {
  // Boireplate to select the protocol and import the configuration
  val gitProtocol = GitProtocol()
  implicit val conf = GatlingGitConfiguration()

  // Feeder definition: the data used for the scenario will be loaded from "data/requests.json"
  val feeder = jsonFile("data/requests.json").circular

  // Scenario definition: 
  val replayCallsScenario: ScenarioBuilder =
    scenario("Git commands") // What's the scenario's name?
      .forever {  // How many time do I need to run though the feed?
        feed(feeder) // Where shall I get my data?
          .exec(new GitRequestBuilder(GitRequestSession("${cmd}", "${url}"))) // Build a Git request
      }

  setUp(
    replayCallsScenario.inject(
      // Traffic shape definition....pretty self explanatory
      nothingFor(4 seconds),
      atOnceUsers(10),
      rampUsers(10) during (5 seconds),
      constantUsersPerSec(20) during (15 seconds),
      constantUsersPerSec(20) during (15 seconds) randomized
    ))
    .protocols(gitProtocol) // Which protocol should I use?
    .maxDuration(60 seconds) // How long should I run the scenario for?
}

That is how the feeder looks like:

[
  {
    "url": "ssh://admin@localhost:29418/loadtest-repo.git",
    "cmd": "clone"
  },
  {
    "url": "http://localhost:8080/loadtest-repo.git",
    "cmd": "fetch"
  },
  {
    "url": "http://localhost:8080/loadtest-repo.git",
    "cmd": "push",
    "ref-spec": "HEAD:refs/for/master"
  }
]

Here another example reproducing the creation of a WIP change using the REST API:

class WIPWorkflow extends Simulation {

  // Configuration bolierplate
  implicit val conf: GatlingGitConfiguration = GatlingGitConfiguration()
  val baseUrl = "https://review.gerrithub.io"
  val username: String = conf.httpConfiguration.userName
  val password: String = conf.httpConfiguration.password
  val httpProtocol: HttpProtocolBuilder = http
    .baseUrl(baseUrl)
    .userAgentHeader("Gatling test")
  val request_headers: Map[String, String] = Map(
    "Content-Type" -> "application/json"
  )
  
  val scn = scenario("WIP Workflow") // What's the name of my scenario?
    .exec(
      http("Create WIP change")
        .post("/a/changes/") // Which url and which HTTP verb should I use? 
        .headers(request_headers) 
        .basicAuth(username, password) // How do I authenticate?
        .body( // What's the body of my request?
          StringBody("""{
            "project" : "GerritForge/sandbox/e2e-tests",
            "subject" : "Let's test this Gerrit! Create WIP changes!",
            "branch" : "master",
            "work_in_progress": "true",
            "status" : "NEW"
          }""")
        )
        .check(status.is(201)) // What's the response code I expect?
    )

  setUp(scn.inject(
    atOnceUsers(1) // Traffic profile
  )
  ).protocols(httpProtocol)
}

Jenkins integration

Running load tests can be tedious and time-consuming, but yet essential to spot any possible performance regression in your application.
Providing the least possible friction is essential to incentivize people in running them. If you are already using Jenkins in your company, you can leverage the Gatling plugin to scale your load quickly and provide easy access to metrics.

A real use case: Gerrit v3.0 Vs Gerrit v3.1 load test…in production!

Let’s go through a real case scenario to show how useful and easy to read are the metrics provided by Gatling.

The closest environment to your production one is…production!

gerrithub.io runs Gerrit in a multi-site configuration, and this gives us the luxury of doing canary releases, only upgrading a subset of the master nodes running Gerrit. One of the significant advantages is that we can run A/B tests in production.

That allows us also to run meaningful load tests against the production environment. See below a simplified picture of our Gerrit setup where it is possible to see the canary server with a higher Gerrit version.

Screenshot 2019-12-14 at 16.26.52.png

We ran against 2 servers in Germany the same load tests which:

  • Create 100 chained Change Sets via REST API
  • Submit all the changes together via REST API

We then compared the server-side and client-side metrics to see if a good job has been done with the latest Gerrit version.

Server-side metrics

Server-side metrics come from the Prometheus exporter plugin. The image is showing the HTTP requests mean time:

Screenshot 2019-12-14 at 16.32.12.png

We can see the improvement in the latest Gerrit version. The mean requests time is almost halved and, of course, the overall duration is decreased.

Client-side metrics

Let’s see what is going on on the client-side using the metrics provided by Gatling.

Among all the metrics we are going to focus on one step of the test, since we have more data points about it, the creation of the change:

Screenshot 2019-12-14 at 16.36.45.png

We can already see from the overall report the reduction of the response time in the latest Gerrit version:

Screenshot 2019-12-14 at 16.38.05.png

If we look in-depth to all the response times, we can see that the distribution of the response times is pretty much the same, but the scale is different….again we confirmed the result we previously encountered.

Screenshot 2019-12-14 at 16.39.17.png

What can we say…Good job Gerrit community!

Wrapping up

(You can see my presentation about Gatling in the last Gerrit User summit in Sunnyvale here <- add this when the talk will be sharable)

I have touched superficially several topics in this blog posts:

  • simplicity and maintainability provided by the Gatling DSL
  • Integration with Jenkins
  • Gatling extensions and reuse of the statistic engine
  • Example scenarios

I would like to write more in-depth about all these topics in some follow-up blog posts. Feel free to vote for the topic you are more interested in or suggest new ones in the comments section of this post.
If you need any help in setting up your scenario or understand how to run load tests against your Gerrit installation effectively, GerritForge can help you.

Fabio Ponciroli (aka Ponch) – GerritForge
Gerrit Code Review and Gatling Contributor