Summary of the Gerrit User Summit & Hackathon in Sunnyvale

sunnyvale-gerritforge-live.jpg

After months of reviews and contributions by different speakers and attendees, the summary of the last Gerrit User Summit & Hackathon in Sunnyvale CA has been published on the Gerrit Code Review News page.

High-performance Summit in numbers

The Gerrit User Summit 2019 has ended, with highest score of achievements
in the history of the 11 years of the entire Gerrit open-source project:

  • Two dates and locations in a 12-months period: Gothenburg (Sweden) and
    Sunnyvale (California).
  • Four Gerrit releases delivered: v2.15.16, v2.16.11, v3.0.2, v3.1.0
  • 127 people registered across the two locations,
    87 people attended on-site (70% turnout) and 38 people followed the event
    remotely at different times using the live streaming coverage
    provided by GerritForge.
  • 373 changes merged (204 in Gothenburg, 169 in Sunnyvale).
  • 32 developers attended the Hackathons, 8 of them have never contributed or
    attended an event before.
  • The highest performing version of Gerrit v3.1.0 released, with over
    2x git and REST-API performance compared to v3.0.x.
  • 22 talks presented across Gothenburg and Sunnyvale, with 6 new speakers
    that have never presented before at the Summit.

The performance of the Summit is yet again another evidence of the continuous
growth of the community and the increased synergies with the JGit, OpenStack/Zuul
and the Tuleap open-source projects.

Read the full Summit and Hackathon summary on the Gerrit Code Review web-site.

Happy New Year, Gerrit Code Review

It has been a hectic and productive year for ourselves at GerritForge and the Gerrit Code Review Community.
We want to take this opportunity to recap some of the milestones of the 2019 and the exciting perspectives for 2020 and beyond.

Gerrit Code Review, 2019 in numbers

gerrit-2019-commits.png

Gerrit had over 120+ contributors from all around the world coming from 33 different companies and organisations, which is excellent. There is a robust 6% increase in the number of commits (+231 commits) but a reduction in the number of contributors (-7 authors).

With regards to the overall trend of commits during the year, the success of the Gerrit User Summit 2019 in Sunnyvale is visible, with an increase of the rate of commits around October/November.

Top-three projects of the 2019

  1. Gerrit (1,626 commits) is, of course, the most active project. However, it is visibly down in terms of number of commits from 2018 (-19%). That is a consequence of the shift of focus to the other two key components listed below, which are available as plugins and then not accounted for the overall gerrit core repository statistics.
  2. Checks (315 commits) is the brand-new 1st class CI integration API for external build systems, such as Jenkins and Zuul. It is incredible how in just 12 months it has become robust and fully mature. It is currently used for the validation of all changes on the Gerrit project.
  3. Multi-site (234 commits) is the long-awaited support for Gerrit that everyone has been waiting for years. It is finally available for all active and supported versions (from 2.16+ onwards).

Top-three companies contributing to Gerrit

gerrit-contributors-2019.png

  1. Google is, with no surprise, still the top contributor of the Gerrit project overall. It is basically stable from 2018 (around 43%) as a confirmation of the continued commitment to the project.
  2. GerritForge is growing significantly in the contribution to the project, with exactly half of the contributions of Google. This is a significant result from 2018 with a 7% growth of involvement.
  3. CollabNet is sliding to the 3rd position (it was 2nd in 2018) with a 3% decrease of contributions. As noticeable mention, however, David Pursehouse from CollabNet is still the number #1 maintainer in terms of number of commits.

Even if it is outside the top#3 contributors companies, SAP deserves a special mention for its continuous involvement in the JGit project, which is at the basis of Gerrit engine, and its fantastic engagement in improving the Gerrit CI system and integrating it with the checks plugin.

Top-three achievements from GerritForge

The outstanding results of contributions of GerritForge in 2019 have been focused on three major topics.

Gerrit multi-site, released and production ready

We released the Gerrit Multi-Site plugin, allowing seamless balancing in a distributed environment, a technologically highly advanced development, crucial for very distributed companies. See https://gerrit.googlesource.com/plugins/multi-site for more information.

Gerrit User Summits in Europe, USA and streaming

We successfully organised and executed the Gerrit User Group in Europe and the US. The event was very well received by the community with an overall attendance of some 87 on-site and 38 in streaming. Have a look at https://gitenterprise.me/2019/12/23/gerrit-user-summit-survey/ for interesting feedback on those from the attendees.
We opened our own local office in Sunnyvale, in the heart of Silicon Valley. A crucial move to better serve our ever-expanding US customer base.

Gerrit Analytics for the Android Open-Source Project

We kickstarted the Gerrit Analytics for the Android open-source project initiative: after the successful adoption of the automatic collection of code metrics on the Gerrit project (see https://analytics.gerrithub.io) the Android team asked GerritForge to start working on extracting the same metrics from their code.

What’s coming in 2020

Gerrit v3.2 is currently under development and it is planned to be released around April/May 2020. It represents a major milestone for the Gerrit project with the support for Java 11 and large JVM heaps, up to hundreds of GBytes. Gerrit v3.2 is definitely the release that everyone that has a big repository (mono-repos) should target as next upgrade. See the Gerrit .roadmap at https://www.gerritcodereview.com/roadmap.html for more details about the planned features.

More work and improvements on the checks plugin, with the aim of fully integrating it into everyone’s user-journey and their CI/CD pipeline. Our first blog-post of 2020 will be how to use Jenkins and Checks plugin together with GerritHub.io.

Multi-site and HA will become more integrated with Gerrit, with the aim of moving parts of their technologies (e.g. global ref-db) into JGit and thus used in Gerrit core.

The Gerrit User Summit 2020 will continue the experiment of cross-pollination with other communities, after the success of the interactions with the JGit and OpenStack communities in 2019. Bazel is the next target, as it is used as the de-facto standard build system for Gerrit and its plugins.


 

Again, Best wishes from your friends at GerritForge and looking forward to a continuing successful partnership in the coming years.

Luca Milanesio
Gerrit Maintainer, Release Manager and member of the ESC.

Gerrit User Summit Survey

The 2019 has been an exceptional year, with the introduction of the next generation of Gerrit Code Review v3 releases and the largest ever Gerrit User Summit in the whole history of 11 years of the project.

As a community we want to improve even further and make the project and the community even better. Collecting metrics has been key for the improvements of the Gerrit product and its performance and, similarly, collecting feedback from the community events is the key to grow and increase the participation and sharing of the experiences about Gerrit Code Review.

Survey results

We have run a survey directed to all of those who have attended the two Gerrit User Summits this year, in Gothenburg and Sunnyvale. See below the executive summary of the results.

Did you achieve your objectives at the Summit?

Screenshot 2019-12-23 at 08.41.14

All of the attendees achieved their objective, which were different for the people, depending on their position and role in the community.

  • Getting the latest news of what’s happening in the Gerrit community and open-source product
  • Meeting the existing members of the community and welcome new contributors
  • Networking with the other Gerrit admin and users around the world
  • Influencing with ideas the future Gerrit roadmap

Overall, how would you rate the event?

Screenshot 2019-12-23 at 08.47.44

Over 76% of the people rated the event very good or excellent. However, as we strive for improvement, there is a substantial 24% of of people that are looking for a better event next year.

What did you like/dislike?

The positives of the event have been:

  • Presentation of the Gerrit roadmap and associated discussions
  • Successful mix of topics, including Zuul and JGit
  • People, atmosphere, friendship and networking
  • High quality of the talks and content

The not so positive sides where:

  • The summit covering the weekend
  • Too focused on Gerrit contributors and admins, no space for users
  • There was too much people for the chosen location
  • The talks and discussions went over the planned schedule

How organized was the event?

Screenshot 2019-12-23 at 08.56.17

89% of the people considered the event very well organized, whilst 11% are looking for improvement, possibly with a bigger venue and better timing.

What topics would you like to see covered next year?

  • Evolution of the User-Interface, roundtable with developers, user-journeys
  • Migration talks and discussions
  • CI/CD integration
  • Monitoring
  • Load testing
  • GitHub integration and pull-requests
  • Gerrit with large clusters
  • User-stories on using Gerrit

Would you like to have a workshop next year?

Screenshot 2019-12-23 at 09.02.56

The vast majority of people would like the next year event to be more informative, including a workshop for learning some of the features of Gerrit Code Review.

What would be the best time for the Summit next year?

Screenshot 2019-12-23 at 09.04.47

For the majority of people (75%) the best time for next year event would be two days during the week, rather than having it again over the weekend.


Thanks everyone again for attending the Gerrit User Summit 2019 in Gothenburg and Sunnyvale, and thanks to GerritForge, Volvo Cars and Google for sponsoring it. We are looking forward to seeing you next year.

Luca Milanesio (GerritForge)
Gerrit Code Review Maintainer, Release Manager and ESC Member.

 

Stress your Gerrit with Gatling

As a Gerrit administrator, making sure there is no performance impact while upgrading from one version to another can be difficult.

It is essential to:

  • have a smooth and maintainable way to reproduce traffic profiles to stress your server
  • easily interpret the results of your tests

Tools like wrk and ab are simple and good to run simple benchmarking tests, but when it comes to more complex scenarios and collection of client-side metrics, they are not the best tools to use.

Furthermore, they only support Close Workload Models, which might not always fit the behaviour of your system.

For those reasons in GerritForge we started to look at more sophisticated tools, and we started adopting Gatling, an open-source load testing framework.

The tool

The Gatling homepage describes it this way:

“Gatling is a highly capable load testing tool. It is designed for ease of use, maintainability and high performance…

Out of the box, Gatling comes with excellent support of the HTTP protocol…..

As the core engine is actually protocol-agnostic, it is perfectly possible to implement support for other protocols…

Based on an expressive DSL, the scenarios are self-explanatory. They are easy to maintain and can be kept in a version control system…”
In this article, we focus on the maintainability and protocol agnosticism of the tool.

What about Git?

Gatling natively supports HTTP protocol, but since the core engine is protocol-agnostic, it was easy to write an extension to implement the Git protocol. I started working on the Gatling git extension in August during a Gerrit hackathon in Sweden, and I am happy to see that is starting to get traction in the community.

This way we ended up among the official Gatling extension on the official Gatling homepage:

Screenshot 2019-12-16 at 15.03.32.png

The code of the git extension if opensource and free to use. It can be found here, and the library can be downloaded from Maven central. In case you want to raise a bug, you can do it here.

Maintainability

Gatling is written in Scala, and it expects the load tests scenarios to be written in Scala. Don’t be scared; there is no need to learn crazy functional programming paradigms, the Gatling DSL does a good job in abstracting the underneath framework. To write a scenario you just have to learn the building blocks made available by the DSL.
Here a couple of snippets extracted from a scenario to understand the DSL is:

class ReplayRecordsFromFeederScenario extends Simulation {
  // Boireplate to select the protocol and import the configuration
  val gitProtocol = GitProtocol()
  implicit val conf = GatlingGitConfiguration()

  // Feeder definition: the data used for the scenario will be loaded from "data/requests.json"
  val feeder = jsonFile("data/requests.json").circular

  // Scenario definition: 
  val replayCallsScenario: ScenarioBuilder =
    scenario("Git commands") // What's the scenario's name?
      .forever {  // How many time do I need to run though the feed?
        feed(feeder) // Where shall I get my data?
          .exec(new GitRequestBuilder(GitRequestSession("${cmd}", "${url}"))) // Build a Git request
      }

  setUp(
    replayCallsScenario.inject(
      // Traffic shape definition....pretty self explanatory
      nothingFor(4 seconds),
      atOnceUsers(10),
      rampUsers(10) during (5 seconds),
      constantUsersPerSec(20) during (15 seconds),
      constantUsersPerSec(20) during (15 seconds) randomized
    ))
    .protocols(gitProtocol) // Which protocol should I use?
    .maxDuration(60 seconds) // How long should I run the scenario for?
}

That is how the feeder looks like:

[
  {
    "url": "ssh://admin@localhost:29418/loadtest-repo.git",
    "cmd": "clone"
  },
  {
    "url": "http://localhost:8080/loadtest-repo.git",
    "cmd": "fetch"
  },
  {
    "url": "http://localhost:8080/loadtest-repo.git",
    "cmd": "push",
    "ref-spec": "HEAD:refs/for/master"
  }
]

Here another example reproducing the creation of a WIP change using the REST API:

class WIPWorkflow extends Simulation {

  // Configuration bolierplate
  implicit val conf: GatlingGitConfiguration = GatlingGitConfiguration()
  val baseUrl = "https://review.gerrithub.io"
  val username: String = conf.httpConfiguration.userName
  val password: String = conf.httpConfiguration.password
  val httpProtocol: HttpProtocolBuilder = http
    .baseUrl(baseUrl)
    .userAgentHeader("Gatling test")
  val request_headers: Map[String, String] = Map(
    "Content-Type" -> "application/json"
  )
  
  val scn = scenario("WIP Workflow") // What's the name of my scenario?
    .exec(
      http("Create WIP change")
        .post("/a/changes/") // Which url and which HTTP verb should I use? 
        .headers(request_headers) 
        .basicAuth(username, password) // How do I authenticate?
        .body( // What's the body of my request?
          StringBody("""{
            "project" : "GerritForge/sandbox/e2e-tests",
            "subject" : "Let's test this Gerrit! Create WIP changes!",
            "branch" : "master",
            "work_in_progress": "true",
            "status" : "NEW"
          }""")
        )
        .check(status.is(201)) // What's the response code I expect?
    )

  setUp(scn.inject(
    atOnceUsers(1) // Traffic profile
  )
  ).protocols(httpProtocol)
}

Jenkins integration

Running load tests can be tedious and time-consuming, but yet essential to spot any possible performance regression in your application.
Providing the least possible friction is essential to incentivize people in running them. If you are already using Jenkins in your company, you can leverage the Gatling plugin to scale your load quickly and provide easy access to metrics.

A real use case: Gerrit v3.0 Vs Gerrit v3.1 load test…in production!

Let’s go through a real case scenario to show how useful and easy to read are the metrics provided by Gatling.

The closest environment to your production one is…production!

gerrithub.io runs Gerrit in a multi-site configuration, and this gives us the luxury of doing canary releases, only upgrading a subset of the master nodes running Gerrit. One of the significant advantages is that we can run A/B tests in production.

That allows us also to run meaningful load tests against the production environment. See below a simplified picture of our Gerrit setup where it is possible to see the canary server with a higher Gerrit version.

Screenshot 2019-12-14 at 16.26.52.png

We ran against 2 servers in Germany the same load tests which:

  • Create 100 chained Change Sets via REST API
  • Submit all the changes together via REST API

We then compared the server-side and client-side metrics to see if a good job has been done with the latest Gerrit version.

Server-side metrics

Server-side metrics come from the Prometheus exporter plugin. The image is showing the HTTP requests mean time:

Screenshot 2019-12-14 at 16.32.12.png

We can see the improvement in the latest Gerrit version. The mean requests time is almost halved and, of course, the overall duration is decreased.

Client-side metrics

Let’s see what is going on on the client-side using the metrics provided by Gatling.

Among all the metrics we are going to focus on one step of the test, since we have more data points about it, the creation of the change:

Screenshot 2019-12-14 at 16.36.45.png

We can already see from the overall report the reduction of the response time in the latest Gerrit version:

Screenshot 2019-12-14 at 16.38.05.png

If we look in-depth to all the response times, we can see that the distribution of the response times is pretty much the same, but the scale is different….again we confirmed the result we previously encountered.

Screenshot 2019-12-14 at 16.39.17.png

What can we say…Good job Gerrit community!

Wrapping up

(You can see my presentation about Gatling in the last Gerrit User summit in Sunnyvale here <- add this when the talk will be sharable)

I have touched superficially several topics in this blog posts:

  • simplicity and maintainability provided by the Gatling DSL
  • Integration with Jenkins
  • Gatling extensions and reuse of the statistic engine
  • Example scenarios

I would like to write more in-depth about all these topics in some follow-up blog posts. Feel free to vote for the topic you are more interested in or suggest new ones in the comments section of this post.
If you need any help in setting up your scenario or understand how to run load tests against your Gerrit installation effectively, GerritForge can help you.

Fabio Ponciroli (aka Ponch) – GerritForge
Gerrit Code Review and Gatling Contributor

Gerrit User Summit LIVE!

Screenshot 2019-10-16 at 06.46.06

The Gerrit User Summit 2019 is going live and allows anyone to join and participate from across the world.

There are only 30 days left for the Gerrit User Summit 2019, the 12th annual event of the Gerrit Code Review community. It is the year of the records, with Gerrit reaching its largest audience ever in its 11 years of history:

  • Over 120 seats
  • People coming from 27 countries
  • 2 major dates and locations, in Sweden and in the USA
  • 20 talks and presentations
  • All seats sold out 2 months before the event

This is also a historical moment for the community because, for the first time since 2011, the JGit and Gerrit contributors will get together and talk to each other face to face, strengthening the cooperation between the two projects.

Do not miss the event, go live

We have received an enormous amount of requests to join the event on-site in Sunnyvale, much more than any previous year: the event was sold out on Eventbrite 2 months before the starting date.

GerritForge has then decided to invest further funding in sponsorship to organise a full live coverage of the event.

How to participate?

GerritForge has launched a new live event broadcasting site, https://live.gerritforge.com.

Watching the event will be FREE OF CHARGE and without adverts, thanks to the sponsorship by GerritForge. To assure the maximum quality of the video, there is a limit of on-line watchers and a pre-registration is needed.

  1. Go to https://live.gerritforge.com
  2. Click on the “Register to Watch” orange button
  3. Enter your full name, e-mail, company name and country of origin
  4. Click “Register to Watch” green button on the bottom of the page

The live event will allow remote attendants to ask questions and interact with the audience in Sunnyvale: it is going to be truly interactive and useful for the whole JGit and Gerrit community.

What to expect from the Sunnyvale event?

The Sunnyvale event includes a huge number of innovations on the JGit and Gerrit projects.

  • The introduction of the Git ref-table for repositories with huge number of refs
  • Support for Git protocol v2 in Gerrit
  • Git / Gerrit plugin for Gatling, for generating consistent end to end and load tests on Gerrit
  • Zuul support for the new Gerrit’s Checks CI integration
  • Introduction of Gerrit Code Review Analytics for the Android open-source project
  • Frictionless and zero downtime upgrades for Gerrit
  • and many more talks and presentations

Your last chance to attend, reserve your live spot now

There are brand-new ways this year to get in touch and be part of the Gerrit User Summit 2019.

Reserve your live spot today by registering at https://live.gerritforge.com and be part of this record event for the JGit and Gerrit Code Review community.

Luca Milanesio
Gerrit maintainer, release manager and ESC member

 

Gerrit User Summit at Volvo Cars

gerrit-user-summit-volvo-cars

The 2019 is a year of the Summit innovations

The Gerrit User Summit 2019 can definitely be defined as truly innovative in its format and audience.

For the first time in the Gerrit history, the Summit is split into two parts. Volvo Cars have hosted the first in Gothenburg (Sweden) while the second will take place from the 11th to the 17th of November at GerritForge Inc. HQ in Sunnyvale, CA (USA).

The Summit has been repeated on both sides of the Atlantic: the European and US communities come from different background and have different needs. The Gerrit Code Review Community is global and is willing to share experiences and receive feedback from both sides.

A truly open Gerrit Hackathon

We are also innovating on the Hackathon perspective, with three new elements:

  1. The Hackathon is now open to everyone, including the people that have never contributed to Gerrit before. Experienced maintainers have paired with newbies to guide through the very first contributions.
  2. The Hackathon at Volvo Cars has been 100% focused in triaging the massive backlog of open issues and fixing as many bugs as possible for the latest three supported branches: stable-3.0, stable-2.16 and stable-2.15.
  3. The OpenStack and Gerrit communities finally have met and started talking and interacting more closely.

Read the full story on gerritcodereview.com/news.html

The full summary of the event has been published on the Gerrit Code Review project news, read what happened in Gothenburg and, if you are in the USA, do not miss the next forthcoming Gerrit User Summit USA in Sunnyvale.

Hurry up as the seats are running out, REGISTER NOW to avoid missing the event.

Luca Milanesio (GerritForge Ltd)
Gerrit Maintainer, Release Manager, ESC member

Gerrit User Summit 2019

gothenburg-sunnyvale

The Gerrit User Summit 2019 is approaching fast, with new exciting features and a brand-new Gerrit v3.0 release to present and discuss together.

The event is FREE but you need to register in advance for the Gerrit User Summit 2019 on Eventbrite.

One Summit, two events

The Gerrit User Summit & Hackathon is composed of two different events and locations, one in Sweden (Europe), hosted and sponsored by Volvo Cars, and another in California (USA) in the new GerritForge Inc. HQ. Having two separate events in two different quarters will allow most of the community around the globe to attend and share their experience and ideas.

Hackathon open to new contributors

The first part of the event is a 5-days Hackathon reserved for the current Gerrit contributors and maintainers plus anyone that is willing to start contributing to the platform. Differently, from the previous years, the community is now welcome even people that have not contributed to Gerrit before but they are willing to do so.

It is a fantastic opportunity for people to join, work side-by-side and pair with the Gerrit maintainers for a whole week. It can be a unique opportunity to implement the features that you always wanted to see in Gerrit and learning how we develop and review our changes.

The Summit

The usual 2-days Users Summit after the Hackathon is opened to all the members of the community or who is willing to adopt Gerrit Code Review in their development process in the near future.

This year there are a number of exciting news:

  • The introduction of an official Gerrit Community Process with an Engineering Steering Committee and Community Managers
  • Gerrit v3.0 and the full migration to NoteDb and PolyGerrit
  • The multi-site plugin goes OpenSource for allowing anyone to run multiple masters on different sites

The full schedule of the event is available on the Gerrit User Summit 2019 site.

Proposing a new talk

More talks and customer stories are scheduled and, if you have something to tell to the rest of the community, you can submit your talk by creating a change and push to Gerrit Summit 2019 repository:

  1. Open the repository commands age at https://gerrit-review.googlesource.com/admin/repos/summit/2019,commands
  2. Click on “CREATE CHANGE”
  3. Select “master” branch, put your description and create the new change
  4. Click on “Edit” on the top-right of the page
  5. Click on “Open” in the mid toolbar and open the page you would like to edit.

For proposing a new session, you need to add one file with the name of your talk into the sessions folder, using the template.md as an example.

Where and when

Volvo Cars HQ at their HQ in Gothenburg (Sweden)

  • 24-28th August 2019 – Gerrit Hackathon Europe
  • 29-30th August 2019 – Gerrit User Summit Europe

GerritForge Inc HQ in Sunnyvale CA (USA)

  • 11-15th November 2019 – Gerrit Hackathon USA
  • 16-17th November 2019 – Gerrit User Summit USA

Thanks to our sponsors

I would like to thank Volvo Cars and Nicholas Mucci for hosting, sponsoring and organizing the Gerrit User Summit Europe in Gothenburg (Sweden) and GerritForge for hosting and sponsoring the events in both Europe and the USA in Sunnyvale CA (USA).

Luca Milanesio
Gerrit Code Review Maintainer and Release Manager
Member of the Engineering Steering Committee

 

 

 

 

GerritHub.io is moving to Gerrit v3.0

It has been a very long journey, from the initial adoption of PolyGerrit at GerritHub to the epic moment where Gerrit historic GWT was dropped with the Gerrit v3.0 last month.

GerritHub.io has always been aligned with the latest and greatest of Gerrit Code Review and thus the moment has come for us to upgrade to v3.0 and drop forever the GWT UI.

PolyGerrit vs. GWT adoption

Screenshot 2019-06-10 at 21.16.48

The PolyGerrit UX was pretty much experimental until the beginning of 2018: the features were incomplete and people needed to go back to the old GWT UI for many of the basic use-cases.

However, things started to change radically in April 2018 when GerritHub.io adopted Gerrit v2.15 which had a 100% functionally complete PolyGerrit UI. The number of users choosing PolyGerrit jumped from 10% to 35% (3.5x times) with a +70% growth in the number of accesses overall. That means that the adoption was mainly driven by users attracted by the new UI.

In the past 12 months, PolyGerrit became the default user-interface and was just renamed as Gerrit UI. Gradually more and more users abandoned the old GWT interface that now represents 30% of the overall accesses.

Timeline of the upgrade

For the 70% of people that are using already using the new Gerrit UI, the upgrade to Gerrit v3.0 would not be noticeable at all:

  • Gerrit v3.0 UI is absolutely identical to the current one in v2.16
  • All existing API and integration points (e.g. Jenkins integration) in Gerrit v3.0 are 100% compatible with v2.16

For the 30% of people that are still using the old GWT UI, things will be very different as their favorite interface will not be available anymore.

The upgrade will happen with zero-downtime across the various GerritHub.io multi-site deployments and will start around mid-June.

Can I still use GWT with GerritHub.io?

The simple answer is NO: Gerrit v3.0 does not contain any GWT code anymore and thus it is impossible for GerritHub.io to bring back the old UI.

The journey to fill the gaps and reach 100% feature and functional equivalence between the old GWT and the new Polymer-based UI took around 6 years, 18k commits and 1M lines of code written by 260+ contributors from 60+ different organizations. It has been tested by hundreds of thousands of developers across the globe and is 100% production-ready and functionally complete.

If you feel that there was “something you could do in the GWT UI and cannot do anymore with the new Polymer-based UI”, please file a bug to the Gerrit Code Review issue tracker and you will get prompt attention and replies from the community.

Can I stay with Gerrit v2.16 on GerritHub.io?

If your organization cannot migrate to Gerrit v3.0, you could still request a dedicated hosting to GerritForge Ltd, which is the company behind GerritHub.io.

Please fill up the GerritForge feedback form and one Sales Representative will come back to you with the possible options and costs associated.

If you fully endorse GerritHub.io with Gerrit v3.0 and start using the new UI, the service will continue to be FREE for public and private repositories, organizations of all types and size. You can optionally purchase Enterprise Support from one of our plans if you require extra help in using and configuring your Gerrit projects with your tools and organization.

Enjoy the future of Gerrit v3.0 with GerritHub.io and GerritForge.

Luca Milanesio, GerritForge Ltd.
Gerrit Code Review Maintainer and Release Manager
Member of the Engineering Steering Committee

Gerrit v3.0 is here

GerritSprintHackathon2019.photo

Gerrit v3.0 has been released during the last Spring Hackathon at Google in Munich involving over 20+ developers for one week.

It can be downloaded from www.gerritcodereview.com/3.0.html and installed on top of any existing Gerrit v2.16/NoteDb installations. Native packages have been distributed through the standard channels and upgrading is as simple as shutting down the service, running the Rpm, Deb or Dnf upgrade command and starting again.

You can also try Gerrit v3.0 using Docker by simply running the following command:

docker run -ti -p 8080:8080 -p 29418:29418 gerritcodereview/gerrit:3.0.0

This article goes through the whole history of the Gerrit v3.0 development and highlights the differences between the previous releases.

Milestone for the Gerrit OpenSource Project

Finally, after 6 years, 18k commits and 1M lines of code written by 260+ contributors from 60+ different organizations, Gerrit v3.0 is finally out.

The event is a fundamental milestone for the project for two reasons:

  • The start of a new journey for Gerrit, without the legacy code of the old GUI based on Google Web Toolkit and without any relational database. Gerrit is now fully based on a Git repository and nothing else.
  • The definition of a clear community organization, with the foundation of a new Engineering Steering Committee and the role of Community Manager.

The new structure will drive the product forward for the years to come and will help to define a clear roadmap to bring back Gerrit at the center of the Software Development Pipeline.

Evolution vs. revolution

When a product release increments the first major number, it typically introduces a series of massive breaking changes and, unfortunately, a period of instability. Gerrit, however, is NOT a typical OpenSource product, because since the beginning it has been based on rigorous Code Review that brought stability and reliability from its initial inception back in 2008. Gerrit v3.0 was developed during the years by following a rigorous backward compatibility rule that has made Gerrit one of the most reliable and scalable Code Review systems on the planet.

For all the existing Gerrit v2.16 installations, the v3.0 will be much more similar to a rather minor upgrade and may not even require any downtime and interruption of the incoming read/write traffic, assuming that you have at least a high-availability setup. How is this possible? Magic? Basically, yes, it’s a “kind of magic” that made this happen, and it is all thanks to the new repository format for storing all the review meta-data: NoteDb.

Last but not least, all the feature that Gerrit v3.0 brings to the table, have been implemented iteratively over the last 6 years and released gradually from v2.13 onwards. Gerrit v3.0 is the “final step” of the implementation that fills the gaps left open in the past v2.16 release.

With regards to statistics of the changes from v2.16 to v3.0, it is clear that the code-base has been basically stabilized and cleaned up, as you can see from the official GerritForge Code Analytics extracted from analytics.gerrithub.io .

  • 1.5k commits from 63 contributors worldwide
  • 62k lines added and 72k lines removed
  • Google, CollabNet, and GerritForge are the top#3 organizations that invested in developing this release

In a nutshell, the Gerrit code-base has shrunk of 10k lines of code, compared to v2.16. So, instead of talking of what’s new in v3.0, we should instead describe what inside the 72k lines removed.

Removal of the GWT UI

The GWT UI, also referred to as “Old UI” has been around since the inception of the project back in 2008.

Gerrit.GWT-UI

Back in 2008, it seemed a good idea to build Gerrit UI on top of GWT, a Web Framework founded by Google two years earlier and aimed at reusing the same Java language for both backend and the Ajax front-end.

However, starting in 2012, things started to change. The interest of the overall community in GWT decreased, as clearly shown by the StackOverflow trends.

Screenshot 2019-05-18 at 23.34.42

In 2015, Andrew Bonventre from the Chromium Project, one of the major users of the Gerrit Code Review platform, apart from the Android Developers, presented the new prototype of the Gerrit Code Review UI, based on the Polymer project, with the code-name of PolyGerrit, and merged as change #72086.

commit ba698359647f565421880b0487d20df086e7f82a
Author: Andrew Bonventre <andybons@google.com>
Date: Wed Nov 4 11:14:54 2015 -0500

Add the skeleton of a new UI based on Polymer, PolyGerrit

This is the beginnings of an experimental new non-GWT web UI developed
using a modern JS web framework, http://www.polymer-project.org/. It
will coexist alongside the GWT UI until it is feature-complete.

The functionality of this change is light years from complete, with
a full laundry list of things that don't work. This change is simply
meant to get the starting work in and continue iteration afterward.

The contents of the polygerrit-ui directory started as the full tree of
https://github.com/andybons/polygerrit at 219f531, plus a few more
local changes since review started. In the future this directory will
be pruned, rearranged, and integrated with the Buck build.

Change-Id: Ifb6f5429e8031ee049225cdafa244ad1c21bf5b5

The PolyGerrit project introduced two major innovations:

  • Gerrit REST-API: for the first time the interaction of the code-review process has been formalized in stable and well-documented REST-API that can be used as “backend contract” for the design of the new GUI
  • The PolyGerrit front-end Team: for the first time, a specific experienced Team focused on user experience and UI workflow was dedicated to rethink and redesign iteratively all the components of the Gerrit Code Review interactions.

The GWT UI and PolyGerrit lived in the same “package” from v2.14 onwards for two years, with the users left with the option to switch between the two. Then in 2018 with v2.16 the PolyGerrit UI became the “default” interface and thus renamed just “Gerrit” UI.

With Gerrit v3.0, the entire GWT code-base in Gerrit has been completely removed with the epic change by David Ostrovsky “Remove GWT UI“, which deleted 33k lines of code in one single commit.

The new Polymer-based UI of Gerrit Code Review is not very different than the one seen in Gerrit v2.16, but includes more bug fixes and is 100% feature complete, including the projects administrations and ACLs configuration.

Screenshot 2019-05-18 at 22.58.13

Removal of ReviewDb

Gerrit v3.0 does not have a DBMS anymore, not even for storing its schema version as it happened in v2.16. This means that almost everything gets stored in the Git repositories.

The journey started back in October 2013, when Shawn Pearce gave to Dave Borowitz the task to convert all the review meta-data managed by Gerrit into a new format inside the Git repository, called NoteDb.

After two years of design and implementation, Dave Borowitz presented NoteDb at the Gerrit User Summit 2015 and called Gerrit v3.0 the release that will be fully working without the need of any other external DBMS (see the full description of the talk at https://storage.googleapis.com/gerrit-talks/summit/2015/NoteDB.pdf).

Google started adopting NoteDb in parallel with ReviewDb on their own internal setup and in June 2017, the old changes table was definitely removed. However, there was more in the todo-list: at the Gerrit User Summit 2017, Dave Borowitz presented the final roadmap to make ReviewDb finally disappear from everyone’s Gerrit server.

Screenshot 2019-05-18 at 23.18.28

In the initial plans, the first version with NoteDb fully working should have been v2.15. However, things went a bit differently and a new minor release was needed in 2018 to make the format really stable and reliable with v2.16.

Gerrit v2.16 is officially the last release that contains both code-bases and allows the migration from ReviewDb to NoteDb.

Dave Borowitz used the hashtag “RemoveReviewDb” to allow anyone to visualize the huge set of commits that removed 35k lines of code complexity from the Gerrit project.

Migrating to Gerrit v3.0, step-by-step

Gerrit v3.0 requires NoteDb as pre-requisite: if you are on v2.16 with NoteDb, the migration to v3.0 is straightforward and can be done with the following simple steps:

  1. Shutdown Gerrit
  2. Upgrade Gerrit war and plugins
  3. Run Gerrit init with the “batch” option
  4. Start Gerrit

If you are running Gerrit in a high-availability configuration, the above process can be executed on the two nodes individually, with a rolling restart and without interrupting the incoming traffic.

If you are running an earlier version of Gerrit and you are still on ReviewDb, then you should upgrade in three steps:

  1. Migrate from your version v2.x (x < v2.16) to v2.16 staying on ReviewDb. Make sure to upgrade through all the intermediate versions. (Example: migrate from v2.13 to v2.14, then from v2.14 to v2.15 and finally from v2.15 to v2.16)
  2. Convert v2.16 from ReviewDb to NoteDb
  3. Migrate v2.16 to v3.0

The leftover of a DBMS stored onto H2 files

Is Gerrit v3.0 completely running without any DBMS at all? Yes and no. There is some leftover that isn’t necessarily associated with the Code Review meta-data and thus did not make sense to be stored in NoteDb.

  • Persistent storage for in-memory caches.
    Some of the Gerrit caches store their status on the filesystem as H2 tables, so that Gerrit can save a lot of CPU time after a restart reusing the previous in-memory cache status.
  • Reviewed flag of changes.
    Represents the flag that enables the “bold” rendering of a change, storing the update status for every user. It is stored by default on the filesystem as H2 table, however, can be alternatively stored on a remote DBMS or potentially managed by a plugin.

New core plugins

Some of the plugins that have been initially distributed only with the Native Packages and Docker versions are now an integral part of the WAR distribution as well:

  • delete-project
    which allows removing a project from Gerrit and the associated changes.
  • gitiles
    a lightweight code-browser created by Dave Borowitz based on JGit
  • plugin-manager
    the interface to discover, download and install Gerrit plugins
  • webhooks
    the HTTP-based remote trigger to schedule remote builds on CI systems or active any other service from a Gerrit event

The above four plugins already existed before Gerrit v3.0, but they were not included in the gerrit.war.

Farewell to Dave Borowitz and the PolyGerrit Team

After having completed the feature parity between GWT and PolyGerrit, the original PolyGerrit Team members left the Gerrit Code Review project.

Their journey came to an end with the release of the new shiny Polymer-based Gerrit UI. The PolyGerrit Team contributed 45k lines of code on 5.3k commits in 4 years.

Then the last event unfolded during the release of Gerrit v3.0: Dave Borowitz announced that he was leaving the Gerrit Code Review project. I defined the event like “Linus Torvalds announcing he was abandoning the Linux Kernel project”.

Dave Borowitz contributed 316k lines of code on 3.6k commits over 36 repositories in 8 years. He helped also the development of the new Gerrit Multi-Site plugin by donating its Zookeeper-based implementation of a global ref-database.

On behalf of GerritForge and the Gerrit Code Review community, I would like to thank all the past contributors and maintainers that made PolyGerrit and NoteDb code-base into Gerrit: Dave, Logan, Kasper, Becky, Viktar, Andrew and Wyatt.

Luca Milanesio – GerritForge
Gerrit Code Review Maintainer, Release Manager
and member of the Engineering Steering Committee

Jenkins ❤︎ Gerrit Code Review, again

Gerrit Code Review has been integrated with Jenkins for over nine years. It was back when Kohsuke was still a Senior Engineer at Sun Microsystem, which was just announced to be acquired by Oracle and his OpenSource CI project was still called Hudson.

Jenkins and Gerrit are the most critical components of the DevOps Pipeline because of their focus on people (the developers), their code and collaboration (the code review) their builds and tests (the Jenkinsfile pipeline) that produce the value stream to the end user.

The integration between code and build is so important that other solutions like GitLab have made it a unique integrated tool and even GitHub has started covering the “last mile” a few months ago by offering powerful actions APIs and workflow to automate build actions around the code collaboration.

Accelerate the CI/CD pipeline

DevOps is all about iteration and fast feedback. That can be achieved by automating the build and verification of the code changes into a target environment, by allowing all the stakeholder to have early access to what the feature will look like and validating the results with speed and quality at the same time.

Every development team wants to make the cycle time smaller and spend less time in tedious work by automating it as much as possible. That trend has created a new explosion of fully automated processes called “Bots” that are more and more responsible for performing those tasks that developers are not interested in doing manually over and over again.

As a result, developers are doing more creative and design work, are more motivated and productive, can address technical debt a lot sooner and allow the business to go faster in more innovative directions.

As more and more companies are adopting DevOps, it becomes more important to be better and faster than your competitors. The most effective way to accelerate is to extract your data, understand where your bottlenecks are, experiment changes and measure progress.

Humans vs. Bots

The Gerrit Code Review project is fully based on an automated DevOps pipeline using Jenkins. We collect the data produced during the development and testing of the platform and extract metrics and graphs around it constantly https://analytics.gerrithub.io thanks to the OpenSource solution Gerrit DevOps Analytics (aka GDA).

By looking at the protocol and code statistics, we founded out that bots are much more hard worker than humans on GerritHub.io, which hosts, apart from the Gerrit Code Review mirrored projects, also many other popular OpenSource.

That should not come as a surprise if you think of how many activities could potentially happen whenever a PatchSet is submitted in Gerrit: style checking, static code analysis, unit and integration testing, etc.

human-vs-bot

We also noticed that most of the activities of the bots are over SSH. We started to analyze what the Bots are doing and see what the impact is on our service and possibly see if there are any improvements we can do.

Build integration, the wrong way

GerritHub has an active site with multiple nodes serving read/write traffic and a disaster recovery site ready to take over whenever the active one has any problem.

Whenever we roll out a new version of Gerrit, using the so-called ping-pong technique, we swap the roles of the two sites (see here for more details).  Within the same site, also, the traffic can jump from one to the other in the same cluster using active failover, based on health, load and availability. The issue is that we end up in a situation like the following:

Basic Use Case Diagram

The “old” instance still served SSH traffic after the switch. We noticed we had loads of long-lived SSH connections. These are mostly integration tools keeping SSH connections open listening to Gerrit events.

Long-lived SSH connections have several issues:

  • SSH traffic doesn’t allow smart routing. Hence we end up with HTTP traffic going on the currently active node and most of the SSH one still on the old one
  • There is no resource pooling since the connections are not released
  • There is the potential loss of events when restarting the connections

That impacts the overall stability of the software delivery lifecycle, extending the feedback loop and slowing your DevOps pipeline down.

Then we started to drill down into the stateful connections to understand why they exist, where are they coming from and, most importantly, which part of the pipeline they belong to.

Jenkins Integration use-case

The Gerrit Trigger plugin for Jenkins is one of the integration tools that has historically been suffering from those problems, and unfortunately, the initial tight integration has become over the years less effective, slow and complex to use.

There are mainly two options to integrate Jenkins with Gerrit:

We use both of them with the Gerrit Code Review project, and we have put together a summary of how they compare to each other:

Gerrit Trigger Plugin Gerrit Code review Plugin Notes
Trigger mechanism Stateful

Jenkins listens for Gerrit events stream

Stateless

Gerrit webhooks notify events to Jenkins

Stateful stream events are consuming resources on both Jenkins and Gerrit
Transport Protocol SSH session on a long-lived stream events connection HTTP calls for each individual stream event – SSH cannot be load-balanced
– SSH connections cannot be pooled or reused
Setup Complexity Hard: requires a node-level and project-level configuration.

No native Jenkinsfile pipeline integration

Easy: no special knowledge required.

Integrates natively with Jenkinsfile and multi-branch pipeline

Configuring the Gerrit Trigger Plugin is more error-prone because requires a lot of parameters and settings.
Systems dependencies Tightly Coupled with Gerrit versions and plugins. Uses Gerrit as a generic Git server, loosely coupled. Upgrade of Gerrit might break the Gerrit Trigger Plugin integration.
Gerrit knowledge Admin: You need to know a lot of Gerrit-specific settings to integrate with Jenkins. User. You only need to know Gerrit clone URL and credentials. The Gerrit Trigger plugin requires special user and permissions to listen to Gerrit stream events.
Fault tolerance to Jenkins restart Missed events: unless you install a server-side DB to capture and replay the events. Transparent: all events are sent as soon as Jenkins is back. Gerrit webhook automatically tracks and retries events transparently.
Tolerance to Gerrit rolling restart Events stuck: Gerrit events are stuck until the connection is reset. Transparent: any of the Gerrit nodes active continue to send events. Gerrit trigger plugin is forced to terminate stream with a watchdog, but will still miss events.
Differentiate actions per stage No flexibility to tailor the Gerrit labels to each stage of the Jenkinsfile pipeline. Full availability to Gerrit labels and comments in the Jenkinsfile pipeline
Multi-branch support Custom: you need to use the Gerrit Trigger Plugin environment variables to checkout the right branch. Native: integrates with the multi-branch projects and Jenkinsfile pipelines, without having to setup anything special.

Gerrit and Jenkins friends again

After so many years of adoption, evolution and also struggles of using them together, finally Gerrit Code Review has the first-class integration with Jenkins, liberating the Development Team from the tedious configuration and BAU management of triggering a build from a change under review.

Jenkins users truly love using Gerrit and the other way around, friends and productive together, again.

Conclusion

Thanks to Gerrit DevOps Analytics (GDA) we managed to find one of the bottlenecks of the Gerrit DevOps Pipeline and making changes to make it faster, more effective and reliable than ever before.

In this case, by just picking the right Jenkins integration plugin, your Gerrit Code Review Master Server would run faster, with less resource utilization. Your Jenkins pipeline is going to be simpler and more reliable with the validation of each change under review, without delays or hiccups.

The Gerrit Code Review plugin for Jenkins is definitively the first-class integration to Gerrit. Give it a try yourself, you won’t believe how easy it is to set up.

Fabio Ponciroli
Gerrit Code Review Contributor, GerritForge.