2023: New Year and opportunities for GerritForge and Gerrit Code Review

Posted on by
Reply

TL;DR: GerritForge has been dedicating its efforts to organising and managing the Gerrit User Summit in London back in November 2022, in conjunction with the release of Gerrit v3.7. The event has been a great success, with a significant presence on-site and record-breaking attendees on the GerritForge TV youtube channel. It has also committed to its promises to research and improve the JGit and Gerrit scalability to large mono-repos, with tens of millions of objects and refs. 2023 will see the finalisation of these efforts with an increase in development efforts and a new JGit Committer for pushing the platform to a new level of performance and scalability and a new innovating system for collecting and optimising the repository metrics automatically. Stay tuned.

Read the full story here below (9 mins read).

2022 has been a critical year for turning the Gerrit Code Review community and development back on track after the COVID-19 pandemic. At GerritForge, we’ve been working hard to make sure that the development, support, and innovation of Gerrit Code Review continue on its main objectives.

Gerrit Code Review v3.6 and v3.7

We have continued to deliver on the development and release of Gerrit Code Review and its plugins, helping the testing and releasing of versions v3.6.0 (May) and v3.7.0 (November).

Some numbers of the past 12 months’ development contributions by individual committers and companies:

  • 3,627 Changes have been merged on 76 projects related to the Gerrit Code Review platform, including JGit
  • 113 committers from 42 different organisations

A special mention to the top #10 contributors: Google (Ben Rohlfs, Edwin Kempin, Chris Pouchet, Dhruv Srivastava, Frank Borden, Milutin Kristofic), GerritForge (Luca Milanesio), Wikimedia (Paladox) and SAP (Matthias Sohn and Thomas Dräbing).

In comparison with 2021, we had 25% fewer changes merged but with more contributors coming from more companies, which is a symptom to a very healthy and thriving ecosystem of maintainers.

GerritForge has committed to resuming the face-to-face user summits, which were suspended since 2020.

The Gerrit User Summit 2022 took place in London, UK the 10-11 of November in a hybrid format, with people having the opportunity to participate either on-site or remotely on GerritForge’s YouTube TV channel.

It was a glorious success, with record-breaking attendance from all around the globe:

  • 50 people registered to attend on-site, 26 of them managed to arrive despite the London tube strike, whilst the others attended remotely
  • 235 people viewed the summit on YouTube with an average view time of 40 mins (one talk)

The summit survey had an outstanding report showing a huge acceptance and appreciation of the event:

  • 82% rated the remote video streaming as “good” or “outstanding”
  • 96% rated the quality of the summit as “good” or “outstanding.”
  • 100% would recommend the summit to a colleague, with 83% strongly recommending it

GerritHub.io SLA gets closer to five-nines.

We have been working hard to make Gerrit more stable and resilient throughout 2022, discovering and fixing many issues in the code base and on the multi-site software architecture.
In 2022, GerritHub.io had only six small hiccups for a total of 19 mins of downtime (SLA = 99.997%) over a 12-month period, a 75% reliability improvement compared to 2021.

We have run extensive RCAs on the causes of the downtime and identified two leading issues, which are explained in the details below.

The “anonymous unlimited query” hole in Gerrit
GerritHub.io has been subject to a 15 mins outage because of anonymous users being able to bring offline all the sites before the system could auto-recover.
Gerrit allows bypassing of all limits set in the ACLs for running queries by simply adding the “no-limit” parameter.
Returning an arbitrary payload without limits could allow a single user to generate a server-side workload for collecting and building a GBytes-sized JSON payload; unfortunately, that option was available to everyone, including anonymous users making any publicly faced Gerrit Code Review installation subject to deny-of-service attacks.
We have identified the issue, reported and fixed it in Gerrit with Change 333304, which has been included in Gerrit v3.3.10, v3.4.4, v3.5.1, and all v3.6.0 or later releases.

More granular monitoring and alerting
We have lowered the threshold of uptime checks on GerritHub.io to 1 minute, giving us the ability to detect and react immediately to 4 smaller hiccups. We have detected a lack of scalability for some specific higher-load projects. Those hiccups have been responsible for 2 mins of downtime over the 2nd part of 2022. Many more projects are also planning to be onboarded on GerritHub.io; hence we do need to address this project-specific capacity needs.

Scaling Gerrit Code Review and JGit beyond its limits

We have been investing a massive effort in building a test environment designed to stress Gerrit and JGit to its limits and identify all the limitations and bottlenecks that prevented us from scaling further.

Scaling the test repository
We have created over the months some test repositories that increased in every dimension:

  • Tens of millions of refs as both refs/changes and refs/heads
  • Millions of delta-chains
  • Tens of millions of Git objects
  • Packfiles of tens of Giga-bytes and packed refs of hundreds of megabytes

For generating a significant load on both client and server side, we have invested more into the aws-gerrit cloud setups and gatling-git performance loading tool.

There were some “well-known” issues and additional surprising ones.

SHA1 complexity and CPU utilization for large entities
JGit has been used SHA1 for identifying uniqueness not just for Git objects but also for other large entities. However, computing SHA1 has become increasingly CPU intensive because of the relatively recent findings about collisions on shattered.io.
We have highlighted two major potential improvements in cooperation with Matthias Sohn (SAP) on the raw SHA1 performance and its application for detecting packed-refs changes on the filesystem.

Commit priority queues
JGit has a custom implementation of priority queues which are intensively used in RevWalk, which has almost quadratic complexity. That isn’t a problem for small to medium chains of commits; however, when the number of commits reaches millions, the performance degradation becomes unbearable.
We have replaced the JGit’s custom implementation with the one provided by the Java JVM library, which has a logarithmic complexity that massively improves its performance with large commit chains.

Unwanted reachability checks
JGit needs to perform a full reachability check whenever a remote unknown client is advertising refs, which makes sense when serving a remote client. However, the cost of full reachability of millions of advertised refs can be a daunting task that may be alleviated if the remote end can be considered trusted.

Fixing JGit bitmaps
Since the introduction of Git bitmap, the whole community has learned how key they are in speeding up the counting and selection during the clone phase.
However, large and unoptimized bitmaps could be so unhelpful for Git that instead of speeding up, they could represent a massive overhead for the system, causing CPU spikes and, eventually, lowering the throughput of the server.
Git bitmaps are compressed using the JavaEWAH library, which is good for memory consumption but evil for CPU utilization: that is the reason why the smaller is best for performance.
We have discovered and fixed a critical issue with the JGit bitmap generation that was causing the inclusion of all commits and BLOBs pointed by annotated tags. Also, we have introduced the ability to inform JGit about the heads that can be excluded from the bitmap, allowing to shorten the creation tens of thousands times (5h generation time for a 2k refs to as little as 60s) and increase its effectiveness by 200%.

Millions of unneeded ref logs
When performing a clone of a repository with millions of heads, JGit created one local reflog file for every remote ref, including the ones there were not actually cloned but just fetched as remote references. This was creating a significant performance gap between JGit and Git, which would instead lazily create the reflog files once they are effectively checked out the first time. Cloning a single branch of a repository with millions of remote refs took around 1h, compared to a few minutes of Git.

All of the findings were included in multiple updates on the following components:

  • JGit changes: all fixes were also provided to stable-5.13, the last supported branch for Java 8, which allows benefiting from these improvements for older versions of Gerrit from v2.16 onwards.
  • pull-replication went through major performance improvements, achieving a 1000x times faster execution time compared to the traditional replication plugin
  • aws-gerrit is going through upgrades for making use of pull-replication plugin, including the support for the bearer token which allows to replicate virtually any repository, including All-Users.git
  • gatling-git: we have upgraded the Gatling version and JGit to the latest stable-5.13 to include the latest performance improvements.
  • git-repo-metrics: we have introduced a brand-new plugin that allows us to keep under control the major dimensions of a repository and therefore graph their increase over time.

GerritForge goals for 2023

We are definitely not done yet with the performance improvements on Gerrit and JGit: there are still significant improvements to be made, and JGit changes to get merged into the mainstream branches.
We believe we are on track to finalize the job and allow a stable and scalable platform for large Git repositories in 2023.

Finalise what we cooked in 2022 for JGit
JGit has a new maintainer, David Ostrovsky, awarded in 2022 as Git committer of the project. GerritForge’s devs are focused to get more reviews and attention to the JGit performance improvements. We are committed to finalising all the open changes related to large repositories.

JGit multi-pack indexes support
There is still a major gap between JGit and Git when dealing with very active repositories: multi-pack indexes. The proliferation of packfiles would eventually lead to a long and painful search-for-reuse phase for BLOBs which could be cut down 100s of times with a multi-pack index.

Git repository optimiser for Gerrit
We have been working on tracking the live information on the Git repository, thanks to the git-repo-metrics plugin. Wouldn’t it be nice to have a tool that can do something with it and automatically?
We would be doing R&D on how to correlate the repository metrics, the Git audit trail, and the performance data for making AI-based decisions on what needs to be improved on the repository.
This work stream is going to be useful for any Git repository, not just the ones powered by Gerrit Code Review. The ‘git-repo-metrics’ and the repository optimiser would also apply to other products, including GitHub and GitLab.

Gerrit v3.8 and projects-specific change numbers
We will finalise the design document for the transition to project-specific change numbers in Gerrit v3.8. That would allow the seamless migration of projects across Gerrit setups without having to worry about changes renumbering anymore.

Gerrit Code Review testing and GerritForge-certified binaries
GerritForge is spending a tremendous amount of time developing test environments and tools for serving the Gerrit community with more stable releases and improving the quality of its code. We want to intensify the effort and also offer our platinum support customers a unique service that includes the GerritForge digital signature and rubber stamp on the binaries of Gerrit Code Review and its plugins that have been successfully tested and validated for being production-ready.
Stay tuned; more details are coming soon …

GerritForge company forecast in 2023

GerritForge Inc. will finalise its roll-out to the USA, and all contracts and services will be run from Sunnyvale, CA and Europe. Over 2022, 60% of the customers and businesses have already been moved, and the operation will be completed over the course of 2023.

We are looking forward to doubling our revenue figures in 2023 and also our contributions to the open-source community, with a main focus on JGit as the driver of performance growth for Gerrit Code Review.

2023 is going to be an incredible year for GerritForge, Gerrit Code Review, and the JGit community altogether.

Happy New start of the Year 2023!

Luca Milanesio (GerritForge)
Gerrit Code Review Maintainer and Release Manager
Member of the Gerrit Engineering Steering Committee

ARM-64 welcomes Gerrit Code Review

Posted on by
Reply

It has been seven years since we first introduced the native packages for Gerrit Code Review. At that time, Docker was just two years old and was starting to pick up momentum: the native packages found their first incarnation in the Gerrit Code Review base on images published on DockerHub.

Since then, Gerrit has always been released as both a “plain vanilla” war file and the equivalent “native packaging” flavors, including the base Docker images.

In the beginning, the release managers mainly used the Gerrit Docker images to simulate pre-configured test environments for testing purposes. However, the adoption spiked up when Kubernetes and AWS Container Services (aka ECS) started to become mainstream.

Nowadays, the Gerrit base images on Dockerhub are pulled millions of times and used as the baseline for creating Enterprise-grade setups, like in the aws-gerrit recipes.

x86 Native Packaging

Even though it wasn’t initially thought off, the native packages ended up mainly in x86-based containers, even though Gerrit is a 100% pure-Java application. How was that possible?

  • Dependencies: gerrit.sh requires some basic OS services to work properly, including the native git command line.
  • Build & testing: the build of gerrit.war and associated testing was always performed on x86, also due to the restrictions of the platform supported by Bazel at that time.

ARM-64 takes off

The ARM processor comes from the historical legacy of Acorn Computers Ltd, a British-born company of the 80s, well known for the BBC Micro project. Back in the 90s, a new joint venture was created with Apple Computer and VLSI Technology to design a brand-new Advanced Risc Machine (aka ARM).

Over time, ARM has created first a 32-bit architecture and subsequently a 64-bit architecture processor, well known and appreciated for its low-power consumption and performance efficiency.

Whilst Intel, with its x86 architecture, dominated the desktop and the server market, the ARM architecture found its natural market space in the low-power devices and, more recently, the mobile phones market.

Apple goes ARM-64, everywhere

Apple initially adopted the ARM architecture on its mobile phones and portable devices, licensing the instruction set to build its microchips in 2010. Ten years later, Apple announced it was also transitioning all the other desktop devices to its ARM-based silicon.

Nowadays, nearly all Apple laptops are on ARM and the other devices are migrating to that too. The applications are rapidly adapting to the new instruction set and getting rebuilt quickly, gaining in both performance and increased battery life.

Cloud services are moving to ARM-64 too.

The move from x86 to ARM-64 brings noticeable advantages in terms of costs and energy efficiency has been recently amplified by the increase in the costs of electricity triggered by the spiked wholesale prices of oil, gas, and other fossil fuel.

GCloud (see T2A offering ) and AWS (see EC2 Graviton) offer now ARM-64 based VMs that promise a healthy 60% reduction of cloud hosting costs and higher throughput and performance per $.

Gerrit has always put high performance and low operating costs at the top of its agenda, allowing companies to keep their development teams highly productive with minimum dedicated staff looking after the system.

Now it is also time to reduce the Gerrit Code Review Cloud hosting costs and increase efficiency while preserving all the benefits that Gerrit brings.

Introducing ARM-64 native packages for Gerrit Code Review

Gerrit Code Review native packages are now available for ARM-64 on all its recent releases, from v3.4.8 onwards. If you are currently running on x86, you can easily switch to ARM-64 transparently by just rebuilding from the base image. All the DockerHub images have been repackaged and republished:

Gerrit Native packages on bare-metal

If you are running Gerrit native packages on bare-metal, you would need to setup the GerritForge’s updated RPM repository:

rpm -i https://gerritforge.com/gerritforge-repo-1-4.noarch.rpm

On Debian, just refresh the repository cache with ‘apt-get update‘ and the new packages will automatically show-up and be available for setup.

Because this is a repackaging of the existing Gerrit release, the 4th number of the package name has been updated, e.g., gerrit-3.4.8-2 is the package name associated with the ARM-64 compatible package.

Ready to switch to ARM-64?

The Gerrit maintainers have already switched to ARM-64 with the recent wave of MacBook upgrades: all the non-EOL Gerrit releases are typically developed on Apple M1 or M2 which are based on ARM-64 rather than x86.

The need to reduce costs and the overall carbon footprint of the Gerrit project would most likely drive the CI agents’ switch to ARM-64 very soon.

Are you ready to switch to ARM-64 to reduce your operating costs and help the planet by reducing your Gerrit consumption?

Q&A

Q1. Are Gerrit on x86 and ARM-64 compatible in terms of data?

Yes, because the underlying data structure is platform-independent and is all based on Git, which is already interoperable cross-platform.

Q2. Can I mix&match x86 and ARM-64 Gerrit nodes?

Yes, by adopting the Gerrit multi-site setup, you can configure and host different sites on different architectures.

Q3. Will Gerrit on x86 continue to be supported?

Yes, because we want to allow the Gerrit administrator to choose the most suitable platform for his users and compatible with his Company’s requirements.

Luca Milanesio – GerritForge
Gerrit Code Review Maintainer
ESC Member
Gerrit Code Review Release Manager

Watch the Gerrit User Summit 2022 online

Posted on by
Reply

The Gerrit User Summit 2022 will start next week, on the 10th of November, at 9:00 GMT at CodeNode in London, 10 South Place London EC2M 7EB.

One of the last seats available on-site can be still yours, REGISTER NOW at at https://www.eventbrite.com/e/gerrit-user-summit-2022-tickets-424995963367.

If you cannot come to London to attend face-to-face, the entire event will be live-streamed on the internet on GerritForge’s live stream channel, already successfully used in the past for the Summit 2019 in Sunnyvale.

How to register

You can pre-register for the online event by visiting https://live.gerritforge.com and clicking the orange button “Register to Watch.”

You would need to provide your name, e-mail, and affiliation (specify “independent” if you are attending just for personal interest) and your consent to be contacted with detailed instructions on the day of the event.

You may also request to be included in future communications about the subsequent events organized by GerritForge Inc. for the Gerrit Code Review community.

On the 10th of November 2022, around one hour before the event, you will receive via e-mail all the instructions for watching the event.

Thanks again for coming to London or watching and engaging with the event remotely.

See you next week at the Gerrit User Summit 2022 !

Luca Milanesio – GerritForge
Gerrit Code Review Maintainer, ESC member and Release manager.

The Gerrit User Summit 2022 is back, save the date!

Posted on by
Reply

Dear fellow Gerrit User,
We are pleased to announce that GerritForge will be organizing this year’s Gerrit User Summit and Hackathon in hybrid mode: face-to-face and online.

The event is FREE; register and reserve your seat now at:
https://www.eventbrite.co.uk/e/gerrit-user-summit-2022-tickets-424995963367

Gerrit User Summit is the event that brings together Gerrit admins, developers, practitioners, and the whole community, in one place, providing attendees with the opportunity to learn, explore, network face-to-face, and help shape the future of Gerrit development and solutions.

After two years of remote meetings and virtual conferences, this year, we are back face-to-face at CodeNode in the heart of the vibrant City of London.

The dates will be:
Nov 7th to 9th – Hackathon
Nov 10th to 11th – User Summit

Shortly we will be publishing the full schedule and logistics for the event.
I look forward to meeting all the community’s friends, face-to-face or virtually, again during the Hackathon & Summit.

Thanks for being a part of the Gerrit community, and we look forward to seeing you in November.

Luca Milanesio
Maintainer, member of the Engineering Steering Committee, and Gerrit Code Review Release Manager

Gerrit Hackathon is back to London

Posted on by
Reply

After two years of remote events and three COVID-19 waves, we are finally back for a new face-to-face hackathon, talking about the future of Gerrit Code Review and coding new and innovative solutions for making Gerrit better, faster and more scalable.

Dates and schedule

The Gerrit hackathon will start on the 9th of May at 9:00 AM for five consecutive days, and will have a daily schedule from 9:00 AM to 7:00 PM BST, with networking and catch-up in the evenings.

For the remote attendees on the US/Pacific time-zone, the schedule will be daily between 7:00 AM to 11:00 AM PDT, which allows 4h of remote interaction with the hackathon in London.

Who is invited to attend the hackathon?

As with every Gerrit hackathon, we have a restricted audience: Gerrit maintainers and contributors are invited to join. We have 10 seats available on-site and 15 seats available remotely, which would allow plenty of people to collaborate and discuss.

The “Alphabet” meeting room of the hackathon

To register to the Gerrit hackathon, add your name and role (“Gerrit Contributor” or “Gerrit Maintainer”) to the attendees sheet. All Gerrit maintainers have edit permissions to the document whilst all other contributors can request permission to edit if they are willing to attend.

Where is the hackathon taking place?

GerritForge will host the Gerrit Hackathon at Huckletree West, Mediaworks, 191 Wood Ln, London W12 7FP. We will be staying at the “Alphabet” meeting room, with a dedicated 10-seats and roundtable, a full-size wall-mounted whiteboard and a permanent online connection and wall-attached screen to interact with all the other remote attendees.

Huckletree West

Huckletree is a creative workspace in West London, based in the heart of White City Place, a thriving new business and cultural district. Alongside the neighboring BBC Studios, Net A Porter Group, and RCA School of Communication, Huckletree West is part of a bold new chapter in the rich creative history of the neighborhood.

For all remote attendees, there will be the ability to connect remotely and interact with the rest of the team on-site during the hackathon hours.

White City and local accommodations

Huckletree West is close to the WestField Shopping Centre in White City, which includes 289 stores, 95 restaurants and Cinemas with 20 screens and almost 3,000 seats.

White City has excellent connections to all parts of London through the London Underground network (Central, Hammersmith&City and Circle lines) and Overground trains, which allow to reach all other parts of the city.

WestField shopping centre – White City

You can look for any Hotel or other accommodation (B&B or Hostels) in other part of London which is covered by the London Underground connections. However, if you are willing to stay local, there are many choices of Hotels and B&B starting from ￡80/night. See below a list of accommodations nearby White City:

Travelling to the hackathon

By airplane: from London Heathrow terminals, take the Piccadilly Line to Central London till Hammersmith, then take the Hammersmith&City line (station is across the street) until Wood Lane station. From London Stansted, take the Stansted Express train to Liverpool Street station and then the Circle Line to Hammersmith until the Wood Lane station.

By train: from the Eurostar Terminal at St. Pancras International, take the Hammersmith&City or Circle line to Edgware Road until the Wood Lane station.

Taxi: you can use the London Black Cab as well as other cheaper alternatives such as Uber or local minicabs companies.

From the Wood Lane station, there is a 6 minutes walk to reach Huckletree West, located in the MediaWorks building on the ground floor.

Route from Wood Lane Station to Huckletree West

Brexit restrictions

The UK has left the European Union the 1st of January 2021, all travellers from EU needs to follow the new rules for business trips. You can check if you need a VISA using the UK Government site and what is the required documentation and insurance required to show at the UK Border.

COVID-19 restrictions

The UK is set to end all COVID-19 restrictions by March 2022, which means there aren’t any vaccination or testing requirements for the attendees to the hackathon. We advise everyone attending face-to-face to take extra precautions and take a lateral-flow test (LFT) or antigen test before traveling to the hackathon, even though it is not required by law or regulations.

Please note that face covering are still mandatory whilst travelling by airplane, train or underground and during taxi rides.

We are excited to meet again the community of Gerrit Code Review maintainers and contributors after so many months. Come and join us in London this year and we can innovate again and help shaping the future of the Gerrit project, together.

Luca Milanesio, GerritForge
Gerrit Code Review Maintainer
Gerrit Code Review Release Manager
Member of the Engineering Steering Committee of the Gerrit Code Review Open-Source project

New year, new JGit contributions coming

Posted on by
Reply

We have shared GerritForge’s goals for improving Gerrit in 2022: most of them will include significant contributions to JGit, the Java-based engine powering Gerrit Code Review’s support for Git data-format and protocol.

GerritForge will contribute many more changes to JGit during 2022, all focused on improving the functionality and performance of large mono-repos. All changes will go through the formal review through Eclipse Foundation’s JGit Gerrit Project.

Lack of knowledge and reviews

The JGit project has suffered from major losses in the past few years, which is clearly shown by the list of top-contributors vs. their recent 12 months activity. I have tried running a “git blame” against all the JGit code-base, which is a heuristic (therefore a rough approximation) of which part of the JGit code has been last written/edited.

  1. (49998 LOC) – Shawn Pearce
  2. (37854 LOC) – Thomas Wolf
  3. (31417 LOC) – Matthias Sohn
  4. (13593 LOC) – David Pursehouse
  5. (13200 LOC) – Christian Halstrick

See below the number of contributions (excluding merges and trivial changes) of the above 5 maintainers in the past 12 months:

  1. Shawn Pearce – 0 changes – He sadly passed away in 2018 
  2. Thomas Wolf – 86 changes
  3. Matthias Sohn – 128 changes
  4. David PurseHouse – 0 changes
  5. Christian Halstrick – 0 changes

The above stats show that the currently active maintainers (Thomas and Matthias) appear in the bit blame of 69k out of the total 390k LOCs.

Thomas and Matthias are doing a fantastic job in keeping up as much as possible with the incoming changes’ pace and reviewing them at their best. At times, though, the incoming change may touch parts of the code they are less familiar with and, therefore, would require more eyes or more time to review.

Breaking the vicious circle

The JGit project is in a dangerous vicious circle.

  1. Incoming changes would take longer to get reviewed and merged.
  2. The lengthy reviews cause detriment to contributors that may lose interest in following up contributions or upload new changes.
  3. The lack of contributions and merged changes would keep the pressure on current maintainers, which fuel point 1. again.

How can GerritForge help break this vicious circle, provide meaningful contributions, and get them merged fast and with proper and thorough reviews?

Keeping the pace of contributions is key to avoid detriment: GerritForge will therefore create a “dev branch” of JGit. All the GerritForge’s contributions to JGit master branch will be part of the dev branch and will go through a rigorous code-review and E2E validation cycle, including the Gatling tests for Gerrit.

Two-steps validation workflow

GerritForge’s workflow for validating JGit changes with Gerrit
  1. A new change is uploaded to the Eclipse Foundation JGit project.
  2. The normal Eclipse Foundation’s CI verification builds the change and, if passes all tests, provides a Verified +1
  3. One of the JGit maintainers, or members of the GerritForge’s contributors, can provide a Code-Review +1 score with the additional description “Approved for dev“
  4. The special “Approved for dev” description triggers the cherry-pick of the Change onto the GerritForge’s JGit dev branch
  5. The new change for review on the JGit dev branch triggers the creation of a Change on gerrit-review.googlesource.com/gerrit with the update of the JGit submodule pointing to the open change.
  6. The JGit submodule update Change triggers the current E2E validation using the Gatling tests, developed and hosted by GerritForge. If all tests are passing, the Gerrit change receives a Verified +1.
  7. The cherry-picked Change on JGit dev branch receives a Verified +1
  8. The cherry-picked Change is merged to the JGit dev branch
  9. The merge of the cherry-picked Change is notified on the original Change with a Code-Review +1 score with description “Merged in dev”.
  10. One of the JGit maintainers can finalise the review and, if all is good, provides the final Code-Review +2 and merge the change on JGit master.

NOTE: The above workflow will only apply to the upcoming changes on the master branch, where we do need to innovate and implement new features at a faster pace. We have to plans to apply the workflow to any stable branches.

Plus and minuses

There are good things on the above workflow, however, there are also risks:

  • Complexity: the secondary dev branch will undergo an E2E validation process with Gerrit, which is obviously complex and it may break at times.
  • Danger of forking: if the JGit maintainers would veto the Change at step 10 the changes already merged in dev would make effectively dev and master branches diverge, which isn’t a good thing and it should be avoided as much as possible.

The augmented lifecycle, which also involves Gerrit E2E tests with Gatling, has also many advantages:

  • Additional E2E validation: incoming changes on JGit would involve an E2E validation with Gerrit against the suite of E2E Gatling tests, which is good feedback and gives more confidence in merging code also on less known parts of the JGit codebase.
  • Increased velocity: speedup validation of new incoming changes and getting them merged to the JGit dev branch, without impacting the pace and quality of reviews from the current JGit maintainers.
  • Gerrit edge release: allows to have a downloadable Gerrit change that includes the JGit dev branch, allowing canary deployments and see how Gerrit behaves with the latest and greatest of JGit code.

From a JGit project’s perspective, the flow of incoming changes will have an additional E2E validation, which is always a good thing. Additionally, it will bring more contributions and inspiration for new innovative changes on the project and attracting more and more talent.

Ready to gear-up contributions on JGit?

The workflow proposed is a starting point; however, we are committed to giving it a go and seeing how it would work in practice and if it will be enough to gear up the contributions to the JGit project.

2022 GOALS for Gerrit

Posted on by
Reply

The year 2021 has been a challenging one because of the COVID-19 global emergency; nevertheless, the Gerrit Code Review project has continued to deliver what the community expected:

GerritForge delivered on the promise of making Gerrit more cloud-native, with a particular focus on AWS, the platform that most users have adopted for running in the cloud. GoogleCloud has also been our focus, assuring a cloud-neutral approach to Gerrit and providing support for events over PubSub.

Focusing on Gerrit unique values

A successful product focuses on what makes it unique and innovative, compared to anything else in the world.

We believe that the key aspects that make Gerrit Code Review THE platform of choice for developing software based on Git repositories are:

  • Large-scale
    Gerrit is THE best platform for developing large-scale projects, huge monorepos, and a large number of changes and refs.
  • Maximum availability
    Large organizations and communities of developers need a platform that is always available, anywhere, anytime, 24×7, and 365 days a year.
  • Performance
    The need to work remotely poses multiple issues, one of them being the increase of network latency. Gerrit multi-site distribution of the repositories and reviews allows anyone, anywhere in the world, to clone, push and review at optimal latency and performance.
  • Quality of tracing of reviews
    Gerrit is based on single-commit code reviews, a winning approach in terms of review accuracy and supporting changes chains, and full traceability of the entire review history and workflow.

Many popular Git code-review tools exist in the Open-Source community; Gerrit is the winning choice when scale, availability, performance, and quality do matter.

GerritForge goals for improving Gerrit in 2022

Scale Gerrit beyond limits

GerritForge and the rest of the community have worked hard to identify the bottlenecks of large mono-repos with Gerrit. Some of them can be mitigated by keeping the Git repository lean and organized, despite the massive amount of push traffic and reviews coming from large teams.

We want to focus on improving at least ten times the following KPIs, without having a significant impact on the overall system performance:

  • Number of changes and refs in a repository: millions of changes and tens of millions of refs
  • Size of the repository: hundreds of GBs

GerritForge will step up its involvement in the JGit project in 2022 and introduce many innovations, some of them already implemented in the C-Git implementation:

  • JGit support for multi-pack index
  • Revamp of JGit cache, allowing the pluggability of high-performance implementations
  • Improvement of JGit bitmaps for large number of refs
  • Support for high-performance large storage systems
  • Introduction of new performance metrics
  • Replace Prolog with native submit rules in the owners plugin

99.999% up-time

GerritForge maintains a free service known as GerritHub.io to demonstrate what Gerrit can do and achieve. GerritHub.io is the most advanced and reliable Open-Source vanilla Gerrit deployment, apart from Google’s.

GerritHub.io uptime in 2021 – checked and reported by PIngdom.com

We achieved an astonishing 99.99% SLA in 2021; we want to push the GerritHub.io uptime further to 99.999%, reducing the annual downtime to just 315s.

In order to reach a five-nines uptime, we will work on:

  • Granular probing and health-checks
  • Advanced repositories performance monitoring and alerting
  • Gerrit limits and deadlines
  • RCAs
  • Multi-site improvements

Goal #3: Increase 1000x times the Gerrit replication performance

GerritForge has presented the innovating pull-replication plugin at the Gerrit Virtual User Summit 2021, showing that it is possible to replicate Git commits and changes meta-data across the globe with msec latency. The pull-replication plugin technology and speed is going to be improved and made available and Open-Sourced to anyone and match and outperform the traditional replication plugin features.

Join the 2022 endeavor


We need YOU and the Gerrit community’s help and support in this 2022 endeavor.

GerritForge has already increased his Team of contributors working on the project, including three Gerrit maintainers and two Gerrit release managers. However, Gerrit’s success is in the cooperation, contribution, and ideas of the whole community of contributors, Gerrit admins, and users.

Let us know what you think about our goals. We are happy to cooperate and work with anyone sharing the same values and goals.

2022 is the year where Gerrit Code Review is pushed beyond its limits even further, making it the MOST innovative tool for large-scale repositories and teams worldwide.

Gerrit: 2021 in review

Posted on by
Reply

Yet another year has passed for the Gerrit Code Review project with many challenges posed by the COVID-19 pandemic, new exciting releases, and the most popular Gerrit User Summit with the largest audience ever in its 12 years of history.

2021 in numbers

  • 93 registered attendees to the Gerrit Virtual User Summit 2021, connecting from 56 companies over 17 countries, 14 talks showcased by 15 presenters over 2 days
  • 1 Gerrit Contributors’ Summit
  • 35 releases of which 2 major versions (v3.4.0 and v3.5.0.1) and 33 patches
  • 107 contributors from 32 organizations, merging 4763 changes to 84 projects

The Gerrit Code Review community has shown resiliency during these difficult times, with outstanding participation in the events organized during the year, all remote and lacking the much-needed face-to-face interaction.

  • Commits: -26%
  • Projects: -16%
  • Contributors: -30%
  • Companies: -41%
  • Average changes/contributor: +10%

The engagement has paid its toll after two years of pandemics with fewer organizations willing to invest time in contributing to Gerrit, possibly also impacted by the uncertainty of the future. 2021 has also been the first whole year of the project without David Pursehouse, one of the Gerrit project’s top #3 contributors. He was used to contributing 1.5k changes per year, which would alone easily justify the drop observed.

On the bright side, the contributors that continued over the year 2021 have shown an increased commitment as the number of active projects and commits has dropped less than the contributors, increasing the change/contributor rate compared to 2020.

Major organisations contributing to Gerrit in 2021

Google is confirmed to be the leading force of the Gerrit Code Review project, with over 62% of the changes merged, while GerritForge continues to be the #1 top contributor from the rest of the community. There are a couple of pleasant special surprises from the contributors.

  • Wikimedia Foundation confirmed to be the #2 top contributor from the community, all provided by Paladox who has been awarded Gerrit Maintainer in November.
  • SAP continues to be a strong contributor, just below Wikimedia Foundation, with Thomas being awarded Gerrit Maintainer in November.
  • Qualcomm is back on the shortlist of the top maintainers, with many new names in the list of contributors, well done!

Top-ten projects with major activity in 2021

  1. gerrit (2,903 changes)
  2. plugins/code-owners (447 changes)
  3. jgit (287 changes)
  4. plugins/task (83 changes)
  5. plugins/multi-site (57 changes)
  6. aws-gerrit (44 changes)
  7. modules/cache-chroniclemap (40 changes)
  8. plugins/checks (39 changes)
  9. plugins/high-availability (39 changes)
  10. plugins/replication (38 changes)

The first surprise is that the code-owners, the emerging star of the Gerrit plugins, received a massive investment of effort from Edwin (Google), who contributed 89% of the changes to it. The code-owners plugin has also been presented at the Gerrit Virtual User Summit 2021 and attracted the community’s attention.

The second surprise is the decline in contributions in the jgit project during the past two years: from 820 changes/year is now down to 374 changes in 2021.

Task is now the #2 plugin project in terms of merged changes in 2021. Qualcomm keeps the project’s full ownership with 98.9% of changes in 2021.

GerritForge confirm their commitment to improving Gerrit Multi-Site, as its plugin is the #3 in terms of changes merged in 2021.

Aws-gerrit is a relatively new project, presented less than two years ago and contributed by GerritForge, who contributed over 99% of the changes. It confirms to be a very active project that has helped the Gerrit Code Review open-source project deploy and test well-known “recipes” of infrastructure setups and see how Gerrit performs and works on those. Many bugs have been detected before the release and identified by the aws-gerrit project and CI integration.

The cache-chroniclemap module confirms to be very active in 2021, with 40 changes all provided by GerritForge. This relatively new module allows existing Gerrit setups to increase the overall performance of all persistent caches, which are vital in reducing the REST-API latency across all Gerrit features.

The checks plugin was deprecated back in 2020. However, it still shows significant changes and investment from Google in supporting the new Gerrit checks-API and UI. However, the rate of contributions is in stiff decline, down from the 324 changes in 2019 when it was still an actively developed project.

The last two plugins projects in the top tens are the replication and high-availability plugins, which has received major contributions from Qualcomm, GerritForge, Google and Ericsson.

Top events in 2021

The Gerrit Code Review community abandoned the idea of a face-to-face event in 2021 because of the continued global pandemic of COVID-19.
Instead, there were two separate virtual events for sharing the news of what is happening on the platform and the expectations from the community.

Virtual Gerrit Contributors’ Summit – 9th of June

The summit was organized by the Gerrit Community Managers and had an amazing audience amongst the contributors. The presentations showed what different teams are working on and reported into the summit notes:

Gerrit Virtual User Summit 2021 – 2-3 of December

It was the first experiment of an entirely Virtual User Summit of the Gerrit Code Review project history. The challenges were multiple, including the limitations of allowing up to 100s of attendees, shortening the overall time to 3h x 2 days, and still allowing some interactions between the audience and the presenters. After two years of silence, we have finally received some user stories of using Gerrit in the wild.

The Summit has received vast overall positive feedback and rated 7.9/10, making it a fantastic achievement. The quality and interest of the talks were scored even higher, reaching 8.2/10.

The talks have been fully recorded and published on the GerritForge TV channel:

It was definitely a lot of information and sharing, which showed that the Gerrit Code Review open-source project is alive and active more than ever.

Gerrit features highlights in 2021

Gerrit Code Review has major innovations developed and decisions made over 2021. See below a short recap of the ones that represent a turning point in the evolution of the Gerrit open-source project. Some of them are considered breaking changes and, therefore, need careful analysis and a planned upgrade path.

Speed up of Gerrit upgrade from v2.7 to the latest version

2021 has seen a significant increase in the cooperation and contributions of Qualcomm to the rest of the Gerrit Code Review community, focussed on the speed-up of the Gerrit upgrade process from v2.7 to v3.5.
The contributions and cooperation have brought many improvements to JGit and Gerrit and will allow many more companies to migrate faster and smoother than ever before.

Goodbye to Java 8

From Gerrit v3.5 onwards, the source code and binaries of Gerrit Code Review won’t be compatible with Java 8 anymore.

JSch SSH library is completed removed from Gerrit Code Review

The quirks and obsolescence of the JSch library has cursed Gerrit’s destiny for years. Thanks to Thomas Wolf (Paranor) JGit moved away from it and rebuilt all its Git/SSH stack on top of Apache Mina. That has allowed to remove the JSch library from the Gerrit dependencies and used the Apache Mina SSHD client stack instead.

ElasticSearch is removed from Gerrit Code Review

On the 2nd of February 2021, Elasticsearch B.V. changed its license model and abandoned the Apache 2.0 open-source license for the new versions of ElasticSearch v8 and over.

Gerrit cannot include or require any commercial product not released under one of the open-source licenses allowed by the project. The ElasticSearch backend has not been widely used in the community anyway, based on a recent survey sent to the community therefore the ESC decided on the 3rd of November that the ElasticSearch backend will be removed from Gerrit core and moved into a libModule.

Submit Requirements waving goodbye to Prolog

The Gerrit Code Review project does not use anymore Prolog rules for the submit rules of the project from the 16th of December. The support for Prolog-less submit rules is now mature and it will be part of the forthcoming v3.6 release in 2022.

What’s coming in 2022?

The future of Gerrit Code Review is bright and full of innovative ideas and improvements on the overall development and CI/CD lifecycle. With the forced remote working of millions of developers worldwide, more and more companies are looking on how to make remote interactions more useful and fruitful, reducing frictions and making the workflow smoother and faster more than ever.

Stay tuned and keep on using and contributing to Gerrit Code Review, one of the most innovative and productive platforms for code review and collaboration.

Gerrit Virtual User Summit 2021 – Summary and Survey Results

Posted on by
Reply

The first Gerrit Virtual User Summit took place online on the 2nd and 3rd of December 2021.
If you have missed the Summit, all the presentations and content are online and is directly pointed by the details of the sessions from the main schedule.
All the recordings of the presentations and associated anonymized Q&A are accessible from GerritForge’s TV channel on YouTube.

The Virtual User Summit format and numbers

The Gerrit Virtual User Summit 2021 was held online to allow most of the community around the globe to attend and share their experience and ideas and avoid the problems with the traveling restrictions due to the COVID-19 pandemic.

Despite the limitations of the remote format, the interest and attendance to the Virtual User Summit has been outstanding, confirming the resiliency of the community and the will to inform and ask questions, share opinions and discuss no matter the difficulties and challenges.

See below some of the numbers of the Summit:

  • 92 registered attendees, with 71 of them showing up at at least one of the two dates (77% of attendance)
  • 17 countries connected (Austria, Canada, China, Czech Republic, Finland, France, Germany, India, Italy, Poland, Portugal, Spain, Sweden, Taiwan, Ukraine, UK, USA)
  • 56 companies
  • 14 sessions in 2 days
  • 15 presenters

Despite the limitations of the remote interaction and the hours, this has been the largest User Summit ever, terrific news for the whole community.

What do people think about the Summit?

All the 71 attendees have received a Survey about the Summit, 24 of them have provided their feedback.

Q1: How would you rate the Gerrit Virtual User Summit 2021 edition?
The average rating has been 7.9 over 10, with the 45% rating 9 to 10 points out of 10.

Q2: What was your main objective in attending the Summit?
Most responses were around “learning about Gerrit’s new features and initiatives” and “getting in touch with the community and its users”.

Q3: Do you consider to have achieved the objective?
88% of the people said “yes”, which seems consistent with the 7.9/10 overall rating of the Summit.

Q4: If your answer was “no”, what is the aspect that was not achieved?
All of them expressed concerns about the inability to have a two-way direct with the other users at the Summit, which is mainly linked to the technical difficulties and limitations of a remote webinar platform.

Q5: How satisfied are you with the quality of the presentations?
The average rating has been 8.2 over 10, with the 55% rating 9 to 10 points out of 10.

Q6: Would you prefer to have your camera on during the Summit as an attendee?
53% answered no, which shows that opinions are divided on the topic of having remote face-to-face communication with such a large audience.

Q7: Would you like to have breakout rooms to interact as part of the Summit?
76% answered yes, giving good confidence of trying that approach in the next Summit, assuming that would be “virtual” again.

Q8: What would be the best way for you to interact with other attendees and speakers during the Virtual Summit?
The majority of the answers were directed to the use of breakout rooms with a moderator, which matches the feedback given on Q7.

Q9: What platform would you suggest to deliver future Virtual Summits?
Most people did not express preferences, while others mentioned Zoom, Google Meet, and GoToWebinar (the one used during this 2021 Summit).

Q10: What did you like the most about the 2021 Summit, and what should be improved in the subsequent Summits?
Most of the people liked the content and quality of the presentations, which matches the high scores given in Q5. Also, the format of 3h per day, split into two different days, looks optimal for compatibility with the existing working day. Registration and attending the event were straightforward, and the browser-based integration with the stream of Q&A was easy.
The proposals for improvements were on the following points:

  • Having more talks about user stories: do people use topics? What they use for CI integration for them? How people vote on changes/topics?
  • Having direct interactions between participants, seeing their faces and talking to them directly
  • More communication and advertisement of the event beforehand
  • More seamless transition between the webinar part and the interactive Q&A session

What’s next for 2022?

With the new Omicron-variant of COVID-19 raging in the UK and the risk of global expansion in 2022, a face-to-face User Summit next year is still possible but unlikely.
The huge success of the Virtual Summit format could lead us to try an improved version in 2022, trying other platforms like Zoom or Google Meet.

Many new exciting features are coming in 2022, including the release of Gerrit v3.6 and the Prolog-less Submit Requirements. We would love to see again what the community thinks about it and interact more with the real-life Gerrit Code Review users and administrators.

Thanks again to those who have participated and presented at the Gerrit Virtual User Summit 2021. See you next year for another exciting and more engaging get-together in 2022!

The Virtual Gerrit User Summit is tomorrow!

Posted on by
Reply

Join the Gerrit Community tomorrow and Friday from 8 am PST for everything related to the Gerrit Code Review Community.

Gerrit provides web based code review and repository management for the Git version control system. Whether you are experienced or new to Gerrit, you should know that it provides a framework you and your teams can use to review code before it becomes part of the code base. Come and take this chance to join and learn about Gerrit Code Review.

Find here the full schedule of the sessions you will have access to.

Register and join the community event!