In two weeks, the Gerrit community is heading back to where it all started.
On 9–10 November, Gerrit User Summit 2026 will take place at Google’s campus in Sunnyvale, California. Gerrit Code Review was born at Google in Silicon Valley, and the last summit Google hosted was organised by Gerrit’s and JGit’s founder, Shawn Pearce, ten years ago.
The summit is free, open to anyone who registers, and hosted by Google in cooperation with GerritForge and the Gerrit community. Two days of talks, demos and conversations with the people who build, run and depend on Gerrit at scale, plus a morning of jj talks and a three-day hackathon hosted and organised by NVIDIA straight afterwards.
The full agenda is now live. Here’s what’s on each day, and why this year’s programme is one not to be missed.
Code review in the age of agents
AI coding agents are writing more code than ever. Someone – or something – still has to review it, validate it and ship it safely.
That question runs right through this year’s programme. Speakers from Google, NVIDIA, ACME Gating, GitLab and GerritForge are tackling how code review holds up when agents join the team: keeping agentic coding safe, managing load when automated traffic spikes, and scaling Git storage for workloads nobody planned for five years ago.
Alongside that, the summit keeps what the community values most: deep technical sessions from the maintainers themselves, real-world stories from teams running Gerrit in production, and plenty of time to talk to the people behind the code.
Day One – Nov 9: Core Architecture, Scaling, and Agentic Workflows
Day one is about Gerrit itself: what’s new, what’s next, and how the teams running it at scale keep it fast.
Luca Milanesio (GerritForge) opens with the latest highlights from the release, and later lays out how Gerrit and JGit are being rebuilt for AI-era storage. Monty Taylor – who helped launch OpenStack and designed the workflows that became Zuul and OpenDev – makes the case for keeping agentic coding safe with the human collaboration tools we already trust. In the afternoon, Hudson River Trading, NVIDIA and GitLab share what they’ve learned running Git and Gerrit under serious load.
9:30 AM: What’s new in Gerrit v3.15 – Luca Milanesio, GerritForge
10:15 AM: Improvements to the replication plugin – Kaushik Lingarkar, NVIDIA
11:05 AM: Gerrit, Zuul and OpenClaw: safe agentic coding via human collaboration tools – Monty Taylor, Zuul and OpenDev
11:50 AM: Multipack index support in JGit – Ivan Frade, Google
1:20 PM: Scaling Gerrit and JGit for the AI era: a pluggable object backend with Cassandra – Luca Milanesio, GerritForge
2:05 PM: From cgit to Gitiles: building repository search for everyone – Devansh Varshney, LibreOffice
2:20 PM: HRT’s Gerrit improvements – Kamen Yotov, Hudson River Trading
3:05 PM: Do I have to approve this again? Sticky Owners in Gerrit – Antonio Barone, GerritForge
4:00 PM: Task quotas in the age of AI – Adithya Chakilam, NVIDIA
4:30 PM: Large object promisors: handling large files natively in Git – Christian Couder, GitLab
Day Two – Nov 10: jj mini-summit and the future of Git scalability
Day two opens with a morning of jj and version control talks, co-hosted with the Jujutsu community, before turning to scaling for the AI era.
Google engineers share how they’re bringing jj into everyday workflows, from a VS Code extension to an open-source commit cloud. Daniele Sassoli and Emily Shaffer show what it takes to add native jj support to Gerrit. After lunch, Zuul creator Jim Blair joins Luca Milanesio on AI development at scale, and Shane McIntosh from the University of Waterloo presents research on keeping Git fast under AI workloads.
The summit closes with an open Q&A with the maintainers – your chance to ask the people steering Gerrit’s future anything you like.
9:45 AM: jj VS Code extension at Google – Kevin Liao, Google
10:15 AM: Version control and code formatting – Danny Hooper, Google
10:45 AM: Where does your metadata live? – Patrick Hiesel, Google
11:30 AM: Bridging the gap: adding native jj support to a Git-based SCM – Daniele Sassoli, GerritForge, and Emily Shaffer, Google
12:00 PM: cloud-jj: an open-source commit cloud for jj – Daniel Sperry, Google
1:30 PM: AI development at scale with Gerrit and Zuul – Jim Blair, Acme Gating, and Luca Milanesio, GerritForge
2:15 PM: SHA-256 in JGit: why, what, and when – Xing Huang, Google
2:45 PM: GHS: sustaining Git performance under AI workloads – Shane McIntosh, University of Waterloo
3:35 PM: Scaling Gerrit builds with Kubernetes and real-time output sync – Shameer Erakkath Saidumuhammed, Hewlett Packard Enterprise
4:05 PM: How to stack changes like a pro – Jacek Centkowski, SoftwareMill
4:20 PM: Q&A with the maintainers
5:20 PM: Social and networking
Doors open at 9:00 AM.
Stay on for the NVIDIA hackathon, 11–13 November
Straight after the summit, NVIDIA is hosting three days of hacking at its Santa Clara offices. This year there’s one shared goal: making partial clones a practical replacement for Git LFS across the Git and Gerrit ecosystem.
If you develop Git or Gerrit, maintain Git infrastructure, or have run LFS or partial clones in production, your experience is wanted. Spots are limited – email gerrit-hackathon-2026@nvidia.com to reserve one.
Join us in Sunnyvale
When: Monday 9 and Tuesday 10 November 2026
Where: Google, Building MP6, 1195 Borregas Ave, Sunnyvale, CA 94089
Whether you run Gerrit for thousands of developers, contribute to the project, or are working out how code review fits into an AI-driven workflow, this is the room to be in. Seats are limited, so if you’re planning to travel, now’s the time to lock it in.
AI-assisted coding and automated agentic development pipelines are overwhelming Git repositories. Rapid packfile accumulation breaks traditional maintenance, causing severe bottlenecks in Git’s “Counting” and “Search-for-Reuse” (SFR) phases. Fast pushes suddenly become agonizing delays.
Our 12-hour stress tests proved that standard solutions fail under scale. Doing nothing leads to server collapse. Scheduled Garbage Collection (GC) is just a band-aid, creating unstable performance until the system finally chokes under heavy traffic.
The solution is adaptive intelligence. Git-at-High-Speed (GHS) uses reinforcement learning to trigger lightweight, proactive repository adjustments. GHS keeps system load flat, drops latencies under 50ms, and entirely eliminates massive GC disruptions.
Introduction
Modern software organisations are rapidly embracing AI-generated code and content, dramatically increasing the volume and velocity of changes flowing into their repositories. As this wave of automated contributions accelerates, version control systems such as Gerrit must sustain ever-increasing data ingestion without degrading developers’ experience or the reliability of CI/CD pipelines. Maintaining high throughput and low latency under continuous, AI-amplified load is no longer optional; it is foundational to engineering productivity and operational stability. As repositories grow and CI/CD activity intensifies, the cost of managing pack files inside Git becomes critical.
Every push or fetch triggers internal Git phases that prepare the objects to send to the client. First, Git performs a Counting phase, where it walks the repository graph to determine which objects need to be included in the pack being generated. It then scans existing pack files to determine whether objects can be reused, a step known as search-for-reuse (SFR). These stages can become bottlenecks when too many pack files accumulate, resulting in slower pushes, fetches, and clones, and higher CPU load.
To understand how different maintenance strategies affect Counting, SFR, and overall system stability, we conducted stress tests that mimic a realistic enterprise workload. Each simulation ran for 12 hours with 70 concurrent users, blending Git read/write operations with REST API calls. We compared three scenarios:
Baseline (no garbage collection) – no attempt is made to repack or compact the repository during the test.
Rule‑based GC (current industry standard) – full garbage collection is scheduled periodically (every 15 minutes) regardless of current load.
GHS (Git‑at‑High‑Speed) – an adaptive agent monitors repository state and chooses between several actions (create bitmaps, repack references, pruning, do nothing, etc) based on reinforcement‑learning rewards.
The core focus of this analysis is how each strategy influences the Counting and search‑for‑reuse metrics over the full 12‑hour window.
Why Counting Time Matters
Before pack generation begins, Git performs a counting phase to determine which objects must be included in the pack sent to the client. During this step, Git walks the commit graph and object references to build the list of objects that need to be transferred. If the repository contains many changes, branches, and references, this traversal can become expensive and increase the time required to prepare the pack. We measure this latency as the counting time. High counting times indicate that Git is spending significant effort discovering the objects to send, which delays the entire operation. Conversely, low counting times indicate that the repository structure allows Git to identify the required objects quickly.
Why Search‑for‑Reuse Matters
When a Git server receives a push or serves a fetch, it may need to create a pack file. Git attempts to reuse objects that are already stored to avoid duplicating data. It does so by scanning the indices of existing pack files. If there are many small pack files, the scan can touch thousands of index entries, dramatically increasing latency. We measure this latency as the search‑for‑reuse time. High SFR times indicate that Git spends too much time searching for existing objects, thereby bottlenecking the entire operation. Conversely, low SFR times indicate the repository’s internal structure is healthy and that operations can proceed quickly.
Workload Model
To evaluate Gerrit under realistic conditions, we modelled three concurrent workloads using Gatling: developer activity, CI/CD clone jobs, and CI/CD fetch operations. Each workload runs with 70 concurrent virtual users, resulting in 210 active clients interacting with Gerrit simultaneously throughout the simulation.
All workloads run continuously for 12 hours while maintaining a constant number of concurrent users. This creates sustained system pressure and avoids the unrealistic burst patterns often seen in short benchmarks.
Developer Simulation
The developer simulation models the typical workflow of engineers interacting with Gerrit. Virtual users continuously perform common code-review activities such as creating changes, uploading new patch sets, rebasing work, reviewing code and submitting approved changes. These operations generate a steady stream of Git pushes and server-side processing, closely resembling the activity of a busy development team.
CI/CD Clone Simulation
The CI/CD clone simulation represents build systems that regularly clone repositories to run builds and tests. Each virtual pipeline repeatedly performs repository clones followed by verification steps. Because clones require Gerrit to generate pack files and search for reusable objects, they place a significant load on the repository storage layer.
CI/CD Fetch Simulation
The CI/CD fetch simulation models pipelines that periodically fetch updates from the main branch to detect new changes. These incremental updates are lighter than full clones but still exercise Gerrit’s pack generation and object lookup mechanisms, contributing to the overall system load.
Controlling Operation Distribution
To keep the workload realistic, each action includes a configurable pause with a small amount of variation. This prevents request bursts and allows the simulations to control the relative frequency of operations such as pushes, submissions and repository fetches, producing a steady and predictable workload over the entire test duration.
Baseline: Unsustainable Growth Without Maintenance
The baseline scenario intentionally performs no maintenance on the repository. Early in each run, the system is in a relatively healthy state: push traffic is stable, CPU load is moderate, SFR and Counting latencies rise steadily and predictably. As the Gatling simulation continues, however, the number of pack files grows unchecked, and the overall system load, driven by the combined cost of counting objects and searching for reuse, begins to climb.
Only once the load crosses a certain threshold does Gerrit begin rejecting HTTP requests. Interestingly, this rising load appears before any HTTP rejections occur. These rejections act as a form of back-pressure: with a fixed pool of HTTP threads, Gerrit limits the number of incoming requests in order to protect the server from complete overload.
This behaviour is clearly visible in the following Counting and SFR graph. As the load builds up, the curve becomes steeper and more unstable. Distinct “waves” start to appear in the data.
Each wave represents a cycle of overload and partial recovery. When the system becomes saturated, Gerrit rejects more requests (shown as red crosses in the graphs). This temporarily reduces the amount of new work entering the system, allowing some ongoing operations to complete and causing Counting and SFR latencies to drop slightly. However, as soon as some headroom becomes available, new requests are accepted again, contention builds up, and latencies begin climbing once more.
The important detail is that each recovery is only partial. The next Counting an SFR peak is higher than the previous one. Over time, the system enters a repeating pattern of build-up, throttling, and renewed overload rather than returning to a stable state.
By the middle of the simulation, Counting and SFR peaks reach hundreds or even thousands of seconds, and latency never returns to the low levels seen at the start of the run.
Without any intervention, the baseline runs invariably end, with the system saturated: pushes and fetches stall, HTTP rejections become constant, and SFR and Counting both continue to grow. The ever‑increasing peaks in the above graph illustrate how an unmanaged repository accumulates pack files until normal operations are no longer feasible.
Key observations:
Trend: The run begins with slow growth, enters a phase of rising oscillations as load builds and then ends in runaway SFR and Counting growth.
Cycles of overload and throttling: SFR and Counting rise in waves, driven by the server alternating between taking on work and aggressively throttling to survive.
Implications for users: Rising SFR and Counting times directly translate into slower pushes and fetches. Developers and CI jobs see spiky latencies that grow from seconds to minutes, and eventually operations time out completely as Gerrit exhausts its HTTP thread pool.
Rule‑Based GC: Periodic Relief, Growing Spikes
To mitigate uncontrolled pack growth, the rule-based scenario performs a full garbage collection every 15 minutes. This approach reflects the most common maintenance strategy used in Git environments. Each GC run repacks objects and temporarily reduces the number of pack files. This produces a saw‑tooth pattern: immediately after a GC run, SFR and Counting times fall dramatically, but as users continue pushing and fetching, new pack files accumulate, and times rise again until the next GC triggers.
The scatter plot for the rule-based runs shows initial latencies of only a few seconds, but as the simulation progresses, they climb into the hundreds of seconds, with later peaks reaching well over a thousand seconds. Each subsequent GC run yields diminishing returns because the maintenance cost grows: from seconds to minutes, and later ones well over an hour. As a result, the time between the end of one GC and the start of the next gradually shrinks, and the system spends an increasing fraction of its time running GC rather than serving user requests. Eventually, GC cycles grow longer than the 15-minute scheduling interval itself. When this happens, the next scheduled GC attempt overlaps with the previous one and is aborted because it cannot obtain exclusive access to the repository.
GC does eventually repack objects, but the system-load graph shows a delayed saw-tooth pattern: each GC cycle briefly relieves pressure, then load builds again as new work accumulates, with progressively higher peaks as as GC executions become so expensive that they can no longer keep pace with incoming traffic.
Key observations:
Temporal instability: The saw‑tooth pattern leads to oscillating performance. Users see periods of good throughput immediately after GC, followed by worsening latencies until the next GC fires. This instability is visible even when the average throughput remains high.
GC cost escalation: Each successive GC run takes longer because the repository is larger and more fragmented. The schedule does not adapt, causing GC to run during peak load and further slow down the system.
Partial improvement: Compared with the baseline, rule‑based GC delays the point of collapse. However, it fails to maintain low Counting and SFR times for more than 12 hours and introduces performance jitter that developers and CI systems will notice.
Alternatives
A natural question at this point is whether alternative strategies, such as geometric repacking, could be used instead of traditional GC.
In practice, when it comes to JGit, which Gerrit relies on, this is not really an option, as geometric repacking is not currently supported.
Even if we tried to run it externally using C Git, it would not integrate well with Gerrit. Packfiles would be replaced abruptly, continuously invalidating JGit caches, and the resulting layout would not be ideal, as C Git does not separate heads and non-heads packfiles.
For these reasons, we focused on approaches that are commonly used in practice and that fit naturally within Gerrit’s execution model and repository structure.
GHS: System load stabiliser and performance accelerator
The GHS approach uses an intelligent agent to monitor repository metrics and decide when any action is required. At each decision point, the agent can either do nothing, create bitmaps, repack references, prune packfiles, and perform other actions. A reinforcement‑learning reward function balances the cost of maintenance against the benefit of reducing future SFR times. As a result, GHS tends to perform less-expensive actions early and avoids the expensive full GCs.
Our GHS runs exhibit remarkably stable SFR and Counting times. The scatter plot below shows most samples concentrated below 500 ms, with only a handful of outliers.
In fact, for long stretches of the simulation, the SFR latencies hover in the 5–50 ms range. To fully appreciate this, it is useful to plot the data on a logarithmic scale.
System‑load graphs remain flat, indicating that maintenance tasks performed by GHS are highly optimised and far less expensive than a full GC cycle.
Key observations:
Flat trend line: Unlike the baseline and rule‑based scenarios, the Counting and SFR metrics under GHS do not grow super‑linearly. Minor fluctuations are present, but the overall trend is remarkably flat.
Proactive actions: GHS performs frequent, inexpensive, locally optimal tasks, avoiding unnecessary overhead and maintaining high read and write throughputs at minimal cost.
Learning and improvement: The agent learns from experience. Chosen actions change as the agent converges on an optimal policy.
Comparative Insights
The three maintenance strategies yield starkly different outcomes.
Without GC, SFR and Counting latencies climb linearly, and the repository becomes unusable after a few hours. Rule‑based GC delays but does not prevent this collapse; it introduces periodic relief but suffers from growing GC costs and oscillating performance.
Not only does GHS keep SFR and Counting times low and stable across the entire 12‑hour simulation, but it does so by leaning much less on system load.
In practical terms, these findings suggest that adaptive optimisations are essential for large Git-based installations: scheduled GC can help, but must be complemented with cheaper, more frequent actions.
GHS is uniquely positioned to decide when such actions are beneficial, especially as modern development workflows, including AI-assisted coding and automated tooling, continue to increase the volume and frequency of changes flowing into repositories.
Conclusion
Search‑for‑reuse and Counting are critical metrics for understanding Git repository health.
Our 12‑hour stress tests show that no maintenance leads to runaway SFR and Counting latencies and system collapse, while periodic GC offers only temporary relief and introduces instability.
In contrast, GHS employs reinforcement‑learning and a palette of actions to keep the repository efficient. The result is a dramatic reduction in SFR and Counting latencies, keeping them as efficient as possible, and sustained throughput throughout the simulation. Importantly, GHS achieves these gains while consuming far less CPU and memory: system‑load graphs remain flat, the agent helps reduce the need for expensive full-GC runs, and the server never approaches the overload thresholds seen in the baseline and rule‑based scenarios.
This lighter footprint translates into lower infrastructure cost, fewer cores and smaller memory footprints are needed to support the same workload, while simultaneously improving user experience. For DevOps engineers and Git administrators, these results highlight the importance of intelligent, adaptive acceleration strategies. Deploying a tool like GHS can transform the developer experience from unpredictable slowdowns to consistent, high‑performance operations, and do so economically by minimising resource consumption.
GerritHub has been steadily growing, with more teams choosing Gerrit-based workflows. With a larger and more diverse set of projects using the platform every day, it becomes increasingly important to keep the service ad-free, open and fair for everyone.
Our analysis shows that less than 0.1% of all the repositories hosted on GerritHub exceeds the 2 GB mark, therefore to ensure we can continue providing high-quality, free hosting for the vast majority of our community, we are introducing a disk size limit for projects on GerritHub. This allows us to guarantee that all repositories under 2 GB remain free for everyone as we scale up the storage capabilities.
Please note that the 2GB limit will not apply to open-source projects, however, they’ll be required to reach out to us for explicit approval.
What does this mean in practice? Repositories imported from GitHub that exceed the 2 GB limit will fail the import into GerritHub, and new pushes that would grow an existing project beyond that threshold will be rejected with a clear, helpful error message explaining why, as follows:
➜ test git:(580cdcf) git push origin HEAD:refs/for/master
...
fatal: Unpack error, check server log
error: remote unpack failed: error Exceeded quota request for project repo/test. Maximum 2GB allowed. If you need a higher limit, reach out to us at GerritForge.com/contact to discuss your needs.
To https://review.gerrithub.io/repo/test
! [remote rejected] HEAD -> refs/for/master (n/a (unpacker error))
error: failed to push some refs to 'https://review.gerrithub.io/repo/test'
This helps keep storage usage under control and ensures that everyone gets a fair use of the common GerritHub.io platform. For most users, especially those working primarily with source code, this limit is generous and should not require any changes to everyday workflows.
This update reflects the healthy and continuous adoption of GerritHub by the community. As usage increases, small operational improvements like this one help us maintain a stable and welcoming environment for all users. We are thankful to everyone who relies on GerritHub for code review and collaboration, and we remain committed to improving the service as it grows. If you require a higher limit than this, please feel free to reach out to us at GerritForge.com/contact to discuss your needs and how we can help you further.
However, the world it was built for – local repositories and teams, along with simple, human-centric CI/CD pipelines – no longer exists.
Today, codebases are massive. Teams are global. And AI is generating code faster than ever before. Our infrastructure hasn’t kept pace, and traditional Git servers are reaching their limits.
Development Has Outgrown Traditional Git
Software development has exploded in both volume and velocity. According to Archive Market Research, the global software repository market will hit $15 billion in revenue in 2025, growing at 15% annually through 2033. GitHub now hosts over 420 million repositories, supporting more than 100 million developers – each pushing more commits, more frequently, and with more automation than ever.
At the same time, 97% of teams now use AI-assisted tools for vibe-coding, AI-agents for creating PRs and commits, automated AI-code-review, and AI-generated documentation. This results in a massive surge in server traffic, particularly read operations from increased test jobs and automated quality-checking processes, which are literally exploding.
Traditional scaling methods – adding more servers, bigger instances, or wider clusters – simply don’t address the core issue: the world has changed since Git was built 20 years ago.
The Breaking Point for Traditional Git Servers
The symptoms are now familiar to most engineering teams:
Slow performance. Git clones and fetches are taking several minutes or even timing out instead of completing swiftly in a few seconds.
Pipeline delays. CI/CD is waiting for repositories to be available for cloning or fetching, which delays the feedback cycle.
Storage capacity strain. Larger monorepos or multi-repo setups are hitting massive BLOB count and hitting ref limits. When storage is hosted on the Cloud or on shared disks, the increase in the directories and I/O transfer makes Git almost unusable.
Rising costs. Over-provisioned infrastructure is trying (and failing) to keep up. The provisional or elastic filesystem limits increase significantly, bringing operational costs to new heights never seen before, and putting more pressure on the budget.
The Archive Market Research report warns: “The rising complexity of software development necessitates efficient and reliable repositories for managing code, dependencies, and artifacts.”
In other words, the systems we depend on to deliver software are now slowing software down.
Scaling Smarter, Not Harder
At GerritForge, we believe that simply throwing more hardware at the problem is no longer enough: it is Git and its utilisation that need to be smarter. We have been researching on the use of AI for optimising the performance and scalability of repositories for over 3 years and shared our findings with the global scientific community.
That’s why we built GHS, an AI-powered accelerator for Git-based SCMs.
We have simulated and demonstrated how traditional tools would collapse and developed GHS, a brand-new model to overcome the traditional limitations of the Git repositories.
GHS learns how your repositories behave, your access patterns, your CI/CD workflows, and your traffic peaks – and, using reinforcement learning, automatically optimises performance in real-time.
With GHS, the whole development team will experience:
Up to 100× faster Git operations. Clones and fetches keep on completing at raw speed, without slowdowns, timeouts or slack.
Real-time auto-optimisation. The Git repositories are automatically monitored and optimised for speed, without manual operations.
Lower operational costs. The CPU and storage do not suffer from the overload of incoming traffic, delivering better performance on existing hardware. There is no need to scale up the infrastructure and costs massively.
Increased reliability. Git servers remain stable even under extreme load, eliminating the need for emergency restarts or maintenance.
Traditional scaling adds more servers. GHS scales using its sophisticated AI model to get the most out of what you already have.
The Gerrit User Summit 2025 made one thing clear: the era of traditional scaling is coming to an end.
As repositories grow and workflows evolve, the next generation of developer infrastructure must be intelligent, simplified & resilient. AI isn’t replacing developers – it’s augmenting them. But if your Git infrastructure can’t keep pace with this new level of activity, your entire SDLC pipeline will suffer.
2025 marks a turning point for software delivery. Traditional Git servers can’t keep up, but that doesn’t mean your team can’t. Because in modern DevOps, speed isn’t just efficiency, it’s a competitive edge.
Slow Git isn’t just annoying – it’s expensive for everyone: developers wait for a CI/CD validation of their changes, SCM admins are wasting their time in fire-fighting SCM slowdown and broken pipelines, IT managers are wasting millions of dollars in CPU and Disk hosting costs.What if your Git operations could be up to 100x faster, and keep up with the new AI-vibe coding landslide of PR and changes?
100x Faster Git, Powered by AI
GHS is an AI-based accelerator for Git SCM that redefines performance:
Up to 100x faster clones and fetches.
CI/CD pipelines that run without SCM barriers.
Adapt automatically to your repository shape and make it faster.
Scale without slowdown even under heavy loads
This isn’t traditional “tuning.” GHS learns your repos and access patterns, then continuously optimises them so your Git server is always running at maximum speed.
How Does GHS Deliver 100x Faster Git?
Measure Everything It collects detailed metrics in realtime on your repositories.
Spot Bottlenecks GHS AI model is trained on recognizing the bottlenecks and take immediate action, before they become a problem
Stay Fast Your Git stays consistently accelerated, not just temporarily boosted.
Why Speed Matters
Developers stop wasting hours on slow fetches and builds.
Release managers push features out faster.
IT leaders reduce infra costs by doing more with less.
Admins no longer fire-fight performance issues.
Every 1% of time saved in Git can add up to days of productivity across a large team. Imagine saving 100x that.
Who Needs 100x Faster Git?
Repositories of all sizes: AI-driven code generation and “vibe-coding” have dramatically accelerated the pace of software delivery.
Enterprises that have adopted the AI-pipeline want that value delivered faster to production.
Any team frustrated with slow CI/CD pipelines.
The GHS Advantage
Transformative speed: not just 2x or 5x faster, but up to 100x
SCM expertise: GerritForge’s decades of enterprise SCM know-how built in.
Proven reliability: Stability and uptime as performance scales.
Get Started Today
You can try GHS free for 30 days and experience the difference for yourself.
So you think PRs are the only way you can do effective code review? Well, then you’ll be surprised to hear that ain’t the case!
If you’ve read my previous article, you’ll be aware that GitHub or GitLab are great for quick and easy integrations with CI/CD tooling and average for issue tracking/planning, but, in my opinion, miss the mark when it comes to facilitating effective code review.
Today, I want to show you a typical Gerrit workflow and explain why I believe it can greatly improve the quality of your reviews.
Let’s start by reiterating that PRs should be small so as to be easily reviewable, and it’s the creator’s job to make sure they are paired with a meaningful description. But don’t take it from me, you can read it directly on GitHub’s blog.
However, splitting out work into easily reviewable chunks is not easy or intuitive when following a PR-based approach as maintaining dependent PRs is cumbersome and error-prone, so let’s see how Gerrit facilitates this for developers.
Relation chains vs PRs
Let’s say that you’ve started working on a new feature. You realize early on that it’ll be a lot easier to implement if you do some refactoring beforehand. So you go ahead, do the refactoring, commit your work, then add the new feature and commit that too, as a separate commit. If you’re working with a PR based workflow, before doing the above you’ve probably created a branch and are now about to push it to the remote to then create a PR from the UI, am I right?
Well, if you’re using Gerrit, you don’t need to do that. If your work is supposed to be merged on main, then you can develop directly on that branch, create those 2 commits and push them with git push origin HEAD:refs/for/main, we’ll decompose this command later, but for now let’s look at the effects of it.
As you can see in the screenshot above, Gerrit detects that you’re pushing two separate commits, so it automatically creates 2 new changes, one for each commit, against the branch you’re working on. There was no need to interact with the UI either.
But how did this happen? HEAD:refs/for/<branch-name> tells the Gerrit backend that you’d like to push your current local changes (HEAD) up for review against branch-name Gerrit then does all the magic for you.
Let’s now take a look at the UI:
There’s quite a lot to unpack here, so let’s go in order. First, Gerrit created a Relation chain for us, that includes both commits. Then, in the commit message, we notice something maybe unexpected, a Change-Id. This is Gerrit’s way of tracking updates to a change. From now on, every time we push a new commit with the same Change-Id, Gerrit will understand it belongs to this change and update it accordingly. You could completely change the commit message, but, as long as you leave the Change-Id unchanged, Gerrit will understand where it belongs.
So let’s try doing that.
How to update changes
The main difference from what most people are used to here is that when you want to update a change, you don’t push a new commit; rather, you amend the existing one
First I need to checkout the change, I can do that by clicking on the DOWNLOAD button(highlighted in the image above), which will show this modal window
This gives me a few options for checking out said change without having to Google how to check out a specific ref or how to cherry-pick each time. So, I copy-paste the “Checkout” option and then run the following on my terminal.
You can see here that I’m not creating a new commit, but rather amending the same one.
Gerrit warns me now that I’ve not amended any files but only the commit message, no problems there, just a handy reminder.
As I’ve not changed the Change-Id, we can see that the change number is still the same. The UI now looks like this:
A few things to notice here. First, I’ve updated the commit message to add a description, great, but surely I wouldn’t want to lose track of the initial state of the change? Can I still see the previous version of this commit? YES!
When you amend your commit, you’re not losing the history of what was there before, you’re merely adding a new patchset to the change. Patchsets are what Gerrit calls each iteration of the commit.
You can also see how the next commit in the relation chain is now marked as an “Indirect relation.” This is because it is not based anymore on the latest version of its parent commit, which as just been updated, so let’s go ahead and fix it. First, we’ll need to select the commit we want to update and then click on the handy “Rebase” button, which will show the following modal
As this is a straightforward case, we’ll confirm the rebase on the parent change.
We can now see that the indirect relation message has disappeared. But how did Gerrit keep track of that? By creating a new patchset which tracks the updated version of the parent commit! Pretty neat, huh?
This leads us to our next topic:
Meaningful commit messages by design
Another problem I see with the PR workflow, or at least how it’s been implemented by the most popular tools today, is that it essentially leaves it up to each individual developer to commit meaningful messages. Whatever merge strategy you pick, neither of them allows for the commit message to be reviewed before it gets merged.
In Gerrit there is no PR description, the commit message IS the “PR description”. Further more, the commit message appears as file in its own right and users can comment on it. This allows reviewers to actually add comments on the message itself to ensure it’s up to scratch with whatever the company standards are.
Submitting the change
Finally, once the comment has been addressed and the change approved, we can click on Submit including parents, and both changes will be merged to the main.
As can be seen by running git log , which will look like this:
Conclusion
The one thing I’d like readers to take away from this article is that no, PRs are not the only way forward and there are solutions to the old problem of wanting to split up code reviews in more manageable chunks.
So let’s quickly review how Gerrit works: You create changes by pushing commits using the “magic” command refs/for/<bran-name> . When you want to update a change, you don’t push a new commit; rather, you amend the existing one. Gerrit keeps track of each update in the form of patchsets so you can see how each commit evolved over time.
You can then chain commits as you wish, allowing you to split your work in easily reviewable units.
Finally, commit messages are treated as first class citizens, they’re like any other file in the change, they can be reviewed and commented on, ensuring that no un-helpful commits are merged to the target branch.
(TL;DR) The Gerrit Code Review project has an ambitious roadmap for 2025 and beyond. Gerrit 3.12 (H1 2025) will focus on JGit performance improvements, X.509 signed commit support, and enhanced Owners and Analytics plugins. Gerrit 3.13 (H2 2025) aims to further optimize push performance, UI usability, and plugin updates, including Kafka and Zookeeper integrations.
The k8s-Gerrit initiative will improve Gerrit’s deployment on Kubernetes, ensuring better scalability and resilience. Looking ahead, Gerrit 4.0 (2026/2027) plans to decouple the review UI from the JGit server, enabling alternative review interfaces and improved flexibility.
The development team is prioritizing significant performance enhancements in JGit, the Java implementation of Git used by Gerrit. Key objectives include:
Speeding Up Conflicting Ref Names on Push: This aims to reduce delays during push operations when ref name conflicts occur.
Enhancing SearchForReuse Latency for Large Monorepos: The goal is to improve latency by at least an order of magnitude, facilitating more efficient operations in extensive monorepositories.
Improving Object Lookup Across Multiple Packfiles: Efforts are underway to accelerate object lookup times by at least tenfold, enhancing performance in repositories with numerous packfiles.
Parallelizing Bitmap Generation: By enabling bitmap generation across multiple cores, the team aims to expedite this process, contributing to overall performance gains.
Gerrit Core Experience Enhancements
A notable feature planned for this release is the support for X.509 signed commits, which will bolster security and authenticity in the code review process.
Owners Plugin Improvements
Enhancements to the Owners Plugin are set to provide clearer insights and streamlined interactions:
Action Requirements Display: Explicitly showing required actions by each owner at the file level.
Detailed Pending Reviews: Offering more comprehensive information on pending reviews by owners.
Easier Contact with File Owners: Facilitating more straightforward communication with file owners.
Analytics Plugin Optimization
The analytics plugin is slated for improvements to enhance speed and usability:
Repo Manifest Discovery: Native support for discovering repository manifests.
Faster Metrics Extraction: Accelerated extraction of metrics for branches, aiding in quicker data analysis.
Gerrit 3.13 (Target: H2 2025)
JGit Performance and Concurrency Enhancements
Building upon previous improvements, version 3.13 aims to further optimize performance:
Optional Connectivity and Collision Checks: Improving push performance by allowing the skipping of certain checks when appropriate.
Customizable Lock-Interval Retries: Providing flexibility in managing lock intervals to enhance concurrency handling.
Read-Only Multi-Pack Index Support: Introducing support for read-only multi-pack indexes to improve repository access efficiency.
Gerrit Core and UI Experience Enhancements
User experience remains a focal point with planned features such as:
File List Filtering: Allowing users to filter the file list in the change review screen for more efficient navigation.
Headless Gerrit Packaging: Offering a version of Gerrit that serves only read/write Git protocols, catering to users seeking a streamlined experience.
Plugin Updates
The roadmap includes updates to key plugins:
Kafka Events-Broker: Upgrading to support Kafka 3.9.0, enhancing event handling capabilities.
Zookeeper Global-Refdb: Updating to support Zookeeper 3.9.3, improving global reference database management.
Replication Plugin Enhancements
Efforts to simplify configuration and improve performance include:
Dynamic Endpoint Management: Introducing APIs for creating and updating replication endpoints dynamically.
UI Integration: Displaying replication status within the user interface for better visibility.
Reduced Latency on Force-Push: Improving replication latency during force-push operations by applying objects with prerequisites.
k8s-Gerrit
The k8s-Gerrit initiative focuses on deploying Gerrit within Kubernetes environments, aiming to enhance scalability, resilience, and ease of management. This approach leverages Kubernetes’ orchestration capabilities to provide automated deployment, scaling, and management of Gerrit instances, facilitating more efficient resource utilization and operational consistency.
Gerrit 4.0 (Target: 2026/2027)
Looking ahead, Gerrit 4.0 is set to introduce significant architectural changes:
Decoupling Gerrit Review UI and JGit Server: This separation will allow the Gerrit UI and JGit server to operate as independent services, providing greater flexibility in deployment and scaling.
Enabling Alternative Review UIs: By decoupling components, the platform will support the integration of other review interfaces, such as pull-request systems, offering users a broader range of tools tailored to their workflows.
The Gerrit community is encouraged to stay engaged with these developments, as the roadmap is subject to change. Contributors planning to work on features not listed are advised to inform the Engineering Steering Committee (ESC) to ensure alignment with the project’s goals.
As 2024 draws to a close, it’s a great time to reflect on the milestones and achievements of the GerritForge team in innovating Git performance with the release of GHS, supporting and advancing the Gerrit Code Review project with 20 releases and over 1.5k commits. This year has been one of exciting developments and initiatives that have made Gerrit stronger, more accessible, and better equipped to meet the needs of modern software teams. Here’s what GerritForge accomplished for the Git and Gerrit Code Review ecosystem in 2024.
2024 in numbers
1,500 changes merged
40 repositories involved in the Gerrit and JGit projects
20 releases
5 conferences, 6 meetups
11 contributors, 4 maintainers
20k man/hours spent in open-source projects
Managing the Releases of Gerrit 3.10 and 3.11
As usual, we continue to ensure the regular releases of new Gerrit versions. This year was no exception with the releases of Gerrit 3.10 and 3.11. These releases introduced new features, performance optimisations, and bug fixes to further solidify Gerrit as the leading code review tool for large-scale projects.
Our team cooperated with the Gerrit Code Review community to ensure these releases were stable, well-documented, and aligned with community needs. From coordinating release schedules to ensuring compatibility with plugins and integrations, GerritForge played a key role in making these versions a reality.
Key Highlights
Gerrit 3.10 brought major improvements in user experience, including refined UI capabilities and enhanced search functionality.
Gerrit 3.11 introduced critical updates to multi-site support, security enhancements, and significant performance boosts.
GerritMeets success in both the USA and Europe
The GerritMeets series has become a cornerstone of community engagement, and 2024 was no exception. GerritForge organised six GerritMeets across several locations, including California, Germany, and the UK, bringing together contributors, users, and maintainers to share knowledge, discuss trends, and explore use cases. Each session covered a diverse range of topics, including:
Best practices for multi-site setups.
Advanced plugin development.
Gerrit performance tuning for large repositories.
Innovations in CI/CD workflows with Gerrit.
These virtual meetups provided a platform for collaboration and learning, reinforcing Gerrit’s community spirit. All the recordings of the 2024 GerritMeets are available as a playlist on the GerritForge TV YouTube channel.
World-class Enterprise Gerrit-as-a-Service on Google Cloud Platform
In 2024, GerritForge expanded Gerrit’s capabilities by adding support for Gerrit-as-a-Service (GaaS) on the Google Cloud Platform (GCP). This initiative makes it easier than ever for organizations to adopt Gerrit without the operational overhead of managing infrastructure.
Benefits of Gerrit-as-a-Service on GCP
Scalability: Leveraging GCP’s powerful infrastructure to scale Gerrit deployments for enterprises of any size.
Simplicity: Reducing setup and maintenance complexity, allowing teams to focus on development and code reviews.
High Availability: Utilizing GCP’s advanced networking and storage capabilities for improved uptime and disaster recovery.
By enabling Gerrit on GCP, GerritForge is broadening the tool’s accessibility, particularly for teams looking for a cloud-native, fully managed solution.
Gerrit User Summit with Qualcomm
This year’s Gerrit User Summit, organized by Qualcomm in collaboration with GerritForge, was a highlight for the community. Held at the Qualcomm HQ in San Diego, CA (USA), the summit offered a chance for Gerrit enthusiasts worldwide to come together in person. The agenda featured:
Keynotes by Gerrit maintainers and industry leaders.
Hands-on hackathon for contributors and users.
Insightful panels on the future of Gerrit.
The collaboration with Qualcomm not only expanded the summit’s reach but also highlighted Gerrit’s growing importance in enterprise environments.
Showcasing GHS and Gerrit Code Review worldwide
This year, GerritForge actively participated in numerous conferences to showcase GHS and expand Gerrit’s visibility, and demonstrate its unique capabilities. These events were a fantastic opportunity to uncover GHS and showcase Gerrit to hundreds of developers who had never encountered it.
At these conferences, we showcased:
GHS: It demonstrates its seamless integration with Git and showcases a whopping 100x performance improvement over a vanilla setup thanks to the power of AI and reinforcement learning.
K8s-Gerrit: Highlighting how Gerrit deployed on Kubernetes provides unparalleled flexibility, performance, and multi-site support.
Through live demos, presentations, and Q&A sessions, we highlighted GHS and Gerrit’s ability to scale, its unique review model, and its role in making software delivery pipelines up to 100x times faster.
K8s-Gerrit Hackathon with SAP
Collaboration was at the forefront of the k8s-Gerrit Hackathon, co-hosted by GerritForge and SAP. The hackathon brought together developers from both SAP and GerritForge teams to tackle the challenges of Kubernetes-based Gerrit deployments and multi-site support.
Outcomes of the Hackathon
Enhanced scalability for k8s-Gerrit deployments.
Breakthroughs in multi-site replication and disaster recovery.
Valuable contributions to the Gerrit codebase and documentation.
The event exemplified the power of open collaboration, pushing Gerrit further into cloud-native development.
Looking Ahead to 2025
As we celebrate the progress made in 2024, we remain focused on the road ahead. GerritForge is committed to:
Public offering of GHS
Gerrit-as-a-Service in Google Cloud
Gerrit Code Review v3.12 and v3.13
More GerritMeets and Gerrit User Summit events
Our gratitude goes out to the Git and Gerrit Code Review communities, contributors, and partners who have made this year a success. Together, we’re building a tool that empowers teams to deliver high-quality code faster and more efficiently. Here’s to an even more impactful 2025!
We’re thrilled to announce that our team will be speaking about our advancements with GerritForge AI Health Service (GHS) at several prestigious conferences in the coming months. These events provide an incredible opportunity to share our innovative AI solutions with a broader audience, engage with industry experts, and showcase how GHS is revolutionizing the way organizations maintain the health and stability of their Gerrit and Git systems.
Our journey begins at the Linux Open Source Summit in Vienna, from the 16th to the 18th of September. This summit is a cornerstone event for the open-source community, and we couldn’t be more excited to discuss how GHS leverages AI to ensure the seamless performance of Git and Gerrit systems, even in the most demanding environments.
Next, we’ll be in Berlin for Git Merge on the 19th and 20th of September. Git Merge is the go-to event for Git enthusiasts and professionals alike, and we’re eager to dive deep into the technical aspects of GHS, sharing insights on how our AI solution optimizes system performance, reduces downtime, and empowers development teams to focus on what they do best—creating great software.
In October, we’re particularly excited about the Gerrit User Summit in San Diego, on the 10th and 11th. This event is especially important to us as it brings together the Gerrit community to discuss the latest developments and best practices. We’ll be showcasing how GHS is enhancing Gerrit environments by providing intelligent and automated health monitoring and ensuring peak performance.
Following that, we’ll speak at the OCX conference in Mainz, from the 22nd to the 24th of October. OCX is known for bringing together top minds in DevOps and open-source technology, making it the perfect venue to highlight how GHS is transforming the management of code review and source control systems with intelligent, automated health monitoring and remediation.
Finally, we’re thrilled to wrap up our conference tour at KubeCon in Salt Lake City, from the 12th to the 15th of November. As one of the most anticipated events in the cloud-native ecosystem, KubeCon offers an unparalleled platform to demonstrate how GHS integrates with Kubernetes environments, ensuring that your SCM systems are always running at peak performance.
These conferences represent more than just speaking engagements for us—they are an opportunity to engage with the community, learn from our peers, and continue pushing the boundaries of what’s possible with AI in software development. We can’t wait to connect with you at these events and share how GHS can make a tangible difference in your organization’s success.
Stay tuned for more updates as we approach these dates, and be sure to catch our sessions if you’re attending any of these events!
Daniele Sassoli GerritForge Engineering Manager Gerrit Code Review Community Manager and Contributor
The Gerrit Code Review Open-Source project has transitioned from using Google Cloud Platform’s Remote Build Execution (RBE) to BuildBuddy’s on-premises to address performance, stability, and latency issues. The migration process included setting up a new Jenkins controller and provisioning BuildBuddy executors on newly provisioned on-premises boxes, which showed significantly reduced build times and a more consistent and reliable performance. After thorough evaluation and community consensus, BuildBuddy was adopted as the new default for Gerrit’s CI/CD pipeline, enhancing overall efficiency and stability.
Historical Context
The Gerrit Code Review project has undergone significant evolution in its build processes to enhance efficiency and performance. This evolution reflects the increasing complexity and demands of modern CI/CD pipelines.
Overview of Gerrit Code Review
Gerrit is a powerful code review tool with a powerful web and command-line interface, all built on top of the Git open-source project. Gerrit codebase is significant and multifaceted, using Python tooling, TypeScript front-end and a Java-based backend. To appreciate the challenges and the need for robust build tools, consider the scope of Gerrit’s codebase and build activity:
Plugins: Gerrit comprises 14 core plugins maintained as git submodules, plus a universe of over 300 community-based plugins developed in multiple languages, from Java to Scala and Groovy.
Java Codebase: The project includes 6011 Java files, with 4765 dedicated to production code, amassing ca. 411,768 lines of code (LoC). Additionally, there are 1246 test files (924 unit tests and 322 integration tests) contributing another ca. 276,632 LoC.
Frontend Codebase: The frontend is built with 110 JavaScript files (ca. 2345 LoC), 733 TypeScript files (ca. 175,765 LoC), 293 HTML files, and 9 CSS files.
Dependencies: Gerrit relies on 135 Java dependencies managed through Maven and 25 NPM dependencies (5 runtime and 20 development).
Gerrit has been founded in 2008 and has over 15 years of code-history, which reflects the evolution of the build tools, Java VMs and front-end technologies used for over a decade. The pre-requisites that you would have to manage in order to build Gerrit are diverse and quite challenging.
Build and Verification Activity
The Gerrit project is highly active, with rigorous commit-level verification processes to ensure code quality and stability. For example, from June 9 to June 23, 2024, Gerrit handled:
Total of changes:
Branch
Number of changes
master
65
stable-3.10
18
stable-3.9
16
stable-3.8
15
stable-3.7
2
stable-3.6
0
stable-3.5
1
stable-3.4
1
Total
118
Total of revisions (patch sets):
Branch
Number of revisions
master
230
stable-3.10
86
stable-3.9
21
stable-3.8
22
stable-3.7
4
stable-3.6
0
stable-3.5
4
stable-3.4
1
Total
368
Total of Gerrit verifications:
Type of verification
number of verifications
Build/Tests
277
Code Style
320
PolyGerrit UI Tests
124
RBE BB Build/Tests
271
Total
992
Evolution of Build Tools
The journey of Gerrit’s build tools reflects its growth and the increasing complexity of its CI/CD requirements:
Apache Maven: Up until version 2.7, Gerrit used Apache Maven as its build tool. Maven, known for its comprehensive project management capabilities, was sufficient during Gerrit’s early stages.
Buck: From version 2.8 to 2.13, Gerrit transitioned to Buck, a build tool designed for faster builds. Buck’s incremental build capabilities helped manage the growing codebase more efficiently than Maven.
Bazel: Since version 2.14, Bazel has been the default build tool for Gerrit. Bazel’s advanced features, including its support for remote caching and execution, provided significant improvements in build performance and scalability.
Transition to Bazel with Remote Execution and Caching
In December 2020, Gerrit Code Review made a significant shift by adopting Bazel with remote execution and caching to address the challenges of long build times. This strategic move aimed to leverage Bazel’s advanced capabilities to enhance the efficiency of the CI processes.
Reasons for the Shift
The primary driver for this transition was the increasing build times due to the growing complexity and size of the Gerrit codebase. The conventional local build processes were becoming a bottleneck, slowing down the development and integration cycles.
Implementation with GCP Remote Build Execution (RBE)
Gerrit integrated Google Cloud Platform’s Remote Build Execution (GCP RBE) as the remote server to support this transition. The integration provided several key benefits:
Reduced Build Times: By offloading build and test tasks to powerful remote servers, build times were significantly reduced.
Efficient Resource Utilization: Local machines were freed from heavy build tasks, allowing developers to continue working without interruptions.
Scalability and Parallelisation: Remote execution and the parallelisation of Gerrit’s Bazel tasks allowed to leverage the scalable cloud resources.
This implementation marked a crucial enhancement in Gerrit’s CI/CD pipeline, setting the stage for further optimisations and improvements in the build process.
Motivation to find RBE alternatives
The RBE implementation on Google Cloud has served the Gerrit Code Review project successfully for many years; however, the needs of the project grew over time and the CI/CD infrastructure had to satisfy additional requirements.
Stability: Google Cloud is SaaS solution which could be flaky at times, whilst the project needed a stable deployment with full control on its stability not influenced by external factors.
Latency between the controller and the executors: the latency between the main CI/CD controller (Jenkins) and the RBE executors paid a significant price for shorter builds like the Code-Style checks, whilst a localised data processing resulted in faster build times and quicker feedback cycles.
Predictability: Consistent and reliable performance is crucial for efficient CI/CD workflows.
Moving to BuildBuddy RBE
BuildBuddy is an open-core Bazel build event viewer, result store, remote cache, and remote build execution platform that provided many new benefits to the Gerrit Code Review builds:
Integration and Customisation: the integration with the existing CI/CD pipelines was straightforward.
Open Source Community: BuildBuddy, being open-core, benefits from community-driven innovation and collaborative support.
Enterprise Features: BuildBuddy Enterprise offers advanced features for companies that need robust capabilities:
OpenID Connect Auth Support: Integrates with Google OAuth.
Configurable Bazel Caches TTL: Allows setting TTL for build results and cache with support for persistent build artifact storage.
High Availability: Configurations for high availability also on-premises
Control and Stability: On-premise deployment offers full control and enhanced stability by minimizing reliance on external factors.
Very Low Latency: Localized data processing results in faster build times and quicker feedback cycles: we could locate the executors and the Jenkins controller in the same data-centre with micro-seconds network latency.
Predictable Performance: Consistent and reliable performance is crucial for efficient CI/CD workflows, thanks to the dedicated always-on executors.
BuildBuddy RBE allowed more development efficiency and reliability for the Gerrit Code Review project, making it a compelling choice for optimizing CI/CD processes while leveraging the benefits of open-source software and robust enterprise features.
What was the migration plan ?
To clarify a few points for a better understanding of the this section:
Scope of Bazel RBE Execution: Bazel RBE is executed only in the Gerrit project and its core plugins (git submodules). It is not executed in non-core plugins, such as pull-replication, high-availability, multi-site, etc.
Branch Support: From a CI/CD perspective, only the master branch and the last three stable branches are supported for Gerrit project, core and non-core plugins. At the time of the migration, these branches were master, stable-3.7, stable-3.8, and stable-3.9.
The initial phase of the migration aimed to assess the reliability and stability of BuildBuddy RBE. A priority in this phase was to maintain the current CI/CD process while simultaneously evaluating BuildBuddy RBE without any disruptions.
To achieve this phase, several updates and new services were implemented:
Provisioning BuildBuddy Executors: A cloud host was provisioned with the following specifications: 128 CPUs, Intel(R) Xeon(R) Gold 6438Y+, 128GB RAM, and SSD. This host runs 3 BuildBuddy executors (as docker containers).
Setting up a new Gerrit CI Server: A new Jenkins server was set up to run build jobs against BuildBuddy RBE on the Gerrit master branch. This server is not accessible from outside.
Registering a new Gerrit verification: A new verification named RBE BB Build/Tests was added to gerrit-review.googlesource.com to trigger builds on the new Gerrit CI server whenever a new revision was created on the Gerrit master branch.
Figure 1: Architecture Migration diagram with default CI flow and new BuildBuddy CI flow:
How is the new CI/CD flow?
In the default CI flow, when a user creates a new revision (patch set) in Gerrit master, or stable-3.7 or stable-3.8 or stable-3.9 branches, a set of verification jobs trigger Jenkins jobs. These verification jobs include:
RBE GCP Build/Tests: Builds the codebase and executes all the unit/integration tests on GCP RBE.
Code Style: Checks Java and Bazel formatting, and JavaScript lint.
Build/Tests: Builds the codebase and executes one single no-op test.
PolyGerrit UI Tests: Executes unit/integration tests for PolyGerrit UI.
If any of the verification jobs fail, the verification status of the revision is marked with a -1.
As mentioned earlier, the intention when testing the reliability and stability of BuildBuddy RBE was to avoid interfering with the default CI/CD flow. To achieve this, a new verification job called RBE BB Build/Tests was added. This verification triggers a Jenkins job on the new Gerrit CI, which builds the codebase and executes unit/integration tests on BuildBuddy RBE. This setup allowed the default flow and the BuildBuddy RBE flow to coexist without affecting each other.
It is important to note two things:
Only revisions in the master branch of Gerrit project triggered this new verification job. The data collected from the master branch is sufficient to draw conclusions.
The status of this new verification job does not affect the overall verification status of the revision.
Figure 2: Verification jobs, default ones and the BuildBuddy RBE, triggered in a Gerrit master branch revision:
Once the first phase concluded, it was important to analyze the data to determine if BuildBuddy RBE was reliable and stable enough to proceed to the next phase. In the second phase, the plan was to evaluate the performance of BuildBuddy RBE against GCP RBE. Architecturally, the CI/CD process remained the same as in the first phase, with one key difference: the verification job RBE BB Build/Tests would be triggered when revisions were created for the Gerrit repo on the master, stable-3.7, stable-3.8, and stable-3.9 branches. This was necessary to ensure that BuildBuddy RBE handled the same number of jobs as GCP RBE, allowing for a fair performance comparison.
Data Collection
Before analysing the data, it’s imperative to elucidate our data collection methodology. To procure the build data (build number, execution time in GCP RBE and BB RBE and status), we developed a script in python that employed two APIs:
Build number is a unique number represented by the tuple: (change number, revision number).
All the graphs show builds in chronological order.
The build numbers are not shown in the graphs for readable purposes.
Builds labelled as “RUNNING” or those lacking specification according to the API have been excluded from the calculations.
Key Performance Indicators
Average Build Time: Calculate the average build time for each platform (GCP RBE and BuildBuddy RBE) to understand the typical time it takes to complete a build on each platform.
Percentage of Builds Faster: Determine the percentage of builds that are completed faster on BuildBuddy RBE compared to GCP RBE. This helps assess which platform is more efficient in terms of build time.
Overall Success Rate / Failure Rate: Calculate the overall success and failing rate of builds on BuildBuddy RBE. This considers both successful and failed builds to provide a comprehensive view of platform reliability.
Outliers (>60 minutes): Identify the percentage of builds that exceed a certain threshold, such as 60 minutes in BuildBuddy RBE. This helps pinpoint builds that take exceptionally long and may require investigation or optimization.
Average Build Time Reduction: Determine the average reduction in build time when using BuildBuddy RBE compared to GCP RBE. This quantifies the efficiency improvement gained by using the BuildBuddy platform.
PHASES
As we mentioned above, the migration has been segmented into two distinct phases:
Phase 1: Spanning from December 28th, 2023, to February 9th, 2024, during which RBE BuildBuddy operated against the Gerrit master branch. Phase 2: Commencing from February 10th, 2024, to February 26th, during which RBE BuildBuddy operated against the Gerrit master, stable-3.7, stable-3.8, and stable-3.9 branches.
Phase 1: Evaluate if BuildBuddy RBE offers stability and low latency
To make the data more readable and understandable, I have split the data into 2 graphs:
Figure 3: RBE Successful Build time for Gerrit master between 28th December 2023 to 18th January 2024:
Figure 4: RBE Successful Build time for Gerrit master between 19th January 2024 to 9th February 2024:
Total number of builds:
master
GCP Builds
489
BB Builds
489
Build status:
BB Successful
BB failed
GCP Successful
390
17
GCP Failed
0
82
Initially, 3.47% of BuildBuddy RBE builds failed due to CPU exhaustion caused by running 100 BuildBuddy executors simultaneously. This problem was addressed by reducing the number of executors to 3. BuildBuddy engineers advise running only one executor container per host/node, with each executor capable of handling multiple RBE Actions concurrently. For each action, an executor initiates an isolated runner to execute it. We plan to reassess our configuration in due course.
Average build time when GCP and BuildBuddy builds were successful:
Minutes
GCP Average
18.69
BB Average
10.2
Where the average build time reduction is 8.49 minutes and 96.4% (376 out of 390 builds) of BuildBuddy builds are faster than GCP builds.
We discovered that 1.5% of BuildBuddy successful builds were outliers. This was due to the need for a restart of the new Gerrit CI server, which caused temporary disruptions.
change_number
REVISION_NUMBER
GCP RBE MINUTES
BB RBE MINUTES
400398
1
6.7
868.68
399657
11
13.7
1293.55
399657
14
21.45
137.47
400958
2
14.52
154.3
247812
7
26.62
67.17
406597
1
14.18
79.55
Average time when GCP and BB Failed:
Minutes
GCP Average
17.68
BB Average
23.29
Conclusions:
Assessing performance and stability, the results were promising, with the BuildBuddy platform showcasing superior performance, as highlighted in the table “Average build time when GCP and BB Successful”. Additionally, issues with BuildBuddy failing builds during successful GCP builds were addressed, primarily stemming from resolved configuration problems. Although outliers represented a mere 1.5%, their significance was negligible. However, despite these favourable outcomes, caution was warranted due to the higher volume of builds in GCP compared to BuildBuddy, attributed to GCP’s operation across stable branches.
Phase 2: Compare BuildBuddy RBE with GCP RBE based on performance
To make the data more readable and understandable, The data has been splitted into 4 graphs:
Figure 5: RBE Successful Build time for Gerrit master:
Figure 6: RBE Successful Build time for Gerrit stable-3.9:
Figure 7: RBE Successful Build time for Gerrit stable-3.8:
Figure 8: RBE Successful Build time for Gerrit stable-3.7:
Successful BB Build status / Successful GCP Build status:
master
stable-3.9
stable-3.8
stable-3.7
Total
Builds
119
26
6
11
162
Average time when GCP and BB Successful:
Minutes
GCP Average
13.91
BB Average
8.45
Where the average build time reduction is 5.46 minutes and 90.74% (147 out of 162 builds) of BuildBuddy builds are faster than GCP builds.
Failed BB Build states / Failed GCP Build status:
master
stable-3.9
stable-3.8
stable-3.7
Total
Builds
30
12
1
1
44
Failed BB Build status / Successful GCP Build status:
master
stable-3.9
stable-3.8
stable-3.7
Total
Builds
1
2
0
0
3
It is worth noting that 1.14% of BuildBuddy builds failed.
Average time when GCP and BB builds failed
Minutes
GCP Average
10.96
BB Average
9.43
Conclusions:
The findings indicated that the BuildBuddy scenario demonstrated a more consistent performance, due to the on-premises allocated resources, as emphasised in the table “Average build time when GCP and BB Successful,” with comparable volumes of builds. Moreover, the stability remained highly consistent, evident from the table “Failed BB Build status / Successful GCP Build status,” alongside the absence of outliers.
Gerrit code review community decision
On February 27, 2024, the collected data was shared with the Gerrit code review open-source community. After careful consideration and thorough analysis, BuildBuddy was found to demonstrate remarkable stability. While it cannot be definitively stated that BuildBuddy surpasses GCP in all aspects, it notably outperforms GCP in terms of latency. Given its superior latency performance and strong stability, the decision was made to adopt BuildBuddy to replace GCP in the CI/CD pipeline.
Configured Gerrit CI to support both core and non-core plugin jobs, ensuring external visibility.
Unregistered the Gerrit verification RBE GCP Build/Tests on gerrit-review.googlesource.com.
Figure 9: Default Architecture diagram with BuildBuddy CI/CD flow as default CI/CD flow:
Final Conclusions
Following the completion of the migration, data on BuildBuddy RBE was collected from May 1, 2024, to June 24, 2024, to validate all assumptions. Subsequent statistical analysis yielded the following results:
Figure 10: Successful Builds:
Builds
465
Mean
13.62 min
Median
10.47 min
Standard dev
10.79 min
A higher standard deviation indicates that the build times are spread out over a wide range, meaning there is a lot of variability in the times
Q3
15.23 min
75% of builds are completed in less than 15.23 minutes.
Figure 11: Failed Builds:
Builds
105
Mean
7.72 min
Median
6.22 min
Standard dev
7.56 min
Q3
7.7 min
75% of builds are completed in less than 7.7 minutes.
While we are satisfied with our current results, we recognize the need for improvements in our successful builds. Our next step will be to analyze all the build data provided by the BuildBuddy dashboard, including target-level metrics, timing, artifacts, cache, and executions. This analysis will help us enhance the Bazel configuration and improve build performance.
