Slow Git isn’t just annoying – it’s expensive for everyone: developers wait for a CI/CD validation of their changes, SCM admins are wasting their time in fire-fighting SCM slowdown and broken pipelines, IT managers are wasting millions of dollars in CPU and Disk hosting costs.What if your Git operations could be up to 100x faster, and keep up with the new AI-vibe coding landslide of PR and changes?
100x Faster Git, Powered by AI
GHS is an AI-based accelerator for Git SCM that redefines performance:
Up to 100x faster clones and fetches.
CI/CD pipelines that run without SCM barriers.
Adapt automatically to your repository shape and make it faster.
Scale without slowdown even under heavy loads
This isn’t traditional “tuning.” GHS learns your repos and access patterns, then continuously optimises them so your Git server is always running at maximum speed.
How Does GHS Deliver 100x Faster Git?
Measure Everything It collects detailed metrics in realtime on your repositories.
Spot Bottlenecks GHS AI model is trained on recognizing the bottlenecks and take immediate action, before they become a problem
Stay Fast Your Git stays consistently accelerated, not just temporarily boosted.
Why Speed Matters
Developers stop wasting hours on slow fetches and builds.
Release managers push features out faster.
IT leaders reduce infra costs by doing more with less.
Admins no longer fire-fight performance issues.
Every 1% of time saved in Git can add up to days of productivity across a large team. Imagine saving 100x that.
Who Needs 100x Faster Git?
Repositories of all sizes: AI-driven code generation and “vibe-coding” have dramatically accelerated the pace of software delivery.
Enterprises that have adopted the AI-pipeline want that value delivered faster to production.
Any team frustrated with slow CI/CD pipelines.
The GHS Advantage
Transformative speed: not just 2x or 5x faster, but up to 100x
SCM expertise: GerritForge’s decades of enterprise SCM know-how built in.
Proven reliability: Stability and uptime as performance scales.
Get Started Today
You can try GHS free for 30 days and experience the difference for yourself.
First, a huge thank you to the OpenInfra Foundation for hosting this event in Paris. Their invitation to have the Gerrit User Summit join the rest of the community set the stage for a truly collaborative and impactful gathering.
Paris last weekend wasn’t just a conference; it was a reunion. Fourteen years after the last GitTogether at Google’s Mountain View HQ, the “Git and Gerrit, together again” spirit was electric.
On October 18-19, luminaries from the early days (Scott, Martin, Luca, and many others) reconvened, sharing the floor with the new generation of innovators. The atmosphere was intense, filled with the same collaborative energy of 2011, but focused on a new set of challenges. The core question: how to evolve Git and Gerrit for the next decade of software development, a future dominated by AI, massive scale, and an urgent demand for smarter workflows.
Here are the key dispatches from the summit floor.
A Historic Reunion, A Shared Future
This event was a powerful reminder that the open-source spirit of cross-pollination is alive and well. The discussions were invigorated by the “fresh air” from new-school tools like GitButler and Jujutsu (JJ), which are fundamentally rethinking the developer experience.
In a significant show of industry-wide collaboration, we were delighted to have GitLab actively participating. Patrick’s technical presentation on the status of reftable was a highlight, but his engagement in discussions on collaborative solutions moving forward with the Gerrit community truly set the tone. It’s clear that the challenges ahead are shared by all platforms, and the solutions will be too.
Scaling Git in the Age of AI
The central theme was scale. In this rapidly accelerating AI era, software repositories are growing at an unprecedented rate across all platforms—Gerrit, GitHub, and GitLab alike. This isn’t a linear increase; it’s an explosion, and it’s pushing SCM systems to their breaking point.
The consensus was clear: traditional vertical and horizontal scaling is no longer enough. The community is now in a race to explore new techniques—from the metal up—to improve performance, slash memory usage, and make core Git operations efficient at a scale we’ve never seen before. This summit was a rare chance for maintainers from different ecosystems to align on these shared problems and forge collaborative paths to solutions.
Dispatches from the Front Lines: NVIDIA and Qualcomm
This challenge isn’t theoretical. We heard powerful testimonials from industry giants NVIDIA and Qualcomm, who are on the front lines of the AI revolution.
They shared fascinating and sobering insights into the repository explosion they are actively managing. Their AI workflows—encompassing massive datasets, huge model binaries, and unprecedented CI/CD activity—are generating data on a scale that is stressing even the most robust SCM systems. Their presentations detailed the unique challenges and innovative approaches they are pioneering to tackle this data gravity, providing invaluable real-world context that fueled the summit’s technical deep dives.
Beyond the Pull Request: The Quest for a ‘Commit-First’ World
One of the most passionate debates centered on the developer workflow itself. The wider Git community increasingly recognizes that the traditional, monolithic “pull request” model is ill-suited to the “change-focused” code review that platforms like Gerrit have championed for years.
The benefits of a change-based workflow, cleaner history, better hygiene, and higher-quality atomic changes—are driving a growing interest in standardizing a persistent Change-ID for each commit. This would make structured, atomic reviews a first-class citizen in Git itself. The collaboration at the summit between the Gerrit community, GitButler, JJ, and other Git contributors on defining this standard was a major breakthrough.
This shift is being powered by tools like GitButler and JJ, which are built on a core philosophy: Workflow Over Plumbing. Modifying commits, rebasing, and resolving conflicts remain intimidating hurdles for many developers. The Git command line can be complex and unintuitive. These new tools abstract that complexity away, guiding users through commit management in a way that feels natural. The result is faster iteration, higher confidence, and a far better developer experience.
AI and the Evolving Craft of Code Review
Finally, no technical summit in 2025 would be complete without a deep dive into AI. The arrival of AI-assisted coding is fundamentally shifting the dynamic between author and reviewer.
Engineers at the summit expressed a cautious optimism. On one hand, AI is a powerful tool to accelerate reviews, improve consistency, and bolster safety. On the other, everyone is aware of the trade-offs. Carelessly used, AI-generated code can weaken knowledge sharing, blur IP boundaries, and erode a team’s deep, institutional understanding of its own codebase.
The challenge going forward is not to replace the human in the loop, but to strengthen the craft of collaborative review by integrating AI as a true co-pilot.
A Path to 100x Scale: The GHS Initiative
The most forward-looking discussions at the summit centered on how to achieve the massive scale required. One of the most promising solutions presented was GHS (Git-at-High-Speed). This innovative approach is not just an incremental improvement; it’s a strategic initiative designed to increase SCM throughput by as much as 100x.
The project’s vision is to enable platforms like Gerrit, GitLab, and GitHub Enterprise to handle the explosive repository growth and build traffic generated by modern AI workflows. By re-architecting key components for hyper-scalability, GHS represents a concrete path forward, ensuring that the industry’s most critical SCMs can meet the unprecedented demands of the AI-driven future.
The Road from Paris
The Gerrit User Summit 2025 was more than a look back at the “glorious days.” It was a statement. The Git and Gerrit communities are unified, energized, and actively building the next generation of SCM. The spirit of GItTogether 2011 is back, but this time it’s armed with 14 years of experience and a clear-eyed view of the challenges and opportunities ahead.
I started my business 17 years ago, with a simple mission: spread the use of Git into large enterprises worldwide. The goal was simple and challenging: convincing large companies to trust Git and fill the gaps separating them from migrating from their legacy systems.
The first product that came out of it was, not surprisingly, Git Enterprise. You can still access its official launch YouTube video (https://youtu.be/unJxlD2aopY), which attracted over 5k views and interest. Since then, my blog has been called GitEnterprise. My mission to work with large companies intensified, and I successfully reached my first client, who agreed to abandon Subversion, which was the de facto standard at that time, and adopt Git after seeing my product.
I didn’t know that Google had started with a similar idea for a different purpose: making Git suitable for large enterprises to cooperate on developing the Android Operating System. The Google project was called Gerrit Code Review, inspired by the Gerrit Ritvield Code Review tool built by Guido Van Rossum after he joined Google.
I donated all the code of GitEnterprise to the Gerrit Code Review project and joined the endeavour under the leadership of Shawn Pearce, the Gerrit project founder. Inspired by the new partnership, I also changed the name of my company to GerritForge to represent its new mission.
Fast forward to today, and a lot has changed. Gerrit is a complex ecosystem comprising a central Gerrit with its core plugins and a constellation of 200+ plugins created by a universe of over 800 contributors. GerritForge has also grown and started expanding to new countries, opening an HQ in Sunnyvale, California. After the passing of Shawn Pearce in 2018, who died from lung cancer, we also took over the management of the global community events and local meetups, to serve the need of coordinating and fostering collaboration amongst contributors and users worldwide. We owe it to honour Shawn’s legacy to make his project successful. He used to call Gerrit Code Review his “130% Google side-project”, as it was more of a mission than a $DAY_JOB for him.
The 100% pure Open-Source model I adopted over 17 years ago must be upgraded to tackle the new challenges of managing and supporting a larger organization and community, and ensuring that it will continue to innovate and evolve in the years to come.
Re-licensing GerritForge plugins
GerritForge has helped many organizations worldwide get the most out of Gerrit Code Review and benefit from a 100% Open-Source business model while relying on rock-solid Enterprise Support with quality gates, longer-term maintenance, and strict SLAs.
However, the marketplace is evolving, and fewer companies are willing to invest in pure 100% Open-Source communities. On the other hand, the recent threats from Cybercriminals targeting the supply chain have driven Enterprises to request a higher level of software and supplier certification, which may interfere with a 100% pure Open-Source contribution model.
After exploring many other options, including joining software foundations or traditional contracting, I have decided to switch from a pure Apache 2.0 Open-Source to a mixed Open-Core contribution model for creating a rock-solid Gerrit distribution that can satisfy the more stringent requirements of certification and compliance.
I have called this new software Gerrit Enterprise to avoid confusion with the Gerrit core, which remains 100% Open-Source.
GerritForge confirms its commitment to the Gerrit core and core-plugins Open-Source contributions under the current AOSP Apache 2.0 license. Some of the plugins developed almost exclusively by GerritForge will continue to be developed as a Commercial Product with a new software licensing model.
To explain the re-licensing policy with a formula: Gerrit Enterprise [Commercial Product] = Gerrit Core [Open-Source Apache 2.0] + GerritForge plugins [Commercial Product].
The licensing model we have chosen is called “Business Software License”, aka BSL.
What is BSL, and what does it mean
The term BSL or “BSL license” was used for the first time by Cockroach Labs (the creators of CockroachDB) back in 2019. “BSL license” most commonly refers to the Business Source License. This source-available license allows for public viewing and non-production use of source code but restricts commercial or production use unless a specific “additional use grant” is provided by the licensor. After a predetermined period, the BSL automatically converts to a fully Open-Source license. This model balances software developers’ need for revenue with the benefits of Open-Source, making software source-available while ensuring it eventually becomes Open-Source.
In a nutshell, the GerritForge plugins released under BSL will have more restrictions compared to the current Apache 2.0 Open-Source:
The requirement to receive a formal BSL License by GerritForge.
The Software License will be FREE of charge for small businesses, non-profit organizations, and non-productive environments.
Does not allow bundling, reselling, or reusing in other products and services without the written consent of GerritForge
All contributors to the plugins under BSL will have to sign a specific CLA with GerritForge
The source code cannot be modified, fully or in part, or reused in source or binary form in any other Open-Source project before the expiration of the BSL
After 5 years of the GerritForge’s plugins released date under BSL, the code will be automatically released as Apache 2.0 to the Gerrit-Review main repository.
The BSL model allows us to continue to develop our plugins in the Open and support the current versions under Apache 2.0, but it will ensure that we are in control of the way the software is certified, distributed, and guaranteed to all the users who benefit from it.
This initiative fosters a virtuous cycle of mutual benefit. The support from companies adopting Gerrit and GerritForge plugins will directly fuel the community, allowing us to reinvest in core software development, key summits, and local meetups, ensuring the ecosystem thrives.
Which GerritForge plugins are going to be under BSL?
The plugins chosen to be converted to a BSL license are the ones that are developed and maintained by GerritForge for at least 80% of the code in the past 24 months.
Also, GerritForge has decided to leave as Open-Source under the Apache 2.0 license the ones that are widely used by the community or reused in other projects.
The major plugins involved in the re-licensing as BSL are:
Gerrit Multi-Site, with its associated components, Kafka/Kinesis/PubSub brokers, Zookeeper/DynamoDb refdb, Healthcheck-Enterprise, and Websession broker
Pull-replication plugin
GitHub and Virtualhost plugins, which are the basis of GerritHub.io
Cache-Chroniclemap lib module
Gerrit and Git analytics, including git-repo-metrics
AWS deployment for Gerrit
Other minor plugins may be added at a later time
The re-licensing means that the new development will happen on the GerritForge organisation (https://github.com/gerritforge) on GitHub and GerritHub (https://review.gerrithub.io), and the current master branch will be deprecated on Gerrit-Review.
What happens to the current GerritForge plugins on the stable branches?
GerritForge confirms its commitment to continue supporting all the existing plugins, either from GerritForge or other contributors, on the stable and supported non-EOL versions (see https://www.gerritcodereview.com/support.html#supported-versions) on the current Google-owned Gerrit-Review site.
What happens to the current GerritForge plugins on the unsupported EOL branches?
What if I am using those plugins in production now?
The current version of the plugins you have installed in your production environment will continue to work and be under Apache 2.0 on the Gerrit-Review site. Once you upgrade to the newer version of Gerrit Code Review (e.g., Gerrit v3.13 or later), you must contact GerritForge (www.gerritforge.com/contact) to obtain a license key before proceeding with the upgrade in production.
What if I am a GerritForge customer and want to upgrade to a new BSL version?
All GerritForge customers have automatic access to the BSL version of the plugins under support. Their Gerrit Enterprise Support contracts and associated policies have not changed. The BSL will not have any impact on their budget or the service they receive today.
What if I have both Open-Source and BSL commercial plugins from GerritForge installed?
You need to obtain a BSL license for the ones released by GerritForge, and you can use them alongside any other Open-Source plugin. The Open-Source plugins are released under the Apache 2.0 license and hosted on Gerrit-Review, and are not subject to any BSL license.
Questions or Concerns?
I am happy to address your concerns and answer any questions you may have about the move we are making for GerritForge; feel free to ask questions on this blog post or get in touch with me through https://gerritforge.com/contact.
Our goal is to continue to exist and support the Gerrit Code Review community, today and for the foreseeable future. GerritForge BSL will allow us to continue to innovate and bring even more contributions and features to Gerrit core and its plugins.
Thank you for supporting us in the past and in the near future!
Luca Milanesio GerritForge co-Founder and CEO Gerrit Code Review Maintainer & Release Manager Member of the Gerrit Code Review Engineering Steering Committee
(TL;DR) The Gerrit Code Review project has an ambitious roadmap for 2025 and beyond. Gerrit 3.12 (H1 2025) will focus on JGit performance improvements, X.509 signed commit support, and enhanced Owners and Analytics plugins. Gerrit 3.13 (H2 2025) aims to further optimize push performance, UI usability, and plugin updates, including Kafka and Zookeeper integrations.
The k8s-Gerrit initiative will improve Gerrit’s deployment on Kubernetes, ensuring better scalability and resilience. Looking ahead, Gerrit 4.0 (2026/2027) plans to decouple the review UI from the JGit server, enabling alternative review interfaces and improved flexibility.
The development team is prioritizing significant performance enhancements in JGit, the Java implementation of Git used by Gerrit. Key objectives include:
Speeding Up Conflicting Ref Names on Push: This aims to reduce delays during push operations when ref name conflicts occur.
Enhancing SearchForReuse Latency for Large Monorepos: The goal is to improve latency by at least an order of magnitude, facilitating more efficient operations in extensive monorepositories.
Improving Object Lookup Across Multiple Packfiles: Efforts are underway to accelerate object lookup times by at least tenfold, enhancing performance in repositories with numerous packfiles.
Parallelizing Bitmap Generation: By enabling bitmap generation across multiple cores, the team aims to expedite this process, contributing to overall performance gains.
Gerrit Core Experience Enhancements
A notable feature planned for this release is the support for X.509 signed commits, which will bolster security and authenticity in the code review process.
Owners Plugin Improvements
Enhancements to the Owners Plugin are set to provide clearer insights and streamlined interactions:
Action Requirements Display: Explicitly showing required actions by each owner at the file level.
Detailed Pending Reviews: Offering more comprehensive information on pending reviews by owners.
Easier Contact with File Owners: Facilitating more straightforward communication with file owners.
Analytics Plugin Optimization
The analytics plugin is slated for improvements to enhance speed and usability:
Repo Manifest Discovery: Native support for discovering repository manifests.
Faster Metrics Extraction: Accelerated extraction of metrics for branches, aiding in quicker data analysis.
Gerrit 3.13 (Target: H2 2025)
JGit Performance and Concurrency Enhancements
Building upon previous improvements, version 3.13 aims to further optimize performance:
Optional Connectivity and Collision Checks: Improving push performance by allowing the skipping of certain checks when appropriate.
Customizable Lock-Interval Retries: Providing flexibility in managing lock intervals to enhance concurrency handling.
Read-Only Multi-Pack Index Support: Introducing support for read-only multi-pack indexes to improve repository access efficiency.
Gerrit Core and UI Experience Enhancements
User experience remains a focal point with planned features such as:
File List Filtering: Allowing users to filter the file list in the change review screen for more efficient navigation.
Headless Gerrit Packaging: Offering a version of Gerrit that serves only read/write Git protocols, catering to users seeking a streamlined experience.
Plugin Updates
The roadmap includes updates to key plugins:
Kafka Events-Broker: Upgrading to support Kafka 3.9.0, enhancing event handling capabilities.
Zookeeper Global-Refdb: Updating to support Zookeeper 3.9.3, improving global reference database management.
Replication Plugin Enhancements
Efforts to simplify configuration and improve performance include:
Dynamic Endpoint Management: Introducing APIs for creating and updating replication endpoints dynamically.
UI Integration: Displaying replication status within the user interface for better visibility.
Reduced Latency on Force-Push: Improving replication latency during force-push operations by applying objects with prerequisites.
k8s-Gerrit
The k8s-Gerrit initiative focuses on deploying Gerrit within Kubernetes environments, aiming to enhance scalability, resilience, and ease of management. This approach leverages Kubernetes’ orchestration capabilities to provide automated deployment, scaling, and management of Gerrit instances, facilitating more efficient resource utilization and operational consistency.
Gerrit 4.0 (Target: 2026/2027)
Looking ahead, Gerrit 4.0 is set to introduce significant architectural changes:
Decoupling Gerrit Review UI and JGit Server: This separation will allow the Gerrit UI and JGit server to operate as independent services, providing greater flexibility in deployment and scaling.
Enabling Alternative Review UIs: By decoupling components, the platform will support the integration of other review interfaces, such as pull-request systems, offering users a broader range of tools tailored to their workflows.
The Gerrit community is encouraged to stay engaged with these developments, as the roadmap is subject to change. Contributors planning to work on features not listed are advised to inform the Engineering Steering Committee (ESC) to ensure alignment with the project’s goals.
We’re thrilled to announce that our team will be speaking about our advancements with GerritForge AI Health Service (GHS) at several prestigious conferences in the coming months. These events provide an incredible opportunity to share our innovative AI solutions with a broader audience, engage with industry experts, and showcase how GHS is revolutionizing the way organizations maintain the health and stability of their Gerrit and Git systems.
Our journey begins at the Linux Open Source Summit in Vienna, from the 16th to the 18th of September. This summit is a cornerstone event for the open-source community, and we couldn’t be more excited to discuss how GHS leverages AI to ensure the seamless performance of Git and Gerrit systems, even in the most demanding environments.
Next, we’ll be in Berlin for Git Merge on the 19th and 20th of September. Git Merge is the go-to event for Git enthusiasts and professionals alike, and we’re eager to dive deep into the technical aspects of GHS, sharing insights on how our AI solution optimizes system performance, reduces downtime, and empowers development teams to focus on what they do best—creating great software.
In October, we’re particularly excited about the Gerrit User Summit in San Diego, on the 10th and 11th. This event is especially important to us as it brings together the Gerrit community to discuss the latest developments and best practices. We’ll be showcasing how GHS is enhancing Gerrit environments by providing intelligent and automated health monitoring and ensuring peak performance.
Following that, we’ll speak at the OCX conference in Mainz, from the 22nd to the 24th of October. OCX is known for bringing together top minds in DevOps and open-source technology, making it the perfect venue to highlight how GHS is transforming the management of code review and source control systems with intelligent, automated health monitoring and remediation.
Finally, we’re thrilled to wrap up our conference tour at KubeCon in Salt Lake City, from the 12th to the 15th of November. As one of the most anticipated events in the cloud-native ecosystem, KubeCon offers an unparalleled platform to demonstrate how GHS integrates with Kubernetes environments, ensuring that your SCM systems are always running at peak performance.
These conferences represent more than just speaking engagements for us—they are an opportunity to engage with the community, learn from our peers, and continue pushing the boundaries of what’s possible with AI in software development. We can’t wait to connect with you at these events and share how GHS can make a tangible difference in your organization’s success.
Stay tuned for more updates as we approach these dates, and be sure to catch our sessions if you’re attending any of these events!
Daniele Sassoli GerritForge Engineering Manager Gerrit Code Review Community Manager and Contributor
TL;DR: GerritForge has been dedicating its efforts to organising and managing the Gerrit User Summit in London back in November 2022, in conjunction with the release of Gerrit v3.7. The event has been a great success, with a significant presence on-site and record-breaking attendees on the GerritForge TV youtube channel. It has also committed to its promises to research and improve the JGit and Gerrit scalability to large mono-repos, with tens of millions of objects and refs. 2023 will see the finalisation of these efforts with an increase in development efforts and a new JGit Committer for pushing the platform to a new level of performance and scalability and a new innovating system for collecting and optimising the repository metrics automatically. Stay tuned.
Read the full story here below (9 mins read).
2022 has been a critical year for turning the Gerrit Code Review community and development back on track after the COVID-19 pandemic. At GerritForge, we’ve been working hard to make sure that the development, support, and innovation of Gerrit Code Review continue on its main objectives.
Gerrit Code Review v3.6 and v3.7
We have continued to deliver on the development and release of Gerrit Code Review and its plugins, helping the testing and releasing of versions v3.6.0 (May) and v3.7.0 (November).
3,627 Changes have been merged on 76 projects related to the Gerrit Code Review platform, including JGit
113 committers from 42 different organisations
A special mention to the top #10 contributors: Google (Ben Rohlfs, Edwin Kempin, Chris Pouchet, Dhruv Srivastava, Frank Borden, Milutin Kristofic), GerritForge (Luca Milanesio), Wikimedia (Paladox) and SAP (Matthias Sohn and Thomas Dräbing).
In comparison with 2021, we had 25% fewer changes merged but with more contributors coming from more companies, which is a symptom to a very healthy and thriving ecosystem of maintainers.
GerritForge has committed to resuming the face-to-face user summits, which were suspended since 2020.
It was a glorious success, with record-breaking attendance from all around the globe:
50 people registered to attend on-site, 26 of them managed to arrive despite the London tube strike, whilst the others attended remotely
235 people viewed the summit on YouTube with an average view time of 40 mins (one talk)
The summit survey had an outstanding report showing a huge acceptance and appreciation of the event:
82% rated the remote video streaming as “good” or “outstanding”
96% rated the quality of the summit as “good” or “outstanding.”
100% would recommend the summit to a colleague, with 83% strongly recommending it
GerritHub.io SLA gets closer to five-nines.
We have been working hard to make Gerrit more stable and resilient throughout 2022, discovering and fixing many issues in the code base and on the multi-site software architecture. In 2022, GerritHub.io had only six small hiccups for a total of 19 mins of downtime (SLA = 99.997%) over a 12-month period, a 75% reliability improvement compared to 2021.
We have run extensive RCAs on the causes of the downtime and identified two leading issues, which are explained in the details below.
The “anonymous unlimited query” hole in Gerrit GerritHub.io has been subject to a 15 mins outage because of anonymous users being able to bring offline all the sites before the system could auto-recover. Gerrit allows bypassing of all limits set in the ACLs for running queries by simply adding the “no-limit” parameter. Returning an arbitrary payload without limits could allow a single user to generate a server-side workload for collecting and building a GBytes-sized JSON payload; unfortunately, that option was available to everyone, including anonymous users making any publicly faced Gerrit Code Review installation subject to deny-of-service attacks. We have identified the issue, reported and fixed it in Gerrit with Change 333304, which has been included in Gerrit v3.3.10, v3.4.4, v3.5.1, and all v3.6.0 or later releases.
More granular monitoring and alerting We have lowered the threshold of uptime checks on GerritHub.io to 1 minute, giving us the ability to detect and react immediately to 4 smaller hiccups. We have detected a lack of scalability for some specific higher-load projects. Those hiccups have been responsible for 2 mins of downtime over the 2nd part of 2022. Many more projects are also planning to be onboarded on GerritHub.io; hence we do need to address this project-specific capacity needs.
Scaling Gerrit Code Review and JGit beyond its limits
We have been investing a massive effort in building a test environment designed to stress Gerrit and JGit to its limits and identify all the limitations and bottlenecks that prevented us from scaling further.
Scaling the test repository We have created over the months some test repositories that increased in every dimension:
Tens of millions of refs as both refs/changes and refs/heads
Millions of delta-chains
Tens of millions of Git objects
Packfiles of tens of Giga-bytes and packed refs of hundreds of megabytes
For generating a significant load on both client and server side, we have invested more into the aws-gerrit cloud setups and gatling-git performance loading tool.
There were some “well-known” issues and additional surprising ones.
SHA1 complexity and CPU utilization for large entities JGit has been used SHA1 for identifying uniqueness not just for Git objects but also for other large entities. However, computing SHA1 has become increasingly CPU intensive because of the relatively recent findings about collisions on shattered.io. We have highlighted two major potential improvements in cooperation with Matthias Sohn (SAP) on the raw SHA1 performance and its application for detecting packed-refs changes on the filesystem.
Commit priority queues JGit has a custom implementation of priority queues which are intensively used in RevWalk, which has almost quadratic complexity. That isn’t a problem for small to medium chains of commits; however, when the number of commits reaches millions, the performance degradation becomes unbearable. We have replaced the JGit’s custom implementation with the one provided by the Java JVM library, which has a logarithmic complexity that massively improves its performance with large commit chains.
Unwanted reachability checks JGit needs to perform a full reachability check whenever a remote unknown client is advertising refs, which makes sense when serving a remote client. However, the cost of full reachability of millions of advertised refs can be a daunting task that may be alleviated if the remote end can be considered trusted.
Fixing JGit bitmaps Since the introduction of Git bitmap, the whole community has learned how key they are in speeding up the counting and selection during the clone phase. However, large and unoptimized bitmaps could be so unhelpful for Git that instead of speeding up, they could represent a massive overhead for the system, causing CPU spikes and, eventually, lowering the throughput of the server. Git bitmaps are compressed using the JavaEWAH library, which is good for memory consumption but evil for CPU utilization: that is the reason why the smaller is best for performance. We have discovered and fixed a critical issue with the JGit bitmap generation that was causing the inclusion of all commits and BLOBs pointed by annotated tags. Also, we have introduced the ability to inform JGit about the heads that can be excluded from the bitmap, allowing to shorten the creation tens of thousands times (5h generation time for a 2k refs to as little as 60s) and increase its effectiveness by 200%.
Millions of unneeded ref logs When performing a clone of a repository with millions of heads, JGit created one local reflog file for every remote ref, including the ones there were not actually cloned but just fetched as remote references. This was creating a significant performance gap between JGit and Git, which would instead lazily create the reflog files once they are effectively checked out the first time. Cloning a single branch of a repository with millions of remote refs took around 1h, compared to a few minutes of Git.
All of the findings were included in multiple updates on the following components:
aws-gerrit is going through upgrades for making use of pull-replication plugin, including the support for the bearer token which allows to replicate virtually any repository, including All-Users.git
gatling-git: we have upgraded the Gatling version and JGit to the latest stable-5.13 to include the latest performance improvements.
git-repo-metrics: we have introduced a brand-new plugin that allows us to keep under control the major dimensions of a repository and therefore graph their increase over time.
GerritForge goals for 2023
We are definitely not done yet with the performance improvements on Gerrit and JGit: there are still significant improvements to be made, and JGit changes to get merged into the mainstream branches. We believe we are on track to finalize the job and allow a stable and scalable platform for large Git repositories in 2023.
Finalise what we cooked in 2022 for JGit JGit has a new maintainer, David Ostrovsky, awarded in 2022 as Git committer of the project. GerritForge’s devs are focused to get more reviews and attention to the JGit performance improvements. We are committed to finalising all the open changes related to large repositories.
JGit multi-pack indexes support There is still a major gap between JGit and Git when dealing with very active repositories: multi-pack indexes. The proliferation of packfiles would eventually lead to a long and painful search-for-reuse phase for BLOBs which could be cut down 100s of times with a multi-pack index.
Git repository optimiser for Gerrit We have been working on tracking the live information on the Git repository, thanks to the git-repo-metrics plugin. Wouldn’t it be nice to have a tool that can do something with it and automatically? We would be doing R&D on how to correlate the repository metrics, the Git audit trail, and the performance data for making AI-based decisions on what needs to be improved on the repository. This work stream is going to be useful for any Git repository, not just the ones powered by Gerrit Code Review. The ‘git-repo-metrics’ and the repository optimiser would also apply to other products, including GitHub and GitLab.
Gerrit v3.8 and projects-specific change numbers We will finalise the design document for the transition to project-specific change numbers in Gerrit v3.8. That would allow the seamless migration of projects across Gerrit setups without having to worry about changes renumbering anymore.
Gerrit Code Review testing and GerritForge-certified binaries GerritForge is spending a tremendous amount of time developing test environments and tools for serving the Gerrit community with more stable releases and improving the quality of its code. We want to intensify the effort and also offer our platinum support customers a unique service that includes the GerritForge digital signature and rubber stamp on the binaries of Gerrit Code Review and its plugins that have been successfully tested and validated for being production-ready. Stay tuned; more details are coming soon …
GerritForge company forecast in 2023
GerritForge Inc. will finalise its roll-out to the USA, and all contracts and services will be run from Sunnyvale, CA and Europe. Over 2022, 60% of the customers and businesses have already been moved, and the operation will be completed over the course of 2023.
We are looking forward to doubling our revenue figures in 2023 and also our contributions to the open-source community, with a main focus on JGit as the driver of performance growth for Gerrit Code Review.
Dear fellow Gerrit User, We are pleased to announce that GerritForge will be organizing this year’s Gerrit User Summit and Hackathon in hybrid mode: face-to-face and online.
Gerrit User Summit is the event that brings together Gerrit admins, developers, practitioners, and the whole community, in one place, providing attendees with the opportunity to learn, explore, network face-to-face, and help shape the future of Gerrit development and solutions.
After two years of remote meetings and virtual conferences, this year, we are back face-to-face at CodeNode in the heart of the vibrant City of London.
The dates will be: Nov 7th to 9th – Hackathon Nov 10th to 11th – User Summit
Shortly we will be publishing the full schedule and logistics for the event. I look forward to meeting all the community’s friends, face-to-face or virtually, again during the Hackathon & Summit.
Thanks for being a part of the Gerrit community, and we look forward to seeing you in November.
Luca Milanesio Maintainer, member of the Engineering Steering Committee, and Gerrit Code Review Release Manager
After two years of remote events and three COVID-19 waves, we are finally back for a new face-to-face hackathon, talking about the future of Gerrit Code Review and coding new and innovative solutions for making Gerrit better, faster and more scalable.
Dates and schedule
The Gerrit hackathon will start on the 9th of May at 9:00 AM for five consecutive days, and will have a daily schedule from 9:00 AM to 7:00 PM BST, with networking and catch-up in the evenings.
For the remote attendees on the US/Pacific time-zone, the schedule will be daily between 7:00 AM to 11:00 AM PDT, which allows 4h of remote interaction with the hackathon in London.
Who is invited to attend the hackathon?
As with every Gerrit hackathon, we have a restricted audience: Gerrit maintainers and contributors are invited to join. We have 10 seats available on-site and 15 seats available remotely, which would allow plenty of people to collaborate and discuss.
The “Alphabet” meeting room of the hackathon
To register to the Gerrit hackathon, add your name and role (“Gerrit Contributor” or “Gerrit Maintainer”) to the attendees sheet. All Gerrit maintainers have edit permissions to the document whilst all other contributors can request permission to edit if they are willing to attend.
Where is the hackathon taking place?
GerritForge will host the Gerrit Hackathon at Huckletree West, Mediaworks, 191 Wood Ln, London W12 7FP. We will be staying at the “Alphabet” meeting room, with a dedicated 10-seats and roundtable, a full-size wall-mounted whiteboard and a permanent online connection and wall-attached screen to interact with all the other remote attendees.
Huckletree West
Huckletree is a creative workspace in West London, based in the heart of White City Place, a thriving new business and cultural district. Alongside the neighboring BBC Studios, Net A Porter Group, and RCA School of Communication, Huckletree West is part of a bold new chapter in the rich creative history of the neighborhood.
For all remote attendees, there will be the ability to connect remotely and interact with the rest of the team on-site during the hackathon hours.
White City and local accommodations
Huckletree West is close to the WestField Shopping Centre in White City, which includes 289 stores, 95 restaurants and Cinemas with 20 screens and almost 3,000 seats.
White City has excellent connections to all parts of London through the London Underground network (Central, Hammersmith&City and Circle lines) and Overground trains, which allow to reach all other parts of the city.
WestField shopping centre – White City
You can look for any Hotel or other accommodation (B&B or Hostels) in other part of London which is covered by the London Underground connections. However, if you are willing to stay local, there are many choices of Hotels and B&B starting from ￡80/night. See below a list of accommodations nearby White City:
The UK has left the European Union the 1st of January 2021, all travellers from EU needs to follow the new rules for business trips. You can check if you need a VISA using the UK Government site and what is the required documentation and insurance required to show at the UK Border.
COVID-19 restrictions
The UK is set to end all COVID-19 restrictions by March 2022, which means there aren’t any vaccination or testing requirements for the attendees to the hackathon. We advise everyone attending face-to-face to take extra precautions and take a lateral-flow test (LFT) or antigen test before traveling to the hackathon, even though it is not required by law or regulations.
Please note that face covering are still mandatory whilst travelling by airplane, train or underground and during taxi rides.
We are excited to meet again the community of Gerrit Code Review maintainers and contributors after so many months. Come and join us in London this year and we can innovate again and help shaping the future of the Gerrit project, together.
Luca Milanesio, GerritForge Gerrit Code Review Maintainer Gerrit Code Review Release Manager Member of the Engineering Steering Committee of the Gerrit Code Review Open-Source project
This year’s program will offer a keynote, six presentation sessions and seven ‘lightning’ (10-minute) talks distributed in two days, so that you can share your ideas, research, demonstrations, live demos, and network with the Gerrit Community and those interested in learning and adopting Gerrit Code Review in their development process.
Submit your presentation proposal by creating a change to the Gerrit Summit 2021 repository by following those steps:
Click “CREATE CHANGE” button and specify the branch (master) and the headline of your talk
Click on “EDIT” button on the top-right to edit your change
Click on the “ADD/OPEN/UPLOAD” button and enter the filename for your talk (e.g. sessions/super-duper-repos.md for a talk or lightning-talks/mini-session.md for a lightning talk) upload the text for your talk by dragging the markdown text into the window.
Click the “PUBLISH EDIT” button on the top-right of the change screen
Click on the “MARK AS ACTIVE” button on the top-right of the change screen
Your talk will then be reviewed by the community and, when accepted, merged into the Gerrit User Summit 2021 site.
GerritForge keeps working to improve performance using Gerrit with very large mono-repos with millions of refs, hundreds of GBs and tens of millions of objects.
On the second day of the virtual Gerrit User Summit 2021, Luca will present the work done over the past two years to overcome significant difficulties when:
Reducing the overhead of refs advertisement
Speeding-up clones by a 10x factor
Reducing the system load when accessing change notes
Increasing performance of replication
Surviving the deadly “search-for-reuse” phase during git-upload-pack
Register here to join the user summit on December 2nd and 3rd.
GerritForge Blog 