Category Archives: aws

2023: New Year and opportunities for GerritForge and Gerrit Code Review

Posted on by
Reply

TL;DR: GerritForge has been dedicating its efforts to organising and managing the Gerrit User Summit in London back in November 2022, in conjunction with the release of Gerrit v3.7. The event has been a great success, with a significant presence on-site and record-breaking attendees on the GerritForge TV youtube channel. It has also committed to its promises to research and improve the JGit and Gerrit scalability to large mono-repos, with tens of millions of objects and refs. 2023 will see the finalisation of these efforts with an increase in development efforts and a new JGit Committer for pushing the platform to a new level of performance and scalability and a new innovating system for collecting and optimising the repository metrics automatically. Stay tuned.

Read the full story here below (9 mins read).

2022 has been a critical year for turning the Gerrit Code Review community and development back on track after the COVID-19 pandemic. At GerritForge, we’ve been working hard to make sure that the development, support, and innovation of Gerrit Code Review continue on its main objectives.

Gerrit Code Review v3.6 and v3.7

We have continued to deliver on the development and release of Gerrit Code Review and its plugins, helping the testing and releasing of versions v3.6.0 (May) and v3.7.0 (November).

Some numbers of the past 12 months’ development contributions by individual committers and companies:

  • 3,627 Changes have been merged on 76 projects related to the Gerrit Code Review platform, including JGit
  • 113 committers from 42 different organisations

A special mention to the top #10 contributors: Google (Ben Rohlfs, Edwin Kempin, Chris Pouchet, Dhruv Srivastava, Frank Borden, Milutin Kristofic), GerritForge (Luca Milanesio), Wikimedia (Paladox) and SAP (Matthias Sohn and Thomas Dräbing).

In comparison with 2021, we had 25% fewer changes merged but with more contributors coming from more companies, which is a symptom to a very healthy and thriving ecosystem of maintainers.

GerritForge has committed to resuming the face-to-face user summits, which were suspended since 2020.

The Gerrit User Summit 2022 took place in London, UK the 10-11 of November in a hybrid format, with people having the opportunity to participate either on-site or remotely on GerritForge’s YouTube TV channel.

It was a glorious success, with record-breaking attendance from all around the globe:

  • 50 people registered to attend on-site, 26 of them managed to arrive despite the London tube strike, whilst the others attended remotely
  • 235 people viewed the summit on YouTube with an average view time of 40 mins (one talk)

The summit survey had an outstanding report showing a huge acceptance and appreciation of the event:

  • 82% rated the remote video streaming as “good” or “outstanding”
  • 96% rated the quality of the summit as “good” or “outstanding.”
  • 100% would recommend the summit to a colleague, with 83% strongly recommending it

GerritHub.io SLA gets closer to five-nines.

We have been working hard to make Gerrit more stable and resilient throughout 2022, discovering and fixing many issues in the code base and on the multi-site software architecture.
In 2022, GerritHub.io had only six small hiccups for a total of 19 mins of downtime (SLA = 99.997%) over a 12-month period, a 75% reliability improvement compared to 2021.

We have run extensive RCAs on the causes of the downtime and identified two leading issues, which are explained in the details below.

The “anonymous unlimited query” hole in Gerrit
GerritHub.io has been subject to a 15 mins outage because of anonymous users being able to bring offline all the sites before the system could auto-recover.
Gerrit allows bypassing of all limits set in the ACLs for running queries by simply adding the “no-limit” parameter.
Returning an arbitrary payload without limits could allow a single user to generate a server-side workload for collecting and building a GBytes-sized JSON payload; unfortunately, that option was available to everyone, including anonymous users making any publicly faced Gerrit Code Review installation subject to deny-of-service attacks.
We have identified the issue, reported and fixed it in Gerrit with Change 333304, which has been included in Gerrit v3.3.10, v3.4.4, v3.5.1, and all v3.6.0 or later releases.

More granular monitoring and alerting
We have lowered the threshold of uptime checks on GerritHub.io to 1 minute, giving us the ability to detect and react immediately to 4 smaller hiccups. We have detected a lack of scalability for some specific higher-load projects. Those hiccups have been responsible for 2 mins of downtime over the 2nd part of 2022. Many more projects are also planning to be onboarded on GerritHub.io; hence we do need to address this project-specific capacity needs.

Scaling Gerrit Code Review and JGit beyond its limits

We have been investing a massive effort in building a test environment designed to stress Gerrit and JGit to its limits and identify all the limitations and bottlenecks that prevented us from scaling further.

Scaling the test repository
We have created over the months some test repositories that increased in every dimension:

  • Tens of millions of refs as both refs/changes and refs/heads
  • Millions of delta-chains
  • Tens of millions of Git objects
  • Packfiles of tens of Giga-bytes and packed refs of hundreds of megabytes

For generating a significant load on both client and server side, we have invested more into the aws-gerrit cloud setups and gatling-git performance loading tool.

There were some “well-known” issues and additional surprising ones.

SHA1 complexity and CPU utilization for large entities
JGit has been used SHA1 for identifying uniqueness not just for Git objects but also for other large entities. However, computing SHA1 has become increasingly CPU intensive because of the relatively recent findings about collisions on shattered.io.
We have highlighted two major potential improvements in cooperation with Matthias Sohn (SAP) on the raw SHA1 performance and its application for detecting packed-refs changes on the filesystem.

Commit priority queues
JGit has a custom implementation of priority queues which are intensively used in RevWalk, which has almost quadratic complexity. That isn’t a problem for small to medium chains of commits; however, when the number of commits reaches millions, the performance degradation becomes unbearable.
We have replaced the JGit’s custom implementation with the one provided by the Java JVM library, which has a logarithmic complexity that massively improves its performance with large commit chains.

Unwanted reachability checks
JGit needs to perform a full reachability check whenever a remote unknown client is advertising refs, which makes sense when serving a remote client. However, the cost of full reachability of millions of advertised refs can be a daunting task that may be alleviated if the remote end can be considered trusted.

Fixing JGit bitmaps
Since the introduction of Git bitmap, the whole community has learned how key they are in speeding up the counting and selection during the clone phase.
However, large and unoptimized bitmaps could be so unhelpful for Git that instead of speeding up, they could represent a massive overhead for the system, causing CPU spikes and, eventually, lowering the throughput of the server.
Git bitmaps are compressed using the JavaEWAH library, which is good for memory consumption but evil for CPU utilization: that is the reason why the smaller is best for performance.
We have discovered and fixed a critical issue with the JGit bitmap generation that was causing the inclusion of all commits and BLOBs pointed by annotated tags. Also, we have introduced the ability to inform JGit about the heads that can be excluded from the bitmap, allowing to shorten the creation tens of thousands times (5h generation time for a 2k refs to as little as 60s) and increase its effectiveness by 200%.

Millions of unneeded ref logs
When performing a clone of a repository with millions of heads, JGit created one local reflog file for every remote ref, including the ones there were not actually cloned but just fetched as remote references. This was creating a significant performance gap between JGit and Git, which would instead lazily create the reflog files once they are effectively checked out the first time. Cloning a single branch of a repository with millions of remote refs took around 1h, compared to a few minutes of Git.

All of the findings were included in multiple updates on the following components:

  • JGit changes: all fixes were also provided to stable-5.13, the last supported branch for Java 8, which allows benefiting from these improvements for older versions of Gerrit from v2.16 onwards.
  • pull-replication went through major performance improvements, achieving a 1000x times faster execution time compared to the traditional replication plugin
  • aws-gerrit is going through upgrades for making use of pull-replication plugin, including the support for the bearer token which allows to replicate virtually any repository, including All-Users.git
  • gatling-git: we have upgraded the Gatling version and JGit to the latest stable-5.13 to include the latest performance improvements.
  • git-repo-metrics: we have introduced a brand-new plugin that allows us to keep under control the major dimensions of a repository and therefore graph their increase over time.

GerritForge goals for 2023

We are definitely not done yet with the performance improvements on Gerrit and JGit: there are still significant improvements to be made, and JGit changes to get merged into the mainstream branches.
We believe we are on track to finalize the job and allow a stable and scalable platform for large Git repositories in 2023.

Finalise what we cooked in 2022 for JGit
JGit has a new maintainer, David Ostrovsky, awarded in 2022 as Git committer of the project. GerritForge’s devs are focused to get more reviews and attention to the JGit performance improvements. We are committed to finalising all the open changes related to large repositories.

JGit multi-pack indexes support
There is still a major gap between JGit and Git when dealing with very active repositories: multi-pack indexes. The proliferation of packfiles would eventually lead to a long and painful search-for-reuse phase for BLOBs which could be cut down 100s of times with a multi-pack index.

Git repository optimiser for Gerrit
We have been working on tracking the live information on the Git repository, thanks to the git-repo-metrics plugin. Wouldn’t it be nice to have a tool that can do something with it and automatically?
We would be doing R&D on how to correlate the repository metrics, the Git audit trail, and the performance data for making AI-based decisions on what needs to be improved on the repository.
This work stream is going to be useful for any Git repository, not just the ones powered by Gerrit Code Review. The ‘git-repo-metrics’ and the repository optimiser would also apply to other products, including GitHub and GitLab.

Gerrit v3.8 and projects-specific change numbers
We will finalise the design document for the transition to project-specific change numbers in Gerrit v3.8. That would allow the seamless migration of projects across Gerrit setups without having to worry about changes renumbering anymore.

Gerrit Code Review testing and GerritForge-certified binaries
GerritForge is spending a tremendous amount of time developing test environments and tools for serving the Gerrit community with more stable releases and improving the quality of its code. We want to intensify the effort and also offer our platinum support customers a unique service that includes the GerritForge digital signature and rubber stamp on the binaries of Gerrit Code Review and its plugins that have been successfully tested and validated for being production-ready.
Stay tuned; more details are coming soon …

GerritForge company forecast in 2023

GerritForge Inc. will finalise its roll-out to the USA, and all contracts and services will be run from Sunnyvale, CA and Europe. Over 2022, 60% of the customers and businesses have already been moved, and the operation will be completed over the course of 2023.

We are looking forward to doubling our revenue figures in 2023 and also our contributions to the open-source community, with a main focus on JGit as the driver of performance growth for Gerrit Code Review.

2023 is going to be an incredible year for GerritForge, Gerrit Code Review, and the JGit community altogether.

Happy New start of the Year 2023!

Luca Milanesio (GerritForge)
Gerrit Code Review Maintainer and Release Manager
Member of the Gerrit Engineering Steering Committee

ARM-64 welcomes Gerrit Code Review

Posted on by
Reply

It has been seven years since we first introduced the native packages for Gerrit Code Review. At that time, Docker was just two years old and was starting to pick up momentum: the native packages found their first incarnation in the Gerrit Code Review base on images published on DockerHub.

Since then, Gerrit has always been released as both a “plain vanilla” war file and the equivalent “native packaging” flavors, including the base Docker images.

In the beginning, the release managers mainly used the Gerrit Docker images to simulate pre-configured test environments for testing purposes. However, the adoption spiked up when Kubernetes and AWS Container Services (aka ECS) started to become mainstream.

Nowadays, the Gerrit base images on Dockerhub are pulled millions of times and used as the baseline for creating Enterprise-grade setups, like in the aws-gerrit recipes.

x86 Native Packaging

Even though it wasn’t initially thought off, the native packages ended up mainly in x86-based containers, even though Gerrit is a 100% pure-Java application. How was that possible?

  • Dependencies: gerrit.sh requires some basic OS services to work properly, including the native git command line.
  • Build & testing: the build of gerrit.war and associated testing was always performed on x86, also due to the restrictions of the platform supported by Bazel at that time.

ARM-64 takes off

The ARM processor comes from the historical legacy of Acorn Computers Ltd, a British-born company of the 80s, well known for the BBC Micro project. Back in the 90s, a new joint venture was created with Apple Computer and VLSI Technology to design a brand-new Advanced Risc Machine (aka ARM).

Over time, ARM has created first a 32-bit architecture and subsequently a 64-bit architecture processor, well known and appreciated for its low-power consumption and performance efficiency.

Whilst Intel, with its x86 architecture, dominated the desktop and the server market, the ARM architecture found its natural market space in the low-power devices and, more recently, the mobile phones market.

Apple goes ARM-64, everywhere

Apple initially adopted the ARM architecture on its mobile phones and portable devices, licensing the instruction set to build its microchips in 2010. Ten years later, Apple announced it was also transitioning all the other desktop devices to its ARM-based silicon.

Nowadays, nearly all Apple laptops are on ARM and the other devices are migrating to that too. The applications are rapidly adapting to the new instruction set and getting rebuilt quickly, gaining in both performance and increased battery life.

Cloud services are moving to ARM-64 too.

The move from x86 to ARM-64 brings noticeable advantages in terms of costs and energy efficiency has been recently amplified by the increase in the costs of electricity triggered by the spiked wholesale prices of oil, gas, and other fossil fuel.

GCloud (see T2A offering ) and AWS (see EC2 Graviton) offer now ARM-64 based VMs that promise a healthy 60% reduction of cloud hosting costs and higher throughput and performance per $.

Gerrit has always put high performance and low operating costs at the top of its agenda, allowing companies to keep their development teams highly productive with minimum dedicated staff looking after the system.

Now it is also time to reduce the Gerrit Code Review Cloud hosting costs and increase efficiency while preserving all the benefits that Gerrit brings.

Introducing ARM-64 native packages for Gerrit Code Review

Gerrit Code Review native packages are now available for ARM-64 on all its recent releases, from v3.4.8 onwards. If you are currently running on x86, you can easily switch to ARM-64 transparently by just rebuilding from the base image. All the DockerHub images have been repackaged and republished:

Gerrit Native packages on bare-metal

If you are running Gerrit native packages on bare-metal, you would need to setup the GerritForge’s updated RPM repository:

rpm -i https://gerritforge.com/gerritforge-repo-1-4.noarch.rpm

On Debian, just refresh the repository cache with ‘apt-get update‘ and the new packages will automatically show-up and be available for setup.

Because this is a repackaging of the existing Gerrit release, the 4th number of the package name has been updated, e.g., gerrit-3.4.8-2 is the package name associated with the ARM-64 compatible package.

Ready to switch to ARM-64?

The Gerrit maintainers have already switched to ARM-64 with the recent wave of MacBook upgrades: all the non-EOL Gerrit releases are typically developed on Apple M1 or M2 which are based on ARM-64 rather than x86.

The need to reduce costs and the overall carbon footprint of the Gerrit project would most likely drive the CI agents’ switch to ARM-64 very soon.

Are you ready to switch to ARM-64 to reduce your operating costs and help the planet by reducing your Gerrit consumption?

Q&A

Q1. Are Gerrit on x86 and ARM-64 compatible in terms of data?

Yes, because the underlying data structure is platform-independent and is all based on Git, which is already interoperable cross-platform.

Q2. Can I mix&match x86 and ARM-64 Gerrit nodes?

Yes, by adopting the Gerrit multi-site setup, you can configure and host different sites on different architectures.

Q3. Will Gerrit on x86 continue to be supported?

Yes, because we want to allow the Gerrit administrator to choose the most suitable platform for his users and compatible with his Company’s requirements.

Luca Milanesio – GerritForge
Gerrit Code Review Maintainer
ESC Member
Gerrit Code Review Release Manager

2022 GOALS for Gerrit

Posted on by
Reply

The year 2021 has been a challenging one because of the COVID-19 global emergency; nevertheless, the Gerrit Code Review project has continued to deliver what the community expected:

GerritForge delivered on the promise of making Gerrit more cloud-native, with a particular focus on AWS, the platform that most users have adopted for running in the cloud. GoogleCloud has also been our focus, assuring a cloud-neutral approach to Gerrit and providing support for events over PubSub.

Focusing on Gerrit unique values

A successful product focuses on what makes it unique and innovative, compared to anything else in the world.

We believe that the key aspects that make Gerrit Code Review THE platform of choice for developing software based on Git repositories are:

  • Large-scale
    Gerrit is THE best platform for developing large-scale projects, huge monorepos, and a large number of changes and refs.
  • Maximum availability
    Large organizations and communities of developers need a platform that is always available, anywhere, anytime, 24×7, and 365 days a year.
  • Performance
    The need to work remotely poses multiple issues, one of them being the increase of network latency. Gerrit multi-site distribution of the repositories and reviews allows anyone, anywhere in the world, to clone, push and review at optimal latency and performance.
  • Quality of tracing of reviews
    Gerrit is based on single-commit code reviews, a winning approach in terms of review accuracy and supporting changes chains, and full traceability of the entire review history and workflow.

Many popular Git code-review tools exist in the Open-Source community; Gerrit is the winning choice when scale, availability, performance, and quality do matter.

GerritForge goals for improving Gerrit in 2022

Scale Gerrit beyond limits

GerritForge and the rest of the community have worked hard to identify the bottlenecks of large mono-repos with Gerrit. Some of them can be mitigated by keeping the Git repository lean and organized, despite the massive amount of push traffic and reviews coming from large teams.

We want to focus on improving at least ten times the following KPIs, without having a significant impact on the overall system performance:

  • Number of changes and refs in a repository: millions of changes and tens of millions of refs
  • Size of the repository: hundreds of GBs

GerritForge will step up its involvement in the JGit project in 2022 and introduce many innovations, some of them already implemented in the C-Git implementation:

  • JGit support for multi-pack index
  • Revamp of JGit cache, allowing the pluggability of high-performance implementations
  • Improvement of JGit bitmaps for large number of refs
  • Support for high-performance large storage systems
  • Introduction of new performance metrics
  • Replace Prolog with native submit rules in the owners plugin

99.999% up-time

GerritForge maintains a free service known as GerritHub.io to demonstrate what Gerrit can do and achieve. GerritHub.io is the most advanced and reliable Open-Source vanilla Gerrit deployment, apart from Google’s.

GerritHub.io uptime in 2021 – checked and reported by PIngdom.com

We achieved an astonishing 99.99% SLA in 2021; we want to push the GerritHub.io uptime further to 99.999%, reducing the annual downtime to just 315s.

In order to reach a five-nines uptime, we will work on:

  • Granular probing and health-checks
  • Advanced repositories performance monitoring and alerting
  • Gerrit limits and deadlines
  • RCAs
  • Multi-site improvements

Goal #3: Increase 1000x times the Gerrit replication performance

GerritForge has presented the innovating pull-replication plugin at the Gerrit Virtual User Summit 2021, showing that it is possible to replicate Git commits and changes meta-data across the globe with msec latency. The pull-replication plugin technology and speed is going to be improved and made available and Open-Sourced to anyone and match and outperform the traditional replication plugin features.

Join the 2022 endeavor


We need YOU and the Gerrit community’s help and support in this 2022 endeavor.

GerritForge has already increased his Team of contributors working on the project, including three Gerrit maintainers and two Gerrit release managers. However, Gerrit’s success is in the cooperation, contribution, and ideas of the whole community of contributors, Gerrit admins, and users.

Let us know what you think about our goals. We are happy to cooperate and work with anyone sharing the same values and goals.

2022 is the year where Gerrit Code Review is pushed beyond its limits even further, making it the MOST innovative tool for large-scale repositories and teams worldwide.

2021: What’s cooking in GerritForge

Posted on by
Reply

Our mission is to improve the Gerrit Code Review platform, with the Open-Source community and in full transparency. Everything we do is published, is open, can be analysed and scrutinised by any member of the community at any time.

In this article, we are publishing what GerritForge is planning to do in the forthcoming 2021, a pivotal year that will see the entire world bouncing back and looking at the future of Technology with a different light.

Goal: cloud-native Gerrit Code Review

Since our initial post in 2019 about our plans for the future, we have been making steady progress toward a unified global Gerrit platform, highly available and distributed across the globe. Since then, a lot of steps have been made, many of them happened in 2020.

Thanks to SAP and GerritForge, we have now two Open-Source standard deployments on the Cloud:

  1. K8s-Gerrit – Helm-charts for deploying Gerrit Code Review to a Kubernetes Cluster (beta)
  2. AWS-Gerrit – CloudFormation templates for creating a production-like setup of Gerrit HA and Multi-Site to ECS on the AWS platform.

Even if Gerrit can deployed now into a Cloud environment, that does not mean that Gerrit is cloud-native. The main requirements for making an application cloud-native is captured by the 12-factor principles. We will be tackling some of them in our 2021 roadmap.

Cloud-native backing services

Thanks to the advances in the Gerrit Multi-Site plugin in 2020, it is today possible to have a geographically distributed cluster of Gerrit servers around the globe. The services that are needed for coordinating the events across Gerrit nodes and ensuring global consistency are:

  1. Event-broker (Kafka)
  2. Global ref-db (Zookeeper)

The price to pay for a global Gerrit Multi-Site is currently quite high, because one may argue that actually managing a cluster of Kafka brokers or Zookeeper nodes can be even more challenging that managing Gerrit itself.

A cloud-native application should be fully independent from its backing services and be opened to swapping one specific implementation (e.g. Kafka) with a serviced off-the-shelf solution (e.g. AWS Kinesis, GCP PubSub).

As GerritForge we are joining the forces with the Gerrit Code Review community in 2021 for refactoring the Gerrit Event system and making it more Cloud-native, using open-standards for representing and publishing events across the network.

The immediate actions are:

  1. Build a new protobuf-based events representation in Gerrit v3.4
  2. Abstract the serialisation of events from their transport
  3. Implement different events brokers and listeners, other than the Kafka. (e.g. NATS)
  4. Provide a AWS Kinesis-stream event broker connector
  5. Provide a GCP PubSub broker connector in cooperation with Google

Gerrit performance across sites

Making a stateful application like Gerrit Code Review a truly cloud-native and distributed platform brings some challenges in terms of latency and performance.

The current Gerrit Multi-Site architecture still relies heavily on the traditional (push) replication plugin, which has been working nicely for over a decade for a Gerrit primary/replica setup. When the repository size increases and the need for global consistency becomes more stringent, relying on a traditional async git push is not enough anymore: you need a lot more tools at your service.

GerritForge has introduced in 2020 the pull-replication plugin, which enables to invert the replication logic: instead of having one Gerrit server to push everything to everyone else, you just tell the other nodes of what’s new in the repository, and it will be their responsibility to fetch only the stuff that has changed.

In 2021 we are going to improve the plugin with native JGit client protocol v2 support and features that will go well beyond the limits of the git protocol:

  1. BLOB and ref-update on remote git pulls. The remote Gerrit node can receive directly the content of what has changed, directly in its payload. That allows to avoid the expensive git pull operation especially for NoteDb related updates.
  2. Event broker integration for git pulls. Instead of relying on REST-API, the pull replication plugin can produce pull replication events in the event broker.

The implementation of those improvements will allow to drastically reduce the incidence of the traditional (push) replication activity and make the Gerrit user experience more seamless across sites.

CI integration

Integrate Gerrit in more seamlessly with other cloud-native services, such as CI/CD. Gerrit v3.4 is planning to introduce a brand-new style of integration which would not require anymore any 3rd party adaptation or API support.

GerritForge is committed to endorse this new module and allow Gerrit v3.4 to be fully connected with Jenkins on day-1 and potentially other cloud-based CI/CD systems.

Git reftable: provide more performance for mono-repos

GerritForge will experiment and test at scale the introduction of Google’s Git reftable, already included from Gerrit v3.2 onwards, to assess its performance impact on large mono-repos, with hundreds of thousands of refs.

Have your say on Gerrit future

This is our GerritForge shopping list for 2021, wide open for discussion, contributions, amendments, and voting.

Why should you care about what we do? Simply because the union of intents and cross-pollination of ideas make the Open-Source products and community stronger: if we join forces in building a solution that is a common interest for all of us, we will all benefit from it.

Your feedback is welcome and precious, feel free to have your say and comment this post with your ideas, likes and suggestions on how to make them even better.

The GerritForge Team.