Gerrit User Summit 2019

gothenburg-sunnyvale

The Gerrit User Summit 2019 is approaching fast, with new exciting features and a brand-new Gerrit v3.0 release to present and discuss together.

The event is FREE but you need to register in advance for the Gerrit User Summit 2019 on Eventbrite.

One Summit, two events

The Gerrit User Summit & Hackathon is composed of two different events and locations, one in Sweden (Europe), hosted and sponsored by Volvo Cars, and another in California (USA) in the new GerritForge Inc. HQ. Having two separate events in two different quarters will allow most of the community around the globe to attend and share their experience and ideas.

Hackathon open to new contributors

The first part of the event is a 5-days Hackathon reserved for the current Gerrit contributors and maintainers plus anyone that is willing to start contributing to the platform. Differently, from the previous years, the community is now welcome even people that have not contributed to Gerrit before but they are willing to do so.

It is a fantastic opportunity for people to join, work side-by-side and pair with the Gerrit maintainers for a whole week. It can be a unique opportunity to implement the features that you always wanted to see in Gerrit and learning how we develop and review our changes.

The Summit

The usual 2-days Users Summit after the Hackathon is opened to all the members of the community or who is willing to adopt Gerrit Code Review in their development process in the near future.

This year there are a number of exciting news:

  • The introduction of an official Gerrit Community Process with an Engineering Steering Committee and Community Managers
  • Gerrit v3.0 and the full migration to NoteDb and PolyGerrit
  • The multi-site plugin goes OpenSource for allowing anyone to run multiple masters on different sites

The full schedule of the event is available on the Gerrit User Summit 2019 site.

Proposing a new talk

More talks and customer stories are scheduled and, if you have something to tell to the rest of the community, you can submit your talk by creating a change and push to Gerrit Summit 2019 repository:

  1. Open the repository commands age at https://gerrit-review.googlesource.com/admin/repos/summit/2019,commands
  2. Click on “CREATE CHANGE”
  3. Select “master” branch, put your description and create the new change
  4. Click on “Edit” on the top-right of the page
  5. Click on “Open” in the mid toolbar and open the page you would like to edit.

For proposing a new session, you need to add one file with the name of your talk into the sessions folder, using the template.md as an example.

Where and when

Volvo Cars HQ at their HQ in Gothenburg (Sweden)

  • 24-28th August 2019 – Gerrit Hackathon Europe
  • 29-30th August 2019 – Gerrit User Summit Europe

GerritForge Inc HQ in Sunnyvale CA (USA)

  • 11-15th November 2019 – Gerrit Hackathon USA
  • 16-17th November 2019 – Gerrit User Summit USA

Thanks to our sponsors

I would like to thank Volvo Cars and Nicholas Mucci for hosting, sponsoring and organizing the Gerrit User Summit Europe in Gothenburg (Sweden) and GerritForge for hosting and sponsoring the events in both Europe and the USA in Sunnyvale CA (USA).

Luca Milanesio
Gerrit Code Review Maintainer and Release Manager
Member of the Engineering Steering Committee

 

 

 

 

GerritHub.io is moving to Gerrit v3.0

It has been a very long journey, from the initial adoption of PolyGerrit at GerritHub to the epic moment where Gerrit historic GWT was dropped with the Gerrit v3.0 last month.

GerritHub.io has always been aligned with the latest and greatest of Gerrit Code Review and thus the moment has come for us to upgrade to v3.0 and drop forever the GWT UI.

PolyGerrit vs. GWT adoption

Screenshot 2019-06-10 at 21.16.48

The PolyGerrit UX was pretty much experimental until the beginning of 2018: the features were incomplete and people needed to go back to the old GWT UI for many of the basic use-cases.

However, things started to change radically in April 2018 when GerritHub.io adopted Gerrit v2.15 which had a 100% functionally complete PolyGerrit UI. The number of users choosing PolyGerrit jumped from 10% to 35% (3.5x times) with a +70% growth in the number of accesses overall. That means that the adoption was mainly driven by users attracted by the new UI.

In the past 12 months, PolyGerrit became the default user-interface and was just renamed as Gerrit UI. Gradually more and more users abandoned the old GWT interface that now represents 30% of the overall accesses.

Timeline of the upgrade

For the 70% of people that are using already using the new Gerrit UI, the upgrade to Gerrit v3.0 would not be noticeable at all:

  • Gerrit v3.0 UI is absolutely identical to the current one in v2.16
  • All existing API and integration points (e.g. Jenkins integration) in Gerrit v3.0 are 100% compatible with v2.16

For the 30% of people that are still using the old GWT UI, things will be very different as their favorite interface will not be available anymore.

The upgrade will happen with zero-downtime across the various GerritHub.io multi-site deployments and will start around mid-June.

Can I still use GWT with GerritHub.io?

The simple answer is NO: Gerrit v3.0 does not contain any GWT code anymore and thus it is impossible for GerritHub.io to bring back the old UI.

The journey to fill the gaps and reach 100% feature and functional equivalence between the old GWT and the new Polymer-based UI took around 6 years, 18k commits and 1M lines of code written by 260+ contributors from 60+ different organizations. It has been tested by hundreds of thousands of developers across the globe and is 100% production-ready and functionally complete.

If you feel that there was “something you could do in the GWT UI and cannot do anymore with the new Polymer-based UI”, please file a bug to the Gerrit Code Review issue tracker and you will get prompt attention and replies from the community.

Can I stay with Gerrit v2.16 on GerritHub.io?

If your organization cannot migrate to Gerrit v3.0, you could still request a dedicated hosting to GerritForge Ltd, which is the company behind GerritHub.io.

Please fill up the GerritForge feedback form and one Sales Representative will come back to you with the possible options and costs associated.

If you fully endorse GerritHub.io with Gerrit v3.0 and start using the new UI, the service will continue to be FREE for public and private repositories, organizations of all types and size. You can optionally purchase Enterprise Support from one of our plans if you require extra help in using and configuring your Gerrit projects with your tools and organization.

Enjoy the future of Gerrit v3.0 with GerritHub.io and GerritForge.

Luca Milanesio, GerritForge Ltd.
Gerrit Code Review Maintainer and Release Manager
Member of the Engineering Steering Committee

Gerrit v3.0 is here

GerritSprintHackathon2019.photo

Gerrit v3.0 has been released during the last Spring Hackathon at Google in Munich involving over 20+ developers for one week.

It can be downloaded from www.gerritcodereview.com/3.0.html and installed on top of any existing Gerrit v2.16/NoteDb installations. Native packages have been distributed through the standard channels and upgrading is as simple as shutting down the service, running the Rpm, Deb or Dnf upgrade command and starting again.

You can also try Gerrit v3.0 using Docker by simply running the following command:

docker run -ti -p 8080:8080 -p 29418:29418 gerritcodereview/gerrit:3.0.0

This article goes through the whole history of the Gerrit v3.0 development and highlights the differences between the previous releases.

Milestone for the Gerrit OpenSource Project

Finally, after 6 years, 18k commits and 1M lines of code written by 260+ contributors from 60+ different organizations, Gerrit v3.0 is finally out.

The event is a fundamental milestone for the project for two reasons:

  • The start of a new journey for Gerrit, without the legacy code of the old GUI based on Google Web Toolkit and without any relational database. Gerrit is now fully based on a Git repository and nothing else.
  • The definition of a clear community organization, with the foundation of a new Engineering Steering Committee and the role of Community Manager.

The new structure will drive the product forward for the years to come and will help to define a clear roadmap to bring back Gerrit at the center of the Software Development Pipeline.

Evolution vs. revolution

When a product release increments the first major number, it typically introduces a series of massive breaking changes and, unfortunately, a period of instability. Gerrit, however, is NOT a typical OpenSource product, because since the beginning it has been based on rigorous Code Review that brought stability and reliability from its initial inception back in 2008. Gerrit v3.0 was developed during the years by following a rigorous backward compatibility rule that has made Gerrit one of the most reliable and scalable Code Review systems on the planet.

For all the existing Gerrit v2.16 installations, the v3.0 will be much more similar to a rather minor upgrade and may not even require any downtime and interruption of the incoming read/write traffic, assuming that you have at least a high-availability setup. How is this possible? Magic? Basically, yes, it’s a “kind of magic” that made this happen, and it is all thanks to the new repository format for storing all the review meta-data: NoteDb.

Last but not least, all the feature that Gerrit v3.0 brings to the table, have been implemented iteratively over the last 6 years and released gradually from v2.13 onwards. Gerrit v3.0 is the “final step” of the implementation that fills the gaps left open in the past v2.16 release.

With regards to statistics of the changes from v2.16 to v3.0, it is clear that the code-base has been basically stabilized and cleaned up, as you can see from the official GerritForge Code Analytics extracted from analytics.gerrithub.io .

  • 1.5k commits from 63 contributors worldwide
  • 62k lines added and 72k lines removed
  • Google, CollabNet, and GerritForge are the top#3 organizations that invested in developing this release

In a nutshell, the Gerrit code-base has shrunk of 10k lines of code, compared to v2.16. So, instead of talking of what’s new in v3.0, we should instead describe what inside the 72k lines removed.

Removal of the GWT UI

The GWT UI, also referred to as “Old UI” has been around since the inception of the project back in 2008.

Gerrit.GWT-UI

Back in 2008, it seemed a good idea to build Gerrit UI on top of GWT, a Web Framework founded by Google two years earlier and aimed at reusing the same Java language for both backend and the Ajax front-end.

However, starting in 2012, things started to change. The interest of the overall community in GWT decreased, as clearly shown by the StackOverflow trends.

Screenshot 2019-05-18 at 23.34.42

In 2015, Andrew Bonventre from the Chromium Project, one of the major users of the Gerrit Code Review platform, apart from the Android Developers, presented the new prototype of the Gerrit Code Review UI, based on the Polymer project, with the code-name of PolyGerrit, and merged as change #72086.

commit ba698359647f565421880b0487d20df086e7f82a
Author: Andrew Bonventre <andybons@google.com>
Date: Wed Nov 4 11:14:54 2015 -0500

Add the skeleton of a new UI based on Polymer, PolyGerrit

This is the beginnings of an experimental new non-GWT web UI developed
using a modern JS web framework, http://www.polymer-project.org/. It
will coexist alongside the GWT UI until it is feature-complete.

The functionality of this change is light years from complete, with
a full laundry list of things that don't work. This change is simply
meant to get the starting work in and continue iteration afterward.

The contents of the polygerrit-ui directory started as the full tree of
https://github.com/andybons/polygerrit at 219f531, plus a few more
local changes since review started. In the future this directory will
be pruned, rearranged, and integrated with the Buck build.

Change-Id: Ifb6f5429e8031ee049225cdafa244ad1c21bf5b5

The PolyGerrit project introduced two major innovations:

  • Gerrit REST-API: for the first time the interaction of the code-review process has been formalized in stable and well-documented REST-API that can be used as “backend contract” for the design of the new GUI
  • The PolyGerrit front-end Team: for the first time, a specific experienced Team focused on user experience and UI workflow was dedicated to rethink and redesign iteratively all the components of the Gerrit Code Review interactions.

The GWT UI and PolyGerrit lived in the same “package” from v2.14 onwards for two years, with the users left with the option to switch between the two. Then in 2018 with v2.16 the PolyGerrit UI became the “default” interface and thus renamed just “Gerrit” UI.

With Gerrit v3.0, the entire GWT code-base in Gerrit has been completely removed with the epic change by David Ostrovsky “Remove GWT UI“, which deleted 33k lines of code in one single commit.

The new Polymer-based UI of Gerrit Code Review is not very different than the one seen in Gerrit v2.16, but includes more bug fixes and is 100% feature complete, including the projects administrations and ACLs configuration.

Screenshot 2019-05-18 at 22.58.13

Removal of ReviewDb

Gerrit v3.0 does not have a DBMS anymore, not even for storing its schema version as it happened in v2.16. This means that almost everything gets stored in the Git repositories.

The journey started back in October 2013, when Shawn Pearce gave to Dave Borowitz the task to convert all the review meta-data managed by Gerrit into a new format inside the Git repository, called NoteDb.

After two years of design and implementation, Dave Borowitz presented NoteDb at the Gerrit User Summit 2015 and called Gerrit v3.0 the release that will be fully working without the need of any other external DBMS (see the full description of the talk at https://storage.googleapis.com/gerrit-talks/summit/2015/NoteDB.pdf).

Google started adopting NoteDb in parallel with ReviewDb on their own internal setup and in June 2017, the old changes table was definitely removed. However, there was more in the todo-list: at the Gerrit User Summit 2017, Dave Borowitz presented the final roadmap to make ReviewDb finally disappear from everyone’s Gerrit server.

Screenshot 2019-05-18 at 23.18.28

In the initial plans, the first version with NoteDb fully working should have been v2.15. However, things went a bit differently and a new minor release was needed in 2018 to make the format really stable and reliable with v2.16.

Gerrit v2.16 is officially the last release that contains both code-bases and allows the migration from ReviewDb to NoteDb.

Dave Borowitz used the hashtag “RemoveReviewDb” to allow anyone to visualize the huge set of commits that removed 35k lines of code complexity from the Gerrit project.

Migrating to Gerrit v3.0, step-by-step

Gerrit v3.0 requires NoteDb as pre-requisite: if you are on v2.16 with NoteDb, the migration to v3.0 is straightforward and can be done with the following simple steps:

  1. Shutdown Gerrit
  2. Upgrade Gerrit war and plugins
  3. Run Gerrit init with the “batch” option
  4. Start Gerrit

If you are running Gerrit in a high-availability configuration, the above process can be executed on the two nodes individually, with a rolling restart and without interrupting the incoming traffic.

If you are running an earlier version of Gerrit and you are still on ReviewDb, then you should upgrade in three steps:

  1. Migrate from your version v2.x (x < v2.16) to v2.16 staying on ReviewDb. Make sure to upgrade through all the intermediate versions. (Example: migrate from v2.13 to v2.14, then from v2.14 to v2.15 and finally from v2.15 to v2.16)
  2. Convert v2.16 from ReviewDb to NoteDb
  3. Migrate v2.16 to v3.0

The leftover of a DBMS stored onto H2 files

Is Gerrit v3.0 completely running without any DBMS at all? Yes and no. There is some leftover that isn’t necessarily associated with the Code Review meta-data and thus did not make sense to be stored in NoteDb.

  • Persistent storage for in-memory caches.
    Some of the Gerrit caches store their status on the filesystem as H2 tables, so that Gerrit can save a lot of CPU time after a restart reusing the previous in-memory cache status.
  • Reviewed flag of changes.
    Represents the flag that enables the “bold” rendering of a change, storing the update status for every user. It is stored by default on the filesystem as H2 table, however, can be alternatively stored on a remote DBMS or potentially managed by a plugin.

New core plugins

Some of the plugins that have been initially distributed only with the Native Packages and Docker versions are now an integral part of the WAR distribution as well:

  • delete-project
    which allows removing a project from Gerrit and the associated changes.
  • gitiles
    a lightweight code-browser created by Dave Borowitz based on JGit
  • plugin-manager
    the interface to discover, download and install Gerrit plugins
  • webhooks
    the HTTP-based remote trigger to schedule remote builds on CI systems or active any other service from a Gerrit event

The above four plugins already existed before Gerrit v3.0, but they were not included in the gerrit.war.

Farewell to Dave Borowitz and the PolyGerrit Team

After having completed the feature parity between GWT and PolyGerrit, the original PolyGerrit Team members left the Gerrit Code Review project.

Their journey came to an end with the release of the new shiny Polymer-based Gerrit UI. The PolyGerrit Team contributed 45k lines of code on 5.3k commits in 4 years.

Then the last event unfolded during the release of Gerrit v3.0: Dave Borowitz announced that he was leaving the Gerrit Code Review project. I defined the event like “Linus Torvalds announcing he was abandoning the Linux Kernel project”.

Dave Borowitz contributed 316k lines of code on 3.6k commits over 36 repositories in 8 years. He helped also the development of the new Gerrit Multi-Site plugin by donating its Zookeeper-based implementation of a global ref-database.

On behalf of GerritForge and the Gerrit Code Review community, I would like to thank all the past contributors and maintainers that made PolyGerrit and NoteDb code-base into Gerrit: Dave, Logan, Kasper, Becky, Viktar, Andrew and Wyatt.

Luca Milanesio – GerritForge
Gerrit Code Review Maintainer, Release Manager
and member of the Engineering Steering Committee

Accelerate with Gerrit DevOps Analytics, in one click!

 

Accelerating your time to market while delivering high-quality products is vital for any company of any size. This fast pacing and always evolving world relies on getting quicker and better in the production pipeline of the products. The whole DevOps and Lean methodologies help to achieve the speed and quality needed by continuously improving the process in a so-called feedback loop. The faster the cycle, the quicker is the ability to achieve the competitive advantage to outperform and beat the competition.

It is fundamental to have a scientific approach and put metrics in place to measure and monitor the progress of the different actors in the whole software lifecycle and delivery pipeline.

Gerrit DevOps Analytics (GDA) to the rescue

We need data to build metrics to design our continuous improvement lifecycle around it. We need to juice information from all the components we use, directly or indirectly, on a daily basis:

  • SCM/VCS (Source and Configuration Management, Version Control System)
    how many commits are going through the pipeline?
  • Code Review
    what’s the lead time for a piece of code to get validated?
    How are people interacting and cooperating around the code?
  • Issue tracker (e.g. Jira)
    how long does it take the end-to-end lifecycle outside the development, from idea to production?

Getting logs from these sources and understanding what they are telling us is fundamental to anticipate delays in deliveries, evaluate the risk of a product release and make changes in the organization to accelerate the teams’ productivity. That is not an easy task.

Gerrit DevOps Analytics (aka GDA) is an OpenSource solution for collecting data, aggregating them based on different dimensions and expose meaningful metrics in a timely fashion.

GDA is part of the Gerrit Code Review ecosystem and has been presented during the last Gerrit User Summit 2018 at Cloudera HQ in Palo Alto. However, GDA is not limited to Gerrit and is aiming at integrating and processing any information coming from other version control and code-review systems, including GitLab, GitHub and BitBucket.

Case study: GDA applied to the Gerrit Code Review project

One of the golden rules of Lean and DevOps is continuous improvement: “eating your dog food” is the perfect way to measure the progress of the solution by using its outcome in our daily life of developing GDA.

As part of the Gerrit project, I have been working with GerritForge to create Open Source tools to develop the GDA dashboards. These are based on events coming from Gerrit and Git, but we also extract data coming from the CI system, the Issue tracker. These tools include the ETL, for the data extraction and the presentation of the data.

As you will see in the examples Gerrit is not just the code review tool itself, but also its plugins ecosystem, hence you might want to include them as well into any collection and processing of analytics data.

Wanna try GDA? You are just one click away.

We made the GDA more accessible to everybody, so more people can play with it and understand its potentials. We create the Gerrit Analytics Wizard plugin so you can have some insights in your data with just one click.

What you can do

With the Gerrit Analytics Wizard you can get started quickly and with only one click you can get:

  • Initial setup with an Analytics playground with some defaults charts
  • Populate the Dashboard with data coming from one or more projects of your choice

The full GDA experience

When using the full GDA experience, you have the full control of your data:

  • Schedule recurring data imports. It is just meant to run a one-off import of the data
  • Create a production ready environment. It is meant to build a playground to explore the potentials of GDA

What components are needed?

To run the Gerrit Analytics Wizard you need:

You can find here more detailed information about the installation.

One click to crunch loads of data

Once you have Gerrit and the GDA Analytics and Wizard plugins installed, chose the top menu item Analytics Wizard > Configure Dashboard.

You land on the Analytics Wizard and can configure the following parameters:

  • Dashboard name (mandatory): name of the dashboard to create
  • Projects prefix (optional): prefix of the projects to import, i.e.: “gerrit” will match all the projects that are starting with the prefix “gerrit”. NOTE: The prefix does not support wildcards or regular expressions.
  • Date time-frame (optional): date and time interval of the data to import. If not specified the whole history will be imported without restrictions of date or time.
  • Username/Password (optional): credentials for Gerrit API, if basic auth is needed to access the project’s data.

Sample dashboard analytics wizard page:

wizard.pngOnce you are done with the configuration, press the “Create Dashboard” button and wait for the Dashboard, tailored to your data, to be created (beware this operation will take a while since it requires to download several Docker images and run an ETL job to collect and aggregate the data).

At the end of the data crunching you will be presented with a Dashboard with some initial Analytics graphs like the one below:

dashboard-e1549490575330.png

You can now navigate among the different charts from different dimensions, through time, projects, people and Teams, uncovering the potentials of your data thanks to GDA!

What has just happened behind the scenes?

When you press the “Create Dashboard” button, loads of magic happens behind the scenes. Several Docker images will be downloaded to run an ElasticSearch and Kibana instance locally, to set up the Dashboard and run the ETL job to import the data. Here a sequence workflow to illustrate the chain of events is happening:

components.png

Conclusion

Getting insights into your data is so important and has never been so simple. GDA is an OpenSource and SaaS (Software as a Service) solution designed, implemented and operated by GerritForge. GDA allows setting up the extraction flows and gives you the “out-of-the-box” solution for accelerating your company’s business right now.

Contact us if you need any help with setting up a Data Analytics pipeline or if you have any feedback about Gerrit DevOps Analytics.

Fabio Ponciroli – Gerrit Code Review Contributor – GerritForge Ltd.

How to migrate Gerrit from v2.15 to v2.16

Time has come to migrate gerrithub.io to the latest Gerrit v2.16, from the outdated v2.15 we had so far. The big change between the two is the full adoption of NoteDB: the internal Gerrit groups were still kept in ReviewDb on v2.15, which forced us to keep a PostgreSQL instance active in production. This means we can finally say goodbye to the ReviewDb 👋 and eliminated yet another SPoF (Single-Point-of-Failure) from the GerritHub high-availability infrastructure.

Migrating to Gerrit v2.16 implies:

  1. Gerrit WAR upgrade
  2. GIT repos upgrade because of a change in the NoteDb format
  3. Change in the database used, from PostgreSQL to H2 (for the schema_version)
  4. Introduction of the new Projects index

The above is a quite complex process and, here at GerritForge, we executed the migration on a running GerritHub.io with 15k of active users avoiding any downtime during the migration.

Architecture

This is the initial architecture we are starting the GerritHub.io v2.15 migration from:

Initial status - 15_01.png

In this setup, we have 2 sites, one in Canada (active) and one in Germany (active for analytics and disaster recovery). The latter is aligned with the active master via replication plugin.

The HA Plugin used between the 2 Canadian nodes is a GerritForge fork enhanced with the ability to align the Lucene Indexes, Caches and Events when sharing repositories via NFS with caching enabled.

NOTE: The original High-Availability plugin is certified and tested on Gerrit v2.14 / ReviewDb only and requires the use of NFS without caching, which requires a direct fiber-channel connection between the Gerrit nodes the disks.

The traffic is routed with HAProxy to the active node. This allows us easy code migrations with no downtimes, using what we call the “ping-pong” technique between the Canadian and the German site, which is inspired by the classical Blue/Green deployment with some adjustments for the peculiarities of the Gerrit multi-site setup.

The migration pattern, in a nutshell, is composed of the following phases:

  1. Upgrade code in Germany
    The Gerrit site in Germany is used for Analytics and thus can be upgraded first with low risk associated.
    German site -> passive, Canadian site -> active
     
  2. Redirect traffic in Germany
    Once the site in Germany is ready and warmed up, the GerritHub users are redirected to it. GerritHub is technically serving the v2.16 user-experience to all users.
    German site -> active, Canadian site -> passive
     
  3. Upgrade code in Canada
    The site in Canada is put offline and upgraded as well.
    German site -> active, Canadian site -> passive
     
  4. Redirect traffic back to Canada
    Once the site in Canada is fully ready and warmed up, the entire user-base is redirected back.
    German site -> passive, Canadian site -> active

Each HAProxy has the same configuration with a primary and 2 backups as follow:

HAProxy CA Primary.png

Timeline of events – 2nd of Jan 2019

2/1/2019 – 8:00 GMT: Starting point of the GerritHub configuration

  • Review-1 – Gerrit 2.15 – active node
  • Review-2 – Gerrit 2.15 –  failover node
  • Review-DE – Gerrit 2.15 – analytics node, used for disaster recovery

2/1/2019 – 10:10 GMT: Upgrade disaster recovery server

  • Stopped all services using Gerrit on review-de (we use the disaster recovery to crunch and serve the analytics dashboard)
  • Disabled replication plugin
  • Stopped Gerrit 2.15 and upgraded to Gerrit 2.16
  • Restarted Gerrit

2/1/2019 – 10:44 GMT: Re-enabled disaster recovery server 

  • Re-Enabled replication from review 1…boom!
    • First issue: mirror option of the replication plugin was set to true, hence all the branches containing the groups on the All-Users repo been dropped from the recovery server. All the Groups were suddenly gone from the disaster recovery server
  • Remove mirror option in replication plugin
  • Re-Enabled replication from review-1…this time everything was ok!
  • Migration re-executed and everything was fine

2/1/2019 – 11:00 GMT: Removed ReviewDB

  • Once we were happy with the replication of the Groups we could remove PostgreSQL

The only information left outside NoteDB is the schema_version table, which contains only one row and it is static. We moved it into H2 by copying the DB from a vanilla 2.16 installation and changing Gerrit Config to use it.

DE 2.16 - 15_01.png

Before the next step, we had to wait for the online reindexing on review-de to finish (~2 hours).

Note: we didn’t consider offline reindexing since it is basically sequential, and it would have been way slower compared to the concurrent online one. Additionally, it does not compute all the Prolog rules in full.

2/1/2019 – 15:15 GMT: Reduce delta between masters

  • Reducing the delta of data between the 2 sites (Canada and Germany) will allow having a shorter read-only window when upgrading the currently active master
  • Manually replicate and reindex misaligned repositories on review-de (see below the effect on the system load)

Screenshot 2019-01-14 at 20.33.10.png

Screenshot 2019-01-14 at 20.33.23.png

  • Pro tip: if you want to check queue status to see, for example, if the replication is still ongoing this command can be used:

    ssh -p 29419 <gerrit_admin_user>@localhost \
                 gerrit show-queue --by-queue --wide

2/1/2019 – 15:50 GMT: Final master catchup

  • Switched on read-only plugin on the active master
  • Service degraded for few minutes (i.e.: Gerrit was read-only), but most of the operations were available, i.e.: Gerrit index/query/plugin/version, git-upload-pack, replication
  • Waited for review-de to catch up with the latest changes that come in review-1 (we monitored it using the above “gerrit show-queue” command)

CA Readonly - 15_01.png

2/1/2019 – 15:54 GMT: Made disaster recovery active

  • Changed HAProxy configuration, and reloaded, to re-direct all the traffic to review-de, which become the active node in the cluster

HAProxy-DE-primary-transition.png

  • See the transition of the traffic to review-de

Screenshot 2019-01-14 at 20.39.22.png

  • Left review-de the whole night as the primary site. This way we also tested the disaster recovery site stability

DE Active - 15_01.png

2/1/2019 – 19:47 GMT: Upgrade review-1 and review-2 to Gerrit 2.16

  • Stopped Gerrit 2.15 and upgraded to Gerrit 2.16
  • Wait for offline reindexing of Projects, Accounts and Groups
  • Started with Gerrit 2.16 with online reindexing of the changesCA 2.16 - 15_01.png

It was possible to see an expected increase in the system load due to the reindexing, lasted for about 2 hours:

System load.png

Furthermore, despite review-1 not being the active node, the HTTP workload grew disproportionately:

HTTP requests.png

This was due to a well-known issue of the high-availability plugin, where the reindexing are forwarded to the passive nodes, creating an excessive workload on them.

3/1/2019 – 10:14 GMT: Made review 1 active

  • We used the same pattern used when upgrading review-de to align the data between masters
  • Changed HAProxy configuration, and reloaded, to re-direct back all the traffic to review-1

 

Final - 15_01.png

Conclusions

Migration was completed and production is back to stable again with the latest and greatest Gerrit v2.16.2 and the full PolyGerrit UI. With the migration of the Groups in NoteDB, ReviewDB leaves the stage completely to NoteDB. PostgreSQL is no more needed, simplifying the overall architecture.

The migration itself was quite smooth, the only issue was due to a plugin misconfiguration, nothing to have with Gerrit core. With the good monitoring we have in place, we managed to spot the issues straight away. Still, we will further automate our release process to avoid these issues from happening again.

Fabio Ponciroli (aka Ponch) – Gerrit Code Review Contributor – GerritForge

New year, free GerritHub: unlimited private reviews with anyone, forever

Today GitHub has announced the extension of its free plan to include unlimited private repositories. This is great because allows a lot more people to start experimenting their side projects and keep them confidential until they ready to be shared publicly.

GerritHub.io allows extending this amazing offer by having a fully-featured code review process on top of their GitHub private repositories and still keep the confidentiality needed for early-stage projects. Differently from GitHub, however, GerritHub allows you to have an unlimited number of reviewers and collaborators, for free, forever.

A wonderful new 2019 is starting with two amazing free offers to allow everyone to experiment and unleash their potential:

  • Free unlimited repos from GitHub, limited to 3 collaborators
  • Free unlimited repos from GerritHub, with unlimited collaborators for reviews

That’s super-cool, how do I start?

Getting started with your private GitHub repositories on GerritHub is easy:

  1. Go to https://review.gerrithub.io
  2. Click the top-right “Sign-in” link
  3. Select “Private” option and click the top-right “Login” button
  4. Enter your GitHub credentials
  5. Allow GerritHub to access in reading/writing your private repositories
  6. Select the GitHub SSH keys and profile into Gerrit, and click the top-right “Next” button
  7. Select the organization and repositories to import into GerritHub, and click the top-right “Import” button
  8. Select the GitHub PRs you want to import into GerritHub for review, and click the top-right “Import Selected” button

Once you’re done with the above steps, you’re up-and-running with GerritHub and you are free to invite collaborators and accept reviews.

You can follow the GerritHub video on YouTube which describes the above process.

I am new to Gerrit Code Review, where do I start?

There is plenty of information on the web about Gerrit Code Review. The best place to start is the project’s tutorial in the documentation.

Alternatively, you can watch the presentation by Shawn Pearce, the Gerrit Code Review project’s founder.

 

Have questions? Get in touch with the Community.

In case of issues, questions, you can get in touch with the Gerrit Code Review Community, and they will be happy to guide you through and provide support.

Want to use Gerrit into your Enterprise?

If you decide to use Gerrit Code Review in your Enterprise and you need the service level compliant with your company standards, you can get in touch with GerritForge which offers the full coverage of the Enterprise Support you will need:

  • Silver: 8×5 Support, with 24h turnaround for P1 issues
  • Gold: 24×7 Support, with 8h turnaround for P1 issues
  • Platinum: 24x7x365 Support, with 4h turnaround for P1 issues

What’s next?

With GitHub and GerritHub you have no excuses anymore to start innovating right now, with free unlimited repositories and free unlimited Gerrit reviewers and contributors.

Go and innovate, the future is now. 

Gerrit User Summit 2018

The Gerrit User Summit 2018 has ended. It has been a truly memorable event and with strong emotions, gratitude and celebration of the success of the Gerrit OpenSource project.

During the hackathon, some major events happened:

  • The release of Gerrit v2.16
  • Support for Kubernetes
  • New plugins contributed by Qualcomm
  • The announcement of a new maintainer, Marco Miller from Ericsson, congrats!

Hackathon and Gerrit v2.16

The hackathon took place at SAP (2 days) and Cloudera (1 day) in Palo Alto with the main focus of completing the migration to PolyGerrit and cut the v2.16 release. Three major milestones of the Gerrit project:

  1. PolyGerrit as default UI for Gerrit Code Review.
  2. Migration of the groups to NoteDb
  3. Support of the Git protocol v2

With regards to the achievement #1, during the hackathon, the commit for removing all the GWT related code from Gerrit master has been finally merged.

PolyGerrit as default Gerrit Code Review UI

The PolyGerrit Team has reached the summit of the long and troubled journey of getting rid of the outdated and Gerrit GWT UI. Kasper and Logan (Google) burned every single bit of their development energy to fix, implement and fill all the remaining gaps in the user journeys.

On Gerrit v2.16, PolyGerrit UI is the default and standard way of interacting with Gerrit Code Review. GWT is still there for allowing people to smoothly adapt to the new interface. However, the next forthcoming version of Gerrit v3.0 will not contain anymore any reference to the old UI. People will still have a few more months to adapt and, from the feedback received on GerritHub.io, they seem to not miss the GWT interface at all.

Well done to the entire PolyGerrit Team, starting from Andrew who kicked off the project back in 2015 to Wyatt, Becky, Victor, Kasper and Logan who brought it up to the usability and portability to multiple devices as it is today. Thanks to their amazing effort, we can all interact with Gerrit Code Review wherever and whenever we want, without losing speed and effectiveness.

Migration of Gerrit groups to NoteDb

One more step towards the complete elimination of ReviewDb has been achieved in Gerrit v2.16: there isn’t anymore any mutable data in the leftover of the (in)famous Gerrit DB. The new version includes a new git repository (All-Groups) which contains all the information related to the groups create through the Gerrit UI.

What’s left in ReviewDb? Not much: only the schema version. When you install Gerrit it is populated with a single row and a single column with the value ‘170’ and will never change. There is no need to share it across master or slaves: just set database.type = h2 and forget about it.

Git protocol v2

It is a major achievement of Gerrit v2.16 as the Git wire protocol v2 solves the performance problems of large repositories: the refs filtering during the advertisement phase.

In a nutshell, if you were fetching a single from a repository with 500k refs, the server would have sent a huge payload of all refs with the associated SHA1s even if at the end of the day you wanted only to fetch a single one. The delay could have been of several seconds if not minutes, which is quite relevant if you are fetching all the times, like in a CI/CD pipeline.

Git protocol v2 reduced the refs advertisement phase by over 70%, which makes Gerrit v2.16 so appealing that you may want to place in the top 5 tasks of your backlog.

Farewell to GWT UI code

Even though Gerrit v2.16 keeps the ability to display the old-fashioned GWT Gerrit UI, the code has been definitely removed on the Gerrit master branch, which will the base for Gerrit v3.0 that will be released in Spring 2019.

This is a memorable moment because represents the final act of making PolyGerrit UI the unified user-experience of Gerrit moving forward.

Data and Insights of the Hackathon

This year the event has been truly focused as well on the analytics side of the Code Review and the entire CI/CD pipeline: was it possibly the location, Cloudera, the world leader in OpenSource BigData and Analytics, giving the right vibrations? Possibly, yes.

Numbers

Over 20 people have attended the Hackathon on-site and, for the first time, remotely via video-conference, from companies and countries all around the world:

  1. Google (USA and Germany)
  2. CollabNet (Japan and Germany)
  3. GerritForge (UK, Ireland, Italy, and Germany)
  4. SAP (Germany)
  5. Qualcomm (USA)
  6. Ericsson (Canada)
  7. Wikimedia Foundation (UK)

Over 300+ changes have been uploaded and 244 of them have been already merged.

They have been working on 44 projects, showing how diverse is today the universe of Gerrit Code Review and its associated plugins and integrations.

Gerrit DevOps Analytics

GerritForge has been hacking on improving the platform that powers the Gerrit Code Review Analytics Platform which has been serving the community for many years.

For the first time, thanks to the introduction of branch-level analytics, the community had data automatically crunched in near-real time of what was happening on the master branch and on the on-going stable 2.16 release.

As a matter of fact, the numbers mentioned before are directly coming from Gerrit DevOps Analytics. Thanks again GerritForge for innovating and improving the Gerrit Code Review platform.

Gerrit Analytics Wizard

For the first time, two new contributors from GerritForge, Tony and Ponch, finalized and demoed a new plugin called ‘analytics-wizard’ that allows anyone with a Gerrit server to set up a mini-version of the Gerrit DevOps Analytics, using Docker, ElasticSearch, and Kibana.

Two new plugins: batch and tasks

Qualcomm has released two new plugins based on their experience of running complex pipelines on a multi-master Gerrit setup. The majority of the code was developed in-house before the hackathon. However, the final release and announced happened this week and I am sure it will allow many other companies to benefit from their experience of validating complex multi-repo changes on a large scale CI system.

Batch

This plugin provides a mechanism for building and previewing sets of proposed updates to multiple projects/refs that should be applied in a batch. These updates are built with Gerrit changes.

Task

The plugin provides a mechanism to manage tasks which need to be performed on changes along with a way to expose and query this information. Tasks are organized hierarchically, and task definitions use Gerrit queries to define which changes each task applies to, and how to define the status criteria for each task. An important use case of the task plugin is to have a common place for CI systems to define which changes they will operate on, and when they will do so.

Gerrit on Kubernetes from SAP and GerritForge

Luke (GerritForge), Matthias (SAP) and Thomas (SAP remotely from Germany) worked on a PoC for supporting Kubernetes deployments.

Last year SAP started using containerized slaves and decided to go cloud native and base our future Gerrit deployments on Kubernetes and leverage project Gardener to support multiple cloud providers. Matthias worked with Thomas to prepare a PoC and a demo the current work in progress and discuss plans moving forward.

The code has been published to the new k8s-gerrit repository and new commits will be subject to the standard Gerrit Code Review process.

Luke prepared an example of Gerrit v2.15 deployment in Kubernetes (single master at the moment) with shared storage and showed how to upgrade easily by leveraging Helm Charts deployments through Tiler.

Even though the two projects started in parallel, they agreed to cooperate and work together with the community to have a unified Kubernetes deployment code-base for installing and upgrading multi-master Gerrit setups. This is definitely an area where we will see very interesting developments very soon.

What about the Summit?

The talks of the summit have been very interesting this year and have covered mostly three main topics: Gerrit roadmap and scalability, sharing of real-life migrations to Gerrit v2.14 and v2.15 and, as previously mentioned, Data Analytics & Insights including research work on using machine-learning for supporting the Code Review process.

All the talks are getting published as we speak on each talk of the Gerrit User Summit Schedule.

All the talks have been recorded and will be published within the next few weeks on the GerritForge TV YouTube channel.

More blog posts will follow with more background on each talk, stay tuned and watch this space to know more about what has been presented and announced at the Summit.

Proposals for the next Gerrit User Summit 2019

The overall feedback on the summit has been very positive, including the Birthday Party organized and sponsored by GerritForge for the 10th anniversary of the Gerrit Code Review project.

During the final closing keynote, many interesting proposals have been made:

  • Gerrit users’ wish-list session.
    Any Gerrit user can feed the backlog of new feature proposing what they will like to see implemented
  • Gerrit plugin hackathon.
    Learn how to implement Gerrit plugin and trying to put together in one or two days something useful using any of the programming languages they wish, including Java, Groovy, Scala or anything else.

More details will come as soon as the closing keynote session recording will be published.

Remembering Shawn Pearce and his legacy

Dave Borowitz, the leader of the Gerrit Code Review project, made a thoughtful and touching speech about Shawn Pearce, the father of the Gerrit 2, who started the project exactly 10 years ago. The cake, the hackathon, the summit, and the v2.16 release have been fully dedicated to his memory as appreciation for his humanity and passion for OpenSource.

Thanks, Shawn. We will continue on your legacy and improve the Gerrit Code Review platform as you would have wanted us to do.

 

 

 

Gerrit User Summit 2018

Gerrit1-high-res.pngWe are pleased to announce that GerritForge, in cooperation with Cloudera and the Gerrit Code Review Community, will be organizing again this year the Gerrit User Summit and Hackathon.

Gerrit User Summit is the event that brings together Gerrit admins, developers, practitioners, the whole community and ecosystem, in one place, providing attendees with the opportunity to learn, explore, network face-to-face and help shape the future of Gerrit development and solutions.

The event will be kindly hosted by Cloudera in their HQ in Palo Alto, in the heart of the Silicon Valley next to the prestigious Stanford University.

Dates

Schedule

Shortly we will be publishing the full agenda and logistics for the event, look forward to that. There are still some available slots for proposing a talk, see the instructions on how to submit your topic.

On behalf of GerritForge and the Gerrit Code Review Community,
Luca Milanesio – GerritForge Ltd

GerritHub.io and the new Reviewer role

Screen Shot 2018-06-21 at 08.57.58

Good news for all the people that wanted to use Gerrit Code Review on top of their GitHub repositories, but so far have been concerned about sharing their profile information: you can now keep your details private and still review other people’s changes on Gerrit.
If you are an EU citizen, you have rights guaranteed by the GDPR and the default role for accessing your GitHub OAuth scope can now be limited to what is the bare minimum.

The default scope explained

When signing up with GerritHub.io, you have the ability to define the scope of access to your personal information held on GitHub:

  • Your e-mail
  • Your membership to organizations and teams
  • Your repositories

The default access requested for doing some activity on Gerrit is: user:email + public:repo + read:org, that allows Gerrit to see your e-mail, clone and push on your behalf to your public repositories and see the list of your teams and organizations. Those permissions are needed when you want to push some code to GerritHub.io, and thus you need to allow Gerrit to resolve your groups (Organizations/Teams) and grant you permissions to push your code to Gerrit and eventually GitHub as well on your behalf.

The problem with GDPR

The default role may not fit with some of the conditions of GDPR for EU where you are not willing to contribute any code but participate in the review of existing changes. That’s why GerritHub.io introduced a new GitHub scope page that allows external reviewers to sign-up with the bare minimum information that Gerrit needs to let you in: your e-mail address.

Why can’t I stay anonymous on Gerrit?

Even though you could create a GitHub account without exposing any private information or exposing your e-mail, you cannot sign-up with Gerrit Code Review if you are not willing to share who you are.
The e-mail address in Gerrit is a crucial property, and without it, some parts of the code-review workflow would stop working. That condition of Gerrit is so pervasive in the architecture because of the requirements of the Android OpenSource Project (code-named AOSP) which is the project historically hosted on the platform: the . See the details of the Android contribution license and process at https://source.android.com/setup/start/licenses.

What if I still want to stay completely anonymous on Gerrit Code Review?

Gerrit has the concept of the “Anonymous Coward” (not an offense, just the default name assigned to it), which is an account that has only an ID without either a full name or an e-mail address. With the anonymous user, however, you cannot review code, for obvious legal reasons and for preventing spamming and abuse.

I am in doubt on what to chose, where to start from?

If you are concerned about sharing your Personal Information and you need to decide which level to use, then just chose “Reviewer” and you can amend your choice and extend your scope at any time later.

GerritHub.io and GDPR

Screen Shot 2018-05-25 at 16.17.49

GerritHub.io has changed some of its key components to be compliant with the new regulations related to the European General Data Protection Rules (aka GDPR). You can find the updated Privacy Policy on the GerritForge Web Site.

Impact on Gerrit Code Review

Gerrit Code Review had a fundamental problem with some of the concepts in the GDPR: accounts could not be removed from the system once created the first time. Yes, you could have disabled them, but their associated personal data would have remained in the Gerrit DB or All-Users git repository.

We have developed a brand-new plugin for Gerrit that enable the following features:

  1. General permission to allow the removal of accounts
    Group of users can be deleted to remove their own accounts data. Gerrit administrators can be delegated to remove other accounts.
  2. New SSH command for account removal
    Users that have been granted permissions, can access a new ‘account’ command to remove their own personal information.
  3. Simple and easy self-service online form to review its Personal Information and self-remove its account.

The new plugin has been shared with the wider Gerrit Code Review community and will be soon public domain for every other publicly available website that wants to host Gerrit and being publicly accessible by EU citizens.

Access and control of your Personal Information

  • Right to Access
    Every personal profile is directly accessible to its owner through the User’s settings in Gerrit Code Review.
  • Data Portability
    Gerrit stores the user’s profile in a specific set of branches of the All-Users repository. GerritHub.io can make the personal branches available to everyone that would like to have direct access to its data and reuse or import them into its system.
  • Privacy by Design
    Gerrit already stores only hashed passwords. Additionally, the GitHub credentials are never passed to Gerrit Code Review and the authentication and profile access delegation process is completely controlled by the user thanks to the OAuth 2.0 and Scope selection process.
  • User Data and “right to be forgotten”
    Thanks to the new Gerrit ‘account’ plugin, every user on GerritHub.io is always in control of its data and can decide to remove its account permanently at any time.
    Note that Gerrit will have to “remember” the account UUID for consistency of all the previous review work done by the user. Every account removed from the platform will become an “Anonymous UUID” and will not be able to be reused anymore in the future.

Questions?

If you still have doubts or questions about Gerrit Code Review and the EU GDPR regulations, you can get in touch with GerritForge Ltd.