Gerrit User Summit: What’s new in 2.15

This week we are honored to publish the amazing talk of Gerrit Ver. 2.15 presented by Dave Borowitz, the “father” of NoteDb review format. It is the very first time that a full roadmap of the migration from the traditional ReviewDb (a relational DBMS) to NoteDb (a pure Git notes-based review store) is drawn and explained in all details.

First steps with Gerrit Ver. 2.15 using Docker

For all of those who want to experiment what Dave has presented at the Gerrit User Summit, there is a Gerrit Ver. 2.15 RC2 Docker image already published on DockerHub on https://hub.docker.com/r/gerritcodereview/gerrit.

See below the simple instruction on how to start a Gerrit 2.15 locally and playing with it.

Step 1 – Run Gerrit Docker image

$ docker run -h localhost -ti -p 8080:8080 -p 29418:29418 gerritcodereview/gerrit:2.15.rc2
...
[2017-11-14 23:12:08,016] [main] INFO  com.google.gerrit.pgm.Daemon : Gerrit Code Review 2.15-rc2 ready

Step 2 – Open Gerrit UX

Open your web-browser at http://localhost:8080 and you will automatically get into the plugin manager selection. At the moment only the core plugins are available, so you can just click on to top right “Go To Gerrit” link.

Screen Shot 2017-11-14 at 23.19.38.png

As soon as Gerrit 2.15 will be officially available, you would be able to discover and install many more plugins on top of your installation.

Step 3 – Create a new Repository

Click on the new top “BROWSE” menu and select “CREATE NEW

Screen Shot 2017-11-14 at 23.20.48.png

Then insert the repository name (e.g. “gerrit-playground”), select “Create initial empty commit” to True and click on “CREATE“.

Screen Shot 2017-11-14 at 23.22.25

Step 4 – Clone the repository

From the repository page select the “HTTP” protocol and click on the “COPY” link next of the “Clone with commit-msg hook” section.

Screen Shot 2017-11-14 at 23.24.56.png

And paste the command on your terminal.

$ git clone http://admin@localhost:8080/a/gerrit-playground && (cd gerrit-playground && curl -Lo `git rev-parse --git-dir`/hooks/commit-msg http://admin@localhost:8080/tools/hooks/commit-msg; chmod +x `git rev-parse --git-dir`/hooks/commit-msg)
Cloning into 'gerrit-playground'...
remote: Counting objects: 2, done
remote: Finding sources: 100% (2/2)
remote: Total 2 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (2/2), done.
Checking connectivity... done.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4695  100  4695    0     0   448k      0 --:--:-- --:--:-- --:--:--  458k

Step 5 – Create a new commit for review

Enter into the new Git cloned repository “gerrit-playground”, add a README.md file.

$ cd gerrit-playground && echo "Hello Gerrit" > README.md && git add .

Then identify yourself as admin@example.com, which is the default Admin e-mail for the Gerrit 2.15 Docker image.

$ git config user.email admin@example.com

And finally create a new commit and push it to Gerrit using “secret” as password

$ git commit -m "Say hello to Gerrit" && git push origin HEAD:refs/for/master
Password: secret
[master b4de540] Say hello to Gerrit
 1 file changed, 1 insertion(+)
 create mode 100644 README.md
Counting objects: 3, done.
Writing objects: 100% (3/3), 303 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: Processing changes: new: 1, done    
remote: 
remote: New Changes:
remote:    Say hello to Gerrit
remote: 
To http://admin@localhost:8080/a/gerrit-playground
 * [new branch]      HEAD -> refs/for/master

Step 6 – Review your Change in Gerrit

Open the Gerrit Change URL mentioned in the Git output (http://localhost:8080/#/c/gerrit-playground/+/1001) in your Web Browser and you can review the code using PolyGerrit UX.

Screen Shot 2017-11-14 at 23.37.43.png

Step 7 – Approve and Submit your Change

Click on “CODE-REVIEW+2” button on the right toolbar to approve the change and then press “SUBMIT” to merge into the master branch.

Screen Shot 2017-11-14 at 23.40.45.png

What’s new in Gerrit Ver. 2.15

Hi, I’m Dave Borowitz, I work for Google, and I’m going to talk about what is new in Gerrit 2.15. We have a six months release cycle, and it has been around 158 days since Ver. 2.14.0 was released in April, and now, just last night at 10 PM, I have release 2.15-RC0.

Gerrit Ver. 2.15 in numbers

Gerrit Ver. 2.15 is aligned with other recent releases:

  • 2,102 commits
  • 53 contributors from all around the world and different companies
  • seven new contributors who have never commit before

I don’t know if any of the new contributors are in this room and if they are, thank you, if not I will thank them remotely.

PolyGerrit frontend

There is a bunch of new stuff going on in Gerrit Ver. 2.15, I will talk only briefly about the frontend, because we had an excellent talk from Logan and Arnab and Justin this morning about what’s happened recently in PolyGerrit.
I want to give you my perspective as a developer that is doing mostly backend work. It’s fascinating to have in Gerrit a modern JavaScript environment to develop in because it has been historically difficult for us to attract frontend developers in the project. There are not just enough GWT developers in the world as there are JavaScript developers and I think that the Gerrit project has suffered from a lack of UI attention in the past because of that.

But now we have a whole team here at Google, we have a lot of external contributors, that are happy to work on modern JavaScript, and PolyGerrit has made some excellent strategies of improving the user experience.

Screen Shot 2017-11-14 at 23.51.35.png

PolyGerrit is also usable. Everyone that has been working in software development always has the attraction of starting from scratch and say “how would have done it differently, “: PolyGerrit follows a little bit that approach.

We understand that we have users that are used to a certain workflow and we have to preserve it. However, when you are writing new code, you have the opportunity to learn from your past mistakes and improve things and maybe make it easier to onboard new people.
There is a little bit of preview of what PolyGerrit looks like in Gerrit Ver. 2.15; I had to cut the screenshot last night to make sure that it would be entirely accurate.
You can see that there is a reasonable rate of improvements, even in the last 24h and I hope that we will keep this peace of development.

Screen Shot 2017-11-14 at 23.52.44.png

Backend

From Ver. 2.15 we support NoteDb, and we suggest that you should convert your existing ReviewDb to NoteDb.
What is the motivation for doing NoteDb? Until now the Gerrit administrators had to be DBMS Administrators as well which is a sort of weird feeling: you care about version control and software development workflow, and, for some reasons, you should take care about being a DB Admin?

The idea behind is that we carry on the data historically stored in ReviewDB and we move it directly into the Git repository.
When you create a new change with a patch-set, a topic and all its meta-data, including comments and reviewers then everything gets stored in the same Git repository with the rest of the committed change data. There are several advantages in storing data and meta-data in this way.
You don’t have to worry about two stores that somehow are going to get eventually inconsistent. If you have to make a backup of your Gerrit Server, you have to take a Database dump and then to have a backup of all of your repositories. If you want to have a consistent backup where everything that is stored in the Database exists in the Git repository and vice-versa, you have to do it while the server is down. There is no other way to do it because you may get new records in the database while you are backing up the Git repositories and you would not see that data reflected.

With NoteDb, it’s all in the Git repository, and it is even better because we changed JGit to be able to write to multiple Git repositories refs atomically atomically. This means you can submit a change and also submit the status of the meta-data at the same time. When you upload a patch-set, we update three refs, either all of them succeed, or all of them fail. There isn’t a state where the change was merged in the branch, but the ReviewDb wasn’t updated, that is just no longer possible. That enables us to have consistent backups.

NoteDb provides a very helpful audit log. We had a lot of data issues in the past where could not understand how a change got into a particular state because in ReviewDb you just update a field with ‘X’ and you forget completely that the field was previously with a value ‘Y.’ In Git the model you append commits to a history graph, so you actually store every operation that has ever happened on NoteDb, and that gives you an understanding on how a change ended in the current state.

Screen Shot 2017-11-14 at 23.54.20

We are thinking about giving extensibility for new features, and this is a kind of optimistic view about the future, plugins will be able to add new data to NoteDb while it wasn’t possible for a plugin to add a new column to an existing table of ReviewDb. I don’t think that we have any plugins that are currently able to leverage this capability and we do not have any extension for it yet, but the data layer supports it.

Whilst is automatically giving new features such as moving changes between Gerrit hosts without having to throw away code review meta-data.

NoteDb roadmap

We never actually pictured in the past a complete NoteDb timeline, which spreads across five long years.

Screen Shot 2017-11-14 at 23.55.18.png

  • 2013
    The very first commit on NoteDb was in December, that was a very long time ago.
  • 2014
    We had an intern, and he wrote all the stuff on inline comments in NoteDb.
  • 2015
    I wrote a thing that it was at the end a good idea in retrospective, but it was a considerable amount of work. It is called the batch update and allows to have a coordinated transaction across two different data-store with a consistent interface. This period is what I called “rewrite every single line of Gerrit”.
  • 2016
    We started migrating googlesource.com; ReviewDb still existed, and it was always the single source of truth in case they got out of sync with NoteDb. Later this year, a few months ago, we moved everything to NoteDb and we don’t use the change table anymore. We have several hundreds of servers nowadays using NoteDb and generated several hundreds of changes to it. That is exciting, we have been running in production for months, and that’s why we believe it could work for other people to run in production.
  • 2017
    Last night, we released Ver. 2.15, the first version of Gerrit where we officially say “we officially support NoteDb and we encourage you to migrate away from ReviewDb.”
  • 2018
    We are going to release Gerrit Ver. 3.0 and Ver. 3.1. The reason for the name ‘3.0’ is because we will not have ReviewDb anymore. There will still be the code and the ability to migrate from ReviewDb to NoteDb, but you would not be able to run Gerrit on ReviewDb. The Ver. 3.1 is the one I am most excited about because in that version we do not have even to support a migration tool. We will then be able to throw away all the ReviewDb code, and that would make me very happy.

How to migrate from ReviewDb to NoteDb

I mentioned that migration for googlesource.com took us a very long time because we’ve found a ton of issues with our data. We discovered all these things while we were running our migration and we fixed them all.

We developed a system that was scanning all ReviewDb, performed an in-memory migration and then compared the result with the changes stored in NoteDb. One example of a bug was that some subjects were truncated in ReviewDb. The subject is supposed to come from the first line of the commit message. We were comparing the data in Git with the data in ReviewDb and they did not match because they were truncated. If we were to require that all the subjects in NoteDb were identical to the ones in ReviewDb we would have never passed because there was this truncation. We could have patched all the existing data but actually what we did is to consider that if the subject in NoteDb starts with the subject in ReviewDb it was then regarded as valid. There were many more bugs of that flavor.

There were also bugs in the NoteDb code that we fixed; it was not just like all related to not good data; my code was far from being bug-free. The reason why I am talking about how much effort we put in making it right is that I want you to feel confident and not think about that this is a so much scary operation on your data. We tested on ourselves, and we fixed a lot of these bugs, and we are still pretty confident that this is a safe operation.

In Ver. 2.15 there are two types of migration options: on-line and off-line. At Google, we are in an exceptional condition because we are always at zero downtime, but that was useful because it allowed us to write a tool for a live migration from ReviewDb to NoteDb while the server is running.

Migration to NoteDb is pretty much similar to the way you do reindex: there is an online reindex and an offline reindex. You can choose to do it offline, and it will be probably faster, but there will be a downtime. Or you can decide to do it online, and it will be slower, but there will be no downtime.

And then in Ver. 3.0 we only are going to support an off-line migration, following the same paradigm of all the other schema upgrades. If you skip between releases, we force you to do to an off-line update, but if you upgrade just one point release at a time, you don’t have to have any downtime for your schema migration. Similarly for NoteDb if you migrate from Ver. 2.14 to Ver. 2.15 and then Ver. 3.0, you won’t have any downtime.

Screen Shot 2017-11-15 at 00.00.20.png

Q: (Han-Wen) Is this process parallel?

A: It is parallel if you do it offline if you do it online it is using a single thread because we are assuming that your server is mostly busy doing other stuff and that’s why you may want to do an online migration in the first place.

Benefits of migrating to NoteDb

There are a lot of incentives to migration to NoteDb; one is that you have new features such as hashtags and others that we implemented only in NoteDb because they were a lot harder in ReviewDb such as the history of all reviewers on a change. NoteDb manages audit natively while on ReviewDb we would have needed to have a new table called reviewers_audit which would have been much harder to implement.

The robot comments introduced in Ver. 2.14, the ability to remove clutter in your dashboard to mark a change as reviewed, are all features that you only have in NoteDb.

What did we learn from migration to NoteDb?

Writing every single line of code just takes a long time, and Gerrit has hundreds of thousands of lines of code. Shawn Pearce, my manager, and the Gerrit project founder at Google, every time he needs to touch NoteDb related code just says “I don’t even recognize this, ” and he is still the contributor #1 in the project. We changed it almost beyond its recognition.

Everything I’ve said so far is about changes; there are also other data besides changes. Accounts have been unconditionally migrated to NoteDb in Ver. 2.15. Is more a git config file format for the accounts that we store in NoteDb, it is not even actually a Note-space format. The account is now a config file that has your name and your e-mail address and the status, which is a new feature in NoteDb. For instance, my account status says that “I having a talk in England”.

New Patch-set comparison

Hi, my name is Alice Kober-Sotzek, and I work at Google. In Ver. 2.15 we have changed the way we compared patch-sets. Let’s imagine we have just a small change with a patchset and two files on it. In the first file we have only the first line modified, and the file consists of one thousand lines. The second file has four lines changed.

Screen Shot 2017-11-15 at 00.02.32.png

Let’s see what happens now when I rebased it to the latest version of master. If I had now to visualize what the patchset 1 consisted of and patchset 2 consisted of, what would I assume it would be? If I had been the author of the change, I would have expected that only one line would have been changed. Let’s just do it and ask Gerrit Ver. 2.14 what the result is.

Screen Shot 2017-11-15 at 00.03.16.png

What’s happening? Why do I have 420 lines changed in my file and ten additions and seven removals on the other?
That was not even touched on. Let’s have a look at the content of my file and what is in there.

In Ver. 2.14 we were just hiding all the differences due to rebasing, and that was it. In Ver. 2.15 things look different though because we try to figure out what happened in between the rebase. All the hanks that we are sure are added by something else, are displayed in a different color. We have not only red and green but orange and blue as well; these are all the ones that were introduced by other changes that were in between rebase.

Screen Shot 2017-11-15 at 00.04.34.png

This feature only works in PolyGerrit, while in GWT was not shown at all.

Can I rely on that and trust what I see there?

The decision we made is that all the hunks marked with orange and blue are the ones we are sure of and you can safely avoid looking at them because they were the ones that happened because of other changes occurring in between rebase.

The ones marked with red and green, we give no guarantee. They could be introduced by other changes, because of conflicts or may be added by the patchset. With that coloring, it is much easier to look at the things that are important.

Screen Shot 2017-11-15 at 00.05.32.png

Killing Draft Changes.

Some of the people think that draft changes are very much a visibility thing so that only my reviewers can see them. Other people use it like a change is not yet ready for review so that I can leave it as draft change until it is ready for being reviewed. You may even just use the server as a store for your changes; rework the code through the Gerrit in-line edit feature until the code is ready and then come up with an absurd number of patchsets. Nobody wanted any of them, but those are the conditions that we ended up with patchset drafts.

Patchsets could have been even deleted so they would never exist. They could just be kept invisible so that you see a gap, but that could be the current patchset: the UI claims that the current patchset is three, but then I do some other operations that say that this patchset is not current anymore, just because the current one is a draft!

Screen Shot 2017-11-15 at 00.06.36.png

Drafts are a kind of mass fraud; the main reason is that they are colliding all these things into a single feature. In Gerrit Ver. 2.15 we killed drafts. Now you have little small features instead of drafts. You have now “Private Changes” which only you and your reviewers can see. There are Work-In-Progress (WIP) changes, that means that while the WIP flag is set nobody gets notifications about it: you can push 30 patchsets, and the reviewers would not get spammed with 30 emails. Last but not least, we introduced a long ago the Change Edit, which can be used as well in conjunction with WIP Changes.

Marking Changes as reviewed

Another thing that we introduced in Ver. 2.15 is the ability to mark changes as you reviewed it. For instance, the one below is a change screen from my dashboard this morning: some changes are highlighted in bold and those other changes are not. I feel like the bold changes are yelling at me and you have to give me your attention just like in e-mails where bolds means “you need to look at me now.” Gerrit Ver. 2.15 when you are using NoteDb allows you to unbold any of them by just clicking a button on the change screen. Or like in an email you wish to remove some changes from your dashboard entirely. There is a function that allows you remove a change unilaterally from your dashboard that the other cannot undo or ignore it, that just makes the change go away.

It was annoying that I could not mark them as reviewed manually and it was really irritating that patchsets disappeared. It is really irritating when I received a review with a bunch of comments, I had to say “done, done, done, done” on each one of them.

Screen Shot 2017-11-15 at 00.10.01.png

And then when I pushed a new patchset, I just forgot to submit all these drafts comments that say “done”. So I added just a push option that says “when you push, publish comments” and all the draft comments will be published automatically. So instead of clicking on all the patchset on that change and check if in any of the patchset I have any draft and if I do, click send on all of them one by one, I can instead just set an option.

Screen Shot 2017-11-15 at 00.10.52.png

It can be specified by the command line, but it is difficult to remember. So there is a user preference with a checkbox which I really encourage you to select in your user preferences screen and it is only available on PolyGerrit.

CCing a Change under review

When someone is getting a co-worker and they want him to be a reviewer for a change, you get an error saying that your co-worker is not a registered user. We have partially solved this problem by adding a CC with an e-mail address, also only available on NoteDb. There are technical and even product reasons why we don’t want to add them as a reviewer, some of them are related to the accountability related to everyone that is working on that change. So people needs to have an account to be a reviewer, but if people just want to look at it or a mailing list, it doesn’t have to be a real user, you can then just CC any e-mail to a code review if you turn the config option to allow this.

Screen Shot 2017-11-15 at 00.11.48.png

Better error messages.

Another inch that Han-Wen scratched was the introduction of better error messages. Sometimes you do a push, and it fails with a very unhelpful error message that says “Prohibited by Gerrit.

It turns out that it is not difficult to check if a user does not have a permission then gives permissions error, so we have included a message saying this user lacks this permission. It is not perfect, so it doesn’t say precisely where in the project hierarchy this permission was coming from but at least says “this is the problem”, it tells you not the solution but at least highlights where the problem is.

Screen Shot 2017-11-15 at 00.12.37.png

Robot comments and automatic fixes.

Robot comments are a feature that we believe it will start to rump up in adoption. With NoteDb, you can suggest fixes as well, and then you have a button that says “apply fixes” which creates a change that applies the fixes.

Many more improvements in the bag.

There is a lot of speed improvements in PolyGerrit, so the changes with a lot of diffs will run a lot faster. An admin can delete comments that really shouldn’t be there. We can explicitly keep track that a change reverts another change so that you can search if that change was reverted. It can even tell you if that was a pure revert at Git level only, or if other changes were sneaked in claiming that this was only a revert which happens way too often I think.
There are a better server consistency checks and a new plugin endpoint for dashboards; there is a new URL scheme as described by Patrick and we are now off the page for putting, even more, new features.

Screen Shot 2017-11-15 at 00.13.15.png

 

Advertisements

Gerrit User Summit: PolyGerrit UX

The second presentation this week from the Gerrit User Summit 2017 is about PolyGerrit, the new Polymer-based UX for Gerrit Code Review, officially feature complete for Code Review in Gerrit Ver. 2.15.

Logan (PolyGerrit and Gerrit Maintainer), Arnab (PolyGerrit UX Designer), Dustin (PolyGerrit User Researcher) and Jason (Gerrit Code Review and Go Language Product Manager) show and discuss the future of UX of Gerrit.

PolyGerrit Overview (Logan Hanks, Google)

We are going to present three topics today:

  1. I am going to talk about the development of PolyGerrit, why we are making PolyGerrit and what we’ve done over the past two years.
  2. Arnab is going to present a new visual design we are going to start rolling out to googlesource.com, and to the next release of Gerrit.
  3. Dustin is going to give some findings from the user research on the survey I believe that many of the people in this room may have received the questionnaire on Gerrit and have responded.

Established November 2015

commit ba698359647f565421880b0487d20df086e7f82a
Author: Andrew Bonventre <andybons@google.com>
Date: Wed Nov 4 11:14:54 2015 -0500

Add the skeleton of a new UI based on Polymer, PolyGerrit
 
This is the beginnings of an experimental new non-GWT web UI developed
using a modern JS web framework, http://www.polymer-project.org/. It
will coexist alongside the GWT UI until it is feature-complete.

That is the very first commit we did on the PolyGerrit project, it was founded by Andrew Bonventre from the Chromium project and the commit message summaries up pretty well.
The idea is to replace the existing UI based on Google Web Toolkit (or GWT) and replace with a new UI built on modern frameworks with straight HTML and JavaScript. Part of the innovation was just not to renew the UI and make it better but also for the Chromium project move to Gerrit.

Based on the Polymer Project

Polymer is pretty essential to PolyGerrit, and that’s why is part of the name. The Polymer project is not a framework but rather a collection of libraries and tools and components that have been developed by Chrome at Google. The idea is the build beautiful web applications on top of the standard web platform. Some critical aspects of the web platform that we use are web components which are built on top of templates that you can import into your HTML file and shadow DOM which is a way of encapsulating your stylesheets into your templates so that you can build reusable components.

Another significant aspect of the Polymer project is the collection of custom elements. They are organized into groups and given chemistry names. We use iron elements pretty heavily; they are the basic building blocks for HTML.
We are starting to use elements from the paper collection which are higher level UI elements that introduce material design. They look nice, they are high level and reusable elements. They let us focus on building code-review without having to rebuild everything from scratch.

Screen Shot 2017-10-19 at 15.50.05.png

Q: Can you explain a bit more on what material design means?

A. (Arnab) Material design is a new design language that Google came out of it in 2015. It’s a library of all the modern components with great look and feel.

Material design elements come with a lot of features like animations and smooth transitions which makes your UI more dynamic, it’s hard to show in slides all that dynamism.

Current status of PolyGerrit

We have been working on PolyGerrit for two years now, and in that time we got a lot of contributions from outside the project, we have a few dozens of contributors, checked in thousands of commits, and we closed almost a thousand issues from our issue tracker.

As far as I know, there is no other significant deployment of PolyGerrit outside of Google and GerritHub.io, but we have been having used in Google for a while, and we see 10k code-reviews a day involving PolyGerrit, at least on a workday. We have got a few major OpenSource projects using it, Chromium has recently transitioned to PolyGerrit, and brings up almost 2000 developers.

I’ve put some other recognizable OpenSource projects like GoLang and also the Gerrit project itself, where we configured gerrit-review.googlesource.com to use PolyGerrit by default. That helps us get early feedback on the functionality and the design of PolyGerrit.

Chromium migration to PolyGerrit

What about Chromium? It is a significant milestone for our Team, and I also think for the Gerrit project. The Chromium project has a very complex system for checking their code and for builds. They have built the system themselves and they used it to run their code-review system them as well based on Rietveld. Unfortunately, that is a sort of unmaintained instance of Rietveld running on AppEngine they did not want to use anymore and also it wasn’t scaling very well for them. There were even commits that they couldn’t review on that system because they were too large.

So a lot we’ve been working at that time, it was allowing Gerrit to come to a state so that Chromium could use it. Part of that was building a nice UI, and part of that was taking features they have been enjoying in Rietveld and bringing them over to Gerrit. Some of them are small things like user account statuses, so that you can tell people that you are not in the office and you can’t do code reviews this week, or having descriptions to your patch-sets so that you can explain what this revision means.

And also they’ve built a lot of new customizations, and they got us flush out a plugins API, and I’ll show one example here. So Chromium has a pretty heavyweight CI system because they have to build Chromium for a lot of different platforms and different architectures and they have a lot of incoming commits. So they need to find a way to verify that a commit that is coming into the code, actually works and in which order to check in those changes, so that is not breaking the build.

Screen Shot 2017-10-19 at 16.06.00.png

One thing that they’ve built for PolyGerrit is a plugin to add these elements to the screen for visibility to what’s going on in their system. The builds that are running for their commits, their status, often the flaky, so that by clicking on these you can see the output of the build. You can also use this UI to start a particular build and chose which architecture you want to run on or re-run things that failed. I think we should be building better integration to Gerrit for these sort of things because those are what big and small projects need, but it is cool that we can do these things with plugins as they are now.

What’s new in Gerrit 2.15 for PolyGerrit

Gerrit 2.14 was the first release that included PolyGerrit but it is just a sort of experimental preview, but that did work well out of the box. Since then, we have fixed a lot of bugs and added more features, we have converted the entire project to ES6 which for JavaScript developers makes their life so much easier and also makes the project more attractive to other contributors.

We made a lot of performance improvements in the application. For example, sometimes we have commits that have thousands of files, and we have to have a way to display them intelligently, we cannot just throw them on the screen. We needed a sort of new navigation for these kinds of things, and we have adapted PolyGerrit to work with any variety of commits of different shapes and sizes.

Some specific things we added that are kind of cute. PolyGerrit offers a reply dialog to manage your reviews, reply to comments, vote on labels, and submit that all as a single batch operation. We added some nice features to this reply dialog since Ver. 2.14, one is being able to preview how your reply will be formatted. Not just to show you how your answer is going to look like but now it renders in the same it would end up in the e-mail because we have HTML e-mails.

Screen Shot 2017-10-19 at 16.06.49.png

We have also reorganized how voting buttons are arranged, there is this beautiful vertical alignment, and you can see the labels that give you the meaning what +1 means for that label. We have a slightly updated patchset selector. That is important because one of the things people struggle with it is navigating a code-review. If there are more patches, understanding where the comments are can be challenging, because of comments land on different patchsets.

Screen Shot 2017-10-19 at 16.08.31.png

We also introduced the feature of resolvable comments, so that you can mark a comment as fixed or done and we can track that for you. You can see that on some patchsets there are some comments you haven’t reply to yet or has a review, and you can verify that everything has been addressed.

The PolyGerrit Admin UI

Another major area that we have been working at is one of the significant features that PolyGerrit has compared to the old version: the admin UI.

A lot of our users had to switch back and forth to do the admin work. We’ve been building up this part of the application. We have rearranged that a little bit, we changed the title by clicking on the browse link on the top menu. Our idea is that you can get to this like the project home page for a repository, where you have to check out instructions and is also the entry point for administering the project.

Screen Shot 2017-10-19 at 16.25.51.png

We have almost all the primary forms for managing groups and projects completed. We have also released a read-only version of the sophisticated project access rules editor. It is read-only mode because we don’t want anybody to risk losing their project.config because of a bug in this: project.config tends to be a very security sensitive thing.

What’s next for PolyGerrit.

We are still working on replacing the GWT UI; we’re getting close. One of the significant features remaining is in-line edit. We’ll be working on that over the next few months, as long as the long tail of the specific missing pieces of the UI. And then we are going to roll out a new visual design which Arnad will be talking about shortly.

We also want to do in the longer term more significant changes to the UI of Gerrit, to show how accessible is to the users, because when people learn how to use Gerrit they love it, but the initial learning curve is a bit difficult at the moment.

PolyGerrit UX New Look (Arnab Banerjee, Google)

We are working on a new look; I’ll just be giving you a quick glimpse of what it looks like. We know that Gerrit and PolyGerrit work very well for the users, but we want to improve its usability as well as the status of the tool.

For improving usability we think there is a long process. We don’t want to break what’s already working for you, and that’s why it will be more an evolutionary process based on user-research and feedback.

However, we started by looking at the aesthetics of PolyGerrit and giving it a new look. A few critical goal that we had is that design should represent what PolyGerrit is and stands for, having a more contained group of information in a way that you know exactly where to find a particular group of information. Also more consistent UI behavior across the board: as soon as you look at the component you see what you can expect what the element is supposed to do. Also, we want to introduce a more up-to-date style of components and fonts, which means: material design.

What should drive the experience we want to create is what we really want to achieve with PolyGerrit.

Screen Shot 2017-10-19 at 16.26.40.png

Confidence: we want you to be confident with the code you are submitting and is going out there.

Scalability: we want PolyGerrit to manage the small as the significant and more complicated change.

Efficiency: we want to increase the efficiency of your team as making the code review process more efficient.

Intelligence: PolyGerrit should be smart enough to do all the repetitive task that you otherwise would have had to do manually. Wisdom: PolyGerrit should be mean to share the knowledge on the team.

PolyGerrit is Blue

One color that captures all of that very very well is blue. So we picked blue and made the primary color of PolyGerrit.
That is going to be the default primary color but you can change it for your team.

Screen Shot 2017-10-19 at 16.28.34.png

That is the new and refreshed look of PolyGerrit.

There are three distinct layouts for PolyGerrit:

  1. The change screen
  2. Dashboards
  3. Admin screen.

PolyGerrit new Change Screen

The change screen is the most used screen in PolyGerrit, so we thought why not start by giving this a new look.
For this presentation, I will just talk about how the new PolyGerrit style works on it. Let’s talk about the visual elements on this page and look at it with a little bit of detail.

Screen Shot 2017-10-19 at 16.27.51.png

One of the things we have added in PolyGerrit is the breadcrumb where you can quickly jump back to wherever you came from. The next is the status, and now we have a clear status indication of the change so that just looking at this state you can understand if the change is it is closed or opened, and the color line adds to this the indication of the change.

Screen Shot 2017-10-19 at 16.30.54.png

Let’s talk about the meta-data of this page, we have now a much more visual structure of spaces, fonts and colors and actions that are now material flat buttons, and they are different from links, which will be all caps and different fonts altogether. So visually you see the difference in what will take you to another space and what will take action. When you click on add assignee, you get this in-line box which let you add assignee, and this is just a more elegant way of putting inline data, it just makes the action so much more clear.
When you click save, you can see the assignee name populated and assigned to the assignee label.

Once you start typing will just suggest the names while you are typing and topics also work in the same way.

Screen Shot 2017-10-19 at 16.31.29.png

Now coming arguably to the most used section of this page which is the files table, and you may notice these two header rows, the file heading and we just combined the information of these two rows into one row.

Let’s look at the elements of this particular section. The first part is the patchset selector. If have both the selected patchset as well as the original patchset, you can see them together side-by-side, so that you don’t have to look all the way to the right to select the referenced patchset.

Once you click on the patchset dropdown, you have a different layout for this different drop-down menu. On the left side you can see the patchset as well as the patchset description, and on the right side, you can look at the comments and the unresolved comments. That makes scanning through this list so much more comfortable, especially if this list is long.

Screen Shot 2017-10-19 at 16.32.17.png

Then adding patchset description also works in the very same way. It just shows the inline edit box, and you just enter the patchset description and say “save.”

Now we have a separate interaction for marking the file as reviewed when you hover on this rows. The mark reviewed shows up on the right-hand side, when you click it you can see a column on the right-hand side of the mark as reviewed and when you click review this text shows up a toggle button.

Screen Shot 2017-10-19 at 16.33.05.png

On the top-right, we have now the table actions, and we just have two actions by default. The first action is “expand all” and the other is “download.” If you click on expand all you see this diff view expanded and you see some icons that appear on the top. So the “expand all” button changes into “collapse all” and you can change from side-by-side to unified diff-view or even change your diff-view preferences.

That is just a tiny step towards what we can do and is just the tip of the iceberg, and I think we’ll reach out to you for more feedback and do more user research for really making PolyGerrit a tool that you’d love as code-review tool.

PolyGerrit User Research (Dustin Smith, Google)

I am a user researcher at Google, and my job is to communicate with our users and take what you tell me and distill what we can do and what changes we need to do to our product.
Being an advocate for all of you in the meetings because I am not a developer I am not a designer and my role is mainly to communicate what you need to our developers and feedback them to our designers. I collect feedback and distill your needs. Arnab will come up with new designs, and we will come back to you and say “how did we do based on your feedback?” Then we will repeat that process to infinity, and that’s how research gets integrated into the design.

So if you are not a member of the Gerrit Google Group  (repo-discuss) , I highly recommend you to join. You can yell at us there; you can say “I really don’t like what’s going on in the product, ” or you can praise us, whatever you like. At some point, you can send us surveys through there and hopefully not a very long survey that helps us aggregate all your needs.

User Survey results on Gerrit Code Review

An example of that we sent out a survey in June and 600 of you responded. We received a lot of nice praise, and that’s great, but we’re also looking at ways to improve Gerrit, and we asked you to rank of what would be your priority.

Screen Shot 2017-10-19 at 16.34.13.png

What you see on the top here is that all the people that answered the survey is interested in improving the navigation of Gerrit, whether it is a patchset navigation, commenting, or other UI navigation. There is also another related to better integration with other tools. We are curing that feedback. Something to keep in mind is that even though there are things at the bottom that are ranked at zero, that doesn’t mean that you don’t want those things but they are relatively listed.

Q (Han-Wen Nienhuys, Google). I am wondering, who is in this group of users you asked because one of the things entirely at the bottom says “improve the reliability of the backend” and then very much at the bottom is “better performance” which is kind of what our Team is continuously worried about. Even yesterday we had a Hackathon and the friends from the Android Team on Sunday morning, and we were all in panic, but I see that “nobody cares about it,” how comes?

A. That’s a fair question, who did we sample? These users were external users of Gerrit, and only 13 of them were PolyGerrit users, so what you’re looking at here, 95% of them are external users. Having said that, why ranking better performance and reliability the lowest, I am not sure how to answer that question.

A. (Gerrit admin from the audience). For us obviously reliability of the backend is important, but the question was “does reliability need improving?” and for us, Gerrit works quite well and we are very happy with it.

A. (Luca Milanesio, GerritForge) I believe that my is very similar to that. First of all, the people you are aking to. You’re not asking Gerrit administrator, but just people that are using as a UI. The people that are using the GUI are saying “reliability is currently 99.99%, do you need to make it 99.99999%? Not really”. What needs improving is what you guys are focused on: usability, discoverability, and leveraging all the power that is out there.

A. That’s a great point and keep in mind that all the items on the bottom of this list don’t mean that people don’t like them just that other aspects that need improvements are ranked above it.
People did not say “I dislike the idea of better performance, ” but more often than not, people selected the options above it before the better performance.

Other highlights from people, maybe people that is one of you, navigation is important “Getting an overview of the change I’m reviewing is hard. When a change is composed of several files, the file navigation feels like a pain”.

We are taking these feedback and then moving forward, and also some complaints about the search itself. The search could be better, for example, “I want to search a commit that includes a particular file.”

And then, integration. I would like more Jenkins integration for passing code coverage reports into Gerrit. If these quotes that I pulled out are jelly with you wonderful, if you like there are things that are missing and you are not being heard, I am the person to talk to. I need to make sure that you are heard. There are also some other options I am going to show you. How many of you are aware of userresearch.google.com?

PolyGerrit Research: call to action

If you’d like to, you can sign up here. What we will do is substantially having a look at the new designs, you can spend an hour with me, maybe 30′ depending on how busy you are, and I’ll show you some new designs, and you’ll get a gift for your time. I am not the only user searcher at Google, and this is not the only product that is supported by this, but I am interested in hearing your feedback about Gerrit and runs studies on here, and you’re welcome to join me.

Screen Shot 2017-10-19 at 16.34.56.png

Today and tomorrow we have a booth that you may have seen already with this banner, and we have a new dashboard design, a new patch navigation system and a new navigation for related changes that we’d loved to have you have a look at. We are not showing them on these slides on purpose because we want you to see the designs and come with a bunch of feedback on your first impression to the UI.

Please come to the booth and talk to any of us, there is Jason Buberel with us (our Product Manager), you’re welcome to come and speak with him as well, and almost all the Googlers here are willing to show some of the new designs.

Questions.

Q. How exactly you come out with a new design, the layout, how to make it better. How do you know that the user experience is better compared to the previous old one?

A. I take what your pinpoints, what you love, and I tell Arnab, and then he comes with a new concept. How do you know if that is better? Typically you generate tasks of what you would usually do with our UI, and we then do a tasks assessment where you see the goal that you go through those tasks and see if they failed. You talk aloud thinking while you go through those tasks and you may say “I really don’t like this new UI” and from that, you do mainly a mini A/B test. You can go through this process for just a few people and you don’t have to build up the entire product that everyone sees.

Q. I remember when a couple of years ago there was the announcement that we are going to have the new PolyGerrit project. We all asked “when the new GUI is going to be available?” and the answer was “six, maybe twelve months.” Now after two years we have something that works that is PolyGerrit, we started using it, and we like it. There are still gaps that are not really on the change screen, which is way better than the old one. I can use it on my Tube in the morning on the mobile phone while the old GUI did not even render or it was so tiny that you could not even see it. The gaps are more on the other parts of the UI. 50% of the users are using Gerrit as a very reliable Git server, have you put on your radar all the other use-cases not directly related to code-review that currently PolyGerrit is not covering?. The on-line editing, the code-browsing, the user-journeys from the code to the review and the way back, integrated search across code and the associated reviews?

A. Something is clearly in the scope of Gerrit, and something is not, but that does not mean we are not interested in it. Source browsing is one example of it, and I don’t think it will be in PolyGerrit but that does not mean that we are not interested in a way to browse source code. We are working on the in-line edit, maybe is not something we looked at the usability of it yet, it is a bit lower on the list of priorities, but are starting with the implementation and Arnad has been helping us with the basic design elements of that. It will be a basic feature at first, but it will look nice and be usable to some extent, and then we iterate.

Q. What about the completely new redesign? Are your efforts directed in completing what is there or are on reinventing everything from the ground up and we have to wait for another two years?

A. We are focused on the immediate terms of completing PolyGerrit because for many technical reasons we want to replace the GWT UI and we want to have more users using PolyGerrit so that we can get feedback to keep going. As we get more and more feedback and more and more iterations on PolyGerrit in the wild, we will come up with a lot more work for us to do. In the near term, we are going to implement what Arnad has presented and while we are doing that Arnad will help us by collecting some more feedback around what you will see at the booth, around the new dashboards and we are going to keep working on that. PolyGerrit is not going to stay static as it is today, it is going to be more iterative process based on Gerrit @Google and the Gerrit master branch. At every release cycle of the Gerrit OpenSource project, we will include some significant improvements on the user experience.

Q. I am just wondering how the new users get represented in your user research. Maybe people coming from GitHub or GitLab but even other people coming from different source control systems such as Perforce or SVN. How do you capture their experience? A lot of us have used Gerrit for a while and understand to a degree, Gerrit. However for other people, at times is a kind of a ping-pong, even though you are a new user for a very short time until you learn Gerrit.

A. On-boarding is definitely on our radar, we take users that have not used Gerrit before, and we ask to perform certain tasks. That is the kind of the recipe: take people that never used the product and working with Arnad on their feedback to make that process smoother.

Q. How many research you have made on the other code-review tools, advantages and disadvantages over Gerrit?

A (Dustin). It is an approach I am willing to take, doing a kind of competitive analysis. I have not started yet, but it is definitely on the roadmap for research.

A. (Jason). If there are other code-review tools that you’ve used and something that worked well, something that Gerrit or PolyGerrit could benefit from their knowledge or usage, please let us know. Nothing is off the table here concerning what we will improve or consider improving. If you used other code review tools, e.g. “hey reviewable.io got this very pretty good thing, you should do something really similar to that”, by all means, please let us know because we are opened to improvements in all areas.

Q. Following-up on this question, there will be talk tomorrow on this, our friends from CollabNet that are contributing to Gerrit Code Review have clients that are using not only Gerrit but even other tools like GitHub and many of them are hearing concerns of people getting lost without their pull requests. Because they are coming from other tools like Atlassian BitBucket, they have the concept that “without a pull request there is no review.” They have as well the misunderstanding that Gerrit forces you to amend the latest commit all the times, which is wrong. Gerrit allows you doing whatever you want. If you want to amend, you can do it; if you want to use feature branches you can do it as well. I remember Martin last year saying that they are using feature branches at Qualcomm and there is nothing wrong with it. CollabNet has developed a very basic UI that allows people that are coming from the pull request workflow to keep on using that paradigm. This UI is a sort of lightweight code review that helps to digest Gerrit gradually. That could be even the key to solve the dilemma raised on a recent discussion in the GoLang project where people asked to stop using Gerrit and instead using GitHub to review contributions.
Following the discussion thread, it emerged that Gerrit is “too good” and does exactly what they need while GitHub doesn’t. However, there are concerns about other potential contributors coming from a GitHub experience having to invest 15′, maybe 30′ or even one day of learning of the Gerrit Code Review workflow and a new UI paradigm. They may decide to give up and avoid contributing at all. If the project would move to GitHub pull requests, this problem would just disappear. Are you guys taking into consideration how to make this transition from GitHub smoother and Gerrit easier to understand for those guys?

A. That’s a great question, and as the formal product manager for the GoLang project, I heard from some members of the community of the project that they were relatively unwilling to contribute. We saw this type of disparity from the casual contributors. Those are the people that find a typo in the docs and just want to fix it quickly. With the GitHub UI is nice because they see the problem, they click Fork, edit the code on a single line or merely a single file, and they create the pull-request with a straightforward review process and gets merged and done. You never had to clone the repository, did not have to figure out the authentication bits, and that is simple. You know, whether we want or not to take Gerrit down a path where we want to make these casual contributions super-simple it is something we are open to if that is what the Gerrit community thinks is a significant capability. At the same time, we do not want to displace GitHub with Gerrit, there is a vast community there for a reason, but there are specific workflows like the casual contributors’ workflow that are certainly easier on GitHub and much more difficult for users that are new to Gerrit. It is an area where if we think there is a need for, we should think about it and consider how we may support such a thing. We are open to hearing more feedback on things like that.

A. I just came from the OpenStack developers’ summit in Colorado a couple of weeks ago, and we were having the same conversation regarding onboarding contributors to OpenStack. Most of the people coming from GitHub are having the same hurdle. One of the things that were discussed out there was having some documentation, some quick-start, cheat-sheet, crash course, for people that are familiar with GitHub. It is not just about enhancing Gerrit code to make it more friendly and port the workflow they are used to use but provide content that helps people, coming over.

A. Dave is one of the tech writers at Google who works explicitly with Code Review. If you have a specific part of the documentation set that you think it needs improving or a brand new part of the documentation set that could be useful to new users, come to him and talk today or during the next days. He will be happy to help.

A. I have started already building a spreadsheet, a kind of rosetta-stone of mapping concepts and commands for mapping one to the other and I’ll show you that. That is just one approach, yes, but there are some pros as well.

 

Gerrit User Summit: Gerrit at Google

Starting from this week, we are going to share one video per week of the amazing talks that were presented at the Gerrit User Summit 2017 in London.

In addition to the YouTube recording, we are during the extraction of the text and publishing it together with the relevant pictures taken from the presenter’s slides, so that people can start digesting the content at small bites.

This week talk is Patrick Hiesel’s presentation on how Gerrit multi-tenant and multi-master setup has been implemented in Google.

Gerrit@Google – Patrick Hiesel, Google

My name is Patrick, and I am going to talk about the setup of Gerrit we are running at Google. I wanted to take you on a journey starting with Gerrit that you all know and making it the system we run at Google, step-by-step; and at the end will have a multi-master and multi-tenant system.

Multi-what?

Multi-tenant is the ability to serve multiple hosts from the same single Java process. Imagine like the same JVM task serving gerrit-review.googlesource.com and gerrit-chromium.googlesource.com.

Multi-master is the ability to have multiple Gerrit servers all over the world. You can contact any one of them for reads and writes.
Most systems have read replicas, which is straightforward, but write replicas is where the juicy meat is.

Multi-tenant

We have gerrit-review.googlesource.com, based on OpenSource Gerrit that you can download right now and have it running hopefully under ten minutes. That is core-Gerrit, and it depends on three things:

  1. JGit: for all the Git stuff
  2. Multiple indexes for the accounts, changes, and other stuff
  3. Caches

All these three components are based on the filesystem in one way or the other.

Now you have a friend that is accessing go-review.googlesource.com, what are you going to do?
The most natural solution is to start another Gerrit instance for it. You can have all of them on one machine, you can give them different ports, very easy, and in the end, they’ll be all based on the filesystem.
All those Gerrit instances do not need to talk to each other; they can just be separate instances operating on separate ports. This is not a multi-tenant system, but only different Gerrit instances on the same host.

Screen Shot 2017-10-10 at 15.35.08.png

You can add another layer on top of it: a servlet engine which receives all the traffic, check which host the traffic is for, and just delegate to the individual host.
To take one step further, have that selection filter doing that for you. Gerrit has a daemon that runs all the functionalities. You can integrate that daemon into the incoming servlet filter. When you can get a request for go-review.googlesource.com and I do not have where to allocate it, you can just launch it, instantiate all the objects and then run the traffic from there. Also, unload instances would work in the same way.
The Gerrit server engine and the selection filter can run in a single JVM.

How Gerrit can conquer the world

So you have a master here in Europe, and you have got one friend on the west coast in the US. He says: “Oh your Gerrit is so slow I have no idea why and I wish I could move to GitHub.” You say: “Hold on, I can do better than that!” and so you put a new master for the person on West coast.

So the key to that is the replication and comes in two sets:

  • Objects that you have to replicate related to all the Git data. JGit is putting objects into the disk, and these are the data you need to replicate correctly and fast.
  • Other stuff that should replicate and fast to provide a pleasant user experience but it really can be best-effort. That is the indexes and the caches.

If it is okay for your master having a 200/600 msec additional latency, then do not replicate the caches. You can have a cold cache in Singapore or the US, and you can reread them without problems.

Screen Shot 2017-10-10 at 15.37.31.png

For the index, replication can be best-effort, but you should make an effort to replicate them. It is still nonetheless a mandatory requirement. One way to achieve that is to use ElasticSearch, but other index implementations that give indexes replication can be used as well.

Multi-tenant and Multi-master together

We talked about a multi-tenant system and then replicate them globally, so we have now a multi-tenant and multi-master system, actually pretty close to what we run at Google.

That is the stack that we run in total. We have a selection filter and two other filters to decide what the traffic is directed. We are also based on JGit, no magic there, we have index and caches we replicate, all our systems are based on filesystem and BigTable.

Some “magic” happens at the Git layer at Google because that is where all the majority consensus across all the cells. When you are pushing anything that is Git and, with NoteDb, anything that is a review is in the repository as well, the system tries to reach the majority of the cells and write the objects to them. When that is acknowledged, you get a green light on the push.
Majority consensus also means that you have it only in so many cells, but don’t have it on all the cells all the times. Some of the replication is happening in the long tail, by replication events eventually get acknowledged by the cells, and then they get written to all the masters.

Our indexes and caches are also replicated, but some of them are just in-memory and a component that gets replicated on top of BigTable.

Redundancy everywhere.

We run five data-centers across three continents (Americas, Europe, Asia-Pacific), with precisely the stack we just saw which gives us a good latency for most of our users worldwide.

Let’s talk about load balancing. We have a system that is multi-master and multi-tenant, and any of the tasks can serve any requests, but just because it can, doesn’t mean that it should.
Maybe it has in cold memory caches, or it is in Singapore, and you are in the US; so the question is what if the biggest machine is not big enough and we want to optimize it?

Screen Shot 2017-10-10 at 15.39.02.png

The idea is that we want to reduce latency that comes out of cold caches and minimize the time the site takes to load.
So you have a request for gerrit-review.googlesource.com, and your instance has a cold cache, and you need to read from disk to memory to serve the request.
You have a fleet of 300 tasks available but you want to serve gerrit-review.googlesource.com from only just five of them. If you serve 300 requests from 300 tasks in a round-robin manner, you pay the latency to load data from disk to memory for every single request. And the second motivation is that you want to distribute the load.

We want a system that can dynamically scale with changing load patterns. We want a system that can optimise the caches, to send a request for a site/repo to the few number of servers and tasks based on two conditions:

  1. serve from one machine as long as it fits on it
  2. server from more machines if you really have to

Level-1 load balancer

In the stack at Google, you saw two load balancing levels. What you see down in the picture is the Gerrit tasks, that contains all the software layers we talked about. We have a user that triggers JSON calls from the browser, with PolyGerrit. The first thing that the JSON request is hitting is the L1 load balancer. The primary routing of your request is by geographic proximity. We have five datacentres at Google; the L1 load balancer picks the one with the lowest latency. When the request goes into the data-center, we have another load balancer which is the one I’ll talk about more, because this is the one where the Gerrit specifics happen.

One thing that L1 is doing as well is managing the spillover of traffic. When a datacentre says “I can handle up to 100 QPS” the L1 load balancer starts redirecting traffic to other datacentres should that threshold be reached.

Level 2 load balancer

Let’s dive into L2 load balancer, we want to know how much traffic we are getting into each Gerrit task, and we want to know in the load balancer where the single request should go, and we want to know that fast!

We added three new components to the architecture:

  • An element to redirect tasks and provide functionality and can report the load we are handling right now. When I mean load it can be anything: QPS (Queries Per Second), metrics, we just want to know from the tasks: what is your current load? We have a system called slicer, which I am going to talk about in a second and it’s added there in the picture.
  • A second component we are adding to the load balancer, with a query interface that responds to the following question: “we have a request for gerrit-googlesource.com, where should it go?”. All of that should be done in memory and should be regularly updated with the new elements in the background so that we don’t add another component of latency by having another RPC.
  • A third part is coordinating everything and is called the assigner, and it takes all the load metrics that we reported generates new assignments and gives them to the query interface.

Introducing the slicer

We have a system that is called slicer. There is a very nice paper that I can recommend, published last fall, that talks about that. It is a load balancer that works on custom keys and can do automatic re-sharding based on new traffic patterns. When your nodes receive more traffic, the slicer will automatically distribute the load or re-shard the whole system. That is a suitable method for local sharding that happens within the data center; we do not use it for inter-datacentre because that is all done via geographic proximity.

Screen Shot 2017-10-10 at 15.40.13.png

The system works with 64-bit keys and gives you a lot of combinations. You can slice the keyspace, for instance, in 400 slices. That gives you 400 ranges, and you can take any of them and assign to one or more tasks. The hostname is my key for instance, and then you hash it, and you end up in the first slice that gets assigned to a single task with an index zero.

What can we do if the load changes? Let’s say that you have key zero that gets assigned to the first range and then the traffic changes. We have two options.

  1. The first option is to assign more tasks, let’s say task 6, and then you round-robin between task 0 and task 6.
  2. The second option is splitting into 600 or 800 slides to get a better grip on each of the keys.

Screen Shot 2017-10-10 at 15.41.20

We can also do that, and then we factor our the load for gerrit-review.googlesource.com and go-review.googlesource.com, and we put them into different hosts.

We do that for Gerrit, and one of the things we want for Gerrit is when we have to split per-host traffic with the affinity on the repository. Caches are based on the project, and because gerrit-android.googlesource.com is a massive host served from a lot of tasks, we don’t want all these tasks just to serve all general traffic for android. We want tasks serving android/project1 from here and android/project2 from there so that we optimise the second layer of caches.

What we do is to mangle these keys together based on both host and project. Before, all these chromium keyspace was served from a single host; when the load increases we just split the keys into Chromium source and the rest of metadata. This is the graph that we obtained after we implemented the load balancer. The load we have on each of the tasks in a single data center is represented by a line with a different color. What you can see is that are all nicely aligned, so that each task is serving precisely the same amount of traffic which is what you want.

Screen Shot 2017-10-10 at 15.42.21.png

What if one project is 100 times the size of the others and we are optimizing on queries per second? The system will just burn resources fast. We had that situation in May, we saw the graphs, and we said “all good, looks nice”; however, people were sending e-mails and raising bugs wondering if the system were serving any traffic at all or if it were down completely.
It turned out that Android had a lot of large repositories, regarding the number of references, and the objects. We were just optimizing the queries per second, but some of the tasks were doing just CPU intensive work, where others were happy with it. Some of them were burning CPU in flames, and others were fine.

So we moved out of the per-request affinity, and we modified the per-repository sharding to optimise all of this.

Warm vs. Cold Cache

There is an extra in the system that is pre-warming caches. What the load balancer can do for you is to tell you that traffic is changing and I need to reconsider how to split the load on the system. For each of the tasks is going to tell “I’m going to give you traffic for gerrit-review.googlesource.com” with a notice of 30 seconds. That time you can use for pre-warm caches.

That is especially nice if you restart your tasks because all the associated in-memory cache gets flushed. The load balancer tells you “oh, this is the list of the tasks I need” and then you can get them all and pre-warm their caches. This graph shows the impact of the cache warmer on our system, on the 99.9% requests latency, really on the long tail of requests latency. That looks nice because we brought the latency down by a third.

Screen Shot 2017-10-10 at 15.43.15.png

What is a task start dying during peak traffic? Imagine that the load balancer is saying “You’re going to handle this” and two seconds later says “I have to reconsider, you’re going to handle that instead”. Again you’re going to watch your system burning on fire, because you’re serving peak traffic and then you’re running close to 100% CPU. That situation causes the load balancer loading and unloading tasks all the time, which is inconvenient. The way we work around this is to make this cache warming a best-effort activity. You can do it if you’re below 50% CPU when you have time to do fancy things, but when you receive peak traffic, you just handle peak traffic without any optimisation made.

Multi-master and multi-tenant outside of Google

The question is: how do we do that in a non-Google setup?
There are plenty of options.

With the new Gerrit release in 2.15, we introduced to a new URL scheme, which includes the project name in the URL. Previously you had gerrit-review.googlesource.com/c/NNN and there was no way to directly know which project this is for and no way to do that load balancing that we just saw.

What we did in 2.15 is just add the project before that, so that extraction for both host and project can be made in a simpler way. You could do the same even before v2.15 but you needed a secondary index lookup, which most opensource load-balancers such as HAProxy or NGINX did not support. And of course, there are lots of products like Google Cloud load balancer, and others that you can use to achieve the same thing.

Wrap-up

We went through a journey where we took OpenSource Gerrit, we added sites selection and got a multi-tenant Gerrit.
Then we took this multi-tenant Gerrit, added replication and obtained multi-master Gerrit.
And then we took that with load balancing and lots of failures and lots of fixes, and we got pretty much the Gerrit that we run at Google, which brings me to the end of this talk 🙂

Q&A

Q: How strategic is Gerrit@Google? Do you have any other code-review systems? If yes, how is used Gerrit vs. the others?

We have another code-review system for internal use only, and Gerrit is used whenever we are doing OpenSource stuff, so for GoLang, Chromium, Android, Gerrit, and whenever the Google Team wants to collaborate with other OpenSource users, or in general with users that are not sitting at Google.

Historically the source at Google was developed in Perforce, and we ported from that to a home-based system called Piper. Around that, we have a tooling ecosystem which is internal. In parallel to that, Google started to do a lot of projects that have nothing to do with the internal search engine and available outside. What we see is that a lot of projects started at Google from scratch were thinking about “what system should we use?”. Many people said: “well, we’re going just to use Git because that’s what we know and we like, ” and when they needed code-review for Git they ended up with us. Gerrit and Git are very popular inside Google.

Q. You have two levels of load balancer. The first one is the location, and the second one is to decide what to do inside the data-center. What about if a location is off? Maybe is not fully off-line but has big problems, or has a very low-percentage of consensus, and some of the locations have not the “latest and greatest” of the repo. Possibly a location that should be “inconvenient for me” actually has the data I want.

You’re talking about replication layer where you have the objects in one location but not in the other. Our replication latency is in the order of seconds, but it may happen that one location is just really slow in getting the objects. That happens from time to time, and we have metrics that says what the replication lag is accounted for. When it exceeds a threshold we just shut the data-center off, which means cut-off the traffic, the data-center will not receive user-traffic anymore but it will still be able to get the replication done, and when the decrease the objects we need to replicate we can send the traffic again.

Cutting off the traffic is happing at the L1 load balancer where we said “don’t send anything there”.

Q. Do all the tasks have the same setup? Or do have a sort of micro-service architecture inside where some of the tasks are more dedicated to this type of operations and other for another type. Serving data from memory in one thing, but calculating diff change is a different type of task.

Not in general. All of our tasks are the same, except for checking access control permissions. We do not go through the whole Gerrit stack but we have only this little task that knows how the project config works and is going to tell us yes or no.

 

 

 

 

Gerrit User Summit 2017 – Sold Out

GerritUserSummit-London-Splash.jpg

The Gerrit User Summit 2017 this year is entirely SOLD OUT, a fantastic result and huge audience increment from last year, coming from 14 different countries.

It is essential for a healthy OpenSource project having a diverse and growing community of users that assures longevity and organic growth of features and ideas.

See you at CodeNode on Monday 2nd of October 8:00 AM and help shape the future of Gerrit Code Review, growing and thriving in the ecosystem of Git Code Review tools.

Gerrit User Summit 2017, 2-3 Oct, London

GerritUserSummit2017-logo.png

New and exciting features are coming for this year Gerrit User Summit, with the launch of Ver. 2.15, NoteDb, high-availability, multi-master and much more.

The Summit will take place for the very first time in Europe, London, the location chosen by the community after a public consultation, the 2nd and 3rd of October at CodeNode (Skills Matter).

There are still a few places available but hurry up and register now at https://gerritusersummit.eventbrite.com.

See below an overview of the topics that will be presented and discussed during the User Summit.

What’s new in Gerrit 2.14.x.

Gerrit v2.14 was released during the last Hackathon in April and has gone through three patch releases. David Pursehouse from CollabNet will give an overview of the new features introduced which would be highly beneficial for all of those who haven’t migrated yet.

Gerrit at Google: Multi-master, Mutli-tenant.

Google is the founder, main contributor and possibly the most advanced user of the Gerrit Code Review: learning from their experience is a unique opportunity to learn and being able to leverage and use the tool at its best.

Patrick Hiesel from Google will go through the insights of their Gerrit Code Review architecture and will provide some of their metrics of scale. In addition to that, he will present some findings from the recent switch of their load-balancing infrastructure and the associated pitfalls encountered.

Google is possibly the only one in the world using Gerrit in a multi-tenant setup, having a unique multi-master installation that serves a constellation of domains and projects, including huge and familiar ones like Android and Chromium.

Standing “on the shoulders of giants” like Google helps a lot in preventing scalability issues as the audience and adoption of Gerrit Code Review grows in large companies: being part of the audience in the talk is a unique opportunity to learn and ask questions directly to the maintainers of their infrastructure.

PolyGerrit: a new UX experience for Gerrit Code Review

Google has invested a lot in reinventing and reengineering the user interface of Gerrit Code Review, which remained mostly unchanged for almost a decade. A new team has been put together in their San Francisco offices with experienced UX developers that leveraged the new Polymer framework of web components.

The result is PolyGerrit, a modern web UX which provides an unprecedented browsing speed and flexible rendering across different devices, including mobile and tablets.

The PolyGerrit Team will be presenting the findings of their user-experience research and show some of the features and insights of the new UX.

Gerrit CI and keeping logs forever.

Gerrit Code Review itself is a large project, involving over 300 developers across the globe and using the most advanced DevOps practices. The CI/CD pipeline has been provided and managed by GerritForge on the https://gerrit-ci.gerritforge.com and Luca Milanesio from GerritForge will present the latest improvements in the pipeline plus an interesting way of collecting and reusing the logs.

Leveraging the logs for identifying the bottlenecks of the CI/CD pipeline is the way to drive improvement. GerritForge leveraged the expertise of his engineers to harvest and organize data and will give it back to the community as powerful dashboards.

Beyond Gerrit.

Gerrit is great. However, it is also quite an important part of a bigger ALM process. Jacek Centkowski from CollabNet will describe how multiple tools can be unified under a single TeamForge umbrella and what are the immediate benefits of it.

What’s coming in Gerrit 2.15

After only four months, we are already close to the v2.15 of Gerrit Code Review, which would be possibly the last one before the step to the v3.0.

Dave Borowitz from Google, principal maintainer of the Gerrit Code Review project, will go through the new features of v2.15 and possibly give a glimpse in what to expect from v3.0.

Mining Gerrit Data to Study Contentious Reviews and Community Evolution

Gerrit Code Review is much more than a tool, it is a way for people working together in companies that are large and mostly distributed across the globe.

Shane McIntosh from McGill University has been running a research lab on this topic. The Software REBELs—a research lab at McGill University—mine code review data to study topics like the impact that code review practices have on software release and design quality. Our more recent work mines code review data to study the reviewing process itself. In this talk, I will describe the results of two empirical studies of data that we collected from the Gerrit instances of the OpenStack project. The first study aims to understand the reviews where reviewers disagree about a patch. The second study follows how the concerns that reviewers raise evolve as the OpenStack community ages and individual reviews accrue experience.

Gerrit Analytics: dashboards, networks, KPI

Gerrit has always been lacking major code analytics features compared to other Git Server tools like GitBlit or GitLab. GerritForge Ltd is filling the gap and adds one important asset to the Gerrit Code Review platform: code review analytics.

We need to harvest and unify the logs and events coming from the different components of the CI/CD pipeline by putting at the center of it the people and teams that are building and discussing the code on Gerrit. The resulting data-lake of information can be later analyzed and correlated to calculate the cycle time of the entire pipeline.

Luca Milanesio from GerritForge will show the new analytics dashboards that are going to be published and provided back to the Team that is developing the Gerrit Code Review project as a precious contribution to the community.

How to extend Gerrit using Scripting Plugins

Gerrit Code Review has a robust set of API that can be used to extend its functionalities and provide a more integrated development workflow for the Teams.

Luca Milanesio from GerritForge will present how to use different scripting tools to extend the capabilities of Gerrit without the need of developing and building a plugin, using Jython, Groovy and Scala.

A new simpler but powerful Gerrit Jenkins plugin

Gerrit Code Review is an essential part of a larger CI/CD pipeline. Most of the times it is used in conjunction with Jenkins, the most popular OpenSource Continuous Integration and Delivery tool.

The integration between Gerrit and Jenkins (Gerrit Trigger Plugin) was developed back in 2010 at Sony and since then has been extended and adopted in thousands of Jenkins installations. However, Jenkins has evolved too and has now a brand new concept and definition of multi-branch pipeline which struggles to be seamlessly integrated with the current Gerrit Trigger Plugin.

Luca Milanesio from GerritForge will present a brand new plugin based on the new Jenkins branch discovery API which works seamlessly with Jenkins multi-branch pipelines and provides a simpler interface with Gerrit by leveraging the new WebHooks.

Diffy with enterprise grade

Since 2012 CollabNet has been working on improving Gerrit integration with TeamForge. Many features have been created to satisfy the needs of enterprise customers. Eryk Szymanski from CollabNet will present features like RBAC, history protection, Git style notifications, quality gates, pull request and code browser which have been implemented on top of vanilla Gerrit.

Q&A with the maintainers

Have you ever wondered why something is working in a certain way? Have you ever wanted to explain any complaint about some parts of Gerrit? Would you give your congratulation to the people that made this project? Would you like to make a feature request or propose new ideas?

This is the moment where you can speak directly face-to-face to the people that are building this project every single day, the Gerrit maintainers.


The event is free for everyone, thanks to the contribution of our sponsors, CollabNet Inc, GerritForge Ltd and Skills Matter Ltd.

SponsorsBanner.png

[Poll] Gerrit User Summit 2017

user-group-crowd.jpg

At the last  User Summit in Mountain View, it has been announced that the Gerrit User Summit 2017 could be hosted in Europe/London rather than the traditional US West Coast location.
The primary rationale behind the proposal is to engage with a wider and more diverse community, who can bring new requirements and needs from of the European Gerrit Community. It would be an opportunity to see more faces and get new ideas.

While the adoption of Gerrit Code Review has been historically high in the USA, the European companies have started using it as the standard tool for their development lifecycle. More and more people from Europe began to participate actively in the mailing list and contributing new ideas and code.

Here is a poll to have your say and express your intention to participate:
https://goo.gl/M7X6rp

The poll will stay open for the next two weeks and then we’ll publish the results.

Thank you in advance for taking the time to respond to the poll and hoping to see you soon at the next Gerrit User Summit 2017.

Luca Milanesio – GerritForge.

Gerrit User Summit 2016 Report

gerritusersummit-2016-google

The 9th edition of the Gerrit User Summit has ended today: a lot of new features and ideas have been presented and experimented in those four days.
Here is a summary of what happened and a description of the main innovations and presentations at the Summit and Hackathon at GooglePlex in Mountain View – CA.

12th-13th of November – The Summit

Gerrit v2.13

David Pursehouse from CollabNet presented what is coming in Gerrit v2.12 and v2.13.
Gerrit v2.13 has been the biggest version of Gerrit ever, with over 2,600 commits from 83 different contributors. That shows that Gerrit is an OpenSource project with a growing Contributors’ Community, getting more diverse over time.

There have been a lot of new features and improvements introduced. The most significant and noteworthy features are …

Git LFS

Ability to store large files outside the Git repository to an externally pluggable repository, such as HDFS or AWS S3.

Gerrit Metrics

New extension point to extract in real-time the performance data of Gerrit’s internal activity and publish them externally to Graphite, ElasticSearch, JConsole or other similar graphing and monitoring tool.

Hooks

There is now the ability to replace the dated hook system with much more powerful in-process-plugins. Existing hooks are then still supported using the “hooks” core plugin. Gerrit is getting more modular and extensible.

Improved review of Merge Commits.

Gerrit can implement effective “merge-requests” lifecycle on top of regular code review. You can now create feature branches exactly as you do with GitHub or GitLab and then submit only the “merge request” as change for reconsideration.

What’s cooking in Gerrit v2.14

PolyGerrit

After eight years, Gerrit GUI needed a complete revamp and modernisation, to compete with all the other web-based Code-Review systems such as GitHub, GitLab, and BitBucket.

Google has allocated a brand new front-end Team, located in San Francisco, to redesign and implement a brand-new user experience. It is much more than a simple rewrite; it is a complete rethinking of how to get the most of your code review functionality with the optimal utilization of space.
The new UX will be more fluid and adaptable to different sized screens, including tablets and even smartphones. It will be then possible to give a usable GUI even when accessing Gerrit “on-the-road.”

Suggested Reviewers

Gerrit is getting smarter and, by using sophisticated heuristics, it can now understand and sort the suggestions of which are the people that are most likely to be interested in reviewing a change. Gerrit allows saving time in browsing for individuals and contributes to the accuracy of the reviews.

New HTML e-mails interaction with Code Reviews

Say goodbye to long ugly diffs in e-mails, and now you can generate nice and tidy e-mails where all the details are carefully formatted for achieving maximum clarity. Additionally, you will be even able to reply in-line to the comments received via e-mail and Gerrit will be able to capture the message back and insert the feedback as code-review messages into the code carefully.

Robot comments

CI and automated systems can contribute comments to the code, without spamming the overall review discussion flow. That contributes to a much cleaner user-interface.
Google uses automated code analysis tools such as Tricium and ShipShape to capture the most common mistakes in coding and event providing the corresponding suggested fixes. All this feedback will be captured and integrated with Gerrit in the change comments GUI. The Change’s author will then be able to review the suggested fix and apply it with a simple click.

The management of robot commits will be fully operational with the new PolyGerrit GUI.

ElasticSearch Indexes

Instead of using the local Lucene Index, Gerrit would use an external ElasticSearch cluster to achieve maximum scalability and share the index across multiple nodes. That is one of the cornerstone feature needed for a multi-master setup.

A lot of time has been spent in making sure that the ElasticSearch support was aligned with the current Lucene implementation, but unfortunately, it took until v2.14 to fill the gap. Still, there are parts of the index not yet implemented in ElasticSearch, such as the on-line reindexing. Hopefully, the v2.14 will see all those gaps resolved and enable a real multi-master shared index setup.

Atomicity with Change-Sets

Basavaraj Karadakal from Juniper Networks presented a real-life implementation of the “submit whole topic” into his organization. We always said that “topics” require some company-wide naming policy to avoid conflicts and unexpected behaviors: Juniper found the solution by introducing a new concept, the “changeset.”

The name isn’t new to people that have been in the SCM long enough to remember IBM Rational ClearCase. Interestingly Basavaraj used that term, possibly to facilitate the understanding and adoption into the company.

The idea behind change set is to have a single change review “leading” the topic submission, stored in a “tracking repository”. One can understand the list of changesets and their status by querying the “tracking repository”. This feature is fascinating because it allows obtaining a full list of “open topics” inside Gerrit, which isn’t currently available as “out of the box” feature.

The overall implementation of change sets is composed of a set of client wrapper scripts and server-side plugin to orchestrate the review status of the changeset based on the review status of each commit included.

We do foresee a lot of feedback in the community around changesets and topics: the discussion started a couple of years ago and everyone agreed that “something was missing” in the current Gerrit topic concept … possibly the “change sets” are the missing piece of the jigsaw to make the “submit whole topic” a feature ready for production use.

Pull Requests in Gerrit

Since v2.13, Gerrit allows full reviews of merge commits. However, the workflow isn’t visible:

  • create a feature branch feature/abc
  • commit changes directly to the feature/abc branch, without requesting any review
  • once the branch is ready for review, merge into its target branch locally and push the merge commit to refs/for/target-branch

Eryk Szymanski, CollabNet, has presented an alternative GUI to make this scenario, typical of GitLib, GitHub, and BitBucket, more accessible for novice users.

The new Pull-Request GUI is based on two components:

  1. REST API, different from Gerrit’s native ones
  2. Pull-Request GUI, not integrated with Gerrit’s GWT or PolyGerrit but available as components of CollabNet’s TeamForge product.

Gerrit Analytics

Luca Milanesio from GerritForge introduced two new plugins in Gerrit 2.14:

  • Kafka stream events: to consume stream events and send them to topics
  • Project analytics: to crunch data from Gerrit repositories to expose aggregated contributors statistics

The vision of Luca Milanesio, GerritForge, is splitting Gerrit statistics into three dimensions:

  • People. What is the list of contributors, maintainers, reviewers? What do they do, how often they commit and which ones are the most active influencers?
  • Projects. How repositories are evolving over time and space and what do they need moving forward regarding storage and performance?
  • Systems. How is Gerrit behaving concerning CPU, Memory, and Network utilization?

The correlation of the metrics collected in these three dimensions allows putting together the KPI that allows understanding better how Gerrit is used and how to improve its current and future performance.
Additionally, you could use them to reward and promote good practices of code-review and fuel continuous innovation across the teams.

In one sentence, Gerrit Analytics will allow to “uncover the hidden value of your Gerrit Code Reviews”.

Plugins for CI Systems

Martin Fick from Qualcomm Innovation Center gave an overview of three new plugins developed for use by CI systems: the batch, manifest, and task plugins, with some modifications to the Gerrit core needed.

The batch plugin provides a mechanism for building and previewing sets of proposed updates to multiple projects/branches/refs that should be applied together. The focus of batch updates tends to be verification (by CI systems). The batch update service provides the tools to build refs by merging changes to temporary “snapshot” refs, which can then be tested extensively, and finally submitted ”as is.”
That represents a different approach to resolving the “topic submission” problem: Martin’s plugin would work for all Gerrit versions from v2.7.

The manifest plugin provides server-side utilities to operate on, and query information about repo manifests (XML) stored in git projects on the current server. This plugin provides APIs to update values in manifests, and to search for manifests with certain values.

The task plugin provides a mechanism to manage tasks which need to be performed on changes along with a way to expose and query this information. Tasks are organized hierarchically, and their definitions use Gerrit queries to define which changes each task applies to, and how to set the status criteria for each task. An important use case of the task plugin is to have a common place for CI systems to define which changes they will operate on, and when they will do so.

Gerrit CI Pipeline

Gerrit has finally a fully-feature CI Pipeline, thanks to GerritForge who invested time and resources to set up the https://gerrit-ci.gerritforge.com web-site.

During the Gerrit Hackathon 2015, Luca Milanesio, Doug Kelly and Khai Do have put together their ideas and skills to introduce an entirely automated and community-driven Jenkins set-up which allows to automatically generate the build jobs from the YAML files archived in the Gerrit-ci-scripts project.

The magic behind the system is that nobody directly controls the definition and execution of the jobs, but it is the collective contribution and code-review of the YAML files that automatically re-configure the Jenkins instance automatically.

Over 30K builds were executed last year, with an average of over 80 builds per day. Thanks to the Gerrit CI it is now possible to discover which plugins are available and stable on different branches of Gerrit and download the pre-built artifacts with a simple click.

What is still missing is an official “distribution site” of the Gerrit plugins; a place where successful plugin builds are installed on real Gerrit instances, tested and released. GerritForge is investing time and money to fill the gap and will provide a premium service called “Gerrit Central” available as an add-on to existing Enterprise Customers. The resulting fixes coming from this process of plugins validation will be contributed back to the community and will be accessible for everyone.

Gerrit Bazel build: a farewell to Buck

From v2.14 Gerrit is moving away from Buck and, after eight months of work of efforts for migrating the build system, David Ostrovsky has finally announced that the build is green and we are ready to move to Bazel, yeah.

Building Gerrit isn’t an easy task, with 35 repositories, 400K Java + 16K JavaScripts lines of code, over 140 dependencies and a complete JavaScript toolchain based on NodeJS. In addition to that, there are over 100 plugins automatically built with our CI system hosted and managed by GerritForge.

Gerrit started initially with Maven, but it was very slow and clumsy. After the London Hackathon in 2013 the project was successfully migrated to Buck, which promised to be much faster and flexible. However, this choice has driven the project build to a swamp of problems and bugs/fixes needed on top of Buck to make the build work.
As a result, a custom version of Buck needed to be built using Ant for potentially every branch of Gerrit development. The Gerrit project has submitted over 100 bugs to the Buck project over the three years of adoption.

Bazel is an OpenSource build tool based on an internal system used in Google and is much more advanced and stable, offering a simpler installation and an advanced sandboxing capability to isolate builds and results.
Differently from Buck, Bazel promises both speed and accuracy, qualities that were not always present at the same time.

During the Hackathon, David posted a milestone change named “Remove Buck build” which is currently under review. We all hope that the change will be merged very soon and give the final farewell to Buck build, and our thanks for having helped and supported us in our journey for over three years.

Gerrit reviews from e-mails

Patrick Hiesel from Google presented a very innovative feature that extends the ability to interact with Gerrit reviews by just answering to e-mails.

The feature is still in the very early stages of development  but the demo showed at the Summit was impressive. When you receive the e-mail notification from Gerrit with the comments of people to your code, you just answer in-line to the e-mail and … the magic happens! Gerrit fetches the e-mail automatically and understands the in-line replies as comments to be added to the reviews.

This new workflow dramatically reduces the cycle time of interactions during the review and represents IMHO the “killer feature” to push Gerrit to the top of the Code Review systems currently on the market.

Let me recap what you have to do currently with Gerrit:

1. Receive a notification e-mail
2. Read the comments
3. Click on the change link
4. Open a browser at the change page
5. Sign-in (assuming that you are using PolyGerrit, it would work on Mobile too)
6. Locate the comment you wanted to reply
7. Click on the comment
8. Click on Reply
9. Enter your Reply
10. Click save
11. Go back to the main change screen
12. Click on Reply
13. Click on Post

… and with the inbound e-mail ingestion in Gerrit it will be:

1. Receive a notification e-mail
2. Read the comments
3. Click on Reply
4. Insert your replies in-line
5. Send

There are issues, though, mostly related to security and trustworthiness of the e-mails as the vehicle for reviews.
We know that e-mails can be SPAMs or can be forged by faking source e-mail addresses. Currently, the prototype shown was only matching the e-mail sender to associate to the reviewer identity, which is fine for the majority of the cases but could raise some concerns regarding compliance.

The reviews coming from e-mails will be appearing with a different style and will be noticeable compared to more secure web-based comments. That would allow judging the comment accordingly and avoid over-reliance on their value.

It has been suggested as well to consider the integration of PGP-signed emails and the matching of the PGP-Key with the one stored in Gerrit for that user: that would be the perfect secure solution to avoid SPAMs or forged e-mails risks. However, Patrick did not commit to implementing the feature at this level for the initial MVP.

Inbound e-mail processing is a feature to keep under the radar as it will cause a wave of adoption of Gerrit because of the speed-up of entire Code Review lifecycle and consequently the time-to-market of the development of projects features.

Keep the conversation flowing with Gerrit at OpenStack

Khai Do, the maintainer of the Gerrit Review at OpenStack, has presented the challenges of managing comments with a large-scale project.

The OpenStack numbers for Code Review are astonishing:

  • Nine servers (1 master + 8 slaves)
  • 1500 repositories
  • 400K changes
  • 20K users
  • 700 events generated every hour

In addition to the scale of the installation, OpenStack allows third-party to connect to Gerrit using stream events and trigger validation builds. This workflow is code-named TCPI at OpenStack. There are currently 180 registered TCPI which generate *a lot* of messages appended to each change under review.

The overload of information associated with a change makes the “discussion flow” hard to follow in Gerrit: each change is swamped by CI messages which make the overall review unreadable.

Khai has developed a plugin that allows avoiding storing all those messages as regular reviews in Gerrit and collect them instead in a relational database. The overall result is then nicely rendered in a different section of the change, making the review discussion visible and flowing again.

The plugin works out of the box on the current version of Gerrit. Well done Khai.

Infinite Gerrit

Haithem Jarraya from GerritForge presented the idea of adopting a much more scalable storage for JGit based on Apache Cassandra. Shawn Pearce experimented the same scenario a few years ago and contributed an initial prototype on GitHub: piggybacking on the same concept, the new experiments are now driven by the need of scaling up Gerrit in space and number of nodes.

Gerrit Multi-Master is getting closer day-by-day, thanks to the joint efforts of all contributors from different points of view:

  • Sessions can be externalized to a flat-file to supports sharing across nodes (Qualcomm contribution)
  • Indexes and Caches can be aligned across nodes through events (Ericsson contribution)
  • ElasticSearch will be soon available as alternate shared indexing engine (Sony & CollabNet contributions)

Haithem now is adding one extra significant piece of the Jigsaw: the ability to have a scalable shared and extensible storage that allows all the master nodes of the cluster to read/write JGit Pack files at maximum speed. Additionally, thanks to Apache Cassandra multi-nodes data sharing and replication. You will be able to dynamically extend and reallocate the size and structure of the storage without impacting the current operations on the Gerrit Cluster.

The Cassandra backend wouldn’t cover the ref-database, which could be implemented with an entirely distributed implementation such as JGit Ketch (still in early development) or a more standard based on Apache Zookeeper.

Accounts and status of NoteDB

The task of migrating all the Code Review meta-data to the Git repository, aka NoteDB, started over two years ago. Edwin Kempin from Google, presented the current status of the implementation, with a focus on the Accounts migration.

In Gerrit v.2.13 there is already a new repository dedicated to the archiving of users, called All-Users. However, some of the information are still stored in Reviewdb, such as the external ids. Accounts will be exposed via REST-API and sensitive information encrypted end-to-end on the repository, using the same technique used for secure.config.

A new “magic ref” concept has been specifically forged for the All-Users use-case: the reference to their branches as refs/users/self. Each user will be able to get its meta-data by only fetching refs/users/self from the All-Users repository; easy isn’t it?

There is still one piece of information that will still stay outside NoteDB: the reviewed flag. The choice was made because of the inherent volatility of the information: NoteDB would have been overkill for a piece of information that lasts a few minutes or maximum hours typically. The default implementation will be H2 but it could be then replaced with something more “multi-master” aware.

What’s left? Gerrit Groups are the very last piece of the jigsaw and then NoteDB will be finally 100% feature complete. Bye bye DataBase, NoteDB is approaching fast!

Zero-Downtime Gerrit Upgrades

Luca Milanesio from GerritForge closed the Summit with a case-study on how GerritHub.io achieved a 100% zero-downtime migration process, with limited disruption to its 10K users.

Apart from Google, the current implementations of Gerrit Code Review around the world do not allow to keep the service running in case of upgrades between major releases. That was true until GerritHub.io recently introduced a well-testing roll-out mechanism inspired by the well known “blue-green” deployment.

The approach adopted by GitHub, in the case of DB upgrades, is to advise the customers up-front and schedule a planned outage of the system. Last time that this happened was last year and the blackout lasted for around an hour. However, GerritHub.io decided to be different and developed a strategy where the old version and the new version can work side-by-side by synchronizing Git repository and reviews, even across different versions of the DB schema and index.

The “preparation phase”, or Step-1 as Luca described it, may take even days or weeks to complete because relies on a low-bandwidth mirroring to avoid service disruption on the current user-base. Once the two instances are aligned, the subsequent steps will take only a few minutes.

During the “flipping” of the blue (old release) with the green (new release) instance, the users will notice that the repositories become read-only and receive a courtesy message saying “Upgrade is in progress, all projects are read-only. Please retry a few minutes later”.

The read-only “service degradation” would last only for a few minutes (2/5 minutes) and wouldn’t cause a service disruption for all the other users that are fetching, cloning or just browsing the reviews.

The Gerrit Hackathon

gerrithackathon-2016

Even though this year we had only two “hacking days,” the overall productivity has been remarkable: 158 changes created, with 82% of them merged during the Hackathon.

Bazel Build

Authors: David Ostrovsky, Luca Milanesio, Edwin Kempin, Han-Wen Nienhuys

Changes: https://gerrit-review.googlesource.com/#/q/project:gerrit+after:2016-11-10+before:2016-11-17+message:bazel

The summary of the changes is perfectly represented by David’s announcement to the Gerrit mailing list:

“I’m pleased to announce, that we are moving from Buck to Bazel build tool [1].
I’m working for a while on the transition now, and we are reaching the feature
parity for Bazel build toolchain.

I think that Bazel is a better tool for us and provides the following advantages:

* all actions are hermetic by design (no arbitrary python code execution)
* bootstrapping doesn’t require Ant
* can be installed on the main operating systems, including Windows
* better support for re-using of build rules across local and remote repositories
* switching to different branches wouldn’t mean rebuilding build tool
* extensible rules
* failed tests are not cached
* flaky tests are re-run multiple times
* error prone static analyzer integration out of the box

I would like to thank Shawn for supporting this project, and reviewers/contributors:
Han-Wen Nienhuys, David Pursehouse and Damien Martin-Guillerez from Bazel team.

Also big thanks to Luca and GerritForge for helping out with the CI and running a *lot* of tests.
Here is my related talk from Gerrit User Summit 2016: [2].

[1] https://bazel.build
[2] http://ostrovsky.org/gerrit/bazel-build-gerrit&#8221;

Allow plugins to define “has” search operands

Author: Martin Fick

Changes: https://gerrit-review.googlesource.com/#/c/91514/

Plugins can define new search operands to extend change searching.
Plugin methods implementing search operands (returning a
`Predicate<ChangeData>`), must be defined on a class implementing
one of the `ChangeQueryBuilder.ChangeOperandsFactory` interfaces
(.e.g., ChangeQueryBuilder.ChangeHasOperandFactory). The specific
`ChangeOperandFactory` class must also be bound to the `DynamicSet` from
a module’s `configure()` method in the plugin.

The new operand, when used in a search would appear as:
operatorName:operandName_pluginName

A sample `ChangeHasOperandFactory` class implementing, and registering, a
new `has:sample_pluginName` operand is shown below:

 @Singleton
 public class SampleHasOperand implements ChangeHasOperandFactory {
 public static class Module extends AbstractModule {
 @Override
 protected void configure() {
 bind(ChangeHasOperandFactory.class)
 .annotatedWith(Exports.named("sample")
 .to(SampleHasOperand.class);
 }
 }

@Override
 public Predicate<ChangeData> create(ChangeQueryBuilder builder)
 throws QueryParseException {
 return new HasSamplePredicate();
 }

Zoekt

Author: Han-Wen Nienhuys

Changes: https://gerrit-review.googlesource.com/#/q/before:2016-11-17+after:2016-11-10+project:zoekt

Han-Wen has officially announced at the Gerrit Hackathon, the general availability of the Zoekt code search engine.

“Hi there,

I’d like to announce “zoekt”, a search engine for source code I wrote over the last month or so. It was made to work well with (large) Git based repositories, so I think some of you may find it useful.

If you’re interested in code search, please try it out, and let me know about your experiences.

http://github.com/google/zoekt
http://gerrit.googlesource.com/zoekt

I didn’t do any work to try and integrate it with Gerrit/Gitiles, but
if someone wants to invest effort in making that happen, I’d be
happy to review CLs. “

ElasticSearch Index

Authors: Hugo Ares, Dariusz Łuksza

Changes: https://gerrit-review.googlesource.com/#/q/before:2016-11-17+after:2016-11-10+project:gerrit+message:elasticsearch

Small steps towards the goal of having ElasticSearch integrated as Gerrit secondary index:
– On-line reindexing (in-progress)
– Accounts indexing (in-progress)

Plugin examples

Author: Martin Fick

Changes: https://gerrit-review.googlesource.com/#/q/before:2016-11-17+after:2016-11-10+project:plugins/examples

Gerrit cookbook has become over the years a mix of different type of plugins functionality. That is good and bad at the same time. A typical “cookbook” should have a set of recipes for obtaining a final dish ready to eat … whilst our was more like a mixed salad with a bit of everything.

Martin started the noteworthy task of reorganizing the cookbook in a set of self-contained example, hopefully,  easier to manage and with a clearer understanding.

Cassandra Repository Manager

Author: Haithem Jarraya

Changes: https://gerrit-review.googlesource.com/#/q/before:2016-11-17+after:2016-11-10+project:libs/modules/repomanager/cassandra

Haithem started a new change (still in draft at the moment) for plugging an alternative JGit DFS implementation to the Gerrit repositories. It is the very first time that such a challenging goal is attempted outside Google data-centers and BigTable/GFS infrastructure. If we succeed in completing the implementation and demonstrating that the resulting architecture is more robust and scalable than the local file-system, it would be a first 100% OpenSource distributed JGit backend ever.

Gerrit LibModules

Author: Luca Milanesio

Changes: https://gerrit-review.googlesource.com/#/q/message:libModule+after:2016-11-10+project:gerrit

Luca discussed how to “plug” an external repository manager into Gerrit: it is something “similar” to a plugin but more fundamental to the system. Differently from a plugin, it needs to satisfy the following requirements:

  • Loaded at start-up time only
  • Cannot be unloaded
  • Must be injected in the primary Gerrit sys injector

It is not the first time that we need to introduce a “modular piece of software” which isn’t part of Gerrit Code Review:

  • Java CryptoProvider – BouncyCastle
  • Encrypted secure store for gerrit.config
  • Servlet filter for authentication and SSO
  • JDBC Interceptor for JavaMelody
  • Multi-hosted Git repository directories

Instead of creating “yet another exception” in Gerrit, we discussed amongst the hackathon members and agreed to come up with a new concept: LibModules.

The rationale behind the name is:

  • They need to be loaded from the same Gerrit class path from /lib
  • They provide extra Guice modules to be added to the sys injector at start-up

The configuration of a set of LibModules in Gerrit is straight-forward:

[gerrit]
installModule = com.mycompany.MyModule
installModule = com.googlesource.gerrit.modules.FancyModule

Now that the change is merged and the new concept has become part of Gerrit, we are expecting a growing number of LibModules to become available.

Removal of references to LocalDiskRepositoryManager

Authors: Hugo Ares, Shawn Pearce

Changes: https://gerrit-review.googlesource.com/#/q/before:2016-11-17+after:2016-10-11+project:gerrit+message:LocalDiskRepositoryManager

Shawn and Hugo started removing the explicit references in Gerrit to the local filesystem for resolving Git repositories. Those changes are needed to fully support an alternative implementation of a Repository Manager not based on a local filesystem anymore.

DynamicBeans for Gerrit Plugins

Author: Martin Fick

Changes: https://gerrit-review.googlesource.com/#/q/before:2016-11-17+after:2016-11-10+project:gerrit+message:DynamicBeans

Until now it has been possible to create new SSH commands or REST API through plugins, which is great but sometimes not enough. Martin came up with a use-case where they need to rather extend the current core commands, without having to necessarily change its code. How can you satisfy this use-case with plugins? Martin has the solution and is called “DynamicBeans”.

It is definitely an interesting use-case that is going to trigger a lot of interest and discussions amongst the Gerrit Community.

Analytics plugin

Author: Luca Milanesio

Changes: https://gerrit-review.googlesource.com/#/q/after:2016-11-10+project:plugins/analytics

One of the things that people are missing in Gerrit, when comparing it to GitHub, is the overall repository statistics:

  • List of contributors with number of commits / adds / remove
  • Project’s statistics of number of commits over time

The new analytics plugin extends the Gerrit project’s SSH and REST API to expose aggregated data extraction. It is extremely fast compared to GitHub because it works purely in streaming and doesn’t actually load the data in memory. The amount of data generated by the analytics can be huge and is intended to be used by an external analysis and dashboard system. Differently from the typical Gerrit’s API, the returned result has one item per line instead of a big unique JSON object. The rationale is that to generate and process a set of JSON elements in streaming, you need to make each record self-contained and processed independently. That would allow as well batch-processing with a parallel system, without having to parse the entire file content all at once.

Example of how to extract the list of contributors out of a Gerrit project via SSH:

$ ssh -p 29418 admin@gerrit.mycompany.com analytics contributors

{"name":"John Doe","email":"john.doe@mycompany.com","num_commits":1,"commits":[{"sha1":"6a1f73738071e299f600017d99f7252d41b96b4b","date":"Apr 28, 2011 5:13:14 AM","merge":false}]}
{"name":"Matt Smith","email":"matt.smith@mycompany.com","num_commits":1,"commits":[{"sha1":"54527e7e3086758a23e3b069f183db6415aca304","date":"Sep 8, 2015 3:11:23 AM","merge":true}]}

Wrap-up and Summit in 2017

It has been a very intense and revolutionary Gerrit User Summit & Hackathon this year, with a lot of new faces, new contributors and new features coming in the next forthcoming releases.

At the summit, Shawn Pearce and Luca Milanesio announced the proposal to organize the event in Europe for next year, possibly in June/July or Oct/Nov 2017. The reason for changing the location is to attract the feedback from more Gerrit Users that have not been heard so far and increase the participation of the community in the future of the of Project.

The summit has been already endorsed by the major organizations driving the Gerrit Code Review Project and other very active sponsors of the OpenSource Community, including CloudBees, CollabNet, eSynergy, Google, GerritForge, SAP and SkillsMatter which has offered to host the event at CodeNode.

Shawn Pearce, the Gerrit Code Review project founder, will send in the next few weeks a Survey to capture the opinion of this proposal from the Gerrit Community.

See you in 2017 for new and amazing features coming on Gerrit Code Review.

Luca Milanesio – GerritForge.

 

 

 

 

 

 

Gerrit User Summit & Hackathon 2015

Gerrit User Summit at Google Mountain View – CA

Gerrit user Summit 2015

Exciting times this year at the Gerrit User Summit and Hackathon 2015: the major contributors and players of the Gerrit community shared experiencing, opinions and news in an intense 7-days event in Google – Mountain View – CA.

The User Summit

The first two days have seen the User Summit at the centre of the stage: scalability, scalability and scalability have been the Leitmotif of the discussion.
GerritHub.io started the saga with the astonishing numbers of the two years growth:

  • 6.5K users (+580%)
  • 41K changes (+1700%)
  • 16K projects (+530%)
  • 500GBytes (+500%)

The two years of live production experience with users coming from GitHub have highlighted problems (replication, repositories sharding, multi-master) and possible solutions, ranging from Gerrit Virtual Private Hosting to the on-premises deployment integrated with BitBucket and GitHub:Enterprise.
Ericsson continued with a very detailed description of how Gerrit is used as “Enterprise Washing Machine” of all code that goes through the development pipeline: scalability and control are the fundamental keywords that were repeatedly mentioned and enforced.
The replication across sites have been massively improved regarding performance and stability since the introduction of Git/HTTPS as the main protocol for replication, as previously advised by GerritForge in 2014.
The first day ended with Google smashing out everyone his hypersonic numbers:

  • 1.6M changes / 3.8M patch-sets
  • 240 Gerrit virtual nodes
  • 2.5M repositories

The second day was all projected on the future of Gerrit, with three very interesting features coming soon.

Gerrit 3.0 and NoteDB

Dave Borowitz presented the status of the replacement of Gerrit DBMS
with a 100% open standard solution based on commit notes, implemented by all OpenSource and Commercial “flavours” of Git. The solution will allow interoperability with other code-review implementations (e.g. Phabricator) and fully enable reviews replication and off-line operations.
The new DBMS-free version of Gerrit will be called the Ver. 3.0 and it will be the next version after current Gerrit master (Ver. 2.13) gets released. released.

Gerrit PluginManager: one plugin to rule them all

Luca Milanesio presented a new vision on how to discover and install plugins on the Gerrit platform: Gerrit PluginManager. We should not bundle more and more plugins with Gerrit, which eventually would lead to an explosion of the WAR file-size. You can now install only a “plugin-manager” which then guide you through the “one-click” set-up of all the others.
Differently from similar solutions such as Jenkins Update Centre, the Gerrit PluginManager is based on the live status of the plugins and their compatibility with Gerrit: as soon as one plugin gets patched and successfully compiled with a version of Gerrit, it will automatically be listed and made available by the PluginManager. Additionally GerritForge will provide a list of certified and guaranteed plugins that have been successfully tested with Gerrit.

PolyGerrit, the new web-component UX for Gerrit.

Andrew Bonventre is the Googler that is driving the transformation of Gerrit UX to the new Polymer platform, however for personal reasons he was not able to attend the Summit. Dave Borowitz replaced him and unveiled that will be the (very) near future of Gerrit UX: no more GWT and complex and black magic transforming Java in obscure JavaScript, the future is coming through web-components, a new emerging and promising standard for the HTML and modern web-browsers.
Despite the UX being at very early stages, Dave was able to showcase a fully functional list of changes and search box powered by Polymer and calling Gerrit REST-API, really fast and promising!

Gertty, the text-only Gerrit Code Review.

On the complete opposite side, why not using Gerrit from an 80×25 text-only console? Gertty is an astonishing 100% char-based console, all based on Gerrit REST-Api and a local SqLite local DB for caching changes. Allows complete off-line operations and synchronisation with Gerrit changes. Productive and effective while you are on the go.

CollabNet and Gerrit tuning cheat-sheet

CollabNet presented a useful four pages brochure to guide through the tuning of a Gerrit set-up for a small, medium or large installation. Based on their experience of running TeamForge SCM, the commercial fork of Gerrit Code Review based on their existing TeamForge ALM proprietary solution, they have been able to experiment the Gerrit default settings and learn how to adjust them to leverage the full power of your setup.
The audience appreciated the effort and encouraged CollabNet to post all the findings as Gerrit reviews to get the code-base better and improve the default settings of the set-up.

Gerrit and Jenkins Workflow dance together with Docker.

Cloudbees presented a very effective demo on how to use Gerrit and Jenkins Workflow plugin to implement a real-life Continuous Delivery Pipeline. The presentation leveraged the use of Docker images as previously presented by Stefano Galarraga in his “Gerrit and Jenkins Continuous Delivery Pipeline for BigData” talk. They both explained and showed how Code Review is key in implementing a successful and smooth code validation and roll-out. Stefano’s presentation made use of the new exciting feature of “topic submission” that enables the grouping and commit of multiple changes across repositories.

The Hackathon, topics and improvements.

Gerrit Hackathon 2015

A lot of code have been pushed during the hackathon:

  • 400 changes merged
  • 3 Gerrit version released

Gerrit metrics

Surely the hot-topic of the hackathon has been the introduction of a new pluggable metrics engine in Gerrit, currently half-merged in master branch. Gustaf demoed on how now is possible to use standard tools such as Graphite and JMX console to extract, display and graph the most relevant Gerrit metrics in real-time. This is similar to what the JavaMelody plugin was providing, with the added value of getting the data outside the Gerrit JVM and analyse with greater detail and a standard monitoring platform.

PolyGerrit

Gerrit master has been officially upgraded to allow the development of the new Polymer-based UX for Gerrit, code-named “PolyGerrit”. Gerrit master will need from now on the installation of NodeJS during development. This is needed for building and packaging the “vulcanised” version of Gerrit UX which contains the basic components of the user interface. At the moment the only thing that will be visible is the demo of list of changes presented by Dave Borowitz at the User Summit, however new changes are coming over and Google announced that they are targeting Q4-2015 for a first internal release of the new UX.

GerritForge CI Verifier

As Gerrit 3.0 will be completely revamped in terms of reviews persistence, the community pushed for having a stricter changes validation on both the old DBMS and new NoteDB based persistence. GerritForge extended the use of the CI system (https://gerrit-ci.gerritforge.com) to cover the validation of every change / patch-set that will be uploaded to Gerrit from now on. This is a substantial improvement on the Code Review workflow of Gerrit itself and will hopefully contribute to a stable and solid Ver. 3.0 release next year.

Externalisation of Gerrit Hooks and Events to plugins

Qualcomm worked at completing the externalisation of Gerrit hooks and stream events into plugins. This change will allow to plug different events providers depending on the type of Gerrit set-up, single node or multi-master. One more important step towards an OpenSource implementation of Gerrit Multi-Master.