Gerrit User Summit LIVE!

Screenshot 2019-10-16 at 06.46.06

The Gerrit User Summit 2019 is going live and allows anyone to join and participate from across the world.

There are only 30 days left for the Gerrit User Summit 2019, the 12th annual event of the Gerrit Code Review community. It is the year of the records, with Gerrit reaching its largest audience ever in its 11 years of history:

  • Over 120 seats
  • People coming from 27 countries
  • 2 major dates and locations, in Sweden and in the USA
  • 20 talks and presentations
  • All seats sold out 2 months before the event

This is also a historical moment for the community because, for the first time since 2011, the JGit and Gerrit contributors will get together and talk to each other face to face, strengthening the cooperation between the two projects.

Do not miss the event, go live

We have received an enormous amount of requests to join the event on-site in Sunnyvale, much more than any previous year: the event was sold out on Eventbrite 2 months before the starting date.

GerritForge has then decided to invest further funding in sponsorship to organise a full live coverage of the event.

How to participate?

GerritForge has launched a new live event broadcasting site, https://live.gerritforge.com.

Watching the event will be FREE OF CHARGE and without adverts, thanks to the sponsorship by GerritForge. To assure the maximum quality of the video, there is a limit of on-line watchers and a pre-registration is needed.

  1. Go to https://live.gerritforge.com
  2. Click on the “Register to Watch” orange button
  3. Enter your full name, e-mail, company name and country of origin
  4. Click “Register to Watch” green button on the bottom of the page

The live event will allow remote attendants to ask questions and interact with the audience in Sunnyvale: it is going to be truly interactive and useful for the whole JGit and Gerrit community.

What to expect from the Sunnyvale event?

The Sunnyvale event includes a huge number of innovations on the JGit and Gerrit projects.

  • The introduction of the Git ref-table for repositories with huge number of refs
  • Support for Git protocol v2 in Gerrit
  • Git / Gerrit plugin for Gatling, for generating consistent end to end and load tests on Gerrit
  • Zuul support for the new Gerrit’s Checks CI integration
  • Introduction of Gerrit Code Review Analytics for the Android open-source project
  • Frictionless and zero downtime upgrades for Gerrit
  • and many more talks and presentations

Your last chance to attend, reserve your live spot now

There are brand-new ways this year to get in touch and be part of the Gerrit User Summit 2019.

Reserve your live spot today by registering at https://live.gerritforge.com and be part of this record event for the JGit and Gerrit Code Review community.

Luca Milanesio
Gerrit maintainer, release manager and ESC member

 

Gerrit User Summit at Volvo Cars

gerrit-user-summit-volvo-cars

The 2019 is a year of the Summit innovations

The Gerrit User Summit 2019 can definitely be defined as truly innovative in its format and audience.

For the first time in the Gerrit history, the Summit is split into two parts. Volvo Cars have hosted the first in Gothenburg (Sweden) while the second will take place from the 11th to the 17th of November at GerritForge Inc. HQ in Sunnyvale, CA (USA).

The Summit has been repeated on both sides of the Atlantic: the European and US communities come from different background and have different needs. The Gerrit Code Review Community is global and is willing to share experiences and receive feedback from both sides.

A truly open Gerrit Hackathon

We are also innovating on the Hackathon perspective, with three new elements:

  1. The Hackathon is now open to everyone, including the people that have never contributed to Gerrit before. Experienced maintainers have paired with newbies to guide through the very first contributions.
  2. The Hackathon at Volvo Cars has been 100% focused in triaging the massive backlog of open issues and fixing as many bugs as possible for the latest three supported branches: stable-3.0, stable-2.16 and stable-2.15.
  3. The OpenStack and Gerrit communities finally have met and started talking and interacting more closely.

Read the full story on gerritcodereview.com/news.html

The full summary of the event has been published on the Gerrit Code Review project news, read what happened in Gothenburg and, if you are in the USA, do not miss the next forthcoming Gerrit User Summit USA in Sunnyvale.

Hurry up as the seats are running out, REGISTER NOW to avoid missing the event.

Luca Milanesio (GerritForge Ltd)
Gerrit Maintainer, Release Manager, ESC member

Gerrit User Summit 2019

gothenburg-sunnyvale

The Gerrit User Summit 2019 is approaching fast, with new exciting features and a brand-new Gerrit v3.0 release to present and discuss together.

The event is FREE but you need to register in advance for the Gerrit User Summit 2019 on Eventbrite.

One Summit, two events

The Gerrit User Summit & Hackathon is composed of two different events and locations, one in Sweden (Europe), hosted and sponsored by Volvo Cars, and another in California (USA) in the new GerritForge Inc. HQ. Having two separate events in two different quarters will allow most of the community around the globe to attend and share their experience and ideas.

Hackathon open to new contributors

The first part of the event is a 5-days Hackathon reserved for the current Gerrit contributors and maintainers plus anyone that is willing to start contributing to the platform. Differently, from the previous years, the community is now welcome even people that have not contributed to Gerrit before but they are willing to do so.

It is a fantastic opportunity for people to join, work side-by-side and pair with the Gerrit maintainers for a whole week. It can be a unique opportunity to implement the features that you always wanted to see in Gerrit and learning how we develop and review our changes.

The Summit

The usual 2-days Users Summit after the Hackathon is opened to all the members of the community or who is willing to adopt Gerrit Code Review in their development process in the near future.

This year there are a number of exciting news:

  • The introduction of an official Gerrit Community Process with an Engineering Steering Committee and Community Managers
  • Gerrit v3.0 and the full migration to NoteDb and PolyGerrit
  • The multi-site plugin goes OpenSource for allowing anyone to run multiple masters on different sites

The full schedule of the event is available on the Gerrit User Summit 2019 site.

Proposing a new talk

More talks and customer stories are scheduled and, if you have something to tell to the rest of the community, you can submit your talk by creating a change and push to Gerrit Summit 2019 repository:

  1. Open the repository commands age at https://gerrit-review.googlesource.com/admin/repos/summit/2019,commands
  2. Click on “CREATE CHANGE”
  3. Select “master” branch, put your description and create the new change
  4. Click on “Edit” on the top-right of the page
  5. Click on “Open” in the mid toolbar and open the page you would like to edit.

For proposing a new session, you need to add one file with the name of your talk into the sessions folder, using the template.md as an example.

Where and when

Volvo Cars HQ at their HQ in Gothenburg (Sweden)

  • 24-28th August 2019 – Gerrit Hackathon Europe
  • 29-30th August 2019 – Gerrit User Summit Europe

GerritForge Inc HQ in Sunnyvale CA (USA)

  • 11-15th November 2019 – Gerrit Hackathon USA
  • 16-17th November 2019 – Gerrit User Summit USA

Thanks to our sponsors

I would like to thank Volvo Cars and Nicholas Mucci for hosting, sponsoring and organizing the Gerrit User Summit Europe in Gothenburg (Sweden) and GerritForge for hosting and sponsoring the events in both Europe and the USA in Sunnyvale CA (USA).

Luca Milanesio
Gerrit Code Review Maintainer and Release Manager
Member of the Engineering Steering Committee

 

 

 

 

Gerrit: OpenSource and Multi-Site

One more recording from the Gerrit User Summit 2018 at Cloudera in Palo Alto.

Luca Milanesio, Gerrit Code Review Maintainer and Release Manager, presented the current status of the support for multi-master and multi-site setups with the standard OpenSource Components, developed by GerritForge and the Gerrit Code Review Community.

Introduction

The focus of this talk is sharing with you one experience that we did with the Gerrit server that we maintained, GerritHub.

First of all, I’m just going to tell you how we went through the journey from a single master-slave installation back in 2013 to a fully multi-site setup across two continents.

The evolution of GerritHub to multi-site

GerritHub was born in November 2013. The idea was straightforward. It was just an idea on how to take a single Gerrit server and put the replication plug-in to push to GitHub.

To implement a good and scalable and reliable architecture, you don’t need to design everything up front. At the beginning of your journey, you don’t know who your users are, how many repos are going to create, what the traffic looks like, what the latency looks like: you know nothing.

You need to start small, and we did back in 2013, with a single Gerrit master located in Germany, because we had no idea of where the users would have come from.

Would the people in Europe like it, or rather the people in the U.S. like it, or again the people in China like it? We did not know. So we started with one in Germany.

Screenshot 2019-03-03 at 00.07.01

Because we wanted to make a self-service system what we did was very simple: a simple plugin called, “The GitHub plug-in”. That was just a wizard to add an entry in the replication config.

You have Gerrit incoming traffic, then you configure replication, plugin and eventually push to GitHub. The only complicated part here is that if you do it as a Gerrit administrator you have to define these remotes in the replication.config but you can express it in an optimized way. On a self-service system, you’ve got 1000s of people then will create 1000s remotes automatically. Luckily, the replication plugin works very well and was able to cope with it very well.

Moving to Canada

Then we evolved. The reason why we changed is that people started saying ‘Listen, GerritHub is cool, I can use it, works well in the morning. Why in the afternoon is so slow?“. Uh oh.

We needed to do some data mining to see precisely who was using it, where they were coming from, and what operations they were doing. Then we realized that we had chosen the wrong location because we decided that we wanted to put the Gerrit master in Germany, but the majority of people are coming from the USA.

Depending on how the backbone between the Atlantic Ocean was performing, GerritHub could be faster or slower. One of the complaints that they were saying is that in the morning, GitHub was slower than GerritHub, but in the afternoon it was exactly the opposite.

We were doing some performance tuning and analyzing the traffic, and even when people were saying that it’s very slow, actually GerritHub was a lot faster than GitHub in terms of throughput. The problem was the number of hops between the end user and GerritHub.

We decided that we needed to move from Germany to the other side of the Atlantic Ocean. We could have done to move the service to the USA but we decided to go with Canada because the latency was precisely the same as hosting in the USA but less expensive.

Screenshot 2019-03-02 at 01.19.34

What we could have done is just to move the Master from Germany to the other side of the Atlantic Ocean, but because, from the beginning, we wanted to give a service that is always available, we decided to keep both zones.

We didn’t want to have any downtime, even in this migration. We wanted, definitely, to do one step at a time. No changes in releases, no changes in configurations, only moving stuff around. Whenever you change something, even if it’s a small release change, you change the function, and that has to be properly communicated.

If we were changing data center and version, when something goes wrong, you would have the doubt of what it is. Would it be the new version that is slower or the new data center that is slower? You don’t know. If you change one thing at a time, it must be that thing that wasn’t working.

We did the migration in two steps.

  • Step-1: The Gerrit master in Germany, still the replication to GitHub, and the new master in Canada was just one extra replication end.

    The traffic was still coming on this side of the master, but it was replicated to both Canada and the other GitHub. Then, when that was stable, so we were doing all the testing, the other master was used as it was at Gerrit slave, but was not a slave, all the nodes were master, with just a different role.

  • Step-2: Flip the switch the Gerrit master is in Canada. When replication was online and everything was aligned, we have put a small read-only plug on the German side, which was making the whole node read-only for a few minutes, to give time to the last replication queue to drain.When the replication queue was drained, we flipped the switch, when it was going to the new master it was already read/write.

The people didn’t change their domain, didn’t notice any difference, apart from the much-improved performance. The feedback was like “Oh my, what have you done to GerritHub since yesterday? It’s so much faster. It has been never so fast before.
Because it was the same version, and we were testing in parallel for quite some time, nobody had a single disruption.

Zero-downtime migration leveraging multi-site

But that was not enough, because we wanted to keep Gerrit always up and running and always up to date with the latest and greatest version. Gerrit is typically released twice a year; however, the code-base is stable in every single commit.
However, we were still forced to do the ping pong between the two data centers when we were doing our roll out. It means that every time that an upgrade was done, users had a few minutes of read-only state. That was not good enough, because we wanted to release more frequently.

When you upgrade Gerrit within the same release, let’s say between 2.15.4 and 2.15.5, the process is really straight forward, because you just replace the .war, restart Gerrit, done.

However, If you don’t have at least two nodes on either side, you need to ping pong between the two different data centers, then apply the read-only window, which isn’t great.
We started with a second server on the central node so each node can deal with the entire traffic of GerritHub. We were not concerned about the German side, because we were just using it as disaster recovery.

Going multi-site: issues

We started doubling the Canadian side with one extra server. Of course, if you do that with version v2.14 which problem do you have?

  • Sessions. So, how do you share the sessions? If you login into one Gerrit server, you create one session, then you go to the other and you don’t have have a session anymore.
  • Caches. That is easier to resolve, you just put the TTL of the cache to a very low value, put some stickiness, you may sort this out. However, cache consistency is another problem and needs to be sorted.
  • Indexes are the very painful one, because, at that time there was no support for ElasticSearch. Now things are different, but back in 2017, it wasn’t there.
    What happens is that every single node has its own index. If an index entry is stale, it’s not right until someone is going to re-index.

The guys from Ericsson were developing a high availability plugin. We said instead of reinventing the wheel, why don’t we use a high availability plugin? We started rolling it out to GerritHub and actually, the configuration is more complex and looks like this one.

So, imagine you’ve got in Canada you’ve got two different masters, still only one in Germany. They align the consistency of the Lucene index and the cache through the HA plugin and they still use the replication plugin.

How do you share the repository between the two? You need to have a shared file system. We use what exactly the same used by Ericsson, NFS.

Screenshot 2019-03-02 at 01.21.36

Then, for exposing the service we needed HAProxy, not just one but at least two. If you put one HAProxy, you’re not HA anymore, because if that HA proxy dies, your service goes down. So, you have two HAProxy, they must have a cross configuration’s, it means that both of them, they can redirect traffic to one master or the second master, it’s not one primary and the second backup: they have exactly the same role. They do exactly the same thing, they contain exactly the same code, they’ve got exactly the same cache, exactly the same index. They’re both running at the same time. They’re both accepting traffic.

This is something similar to what Martin Fick (Qualcomm) did, I believe, last year, with the only difference that they did not use HAProxy but only DNS round-robin.

Adoption of the high-availability plugin

Based on the experience of running this configuration on GerritHub, we started contributing a lot of fixes to the high-availability plugin.

A lot of people are asking “Oh, GerritHub is amazing. Can you sell me GerritHub?”. I reply with “What do you mean exactly?”

GerritHub is just a domain name that I own, with Gerrit 2.15, plus a bunch of plugins: replication plugin, GitHub plugin, the high-availability plugin (we use a fork), the web session flat file and a bunch of scripts to implement the health check.

If you guys want to do your own, the same configuration, you don’t need to buy any commercial product. We don’t sell commercial products. We just put the ideas into the OpenSource community to make it happen. Then if you need Enterprise Support, we can help you implement it.

The need for a Gerrit disaster-recovery site

Then we needed to do something more because we had one problem. OVH had historically been very reliable, but, of course, shit happens sometimes.
It happened one day that OVH network backbone was down for a few hours.

That means that any server that was on that Data-Center was absolutely unreachable. You couldn’t even connect to them, you couldn’t even check their status, zero. We turned the traffic to the disaster recovery side, but then we faced a different challenge because we had only one master.
It means that if something happens to that master, I don’t know, a peak, or whatever, is becoming a little bit unhealthy, then we are going to have an outage. We didn’t want to risk to have an outage in that situation.

So we moved to Germany with two servers, OVH with two servers, and afterward, we migrated to Gerrit v2.15 and NoteDb.

Your disaster recovery side is never really safe until you are going to need and use it. Then use it all the times, on a regular basis. This is what we ended up to implement.

We have now two different data centers, managed by two different cloud providers. One is still OVH with Canada, and the other is Hetzner in Germany. We still have the same configuration, HA plugin over a shared NFS, so this one is completely replicated into the disaster recovery site, and we are using the disaster recovery continuously to make sure that is always healthy and aligned.

Leverage Gerrit-DR site for Analytics

Because we didn’t want to serve actual user traffic on the disaster recovery site, because of the synchronization lag between the two sides, we ended up using for all the data mining activities. There is a lot of things that we do on data, trying to understand how our system performs. There is a universe of data that typically either never looked at or you don’t really extract and process in the right way.

Have you ever noticed how much data Gerrit generates under the logs directory? A tremendous amount of data and that data tells you exactly the stories that you want to know. It tells you exactly how Gerrit works today, what I need to do to makes sure that Gerrit will work tomorrow, how functions are performing for the end users, if something has blown up, it’s all there.

We started really long ago to work on that DevOps Analytics space, and we started providing that metrics and insights data for the Gerrit Code Review project itself and reporting it back to the Gerrit Code Review project through the service https://analytics.gerrithub.io.

Therefore we started using the disaster recovery site for analytics traffic because if I do an extraction and processing of my data on my logging, on my activities, is there really a need for an analysis of the visit of 10 seconds ago or not? A small time lag on data doesn’t make any difference from an analytics’s perspective.

Screenshot 2019-03-03 at 00.19.54.png

We were running Gerrit v2.15 here, so the HA plugin needed to be radically different from the one that it is today. We are still massively on the HA plugin fork, but all the changes have been pushed for review on the high availability.

However, the solution was still not good enough because there were still some problems. The problem is that the HA plugin within the same data center relies on the shared file system. We knew within the same data center that was not a problem. But, what about creating an NFS across data-centers in different continents? It would never work effectively because of the latency limitations.

We then started with a low tech solution: rely on the replication plugin for the Git data in the repository. Then every 30 minutes, there was a cronjob that was checking the consistency between the two and then does a delta re-index between the different sites.

Back in Gerrit v2.14, I needed to do as well a database export and import, because contained the reviews.

But, there was also the timing problem: in case of a disaster occurred, people will have to wait for half an hour to get the data after re-index.
They would have not lost anything because the index can be recreated at any time, but the user experience was not ideal. And also, you’ve got the DNS related issues for going from one zone to the other.

Sharding Gerrit across sites

First of all, we want to leverage the sharding based on the repository, available from Gerrit 2.15, which include the project name in each page or REST-API URLs. That allows achieving basic sharding across different nodes even with a simple OpenSource HAProxy or other Workload Balancer, without having the magic of the Google intelligent Gerrit/Git proxy. Bear in mind this is not pure sharding, because all the nodes keep on having all the repositories available on every node. HAProxy is going to be clever and based on the project name and action, will use the most appropriate backend for the reads and writes operation. In that way, we are making sure that you never push on the same repository on the same branch from two different nodes concurrently.

Screenshot 2019-03-02 at 01.23.16

How magic works? The replication plugin takes care of synching the nodes. Historically with master-slave, the synchronization was unidirectional. With multi-site instead, all masters are replicating to each other. It means that the first master will replicate to the second and the other way around. Thanks to the new URL scheme, Google made our lives so much easier.

Introducing the multi-site plugin

We have already started writing a brand-new plugin called multi-site. Yes, it has a different name, because the direction that it’s taking is completely different in terms of decisions compared to the high-availability plugin, which required having a shared file system. Secondly, the HTTP synchronous communication with the other site was not feasible anymore. What happens if, for instance, a remote Site in India is not reachable for 50 seconds? I still want things events and data to be put into a persistent queue to be eventually sent through the remote site.

Screenshot 2019-03-02 at 01.24.06
For the multi-site broker implementation, were have decided to start with Kafka, because there is already an implementation of the stream events it in Gerrit as a plugin. The multi-site plugin, however, won’t be limited to Kafka but could be extended to support other popular brokers such as NATS.

We will go to the location-aware DNS because we want the users to access the HAProxy that is closer to him. If I am living in Germany maybe it makes sense for me to use only the European servers, but definitely not the servers in California.

You will go by default to a “location-aware” site that is closer to you. However, depending on what you do, you could still be laster on redirected to another server across zones.

For example, if I want to fetch data, I can still fetch everything on my local site. But, if I want to push data, the operation can either be executed locally or forwarded to the remote site, if the target repository has been sharded remotely.

Screenshot 2019-03-02 at 01.24.43

The advantage is the location-awareness and data-locality. The majority of data transfer will happen with my local site because, at the end of the day, the major complaints of people using Gerrit with remote masters is a sluggish GUI.

If you have Gerrit master/slave, for instance, a Gerrit master in San Francisco and your Gerrit slaves in India, you have the problem that everyone from India will still have to access a remote GUI from the server in San Francisco, and thus would experience a very slow GUI.

Even if only 10 percent of your traffic is a write operation, you would suffer from all the performance penalties of using a remote server in San Francisco. However, if I have an intelligent HAProxy, with also an intelligent logic in Gerrit that understands where the traffic needs to go to, I can always talk to my local server is in my zone. Then, depending on what I do, I can use my local server or a remote server. This happens behind the scenes and is completely transparent to the user.

Q&A

Screenshot 2019-03-02 at 01.25.33

Q I just wanted to ask if you’re ignoring SSH?

SSH is a problem in this architecture, right, because HAProxy supports SSH, but has a big problem.

Q: You don’t know what projects, you don’t have any idea of the operation, whether it’s read or write.

Exactly. HAProxy works at transport levels, so it means that it knows that flow of encrypted data going back and forth to the Gerrit server, but cannot see anything in clear and thus cannot make an educated decision based on it. Your data may end up being forwarded to the zone where your traffic is not optimized, and you will end up waiting a little bit more. But, bear in mind, that the majority of complaints about Gerrit master/slave is not really on the speed of the push/pull but rather on the sluggish Gerrit GUI.

Of course, we’ll find as a Community a solution to the SSH problem, I would be more than happy to address that in multi-site as well.

Q: Solution is to get a better LAN because we don’t have any complaints about the GUI across the continents.

Okay, yeah, get a super fast network across the globe, so everyone and everywhere in the world will be super fast accessing a single unique central point. However, not everyone can have a super-fast network. For instance in GerritHub, we don’t own the infrastructure so we cannot control the speed. Similarly, a lot of people are adopting a cloud infrastructure, where you don’t own and control the infrastructure anymore.

Q: Would you also have some Gerrit slaves in this setup. Would there be several masters and some of the slaves? Or everything will become just a master?

In multi-site architecture, everything can be a master. The distinction between master and slave does not exist anymore. Functionally speaking, some of them will be master for certain repositories and slave for other repositories. You can say that every single node is capable of managing any type of operations. But, based on this smart routing, any node can potentially manage everything, but effectively forwards the calls based on where is best to execute them.
If with this simple multi-site solution you can serve the 99% of the traffic locally, and for that one percent, we accept the compromise of going a bit slower, than it could still sound very good.

We implement this architecture last year with a client that is distributed worldwide and it just worked, because it is very simple and very effective. There isn’t any “magic product” involved but simply standard OpenSource components.

We are on a journey, because as Martin said last year in his talk, “Don’t try to do multi-master all at once.” Every single situation is different, every client is different, every installation is different, and all your projects are different. You will always need to compromise, find the trade-off, and tailor the solution to your needs.

Q: I think in most of the cases that I saw up there, you were deploying a Gerrit master in multiple locations? If you want to deploy multi-master in the same sites, like say I want two Gerrit servers in Palo Alto, does that change the architecture at all? Basically, I am referring to shared NFS.

This is actually the point made by Martin: if you have the problem on multi-site, it’s not a problem of Gerrit, it’s a problem of your network because your network is too slow. You should solve the network, you shouldn’t solve this problem in Gerrit.
Actually, the architecture is this one. Imagine you have a super fast network all across the globe, everyone is reaching the same Gerrit in the same way. You have a super fast direct connection with your shared NFS, so you can go and grow your master and scale horizontally.

The answer is, for instance, yes, if your company can do that. Absolutely, I wouldn’t recommend doing multi-site at all. But if you cannot do anything about speed, and you got people that work remotely and unfortunately, you cannot go and put a cable between the U.S. and India, under the sea just for your company, then maybe you want to address the problem in a multi-site fashion way.

One more thing that I wanted to point out, is that the multi-site plugin makes sense even in the same site. Why this picture is better than the previous one? I’m not talking about multi-site, I’m talking about multi-master on the same data center. Why this one is better?

You’ll read and write into both. Read, write, traffic to both. The difference between this one and previous is that there is no shared file system on this one. The previous, even within the same data center, if you use a shared file system, you still have a single point of failure. Because, even if you are willing to buy the most expensive shared file system with the most expensive network and system that exists in the world, they will still fail. Reliability cannot be simply resolved by throwing more money on it.
It’s not the money that makes my system reliable. Machines and disks will fail one day or another. If you don’t plan for failures, you’re going to fail anyway.

Q: Let’s say when you’re doing an upgrade on all of your Gerrit masters right? Do you have an automatic mechanism to upgrade? I’m coming from a perspective where we have our Gerrit on Kubernetes. So, sure, if I’m doing a rolling upgrade, it’s kind of like a piece of cake, HAProxy is taking care of by confd and all that stuff, right?
In your situation, how does an upgrade would look like, especially if you’re trying to upgrade all the masters across the globe? Do you pick and choose which site gets upgraded first and then let the replication plugins catch up after that?

First of all, the rolling upgrade with Kubernetes will work if you do minor upgrades. If you’re doing major Gerrit upgrades, it doesn’t. This is because that changes the data, right? With the minor upgrades, you do exactly the same here. Even if the others in the other zones have a different version, they are still interoperable, right, because they talk exactly to the same schema, exactly to the same API, exactly in the same way.

Q: I guess taking Kubernetes out of the picture, even, so with the current multi-master setup you have, if you’re not doing just a trivial upgrade, how do you approach that usually?

If you are doing a major version upgrade, that is going to orchestrate exactly like the GerritHub upgrade from 2.14 to 2.15. You need to use what I called a ping-pong technique. You basically need to do across data centers.

It’s not fully automated yet. I’m trying to automate it as much as possible and contribute back to the community.

Q: In the multi-master, when you’re doing the major upgrade, and even in the ping pong, so if the schemas changed, and you’re adding the events to replication plugin, are you going to temporarily suspend replication during the period of time? Because the other items on the earlier version don’t understand that schema yet. Can you explain that a little?

Okay, when you do the ping pong that was this morning presentation, what happens is the upgrade on the first node, interruption the traffic there, you do all the testing you want, you are behind with the original master but it catches up with replication and does all the testing.

With regards to the replication event, they are not understood by the client, such as the Jenkins Gerrit trigger plugin. That point was raised this morning as well. If you go to YouTube.com/GerritForgeTV, there is the recording of my talk of last year about a new plugin that is not subject to this fluctuation of Garret version.

Luca Milanesio – Gerrit Code Review Maintainer and Release Manager

Gerrit v2.14 / v2.15 talk at Palo Alto

This week we have published the recording of David Pursehouse (CollabNet) talk about Gerrit v2.14 and v2.15. Even though the releases are not new, there are many improvements were made recently, especially regarding the support for Elasticsearch as an engine to store the Gerrit secondary indexes.

Introduction

Hello, I am David Pursehouse, Gerrit Maintainer and release manager.

So, I’m going to talk about 2.14. Actually, this is the same presentation as I already gave last year. So, some of you may have seen it already. Although, I have made … how do I just … I have made some changes.

Basically, 2.14 isn’t new anymore. it’s already 18 months old, but even last year when I presented this, it was already six months old. And it has been in constant development since then. Even until this month we’ve been adding new stuff or making new releases.

And I’ll also touch a bit on 2.15 because yesterday evening we realized that there isn’t a 2.15 talk in the schedule today or tomorrow. So, I will briefly cover 2.15.

What’s new in v2.14

A number of new features in 2.14. I won’t read them all out because I’ve got separate slides for them, but there are some new features as well, which I’ll talk a bit more about later, which were not in 2.14 last year.

Some notes on important things about upgrading to 2.14, and actually the same things apply if you are upgrading to 2.15, 2.16. We require Java 8, so I assume most of you are already on Java 8 at least, so that shouldn’t be an issue, but these things are things that people run into when they’re doing the upgrades on the mailing list and issue tracker. So, these are the things that people have stumbled on that you should know about.

There’s no more HTTP digest authentication. This was basically because of moving to the accounts to NoteDb and not being able to store them in NoteDb. If you upgrade, it will get migrated. I think there were some issues with this initially, but they’ve been fixed in the bug fix releases. And the main thing for users here is that you can’t see your password in the UI anymore. It doesn’t display it. If you want a new password, you have to reset it, and then it disappears.

Also, there’s a new secondary index for groups, and you need to reindex that before you start the server. Otherwise your groups don’t work, and basically, Gerrit doesn’t work. And there are more details about this in the release notes.

Speaking of which, probably you have already seen this already because we’ve been sending out emails with the links, but we’ve moved the release notes to the new homepage, which is on www.gerritcodereview.com. When I wrote this last year, it was the case that the change would go live immediately after we submit them, but we’ve changed the hosting, and that isn’t the case anymore. Luca was going to fix that, but I don’t think he has. No. That will get fixed soon I hope.

New features in v2.14.

We move onto the actual new features. These are not in any particular order. It’s just a brain dump of the new stuff. Changes can now be assigned to people, and you can search for them with the new operators and make it highlighted. And that’s basically it. There’s no workflow around this. Gerrit doesn’t enforce any workflow for assigned changes. You can use it as you like, although I think there’s been some discussion about changing that to make it a more defined workflow so that people are more aware of what assignable changes are. There’s confusion among people about what does it mean when a change gets assigned to somebody. And right now it means nothing. There’s no enforcement of that in Gerrit.

Changes can now be deleted if they’re open or abandoned. This is kind of similar to previously with draft changes. You could delete a draft change, but now you can delete any change that’s basically not merged. And that can be done by the administrator or the owner of the change, assuming they’ve been given the capability. And this basically deletes the change completely. It deletes the ref. It deletes the metadata. And you can’t delete merged changes for obvious reasons.

The next one is reviewer suggestions. If you click on the box to add a new review, it will suggest based on previous users who have already reviewed your changes. And it can also be filtered. Also, there’s an extension point. Plugins can inject suggestions, although I’m not sure if any plugins actually implement that, but it’s there.

New email templates. Now we’re using a different framework for templating. It supports HTML, and the previous templating engine is going to be removed, actually is removed in 2.16. Still present in 2.14, 2.15, but deprecated.

Inbound email. This is a feature that was implemented by Patrick at Google. If you get a notification of a review from Gerrit, you can reply to that, and it will add the comments on the change. And if you click this link, you get more details about that.

Tagged comments. This is a way that enables you to filter the comments in the UI according to whether they’re by humans or robots, in this case meaning CI. So, for example, on the example here, you can see that all the CI comments are not shown on the right side. I think this only works in the new UI. I’m not … Is that right? I don’t know. Maybe it works in both UIs. I’m not sure.

So, in the new UI in 2.14 there’s been a lot of improvements compared to previously, and it’s basically usable for most of the common things that you want to do like reviewing changes, but there’s still a lot missing. A lot of the admin pages are missing. And you can switch between the two. So, if anything’s missing in PolyGerrit, you go to GWT and do it there. And if you look at this talk that’s linked here, it will give you an overview of the development of PolyGerrit, the idea behind it and so on.

The merge list is a dynamically generated list of commits that are going to be included by a merge commit, and it appears in the UI as a file that you can review, which is really useful if the commit author didn’t use a log option on the merge, so you can see what really is coming in by that merge and you can maybe catch if someone’s done it against the wrong parent or something. This isn’t actually a real file. It doesn’t get included in your commit. It’s just generated on the fly by the server.

Support for Elasticsearch as a secondary index

This is new since the last time I did this talk. And I want to thank Marco and David, who are the guys that really made this work. A lot of work went in by these guys to get Elasticsearch working. This is included since 2.14.8 with minimal support for Version 2 of Elasticsearch. And then there are a few maintenance releases which add support for five and six. If you look at this link here, you’ll see the more detailed list of compatibility between Gerrit and Elasticsearch. There actually was this week a new version of Elasticsearch, which will not be supported in 2.14 as far as planned. I’ve got a change up for review that will support that in 2.15.8, whatever the next one is. And I know Elasticsearch 7 is coming soon, but we haven’t started working on that yet. If we do, it’ll be in 2.15.whatever.

This is another new one since the last presentation. This object size limit is an option in Gerrit that’s been there for a long time. It allows you to prevent people uploading big files, but the problem with that was it was kind of weird to configure it. You couldn’t configure it on a project and have it inherited to the children projects. It was either global or per project. So, now you can configure this and have it inherited, but we made it so it’s disabled by default, so it doesn’t mess up anyone’s existing configurations.

And there was one minor update of LFS in 2.14.1. Actually, we added LFS in 2.13, but then we added support for the file locking. And the support for the Amazon S3 and File-system isn’t new, but previously there were separate plugins, and now we’ve rolled them together into the one single LFS plugin.

Overview of new features in v2.15

And now finally just a brief overview of what’s in 2.15. I’ve taken this from Dave’s talk from last year, but he had a lot of details about NoteDb, which I’m not really qualified to talk about, so I’ll kind of gloss over it a bit. There is a lot more PolyGerrit UI functionality, but it’s still not equal with GWT. You need to wait for 2.16 to get that. The support for NoteDb allows having some things that were not available before. The separation of reviewer and CC in the changes with a history of adding reviewers. Hashtags, actually, were introduced a long time ago, I think in 2.13, but required NoteDb, so it couldn’t really be used. The robot comments, which is basically CI can inject comments into the changes and suggest changes. You can also ignore a change, which means it will just not appear in your dashboard. You can mark it as reviewed, so that it will not be highlighted. There is also a really improved experience in diff between rebased patch sets, and again, in Dave’s presentation from last year, there is a lot more detail about that. So, if you want to know more, you can look at that.

We have a new change workflow. The draft changes are removed and replaced by private and work in progress, and David will talk about that this morning. And there’s a lot of other stuff that I kind of missed, but if you want to see that, you can look at Dave’s talk from last year, which is linked here.

Q&A

Q: We recently migrated to 2.14, and we often see the index and losing the indexing being really slow. How do we diagnose those issues, the indexing issues? I don’t see very good documentation on this.

A: If you have specific problems or issues and you have data to support that, you can submit a bug report or you can submit any questions to the mailing list. I don’t really know if there’s any way you can tweak that to make it perform better, but if you have specific questions, we can take them on the mailing list or in the issue tracker.

Q: Gerrit indexing works very badly when we have close to 2000 open changes. Is there a specific configuration to improve the performance? We are using v2.14.10.

A: I don’t know if there are any improvements from 10 up to the latest patch release. One thing about this 2.14 is we have made a much larger number of maintenance releases than previous releases, mainly because of the Elasticsearch support, but also there have been some pretty serious things fixed in these releases, so you probably should check the release notes and see if anything related to your issue is mentioned after 2.14.10.

Q: You mentioned Java 8. I was wondering if any later Java would also work, or does it specifically have to be Java 8? And generally, what’s the thinking about newer versions of Java support?

A: Yeah. There is work ongoing to support later versions of Java, and David Ostrovsky will tell you about that in his presentation.

David Pursehouse (CollabNet) – Gerrit Maintainer and Release Manager.

Gerrit User Summit 2018

The Gerrit User Summit 2018 has ended. It has been a truly memorable event and with strong emotions, gratitude and celebration of the success of the Gerrit OpenSource project.

During the hackathon, some major events happened:

  • The release of Gerrit v2.16
  • Support for Kubernetes
  • New plugins contributed by Qualcomm
  • The announcement of a new maintainer, Marco Miller from Ericsson, congrats!

Hackathon and Gerrit v2.16

The hackathon took place at SAP (2 days) and Cloudera (1 day) in Palo Alto with the main focus of completing the migration to PolyGerrit and cut the v2.16 release. Three major milestones of the Gerrit project:

  1. PolyGerrit as default UI for Gerrit Code Review.
  2. Migration of the groups to NoteDb
  3. Support of the Git protocol v2

With regards to the achievement #1, during the hackathon, the commit for removing all the GWT related code from Gerrit master has been finally merged.

PolyGerrit as default Gerrit Code Review UI

The PolyGerrit Team has reached the summit of the long and troubled journey of getting rid of the outdated and Gerrit GWT UI. Kasper and Logan (Google) burned every single bit of their development energy to fix, implement and fill all the remaining gaps in the user journeys.

On Gerrit v2.16, PolyGerrit UI is the default and standard way of interacting with Gerrit Code Review. GWT is still there for allowing people to smoothly adapt to the new interface. However, the next forthcoming version of Gerrit v3.0 will not contain anymore any reference to the old UI. People will still have a few more months to adapt and, from the feedback received on GerritHub.io, they seem to not miss the GWT interface at all.

Well done to the entire PolyGerrit Team, starting from Andrew who kicked off the project back in 2015 to Wyatt, Becky, Victor, Kasper and Logan who brought it up to the usability and portability to multiple devices as it is today. Thanks to their amazing effort, we can all interact with Gerrit Code Review wherever and whenever we want, without losing speed and effectiveness.

Migration of Gerrit groups to NoteDb

One more step towards the complete elimination of ReviewDb has been achieved in Gerrit v2.16: there isn’t anymore any mutable data in the leftover of the (in)famous Gerrit DB. The new version includes a new git repository (All-Groups) which contains all the information related to the groups create through the Gerrit UI.

What’s left in ReviewDb? Not much: only the schema version. When you install Gerrit it is populated with a single row and a single column with the value ‘170’ and will never change. There is no need to share it across master or slaves: just set database.type = h2 and forget about it.

Git protocol v2

It is a major achievement of Gerrit v2.16 as the Git wire protocol v2 solves the performance problems of large repositories: the refs filtering during the advertisement phase.

In a nutshell, if you were fetching a single from a repository with 500k refs, the server would have sent a huge payload of all refs with the associated SHA1s even if at the end of the day you wanted only to fetch a single one. The delay could have been of several seconds if not minutes, which is quite relevant if you are fetching all the times, like in a CI/CD pipeline.

Git protocol v2 reduced the refs advertisement phase by over 70%, which makes Gerrit v2.16 so appealing that you may want to place in the top 5 tasks of your backlog.

Farewell to GWT UI code

Even though Gerrit v2.16 keeps the ability to display the old-fashioned GWT Gerrit UI, the code has been definitely removed on the Gerrit master branch, which will the base for Gerrit v3.0 that will be released in Spring 2019.

This is a memorable moment because represents the final act of making PolyGerrit UI the unified user-experience of Gerrit moving forward.

Data and Insights of the Hackathon

This year the event has been truly focused as well on the analytics side of the Code Review and the entire CI/CD pipeline: was it possibly the location, Cloudera, the world leader in OpenSource BigData and Analytics, giving the right vibrations? Possibly, yes.

Numbers

Over 20 people have attended the Hackathon on-site and, for the first time, remotely via video-conference, from companies and countries all around the world:

  1. Google (USA and Germany)
  2. CollabNet (Japan and Germany)
  3. GerritForge (UK, Ireland, Italy, and Germany)
  4. SAP (Germany)
  5. Qualcomm (USA)
  6. Ericsson (Canada)
  7. Wikimedia Foundation (UK)

Over 300+ changes have been uploaded and 244 of them have been already merged.

They have been working on 44 projects, showing how diverse is today the universe of Gerrit Code Review and its associated plugins and integrations.

Gerrit DevOps Analytics

GerritForge has been hacking on improving the platform that powers the Gerrit Code Review Analytics Platform which has been serving the community for many years.

For the first time, thanks to the introduction of branch-level analytics, the community had data automatically crunched in near-real time of what was happening on the master branch and on the on-going stable 2.16 release.

As a matter of fact, the numbers mentioned before are directly coming from Gerrit DevOps Analytics. Thanks again GerritForge for innovating and improving the Gerrit Code Review platform.

Gerrit Analytics Wizard

For the first time, two new contributors from GerritForge, Tony and Ponch, finalized and demoed a new plugin called ‘analytics-wizard’ that allows anyone with a Gerrit server to set up a mini-version of the Gerrit DevOps Analytics, using Docker, ElasticSearch, and Kibana.

Two new plugins: batch and tasks

Qualcomm has released two new plugins based on their experience of running complex pipelines on a multi-master Gerrit setup. The majority of the code was developed in-house before the hackathon. However, the final release and announced happened this week and I am sure it will allow many other companies to benefit from their experience of validating complex multi-repo changes on a large scale CI system.

Batch

This plugin provides a mechanism for building and previewing sets of proposed updates to multiple projects/refs that should be applied in a batch. These updates are built with Gerrit changes.

Task

The plugin provides a mechanism to manage tasks which need to be performed on changes along with a way to expose and query this information. Tasks are organized hierarchically, and task definitions use Gerrit queries to define which changes each task applies to, and how to define the status criteria for each task. An important use case of the task plugin is to have a common place for CI systems to define which changes they will operate on, and when they will do so.

Gerrit on Kubernetes from SAP and GerritForge

Luke (GerritForge), Matthias (SAP) and Thomas (SAP remotely from Germany) worked on a PoC for supporting Kubernetes deployments.

Last year SAP started using containerized slaves and decided to go cloud native and base our future Gerrit deployments on Kubernetes and leverage project Gardener to support multiple cloud providers. Matthias worked with Thomas to prepare a PoC and a demo the current work in progress and discuss plans moving forward.

The code has been published to the new k8s-gerrit repository and new commits will be subject to the standard Gerrit Code Review process.

Luke prepared an example of Gerrit v2.15 deployment in Kubernetes (single master at the moment) with shared storage and showed how to upgrade easily by leveraging Helm Charts deployments through Tiler.

Even though the two projects started in parallel, they agreed to cooperate and work together with the community to have a unified Kubernetes deployment code-base for installing and upgrading multi-master Gerrit setups. This is definitely an area where we will see very interesting developments very soon.

What about the Summit?

The talks of the summit have been very interesting this year and have covered mostly three main topics: Gerrit roadmap and scalability, sharing of real-life migrations to Gerrit v2.14 and v2.15 and, as previously mentioned, Data Analytics & Insights including research work on using machine-learning for supporting the Code Review process.

All the talks are getting published as we speak on each talk of the Gerrit User Summit Schedule.

All the talks have been recorded and will be published within the next few weeks on the GerritForge TV YouTube channel.

More blog posts will follow with more background on each talk, stay tuned and watch this space to know more about what has been presented and announced at the Summit.

Proposals for the next Gerrit User Summit 2019

The overall feedback on the summit has been very positive, including the Birthday Party organized and sponsored by GerritForge for the 10th anniversary of the Gerrit Code Review project.

During the final closing keynote, many interesting proposals have been made:

  • Gerrit users’ wish-list session.
    Any Gerrit user can feed the backlog of new feature proposing what they will like to see implemented
  • Gerrit plugin hackathon.
    Learn how to implement Gerrit plugin and trying to put together in one or two days something useful using any of the programming languages they wish, including Java, Groovy, Scala or anything else.

More details will come as soon as the closing keynote session recording will be published.

Remembering Shawn Pearce and his legacy

Dave Borowitz, the leader of the Gerrit Code Review project, made a thoughtful and touching speech about Shawn Pearce, the father of the Gerrit 2, who started the project exactly 10 years ago. The cake, the hackathon, the summit, and the v2.16 release have been fully dedicated to his memory as appreciation for his humanity and passion for OpenSource.

Thanks, Shawn. We will continue on your legacy and improve the Gerrit Code Review platform as you would have wanted us to do.

 

 

 

Gerrit User Summit: only two weeks away

GerritUserSummitPaloAlto.png

It is exciting times for the Gerrit Code Review project, which is approaching its 10th anniversary. The very first commit is dated 14th of November 2008, exactly ten years back from the official kick-off the 15th of November 2018 of the Gerrit User Summit 2018 in Palo Alto

commit 23571ab1fa7fedc262d6c21510614353e9d8a4dc
Author: Shawn O. Pearce <sop@google.com>
Date: Fri Nov 14 16:56:58 2008 -0800

Initial project setup of Gerrit 2

Gerrit 2 is a ground-up rewrite of Gerrit, using GWT (Google Web
Toolkit) for the client side user interface and a Java servlet
based backend.

Signed-off-by: Shawn O. Pearce <sop@google.com>

An exciting schedule with talks about new features

The program is now complete and includes a lot of interesting talks about what’s new and noteworthy in Gerrit:

There are as well very interesting use-cases of using Gerrit in real-life scenarios:

The summit is kindly hosted by Cloudera, the world’s leader of the BigData OpenSource Platform. It is not a coincidence that there are talks about Gerrit and the DevOps Analytics:

Recordings of the Gerrit User Summit 2017 in London

The best way to get ready to the Summit this year is to watch what happened during the past 12 months, compared to what was announced at the Gerrit User Summit held at Skills Matter in Central London back in 2017, leveraging all the free content available on the GerritForge TV YouTube Channel. There is a lot to learn about Gerrit and its related technologies.

All the talks are now live, including the very interesting board discussion between the European members of the Gerrit Code Review community and the Maintainers of the project.

Don’t miss the dates, 15-16 November

2018 is an exceptional year for the Gerrit Community, and this Summit will most likely be the largest ever in the history of the Git Together and Gerrit User Summits of the past ten years.

Have a safe trip and see you soon see you at Cloudera HQ in Palo Alto Palo Alto.

Luca Milanesio – GerritForge
Gerrit Code Review Maintainer

 

 

 

 

Gerrit User Summit 2018

Gerrit1-high-res.pngWe are pleased to announce that GerritForge, in cooperation with Cloudera and the Gerrit Code Review Community, will be organizing again this year the Gerrit User Summit and Hackathon.

Gerrit User Summit is the event that brings together Gerrit admins, developers, practitioners, the whole community and ecosystem, in one place, providing attendees with the opportunity to learn, explore, network face-to-face and help shape the future of Gerrit development and solutions.

The event will be kindly hosted by Cloudera in their HQ in Palo Alto, in the heart of the Silicon Valley next to the prestigious Stanford University.

Dates

Schedule

Shortly we will be publishing the full agenda and logistics for the event, look forward to that. There are still some available slots for proposing a talk, see the instructions on how to submit your topic.

On behalf of GerritForge and the Gerrit Code Review Community,
Luca Milanesio – GerritForge Ltd

Gerrit User Summit: What’s new in 2.15

This week we are honored to publish the amazing talk of Gerrit Ver. 2.15 presented by Dave Borowitz, the “father” of NoteDb review format. It is the very first time that a full roadmap of the migration from the traditional ReviewDb (a relational DBMS) to NoteDb (a pure Git notes-based review store) is drawn and explained in all details.

First steps with Gerrit Ver. 2.15 using Docker

For all of those who want to experiment what Dave has presented at the Gerrit User Summit, there is a Gerrit Ver. 2.15 RC2 Docker image already published on DockerHub on https://hub.docker.com/r/gerritcodereview/gerrit.

See below the simple instruction on how to start a Gerrit 2.15 locally and playing with it.

Step 1 – Run Gerrit Docker image

$ docker run -h localhost -ti -p 8080:8080 -p 29418:29418 gerritcodereview/gerrit:2.15.rc2
...
[2017-11-14 23:12:08,016] [main] INFO  com.google.gerrit.pgm.Daemon : Gerrit Code Review 2.15-rc2 ready

Step 2 – Open Gerrit UX

Open your web-browser at http://localhost:8080 and you will automatically get into the plugin manager selection. At the moment only the core plugins are available, so you can just click on to top right “Go To Gerrit” link.

Screen Shot 2017-11-14 at 23.19.38.png

As soon as Gerrit 2.15 will be officially available, you would be able to discover and install many more plugins on top of your installation.

Step 3 – Create a new Repository

Click on the new top “BROWSE” menu and select “CREATE NEW

Screen Shot 2017-11-14 at 23.20.48.png

Then insert the repository name (e.g. “gerrit-playground”), select “Create initial empty commit” to True and click on “CREATE“.

Screen Shot 2017-11-14 at 23.22.25

Step 4 – Clone the repository

From the repository page select the “HTTP” protocol and click on the “COPY” link next of the “Clone with commit-msg hook” section.

Screen Shot 2017-11-14 at 23.24.56.png

And paste the command on your terminal.

$ git clone http://admin@localhost:8080/a/gerrit-playground && (cd gerrit-playground && curl -Lo `git rev-parse --git-dir`/hooks/commit-msg http://admin@localhost:8080/tools/hooks/commit-msg; chmod +x `git rev-parse --git-dir`/hooks/commit-msg)
Cloning into 'gerrit-playground'...
remote: Counting objects: 2, done
remote: Finding sources: 100% (2/2)
remote: Total 2 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (2/2), done.
Checking connectivity... done.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4695  100  4695    0     0   448k      0 --:--:-- --:--:-- --:--:--  458k

Step 5 – Create a new commit for review

Enter into the new Git cloned repository “gerrit-playground”, add a README.md file.

$ cd gerrit-playground && echo "Hello Gerrit" > README.md && git add .

Then identify yourself as admin@example.com, which is the default Admin e-mail for the Gerrit 2.15 Docker image.

$ git config user.email admin@example.com

And finally create a new commit and push it to Gerrit using “secret” as password

$ git commit -m "Say hello to Gerrit" && git push origin HEAD:refs/for/master
Password: secret
[master b4de540] Say hello to Gerrit
 1 file changed, 1 insertion(+)
 create mode 100644 README.md
Counting objects: 3, done.
Writing objects: 100% (3/3), 303 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: Processing changes: new: 1, done    
remote: 
remote: New Changes:
remote:    Say hello to Gerrit
remote: 
To http://admin@localhost:8080/a/gerrit-playground
 * [new branch]      HEAD -> refs/for/master

Step 6 – Review your Change in Gerrit

Open the Gerrit Change URL mentioned in the Git output (http://localhost:8080/#/c/gerrit-playground/+/1001) in your Web Browser and you can review the code using PolyGerrit UX.

Screen Shot 2017-11-14 at 23.37.43.png

Step 7 – Approve and Submit your Change

Click on “CODE-REVIEW+2” button on the right toolbar to approve the change and then press “SUBMIT” to merge into the master branch.

Screen Shot 2017-11-14 at 23.40.45.png

What’s new in Gerrit Ver. 2.15

Hi, I’m Dave Borowitz, I work for Google, and I’m going to talk about what is new in Gerrit 2.15. We have a six months release cycle, and it has been around 158 days since Ver. 2.14.0 was released in April, and now, just last night at 10 PM, I have release 2.15-RC0.

Gerrit Ver. 2.15 in numbers

Gerrit Ver. 2.15 is aligned with other recent releases:

  • 2,102 commits
  • 53 contributors from all around the world and different companies
  • seven new contributors who have never commit before

I don’t know if any of the new contributors are in this room and if they are, thank you, if not I will thank them remotely.

PolyGerrit frontend

There is a bunch of new stuff going on in Gerrit Ver. 2.15, I will talk only briefly about the frontend, because we had an excellent talk from Logan and Arnab and Justin this morning about what’s happened recently in PolyGerrit.
I want to give you my perspective as a developer that is doing mostly backend work. It’s fascinating to have in Gerrit a modern JavaScript environment to develop in because it has been historically difficult for us to attract frontend developers in the project. There are not just enough GWT developers in the world as there are JavaScript developers and I think that the Gerrit project has suffered from a lack of UI attention in the past because of that.

But now we have a whole team here at Google, we have a lot of external contributors, that are happy to work on modern JavaScript, and PolyGerrit has made some excellent strategies of improving the user experience.

Screen Shot 2017-11-14 at 23.51.35.png

PolyGerrit is also usable. Everyone that has been working in software development always has the attraction of starting from scratch and say “how would have done it differently, “: PolyGerrit follows a little bit that approach.

We understand that we have users that are used to a certain workflow and we have to preserve it. However, when you are writing new code, you have the opportunity to learn from your past mistakes and improve things and maybe make it easier to onboard new people.
There is a little bit of preview of what PolyGerrit looks like in Gerrit Ver. 2.15; I had to cut the screenshot last night to make sure that it would be entirely accurate.
You can see that there is a reasonable rate of improvements, even in the last 24h and I hope that we will keep this peace of development.

Screen Shot 2017-11-14 at 23.52.44.png

Backend

From Ver. 2.15 we support NoteDb, and we suggest that you should convert your existing ReviewDb to NoteDb.
What is the motivation for doing NoteDb? Until now the Gerrit administrators had to be DBMS Administrators as well which is a sort of weird feeling: you care about version control and software development workflow, and, for some reasons, you should take care about being a DB Admin?

The idea behind is that we carry on the data historically stored in ReviewDB and we move it directly into the Git repository.
When you create a new change with a patch-set, a topic and all its meta-data, including comments and reviewers then everything gets stored in the same Git repository with the rest of the committed change data. There are several advantages in storing data and meta-data in this way.
You don’t have to worry about two stores that somehow are going to get eventually inconsistent. If you have to make a backup of your Gerrit Server, you have to take a Database dump and then to have a backup of all of your repositories. If you want to have a consistent backup where everything that is stored in the Database exists in the Git repository and vice-versa, you have to do it while the server is down. There is no other way to do it because you may get new records in the database while you are backing up the Git repositories and you would not see that data reflected.

With NoteDb, it’s all in the Git repository, and it is even better because we changed JGit to be able to write to multiple Git repositories refs atomically atomically. This means you can submit a change and also submit the status of the meta-data at the same time. When you upload a patch-set, we update three refs, either all of them succeed, or all of them fail. There isn’t a state where the change was merged in the branch, but the ReviewDb wasn’t updated, that is just no longer possible. That enables us to have consistent backups.

NoteDb provides a very helpful audit log. We had a lot of data issues in the past where could not understand how a change got into a particular state because in ReviewDb you just update a field with ‘X’ and you forget completely that the field was previously with a value ‘Y.’ In Git the model you append commits to a history graph, so you actually store every operation that has ever happened on NoteDb, and that gives you an understanding on how a change ended in the current state.

Screen Shot 2017-11-14 at 23.54.20

We are thinking about giving extensibility for new features, and this is a kind of optimistic view about the future, plugins will be able to add new data to NoteDb while it wasn’t possible for a plugin to add a new column to an existing table of ReviewDb. I don’t think that we have any plugins that are currently able to leverage this capability and we do not have any extension for it yet, but the data layer supports it.

Whilst is automatically giving new features such as moving changes between Gerrit hosts without having to throw away code review meta-data.

NoteDb roadmap

We never actually pictured in the past a complete NoteDb timeline, which spreads across five long years.

Screen Shot 2017-11-14 at 23.55.18.png

  • 2013
    The very first commit on NoteDb was in December, that was a very long time ago.
  • 2014
    We had an intern, and he wrote all the stuff on inline comments in NoteDb.
  • 2015
    I wrote a thing that it was at the end a good idea in retrospective, but it was a considerable amount of work. It is called the batch update and allows to have a coordinated transaction across two different data-store with a consistent interface. This period is what I called “rewrite every single line of Gerrit”.
  • 2016
    We started migrating googlesource.com; ReviewDb still existed, and it was always the single source of truth in case they got out of sync with NoteDb. Later this year, a few months ago, we moved everything to NoteDb and we don’t use the change table anymore. We have several hundreds of servers nowadays using NoteDb and generated several hundreds of changes to it. That is exciting, we have been running in production for months, and that’s why we believe it could work for other people to run in production.
  • 2017
    Last night, we released Ver. 2.15, the first version of Gerrit where we officially say “we officially support NoteDb and we encourage you to migrate away from ReviewDb.”
  • 2018
    We are going to release Gerrit Ver. 3.0 and Ver. 3.1. The reason for the name ‘3.0’ is because we will not have ReviewDb anymore. There will still be the code and the ability to migrate from ReviewDb to NoteDb, but you would not be able to run Gerrit on ReviewDb. The Ver. 3.1 is the one I am most excited about because in that version we do not have even to support a migration tool. We will then be able to throw away all the ReviewDb code, and that would make me very happy.

How to migrate from ReviewDb to NoteDb

I mentioned that migration for googlesource.com took us a very long time because we’ve found a ton of issues with our data. We discovered all these things while we were running our migration and we fixed them all.

We developed a system that was scanning all ReviewDb, performed an in-memory migration and then compared the result with the changes stored in NoteDb. One example of a bug was that some subjects were truncated in ReviewDb. The subject is supposed to come from the first line of the commit message. We were comparing the data in Git with the data in ReviewDb and they did not match because they were truncated. If we were to require that all the subjects in NoteDb were identical to the ones in ReviewDb we would have never passed because there was this truncation. We could have patched all the existing data but actually what we did is to consider that if the subject in NoteDb starts with the subject in ReviewDb it was then regarded as valid. There were many more bugs of that flavor.

There were also bugs in the NoteDb code that we fixed; it was not just like all related to not good data; my code was far from being bug-free. The reason why I am talking about how much effort we put in making it right is that I want you to feel confident and not think about that this is a so much scary operation on your data. We tested on ourselves, and we fixed a lot of these bugs, and we are still pretty confident that this is a safe operation.

In Ver. 2.15 there are two types of migration options: on-line and off-line. At Google, we are in an exceptional condition because we are always at zero downtime, but that was useful because it allowed us to write a tool for a live migration from ReviewDb to NoteDb while the server is running.

Migration to NoteDb is pretty much similar to the way you do reindex: there is an online reindex and an offline reindex. You can choose to do it offline, and it will be probably faster, but there will be a downtime. Or you can decide to do it online, and it will be slower, but there will be no downtime.

And then in Ver. 3.0 we only are going to support an off-line migration, following the same paradigm of all the other schema upgrades. If you skip between releases, we force you to do to an off-line update, but if you upgrade just one point release at a time, you don’t have to have any downtime for your schema migration. Similarly for NoteDb if you migrate from Ver. 2.14 to Ver. 2.15 and then Ver. 3.0, you won’t have any downtime.

Screen Shot 2017-11-15 at 00.00.20.png

Q: (Han-Wen) Is this process parallel?

A: It is parallel if you do it offline if you do it online it is using a single thread because we are assuming that your server is mostly busy doing other stuff and that’s why you may want to do an online migration in the first place.

Benefits of migrating to NoteDb

There are a lot of incentives to migration to NoteDb; one is that you have new features such as hashtags and others that we implemented only in NoteDb because they were a lot harder in ReviewDb such as the history of all reviewers on a change. NoteDb manages audit natively while on ReviewDb we would have needed to have a new table called reviewers_audit which would have been much harder to implement.

The robot comments introduced in Ver. 2.14, the ability to remove clutter in your dashboard to mark a change as reviewed, are all features that you only have in NoteDb.

What did we learn from migration to NoteDb?

Writing every single line of code just takes a long time, and Gerrit has hundreds of thousands of lines of code. Shawn Pearce, my manager, and the Gerrit project founder at Google, every time he needs to touch NoteDb related code just says “I don’t even recognize this, ” and he is still the contributor #1 in the project. We changed it almost beyond its recognition.

Everything I’ve said so far is about changes; there are also other data besides changes. Accounts have been unconditionally migrated to NoteDb in Ver. 2.15. Is more a git config file format for the accounts that we store in NoteDb, it is not even actually a Note-space format. The account is now a config file that has your name and your e-mail address and the status, which is a new feature in NoteDb. For instance, my account status says that “I having a talk in England”.

New Patch-set comparison

Hi, my name is Alice Kober-Sotzek, and I work at Google. In Ver. 2.15 we have changed the way we compared patch-sets. Let’s imagine we have just a small change with a patchset and two files on it. In the first file we have only the first line modified, and the file consists of one thousand lines. The second file has four lines changed.

Screen Shot 2017-11-15 at 00.02.32.png

Let’s see what happens now when I rebased it to the latest version of master. If I had now to visualize what the patchset 1 consisted of and patchset 2 consisted of, what would I assume it would be? If I had been the author of the change, I would have expected that only one line would have been changed. Let’s just do it and ask Gerrit Ver. 2.14 what the result is.

Screen Shot 2017-11-15 at 00.03.16.png

What’s happening? Why do I have 420 lines changed in my file and ten additions and seven removals on the other?
That was not even touched on. Let’s have a look at the content of my file and what is in there.

In Ver. 2.14 we were just hiding all the differences due to rebasing, and that was it. In Ver. 2.15 things look different though because we try to figure out what happened in between the rebase. All the hanks that we are sure are added by something else, are displayed in a different color. We have not only red and green but orange and blue as well; these are all the ones that were introduced by other changes that were in between rebase.

Screen Shot 2017-11-15 at 00.04.34.png

This feature only works in PolyGerrit, while in GWT was not shown at all.

Can I rely on that and trust what I see there?

The decision we made is that all the hunks marked with orange and blue are the ones we are sure of and you can safely avoid looking at them because they were the ones that happened because of other changes occurring in between rebase.

The ones marked with red and green, we give no guarantee. They could be introduced by other changes, because of conflicts or may be added by the patchset. With that coloring, it is much easier to look at the things that are important.

Screen Shot 2017-11-15 at 00.05.32.png

Killing Draft Changes.

Some of the people think that draft changes are very much a visibility thing so that only my reviewers can see them. Other people use it like a change is not yet ready for review so that I can leave it as draft change until it is ready for being reviewed. You may even just use the server as a store for your changes; rework the code through the Gerrit in-line edit feature until the code is ready and then come up with an absurd number of patchsets. Nobody wanted any of them, but those are the conditions that we ended up with patchset drafts.

Patchsets could have been even deleted so they would never exist. They could just be kept invisible so that you see a gap, but that could be the current patchset: the UI claims that the current patchset is three, but then I do some other operations that say that this patchset is not current anymore, just because the current one is a draft!

Screen Shot 2017-11-15 at 00.06.36.png

Drafts are a kind of mass fraud; the main reason is that they are colliding all these things into a single feature. In Gerrit Ver. 2.15 we killed drafts. Now you have little small features instead of drafts. You have now “Private Changes” which only you and your reviewers can see. There are Work-In-Progress (WIP) changes, that means that while the WIP flag is set nobody gets notifications about it: you can push 30 patchsets, and the reviewers would not get spammed with 30 emails. Last but not least, we introduced a long ago the Change Edit, which can be used as well in conjunction with WIP Changes.

Marking Changes as reviewed

Another thing that we introduced in Ver. 2.15 is the ability to mark changes as you reviewed it. For instance, the one below is a change screen from my dashboard this morning: some changes are highlighted in bold and those other changes are not. I feel like the bold changes are yelling at me and you have to give me your attention just like in e-mails where bolds means “you need to look at me now.” Gerrit Ver. 2.15 when you are using NoteDb allows you to unbold any of them by just clicking a button on the change screen. Or like in an email you wish to remove some changes from your dashboard entirely. There is a function that allows you remove a change unilaterally from your dashboard that the other cannot undo or ignore it, that just makes the change go away.

It was annoying that I could not mark them as reviewed manually and it was really irritating that patchsets disappeared. It is really irritating when I received a review with a bunch of comments, I had to say “done, done, done, done” on each one of them.

Screen Shot 2017-11-15 at 00.10.01.png

And then when I pushed a new patchset, I just forgot to submit all these drafts comments that say “done”. So I added just a push option that says “when you push, publish comments” and all the draft comments will be published automatically. So instead of clicking on all the patchset on that change and check if in any of the patchset I have any draft and if I do, click send on all of them one by one, I can instead just set an option.

Screen Shot 2017-11-15 at 00.10.52.png

It can be specified by the command line, but it is difficult to remember. So there is a user preference with a checkbox which I really encourage you to select in your user preferences screen and it is only available on PolyGerrit.

CCing a Change under review

When someone is getting a co-worker and they want him to be a reviewer for a change, you get an error saying that your co-worker is not a registered user. We have partially solved this problem by adding a CC with an e-mail address, also only available on NoteDb. There are technical and even product reasons why we don’t want to add them as a reviewer, some of them are related to the accountability related to everyone that is working on that change. So people needs to have an account to be a reviewer, but if people just want to look at it or a mailing list, it doesn’t have to be a real user, you can then just CC any e-mail to a code review if you turn the config option to allow this.

Screen Shot 2017-11-15 at 00.11.48.png

Better error messages.

Another inch that Han-Wen scratched was the introduction of better error messages. Sometimes you do a push, and it fails with a very unhelpful error message that says “Prohibited by Gerrit.

It turns out that it is not difficult to check if a user does not have a permission then gives permissions error, so we have included a message saying this user lacks this permission. It is not perfect, so it doesn’t say precisely where in the project hierarchy this permission was coming from but at least says “this is the problem”, it tells you not the solution but at least highlights where the problem is.

Screen Shot 2017-11-15 at 00.12.37.png

Robot comments and automatic fixes.

Robot comments are a feature that we believe it will start to rump up in adoption. With NoteDb, you can suggest fixes as well, and then you have a button that says “apply fixes” which creates a change that applies the fixes.

Many more improvements in the bag.

There is a lot of speed improvements in PolyGerrit, so the changes with a lot of diffs will run a lot faster. An admin can delete comments that really shouldn’t be there. We can explicitly keep track that a change reverts another change so that you can search if that change was reverted. It can even tell you if that was a pure revert at Git level only, or if other changes were sneaked in claiming that this was only a revert which happens way too often I think.
There are a better server consistency checks and a new plugin endpoint for dashboards; there is a new URL scheme as described by Patrick and we are now off the page for putting, even more, new features.

Screen Shot 2017-11-15 at 00.13.15.png

 

Gerrit User Summit: PolyGerrit UX

The second presentation this week from the Gerrit User Summit 2017 is about PolyGerrit, the new Polymer-based UX for Gerrit Code Review, officially feature complete for Code Review in Gerrit Ver. 2.15.

Logan (PolyGerrit and Gerrit Maintainer), Arnab (PolyGerrit UX Designer), Dustin (PolyGerrit User Researcher) and Jason (Gerrit Code Review and Go Language Product Manager) show and discuss the future of UX of Gerrit.

PolyGerrit Overview (Logan Hanks, Google)

We are going to present three topics today:

  1. I am going to talk about the development of PolyGerrit, why we are making PolyGerrit and what we’ve done over the past two years.
  2. Arnab is going to present a new visual design we are going to start rolling out to googlesource.com, and to the next release of Gerrit.
  3. Dustin is going to give some findings from the user research on the survey I believe that many of the people in this room may have received the questionnaire on Gerrit and have responded.

Established November 2015

commit ba698359647f565421880b0487d20df086e7f82a
Author: Andrew Bonventre <andybons@google.com>
Date: Wed Nov 4 11:14:54 2015 -0500

Add the skeleton of a new UI based on Polymer, PolyGerrit
 
This is the beginnings of an experimental new non-GWT web UI developed
using a modern JS web framework, http://www.polymer-project.org/. It
will coexist alongside the GWT UI until it is feature-complete.

That is the very first commit we did on the PolyGerrit project, it was founded by Andrew Bonventre from the Chromium project and the commit message summaries up pretty well.
The idea is to replace the existing UI based on Google Web Toolkit (or GWT) and replace with a new UI built on modern frameworks with straight HTML and JavaScript. Part of the innovation was just not to renew the UI and make it better but also for the Chromium project move to Gerrit.

Based on the Polymer Project

Polymer is pretty essential to PolyGerrit, and that’s why is part of the name. The Polymer project is not a framework but rather a collection of libraries and tools and components that have been developed by Chrome at Google. The idea is the build beautiful web applications on top of the standard web platform. Some critical aspects of the web platform that we use are web components which are built on top of templates that you can import into your HTML file and shadow DOM which is a way of encapsulating your stylesheets into your templates so that you can build reusable components.

Another significant aspect of the Polymer project is the collection of custom elements. They are organized into groups and given chemistry names. We use iron elements pretty heavily; they are the basic building blocks for HTML.
We are starting to use elements from the paper collection which are higher level UI elements that introduce material design. They look nice, they are high level and reusable elements. They let us focus on building code-review without having to rebuild everything from scratch.

Screen Shot 2017-10-19 at 15.50.05.png

Q: Can you explain a bit more on what material design means?

A. (Arnab) Material design is a new design language that Google came out of it in 2015. It’s a library of all the modern components with great look and feel.

Material design elements come with a lot of features like animations and smooth transitions which makes your UI more dynamic, it’s hard to show in slides all that dynamism.

Current status of PolyGerrit

We have been working on PolyGerrit for two years now, and in that time we got a lot of contributions from outside the project, we have a few dozens of contributors, checked in thousands of commits, and we closed almost a thousand issues from our issue tracker.

As far as I know, there is no other significant deployment of PolyGerrit outside of Google and GerritHub.io, but we have been having used in Google for a while, and we see 10k code-reviews a day involving PolyGerrit, at least on a workday. We have got a few major OpenSource projects using it, Chromium has recently transitioned to PolyGerrit, and brings up almost 2000 developers.

I’ve put some other recognizable OpenSource projects like GoLang and also the Gerrit project itself, where we configured gerrit-review.googlesource.com to use PolyGerrit by default. That helps us get early feedback on the functionality and the design of PolyGerrit.

Chromium migration to PolyGerrit

What about Chromium? It is a significant milestone for our Team, and I also think for the Gerrit project. The Chromium project has a very complex system for checking their code and for builds. They have built the system themselves and they used it to run their code-review system them as well based on Rietveld. Unfortunately, that is a sort of unmaintained instance of Rietveld running on AppEngine they did not want to use anymore and also it wasn’t scaling very well for them. There were even commits that they couldn’t review on that system because they were too large.

So a lot we’ve been working at that time, it was allowing Gerrit to come to a state so that Chromium could use it. Part of that was building a nice UI, and part of that was taking features they have been enjoying in Rietveld and bringing them over to Gerrit. Some of them are small things like user account statuses, so that you can tell people that you are not in the office and you can’t do code reviews this week, or having descriptions to your patch-sets so that you can explain what this revision means.

And also they’ve built a lot of new customizations, and they got us flush out a plugins API, and I’ll show one example here. So Chromium has a pretty heavyweight CI system because they have to build Chromium for a lot of different platforms and different architectures and they have a lot of incoming commits. So they need to find a way to verify that a commit that is coming into the code, actually works and in which order to check in those changes, so that is not breaking the build.

Screen Shot 2017-10-19 at 16.06.00.png

One thing that they’ve built for PolyGerrit is a plugin to add these elements to the screen for visibility to what’s going on in their system. The builds that are running for their commits, their status, often the flaky, so that by clicking on these you can see the output of the build. You can also use this UI to start a particular build and chose which architecture you want to run on or re-run things that failed. I think we should be building better integration to Gerrit for these sort of things because those are what big and small projects need, but it is cool that we can do these things with plugins as they are now.

What’s new in Gerrit 2.15 for PolyGerrit

Gerrit 2.14 was the first release that included PolyGerrit but it is just a sort of experimental preview, but that did work well out of the box. Since then, we have fixed a lot of bugs and added more features, we have converted the entire project to ES6 which for JavaScript developers makes their life so much easier and also makes the project more attractive to other contributors.

We made a lot of performance improvements in the application. For example, sometimes we have commits that have thousands of files, and we have to have a way to display them intelligently, we cannot just throw them on the screen. We needed a sort of new navigation for these kinds of things, and we have adapted PolyGerrit to work with any variety of commits of different shapes and sizes.

Some specific things we added that are kind of cute. PolyGerrit offers a reply dialog to manage your reviews, reply to comments, vote on labels, and submit that all as a single batch operation. We added some nice features to this reply dialog since Ver. 2.14, one is being able to preview how your reply will be formatted. Not just to show you how your answer is going to look like but now it renders in the same it would end up in the e-mail because we have HTML e-mails.

Screen Shot 2017-10-19 at 16.06.49.png

We have also reorganized how voting buttons are arranged, there is this beautiful vertical alignment, and you can see the labels that give you the meaning what +1 means for that label. We have a slightly updated patchset selector. That is important because one of the things people struggle with it is navigating a code-review. If there are more patches, understanding where the comments are can be challenging, because of comments land on different patchsets.

Screen Shot 2017-10-19 at 16.08.31.png

We also introduced the feature of resolvable comments, so that you can mark a comment as fixed or done and we can track that for you. You can see that on some patchsets there are some comments you haven’t reply to yet or has a review, and you can verify that everything has been addressed.

The PolyGerrit Admin UI

Another major area that we have been working at is one of the significant features that PolyGerrit has compared to the old version: the admin UI.

A lot of our users had to switch back and forth to do the admin work. We’ve been building up this part of the application. We have rearranged that a little bit, we changed the title by clicking on the browse link on the top menu. Our idea is that you can get to this like the project home page for a repository, where you have to check out instructions and is also the entry point for administering the project.

Screen Shot 2017-10-19 at 16.25.51.png

We have almost all the primary forms for managing groups and projects completed. We have also released a read-only version of the sophisticated project access rules editor. It is read-only mode because we don’t want anybody to risk losing their project.config because of a bug in this: project.config tends to be a very security sensitive thing.

What’s next for PolyGerrit.

We are still working on replacing the GWT UI; we’re getting close. One of the significant features remaining is in-line edit. We’ll be working on that over the next few months, as long as the long tail of the specific missing pieces of the UI. And then we are going to roll out a new visual design which Arnad will be talking about shortly.

We also want to do in the longer term more significant changes to the UI of Gerrit, to show how accessible is to the users, because when people learn how to use Gerrit they love it, but the initial learning curve is a bit difficult at the moment.

PolyGerrit UX New Look (Arnab Banerjee, Google)

We are working on a new look; I’ll just be giving you a quick glimpse of what it looks like. We know that Gerrit and PolyGerrit work very well for the users, but we want to improve its usability as well as the status of the tool.

For improving usability we think there is a long process. We don’t want to break what’s already working for you, and that’s why it will be more an evolutionary process based on user-research and feedback.

However, we started by looking at the aesthetics of PolyGerrit and giving it a new look. A few critical goal that we had is that design should represent what PolyGerrit is and stands for, having a more contained group of information in a way that you know exactly where to find a particular group of information. Also more consistent UI behavior across the board: as soon as you look at the component you see what you can expect what the element is supposed to do. Also, we want to introduce a more up-to-date style of components and fonts, which means: material design.

What should drive the experience we want to create is what we really want to achieve with PolyGerrit.

Screen Shot 2017-10-19 at 16.26.40.png

Confidence: we want you to be confident with the code you are submitting and is going out there.

Scalability: we want PolyGerrit to manage the small as the significant and more complicated change.

Efficiency: we want to increase the efficiency of your team as making the code review process more efficient.

Intelligence: PolyGerrit should be smart enough to do all the repetitive task that you otherwise would have had to do manually. Wisdom: PolyGerrit should be mean to share the knowledge on the team.

PolyGerrit is Blue

One color that captures all of that very very well is blue. So we picked blue and made the primary color of PolyGerrit.
That is going to be the default primary color but you can change it for your team.

Screen Shot 2017-10-19 at 16.28.34.png

That is the new and refreshed look of PolyGerrit.

There are three distinct layouts for PolyGerrit:

  1. The change screen
  2. Dashboards
  3. Admin screen.

PolyGerrit new Change Screen

The change screen is the most used screen in PolyGerrit, so we thought why not start by giving this a new look.
For this presentation, I will just talk about how the new PolyGerrit style works on it. Let’s talk about the visual elements on this page and look at it with a little bit of detail.

Screen Shot 2017-10-19 at 16.27.51.png

One of the things we have added in PolyGerrit is the breadcrumb where you can quickly jump back to wherever you came from. The next is the status, and now we have a clear status indication of the change so that just looking at this state you can understand if the change is it is closed or opened, and the color line adds to this the indication of the change.

Screen Shot 2017-10-19 at 16.30.54.png

Let’s talk about the meta-data of this page, we have now a much more visual structure of spaces, fonts and colors and actions that are now material flat buttons, and they are different from links, which will be all caps and different fonts altogether. So visually you see the difference in what will take you to another space and what will take action. When you click on add assignee, you get this in-line box which let you add assignee, and this is just a more elegant way of putting inline data, it just makes the action so much more clear.
When you click save, you can see the assignee name populated and assigned to the assignee label.

Once you start typing will just suggest the names while you are typing and topics also work in the same way.

Screen Shot 2017-10-19 at 16.31.29.png

Now coming arguably to the most used section of this page which is the files table, and you may notice these two header rows, the file heading and we just combined the information of these two rows into one row.

Let’s look at the elements of this particular section. The first part is the patchset selector. If have both the selected patchset as well as the original patchset, you can see them together side-by-side, so that you don’t have to look all the way to the right to select the referenced patchset.

Once you click on the patchset dropdown, you have a different layout for this different drop-down menu. On the left side you can see the patchset as well as the patchset description, and on the right side, you can look at the comments and the unresolved comments. That makes scanning through this list so much more comfortable, especially if this list is long.

Screen Shot 2017-10-19 at 16.32.17.png

Then adding patchset description also works in the very same way. It just shows the inline edit box, and you just enter the patchset description and say “save.”

Now we have a separate interaction for marking the file as reviewed when you hover on this rows. The mark reviewed shows up on the right-hand side, when you click it you can see a column on the right-hand side of the mark as reviewed and when you click review this text shows up a toggle button.

Screen Shot 2017-10-19 at 16.33.05.png

On the top-right, we have now the table actions, and we just have two actions by default. The first action is “expand all” and the other is “download.” If you click on expand all you see this diff view expanded and you see some icons that appear on the top. So the “expand all” button changes into “collapse all” and you can change from side-by-side to unified diff-view or even change your diff-view preferences.

That is just a tiny step towards what we can do and is just the tip of the iceberg, and I think we’ll reach out to you for more feedback and do more user research for really making PolyGerrit a tool that you’d love as code-review tool.

PolyGerrit User Research (Dustin Smith, Google)

I am a user researcher at Google, and my job is to communicate with our users and take what you tell me and distill what we can do and what changes we need to do to our product.
Being an advocate for all of you in the meetings because I am not a developer I am not a designer and my role is mainly to communicate what you need to our developers and feedback them to our designers. I collect feedback and distill your needs. Arnab will come up with new designs, and we will come back to you and say “how did we do based on your feedback?” Then we will repeat that process to infinity, and that’s how research gets integrated into the design.

So if you are not a member of the Gerrit Google Group  (repo-discuss) , I highly recommend you to join. You can yell at us there; you can say “I really don’t like what’s going on in the product, ” or you can praise us, whatever you like. At some point, you can send us surveys through there and hopefully not a very long survey that helps us aggregate all your needs.

User Survey results on Gerrit Code Review

An example of that we sent out a survey in June and 600 of you responded. We received a lot of nice praise, and that’s great, but we’re also looking at ways to improve Gerrit, and we asked you to rank of what would be your priority.

Screen Shot 2017-10-19 at 16.34.13.png

What you see on the top here is that all the people that answered the survey is interested in improving the navigation of Gerrit, whether it is a patchset navigation, commenting, or other UI navigation. There is also another related to better integration with other tools. We are curing that feedback. Something to keep in mind is that even though there are things at the bottom that are ranked at zero, that doesn’t mean that you don’t want those things but they are relatively listed.

Q (Han-Wen Nienhuys, Google). I am wondering, who is in this group of users you asked because one of the things entirely at the bottom says “improve the reliability of the backend” and then very much at the bottom is “better performance” which is kind of what our Team is continuously worried about. Even yesterday we had a Hackathon and the friends from the Android Team on Sunday morning, and we were all in panic, but I see that “nobody cares about it,” how comes?

A. That’s a fair question, who did we sample? These users were external users of Gerrit, and only 13 of them were PolyGerrit users, so what you’re looking at here, 95% of them are external users. Having said that, why ranking better performance and reliability the lowest, I am not sure how to answer that question.

A. (Gerrit admin from the audience). For us obviously reliability of the backend is important, but the question was “does reliability need improving?” and for us, Gerrit works quite well and we are very happy with it.

A. (Luca Milanesio, GerritForge) I believe that my is very similar to that. First of all, the people you are aking to. You’re not asking Gerrit administrator, but just people that are using as a UI. The people that are using the GUI are saying “reliability is currently 99.99%, do you need to make it 99.99999%? Not really”. What needs improving is what you guys are focused on: usability, discoverability, and leveraging all the power that is out there.

A. That’s a great point and keep in mind that all the items on the bottom of this list don’t mean that people don’t like them just that other aspects that need improvements are ranked above it.
People did not say “I dislike the idea of better performance, ” but more often than not, people selected the options above it before the better performance.

Other highlights from people, maybe people that is one of you, navigation is important “Getting an overview of the change I’m reviewing is hard. When a change is composed of several files, the file navigation feels like a pain”.

We are taking these feedback and then moving forward, and also some complaints about the search itself. The search could be better, for example, “I want to search a commit that includes a particular file.”

And then, integration. I would like more Jenkins integration for passing code coverage reports into Gerrit. If these quotes that I pulled out are jelly with you wonderful, if you like there are things that are missing and you are not being heard, I am the person to talk to. I need to make sure that you are heard. There are also some other options I am going to show you. How many of you are aware of userresearch.google.com?

PolyGerrit Research: call to action

If you’d like to, you can sign up here. What we will do is substantially having a look at the new designs, you can spend an hour with me, maybe 30′ depending on how busy you are, and I’ll show you some new designs, and you’ll get a gift for your time. I am not the only user searcher at Google, and this is not the only product that is supported by this, but I am interested in hearing your feedback about Gerrit and runs studies on here, and you’re welcome to join me.

Screen Shot 2017-10-19 at 16.34.56.png

Today and tomorrow we have a booth that you may have seen already with this banner, and we have a new dashboard design, a new patch navigation system and a new navigation for related changes that we’d loved to have you have a look at. We are not showing them on these slides on purpose because we want you to see the designs and come with a bunch of feedback on your first impression to the UI.

Please come to the booth and talk to any of us, there is Jason Buberel with us (our Product Manager), you’re welcome to come and speak with him as well, and almost all the Googlers here are willing to show some of the new designs.

Questions.

Q. How exactly you come out with a new design, the layout, how to make it better. How do you know that the user experience is better compared to the previous old one?

A. I take what your pinpoints, what you love, and I tell Arnab, and then he comes with a new concept. How do you know if that is better? Typically you generate tasks of what you would usually do with our UI, and we then do a tasks assessment where you see the goal that you go through those tasks and see if they failed. You talk aloud thinking while you go through those tasks and you may say “I really don’t like this new UI” and from that, you do mainly a mini A/B test. You can go through this process for just a few people and you don’t have to build up the entire product that everyone sees.

Q. I remember when a couple of years ago there was the announcement that we are going to have the new PolyGerrit project. We all asked “when the new GUI is going to be available?” and the answer was “six, maybe twelve months.” Now after two years we have something that works that is PolyGerrit, we started using it, and we like it. There are still gaps that are not really on the change screen, which is way better than the old one. I can use it on my Tube in the morning on the mobile phone while the old GUI did not even render or it was so tiny that you could not even see it. The gaps are more on the other parts of the UI. 50% of the users are using Gerrit as a very reliable Git server, have you put on your radar all the other use-cases not directly related to code-review that currently PolyGerrit is not covering?. The on-line editing, the code-browsing, the user-journeys from the code to the review and the way back, integrated search across code and the associated reviews?

A. Something is clearly in the scope of Gerrit, and something is not, but that does not mean we are not interested in it. Source browsing is one example of it, and I don’t think it will be in PolyGerrit but that does not mean that we are not interested in a way to browse source code. We are working on the in-line edit, maybe is not something we looked at the usability of it yet, it is a bit lower on the list of priorities, but are starting with the implementation and Arnad has been helping us with the basic design elements of that. It will be a basic feature at first, but it will look nice and be usable to some extent, and then we iterate.

Q. What about the completely new redesign? Are your efforts directed in completing what is there or are on reinventing everything from the ground up and we have to wait for another two years?

A. We are focused on the immediate terms of completing PolyGerrit because for many technical reasons we want to replace the GWT UI and we want to have more users using PolyGerrit so that we can get feedback to keep going. As we get more and more feedback and more and more iterations on PolyGerrit in the wild, we will come up with a lot more work for us to do. In the near term, we are going to implement what Arnad has presented and while we are doing that Arnad will help us by collecting some more feedback around what you will see at the booth, around the new dashboards and we are going to keep working on that. PolyGerrit is not going to stay static as it is today, it is going to be more iterative process based on Gerrit @Google and the Gerrit master branch. At every release cycle of the Gerrit OpenSource project, we will include some significant improvements on the user experience.

Q. I am just wondering how the new users get represented in your user research. Maybe people coming from GitHub or GitLab but even other people coming from different source control systems such as Perforce or SVN. How do you capture their experience? A lot of us have used Gerrit for a while and understand to a degree, Gerrit. However for other people, at times is a kind of a ping-pong, even though you are a new user for a very short time until you learn Gerrit.

A. On-boarding is definitely on our radar, we take users that have not used Gerrit before, and we ask to perform certain tasks. That is the kind of the recipe: take people that never used the product and working with Arnad on their feedback to make that process smoother.

Q. How many research you have made on the other code-review tools, advantages and disadvantages over Gerrit?

A (Dustin). It is an approach I am willing to take, doing a kind of competitive analysis. I have not started yet, but it is definitely on the roadmap for research.

A. (Jason). If there are other code-review tools that you’ve used and something that worked well, something that Gerrit or PolyGerrit could benefit from their knowledge or usage, please let us know. Nothing is off the table here concerning what we will improve or consider improving. If you used other code review tools, e.g. “hey reviewable.io got this very pretty good thing, you should do something really similar to that”, by all means, please let us know because we are opened to improvements in all areas.

Q. Following-up on this question, there will be talk tomorrow on this, our friends from CollabNet that are contributing to Gerrit Code Review have clients that are using not only Gerrit but even other tools like GitHub and many of them are hearing concerns of people getting lost without their pull requests. Because they are coming from other tools like Atlassian BitBucket, they have the concept that “without a pull request there is no review.” They have as well the misunderstanding that Gerrit forces you to amend the latest commit all the times, which is wrong. Gerrit allows you doing whatever you want. If you want to amend, you can do it; if you want to use feature branches you can do it as well. I remember Martin last year saying that they are using feature branches at Qualcomm and there is nothing wrong with it. CollabNet has developed a very basic UI that allows people that are coming from the pull request workflow to keep on using that paradigm. This UI is a sort of lightweight code review that helps to digest Gerrit gradually. That could be even the key to solve the dilemma raised on a recent discussion in the GoLang project where people asked to stop using Gerrit and instead using GitHub to review contributions.
Following the discussion thread, it emerged that Gerrit is “too good” and does exactly what they need while GitHub doesn’t. However, there are concerns about other potential contributors coming from a GitHub experience having to invest 15′, maybe 30′ or even one day of learning of the Gerrit Code Review workflow and a new UI paradigm. They may decide to give up and avoid contributing at all. If the project would move to GitHub pull requests, this problem would just disappear. Are you guys taking into consideration how to make this transition from GitHub smoother and Gerrit easier to understand for those guys?

A. That’s a great question, and as the formal product manager for the GoLang project, I heard from some members of the community of the project that they were relatively unwilling to contribute. We saw this type of disparity from the casual contributors. Those are the people that find a typo in the docs and just want to fix it quickly. With the GitHub UI is nice because they see the problem, they click Fork, edit the code on a single line or merely a single file, and they create the pull-request with a straightforward review process and gets merged and done. You never had to clone the repository, did not have to figure out the authentication bits, and that is simple. You know, whether we want or not to take Gerrit down a path where we want to make these casual contributions super-simple it is something we are open to if that is what the Gerrit community thinks is a significant capability. At the same time, we do not want to displace GitHub with Gerrit, there is a vast community there for a reason, but there are specific workflows like the casual contributors’ workflow that are certainly easier on GitHub and much more difficult for users that are new to Gerrit. It is an area where if we think there is a need for, we should think about it and consider how we may support such a thing. We are open to hearing more feedback on things like that.

A. I just came from the OpenStack developers’ summit in Colorado a couple of weeks ago, and we were having the same conversation regarding onboarding contributors to OpenStack. Most of the people coming from GitHub are having the same hurdle. One of the things that were discussed out there was having some documentation, some quick-start, cheat-sheet, crash course, for people that are familiar with GitHub. It is not just about enhancing Gerrit code to make it more friendly and port the workflow they are used to use but provide content that helps people, coming over.

A. Dave is one of the tech writers at Google who works explicitly with Code Review. If you have a specific part of the documentation set that you think it needs improving or a brand new part of the documentation set that could be useful to new users, come to him and talk today or during the next days. He will be happy to help.

A. I have started already building a spreadsheet, a kind of rosetta-stone of mapping concepts and commands for mapping one to the other and I’ll show you that. That is just one approach, yes, but there are some pros as well.