GitHub planned outage

GitHub announced a scheduled downtime of its API starting from this forthcoming Saturday, 21st of March 2015 from 12PM UTC … I have to say that this is really the first time and I am quite surprised. I have always considered GitHub as one of the best examples of continuous deployment and feedback, allowing the transparent roll-out of dozen of changes every week; however sometimes even “The Rich Also Cry”.

What are the implications of this outage for GerritHub.io?

GerritHub.io uses the GitHub API for the following operations:

• Sign-Up and Sign-In to Gerrit Code Review GUI
• Import user profile, repositories and pull requests
• Gerrit groups lookup
• Replication using GitHub OAuth

As all the GitHub API would return 503 (Service Unavailable) the basic Gerrit Code Review functionalities could be eventually impacted.

How can we minimise the impact?

We will be rolling out longer cache TTL and cookie expiry times on Friday 20th of March on Gerrit Code Review, allowing to keep existing sessions for a much longer time up to 2 days validity. Similarly the Group and Accounts caches TTL will be extended in order to fill the GitHub API blackout.

And what about replication?

Whilst we can minimise the impact on Gerrit Code Review which is under our control, we can do little about GitHub availability: the commits pushed to GerritHub.io will be “parked” until GitHub services will be resumed again.

They will still be accessible to your Team but only through the GerritHub.io clone URLs.

What should I do when GitHub services will be resumed?

GitHub has not notified yet the length of his maintenance window but you will be able to receive notifications on its status on https://status.github.com and we will notify the progress and the impact on our services on https://gitenterprise.me, Twitter @gitenterprise and Facebook on https://facebook.com/gitenterprise.

Once the GitHub services will be back and fully operational, we do suggest to sign-in and verify the replication status of your repositories to GitHub, checking the SHA-1 of your branches on GerritHub.io against the corresponding ones on GitHub.

Example on how check the replication status of myorg/myrepo:

$ git ls-remote https://review.gerrithub.io/myorg/myrepo | \
  egrep -e "(heads|tags)" | awk '{print $2"\t"$1}' | \
  sort > /tmp/myrepo.gerrit
$ git ls-remote https://github.com/myorg/myrepo | \
  egrep -e "(heads|tags)" | awk '{print $2"\t"$1}' | \
  sort > /tmp/myrepo.github
$ diff /tmp/myrepo.gerrit /tmp/myrepo.github

What should I do to resync the repositories?

First of all you need to establish which one is the “source of truth”. If you have been using GerritHub.io as main code review, then the answer is always review.gerrithub.io.
In order to resync your GitHub repository, you just need to manually pull from review.gerrithub.io and push to github.com.

Example on how to resync myorg/myrepo:

$ git clone --mirror https://review.gerrithub.io/myorg/myrepo 
$ cd myrepo.git
$ git push --all --tags https://github.com/myorg/myrepo

What should I do if the push to GitHub fails?

There is not a unique answer to this question: if the push fails it means that your GerritHub.io and GitHub.com repositories started diverging. This happens when people pushes directly to GitHub without going any Code Review, which is potentially possible if you have left the permissions doors wide opened on GitHub.

My suggestion is always to check what is in GitHub that has not gone through Gerrit Code Review and, if possible and does not create conflicts, pull that set of commits into your GerritHub.io repository.

Example of pulling changes from a GitHub branch (e.g. mybranch) that are not contained in GerritHub.io:

$ git clone https://review.gerrithub.io/myorg/myrepo 
$ cd myrepo && git checkout mybranch
$ git pull https://github.com/myorg/myrepo mybranch
$ git push origin mybranch

Questions? Doubts? Problems?

If you have any questions or you need any assistance during the outage because you are experiencing problems, feel free to contact our customer support at https://gerritforge.com/support or tweet us at @gitenterprise.

Alternatively for any Gerrit-related problems, the best free source of information is always the Gerrit mailing list at https://groups.google.com/forum/#!forum/repo-discuss.