Locally testing destructive Git actions

Saturday, 24 May, 2014 — git development

My team is responsible for maintaining a production line in our various Git repositories. After releases, we’re responsible for merging the development line to the production line, so any bug fix release starts from the basis of what’s currently running in production.

Making a longer story shorter, we had a release that we cancelled and decided to roll into a future release. But this was after we’d already made the merge to the production branch. We further had a mix of fast-forward commits with no merge commit and non-fast-forward merges across the different repositories that comprise a product release. We want to get our production line back to where production is actually at.

I researched a few strategies, centered around git revert and git reset for how to step forward and wanted to test them out.

Since each Git clone is complete unto itself (and we use a single remote versus multiple points of authority), I have the entire public history. So, it’s very easy to make a copy of a repo, make a bare clone as a “remote” and test changes locally.

Before we start

First, some words about safety. Out of an abundance of caution, I recommend the following before we start the experiment:

• Make sure your remote origin is up-to-date
• Make sure you are up-to-date with the remote origin
• Make sure the remote origin is backed-up
• Disable networking on your machine until the experiment is concluded

I also need your understanding that I’m describing procedures for experimentation, not trying to solve for your problem. We’re going to use git reset below and it is a destructive action. You own your data and are much better equipped to understand your repository and situation. Act with prudence.

If you’re looking for info on addressing various changes to repositories, you’d do worse than having a look at Kai Howelmeyer’s “Git Undo by Example” post.

Local and destructive testing

Our safety pre-check complete, here’s how to test destructive Git repository changes locally:

1. Determine where you’re going to be working on this test. I recommend something like git-experiment underneath /tmp.

   1
   $ mkdir -p /tmp/git-experiement/
   

2. Copy the repository in question to git-experiment, or clone if it’s a small repository. For this example, let’s pretend this repository is named smith. I’m going to refer to this as the local clone.

   1
   $ cp -R /path/to/repo/smith /tmp/git-experiment/smith
   

3. Create a bare clone of smith in a new remotes directory. This will allow us to push the local clone to what we’ll call the remote.

   1
   2
   3
   $ cd /tmp/git-experiment
   $ mkdir remotes
   $ git clone --bare smith remotes/smith.git
   

4. Go into your experimental copy of smith and open the .git/config file and update the url for the remote origin to point to your bare clone of smith.

   1
   2
   $ cd smith/.git
   $ vim config
   

   Before:

   1
   2
   3
   4
   5
   6
   7
   8
   9
   [remote "origin"]
       url = git@example.com:smith.git
       fetch = +refs/heads/*:refs/remotes/origin/*
   [branch "master"]
       remote = origin
       merge = refs/heads/master
   [branch "production"]
       remote = origin
       merge = refs/heads/production
   

   After:

   1
   2
   3
   4
   5
   6
   7
   8
   9
   [remote "origin"]
       url = file:///tmp/git-experiment/remotes/smith.git
       fetch = +refs/heads/*:refs/remotes/origin/*
   [branch "master"]
       remote = origin
       merge = refs/heads/master
   [branch "production"]
       remote = origin
       merge = refs/heads/production
   

5. Go into the remote clone of smith. Instead of .git/config, open config here. Remove the remote origin block. Also remove branch info that ties back to origin. This isn’t strictly necessary, but we’re just going to keep things clean and not run the risk of pushing changes where they shouldn’t go.

   1
   2
   $ cd ../remotes/smith.git
   $ vim config
   

   Before:

   1
   2
   3
   4
   5
   6
   7
   8
   9
   [remote "origin"]
       url = git@example.com:smith.git
       fetch = +refs/heads/*:refs/remotes/origin/*
   [branch "master"]
       remote = origin
       merge = refs/heads/master
   [branch "production"]
       remote = origin
       merge = refs/heads/production
   

   After:

   1
   2
   3
   4
   [branch "master"]
       merge = refs/heads/master
   [branch "production"]
       merge = refs/heads/production
   

6. With this done, clone the bare remote repository as a second working copy, if you’d like to check the workflow once you publish your destructive change.

   1
   2
   $ cd /tmp/git-experiment
   $ git clone remotes/smith.git smith-two
   

7. Since there’s a good chance you’ll want to try a few different approaches of your destructive change, make a copy of your test set-up and set it as read only:

   1
   2
   3
   $ cd /tmp
   $ cp -R git-experiment git-experiment.orig
   $ chmod -R ugo-w git-experiment.orig
   

8. Go into your experimental clone and make your destructive change and see what happens with it. You can safely push to your remote and see how that process works with doing a push and then git fetching or git pulling changes into the second copy.

   In my case, I was experimenting with reverting branch merges and resetting and doing git push --force to publish changes. Below, I’m using HEAD@{3} but this could just as easily be a hash reference or a tag.

   1
   2
   3
   4
   $ cd /tmp/git-experiment/smith
   $ git checkout production
   $ git reset --hard HEAD@{3}
   $ git push --force
   

   The --force is important here, because otherwise, Git believes you to be doing something it doesn’t think you should. And 99 percent of the time, Git would be correct, but we’re explicitly seeking to do something destructive.

9. Go check the results of your experiment in the second local clone:

   1
   2
   3
   $ cd /tmp/git-experiment/smith-two
   $ git checkout production
   $ git pull --rebase
   

10. When you’re done, package everything up with a note or just rm -rf /tmp/git-experiment. If you’re saving the iteration, give it a meaningful name:

    1
    2
    $ cd /tmp
    $ tar -czf git-experiment.git-reset.tgz git-experiment
    

Results and Iteration

The first experiment is complete. You can shake out what to do and discuss it with your team, safely not affecting what other users are seeing on a repository. Determine what your next steps are, then move your original test aside, copy in the back-up and run further iterations.

In particular, you may have other questions to consider, such as determining what happens if you force a merge commit, provide tags, delete branches, and so on. What’s it going to be like publishing the changes? What’s it going to be like trying to fetch or pull the changes? Are there any other workflow considerations? How are you going to inform other users?

Hopefully, these experiments help you sort that out and provide a well-known path for your users. This contained experiment gives you a margin of safety to answer those questions.

Happy experimenting.

Git cloning and branch heads

Saturday, 8 September, 2012 — git development

Last week at work, I was trying to diagnose apparent trouble with a Git repository. Whenever the infrequently updated repository was cloned, we would end up in a different branch than our devel branch. Not knowing if there was a problem in the remote bare repository, I dug around refs/heads and HEAD on both the remote repository and in a fresh local clone. Not seeing any apparent trouble, I Googled.

I came across the following mailing list post that explained my trouble:

… the git protocol does not expose which branch HEAD points to, only which commit. So if two branches point to the same commit, git just takes the first branch and points the local HEAD to that.

How we ended up in that situation is like this:

• Starting with an existing repository, create a branch off of master or whatever other branch you treat as your default HEAD
  • NB: We use devel as a historical artifact
• Publish the new branch to the remote origin
• Clone the repository somewhere else and examine the new clone. There’s a good chance the new repo will be using the new branch instead of master

I created a GitHub repository to demonstrate (although there’s no guarantee you’ll get the new branch on clone).

1
2
3
4
5
6
7
$ git clone https://github.com/base10/branch-head-examples

(add files)

$ git push
$ git branch feature
$ git push -u origin feature

Now, I’m going to look at a fresh clone:

1
2
3
$ git clone git://github.com/base10/branch-head-examples.git clone-example
$ ls clone-example/.git/refs/heads
master

So, I didn’t get feature. If however, in your clone, you saw this …

1
2
$ ls clone-example/.git/refs/heads
feature

… you would just need to do this:

1
2
3
4
$ cd clone-example
$ git checkout master
Branch feature set up to track remote branch master from origin.
Switched to a new branch 'master'

Now, diff the refs/heads files:

1
2
$ diff .git/refs/heads/master .git/refs/heads/feature
$

Getting an unexpected branch from a repository in this state isn’t harmful. It is a little disconcerting if you’re looking at a repository you established, but don’t work with frequently. I suspect this case is more likely to come up under the following circumstances:

• You don’t have a master branch at all (which might be true if you’ve converted a Subversion repository) or master is a dead branch
• You have a mainline of development and create a release branch from the mainline of development
• The repository, with all branches, is pushed
• The repository is later cloned, with no further changes.