165

I was working on a project, a private repo, and suddenly all the commits disappeared and were replaced with a single text file saying

To recover your lost code and avoid leaking it: Send us 0.1 Bitcoin (BTC) to our Bitcoin address 1ES14c7qLb5CYhLMUekctxLgc1FV2Ti9DA and contact us by Email at admin@gitsbackup.com with your Git login and a Proof of Payment. If you are unsure if we have your data, contact us and we will send you a proof. Your code is downloaded and backed up on our servers. If we dont receive your payment in the next 10 Days, we will make your code public or use them otherwise.

At the time of this happening, Google search didn't show up anything, but in an hour or so this started coming up.

I am using SourceTree (always up-to-date) but somehow I doubt that SourceTree is the issue, or that my system (Windows 10) was compromised. I'm not saying it's not that, it's just that I doubt it.

This happened only to one of my repositories (all of them private) and all the others were left untouched. I changed my password, enabled 2 factor authentication, removed one access token that I wasn't using for years and wrote an email to GitLab in the hopes that they could tell me something about where/who the attacker got in.

My password was a weak one that could've been relatively easily cracked via brute-force (it's not a common one but starts with "a" and has only a-z characters in it) and it could be that they just automatically checked if they can access the account and then ran some git commands. It is also possible that my email address and that particular password are on a list of leaked accounts. One might argue that if this is how they got in, they would've simply changed the account credentials but searching the Internet revealed that in these cases GitLab/GitHub will simply restore the credentials for you, and so I assume this is why they didn't do it this way.

Could've also been that old access token, I can't remember what and where I used it for in the past - most likely generated for use on a computer I previously owned, so I doubt that that was the issue.

There are also 4 developers working on it, all having full access to the repository, so their accounts being compromised is also a possibility.

I've scanned my computer with BitDefender and couldn't find anything but I am not doing shady things on the internet so I don't think that me being infected with a malware/trojan is what caused this.

I am waiting for an answer from GitLab and maybe they can shed some light on this. I have the code base on my local Git, so that is not an issue, but I am not pushing the code back to the repository just yet. Also, just in case the code gets published somewhere, I will change any passwords that are to be found in the source (databases, IMAP accounts)

UPDATE

I found out that the code isn't gone. I tried accessing a commit's hash and it worked. So the code is there but there's something wrong with the HEAD. My knowledge on this is very limited but

git reflog

shows all my commits.

What this means to me is that the attackers most likely didn't clone the repositories (would be a logistical nightmare to do this for all the victims, anyway) and that the chances for them going over the source code looking for sensitive data, or of making the code public are low. It also means to me that is not a targeted attack but a random, bulk attack, carried out by a script. I really hope this is the case for our own sake!

UPDATE 2

So, if you do

git checkout origin/master

you will see the attacker's commit

git checkout master

you will see all your files

git checkout origin/master
git reflog # take the SHA of the last commit of yours
git reset [SHA]

will fix your origin/master...but

git status

now will say

HEAD detached from origin/master

still searching for a fix on this

UPDATE 3

If you have the files locally, running

git push origin HEAD:master --force

will fix everything. See Peter's comment

So, the question is what commands will get my repository back to the previously working state assuming you don't have the repo locally, as for how the attacked got in, I am hoping that the answer from GitLab (if any) will help us more.

There is a discussion going on here

The attack targets GitHub, BitBucket and GitLab accounts. Here's the magnitude on GitHub's public repos

  • 1
    Comments are not for extended discussion; this conversation has been moved to chat. – Rory Alsop May 6 at 23:14
  • Update: I sent an abuse complaint to NameCheap (the domain registrar of the email) and they just got back to me stating that they've suspended the domain. – gparyani May 11 at 20:24
  • isn't there some kind of authority for tracking down scammers/hackers? – Stefan Gabos May 13 at 12:43
76

You can use git reflog in a clone and checkout the last commit before this happened.

It happened because .git/config on your webserver (in the directory of the cloned repo) includes the remote URLs and people added username:password in it which should never be the case - people should use SSH, deploy keys or authenticate on each pull. Never store your credentials in a config file. Use the credential helper(s).

Source: https://www.reddit.com/r/git/comments/bk1eco/comment/emg3cxg

hello, it is me , the guy with your backups ..

i will reveal your sins

Here is an article from 2015, its more detailed, https://en.internetwache.org/dont-publicly-expose-git-or-how-we-downloaded-your-websites-sourcecode-an-analysis-of-alexas-1m-28-07-2015/

Article by Internetwache about this: https://en.internetwache.org/dont-publicly-expose-git-or-how-we-downloaded-your-websites-sourcecode-an-analysis-of-alexas-1m-28-07-2015/

To prevent this either block access to directories starting with a dot, see https://github.com/h5bp/html5-boilerplate/blob/master/dist/.htaccess#L528-L551

# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

# Block access to all hidden files and directories with the exception of
# the visible content from within the `/.well-known/` hidden directory.
#
# These types of files usually contain user preferences or the preserved
# state of an utility, and can include rather private places like, for
# example, the `.git` or `.svn` directories.
#
# The `/.well-known/` directory represents the standard (RFC 5785) path
# prefix for "well-known locations" (e.g.: `/.well-known/manifest.json`,
# `/.well-known/keybase.txt`), and therefore, access to its visible
# content should not be blocked.
#
# https://www.mnot.net/blog/2010/04/07/well-known
# https://tools.ietf.org/html/rfc5785

<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteCond %{REQUEST_URI} "!(^|/)\.well-known/([^./]+./?)+$" [NC]
    RewriteCond %{SCRIPT_FILENAME} -d [OR]
    RewriteCond %{SCRIPT_FILENAME} -f
    RewriteRule "(^|/)\." - [F]
</IfModule>

Or separate the .git directory and the data using --separate-git-dir.

--separate-git-dir=<git dir>
Instead of initializing the repository as a directory to either $GIT_DIR or ./.git/, create a text file there containing the path to the actual repository. This file acts as filesystem-agnostic Git symbolic link to the repository.

If this is reinitialization, the repository will be moved to the specified path.

But the best is to rm -rf .git after a deployment - which should just copy a build artefact to the destination using rsync.

https://git-scm.com/docs/git-init#Documentation/git-init.txt---separate-git-dirltgitdirgt

--separate-git-dir=<git dir>
Instead of placing the cloned repository where it is supposed to be, place the cloned repository at the specified directory, then make a filesystem-agnostic Git symbolic link to there. The result is Git repository can be separated from working tree.

https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---separate-git-dirltgitdirgt

https://stackoverflow.com/a/8603156/753676

Information about deploy keys and the credential helpers:

https://developer.github.com/v3/guides/managing-deploy-keys/

Deploy keys are read-only by default, but you can give them write access when adding them to a repository.

https://gist.github.com/zhujunsan/a0becf82ade50ed06115

https://help.github.com/en/articles/caching-your-github-password-in-git

Use git push -u origin master -f && git push --tags -f from your local clone to push all references for master, tags and so on to the remote and then enable 2FA in your account.

If more branches are affected use git push -u --all -f

Also please enable 2FA to decrease the possibility of such attacks.

Please do not forget to change all compromised logins / passwords and revoke any unknown sessions.

  • 8
    Do you have a source for this claim? How are the attackers getting access to the .git/config? If they can access that file, they can presumably access the entire .git folder anyway, an attacker being able to access the entire source code and history is still a significant issue for private repos. – JosephH May 5 at 8:35
  • 2
    There are tools for scanning .git directories. Access to dotfiles and directories should be blocked in general, see the html5boilerplate .htaccess file for the needed rule. With Git you can also separate the location for the data and the .git directory too. Anyway, deploying to a server should be a one way task (rsync and so on) and deploy keys should be used. – Daniel Ruf May 5 at 9:45
  • 3
    it is common sense, but please add to the answer one last step - once you have regained access to your repository go and change any passwords and usernames that you might have had in the source code. or anything else that can be used against you for that matter, though I cannot come up with anything of sorts – Stefan Gabos May 6 at 6:24
  • 2
    It should be mentioned that the .git/config discovering happens on installed servers not on github, in case you wondered why you would need to fix GH settings (you dont] – eckes May 6 at 6:59
  • 3
    This goes to show, once again: Git is not a deployment tool. I shudder every time someone mentions they just use git push to deploy their site... – sleske May 6 at 11:45
50

I doubt that the hackers pushed a "delete all" commit, or else you could simply revert the last commit. Rather, they force-pushed a different commit with the note to the HEAD of the master branch, making it look like your entire commit history is gone.

As others have pointed out, you can easily use a local repo to force push the correct code to the server. Due to the distributed nature of Git, this always works whether or not the server was wiped since every local repo has a complete clone of the server, including both commits and code. Of course, you should make sure the server has been secured first before attempting recovery efforts. :-)

If you don't have a local repo that includes the most recent commit, the commit history (and all associated files) will still exist on the server for a while. However, the server will eventually run git gc, which will clean up those unreachable commits. As of 2013, GitHub said they will run git gc at most once per day but it can also be triggered manually, while BitBucket will run it as needed, or perhaps after each push. GitLab runs it after 200 pushes by default, or it can be triggered manually.

However, even if all of the commits and files are still on the server, you would need to find the hash of the commit so you can restore it. Without a local repo with a reflog, it's hard to find the correct commit to restore. Some ideas that you could try:

  • Pull requests are typically kept forever, so you should be able to look at the most recent pull request merged into the master branch. Just make sure to pick the hash of the merge commit, not the hash of the branch. (GitHub has a green check mark next to the merge commit hash, GitLab shows "merged into master with", not sure about BitBucket).
  • If you have a build server, see what the most recent build of the master branch was (perhaps in the build log?)
  • You may want to check forks of your repo as well. GitHub allows you to see them either in Forks or Network views.

Once you find the correct hash for master, you can restore your server using the following commands (assuming you have a Git remote called 'origin').

git fetch origin <hash>
git checkout master
git reset --hard <hash>
git push --force origin master:master

Note that you should never use git push --force unless you intend to overwrite someone's work.

  • If OP doesn't have the latest data in their local repository anymore, how can they safely pull & merge changes from the server? I imagine the commit history could have been maliciously rewritten on the server. (Welcome to Security.SE, Matt!) – ComFreek May 4 at 9:20
  • 1
    If there are forks, use the newest fork. Or the PR which was recently rebased against the master branch. Otherwise, the data may be lost if the platform does not have its own reflog or backup of all (past) objects. – Daniel Ruf May 4 at 15:55
  • 2
    But a cloned repo does not always contain all references (tags + untracked branches). – Daniel Ruf May 4 at 15:57
  • 1
    @ComFreek The second half of my answer addresses the scenario where the local repo doesn't exist or wasn't up to date. When Git performs a hash, it will hash both the files and the commit history of that branch. Once you locate the most recent commit hash, then you have recovered the files and commit history, assuming they haven't been garbage collected yet. (Git uses SHA-1 hashes, so it would be incredibly difficult to engineer a hash collision.) Once the server has been recovered, all of the local repos should be able to pull/merge as normal. – Matt May 6 at 17:11
6

If more branches are affected, you may need to checkout all branches first with the following command before performing git push -u --all -f

for branch in `git branch -a | grep remotes | grep -v HEAD | grep -v master `; do
   git branch --track ${branch#remotes/origin/} $branch
done

https://gist.github.com/octasimo/66f3cc230725d1cf1421

3

I guess you already know the more obvious, but nevertheless:

  1. Going forward, set up SSH for communicating with GitLab (and any other remote that supports it, for that matter) instead of username+password, as @Daniel Ruf has advised.

  2. Configure a very strong password (on the order of 16+ randomly-generated characters) for your GitLab account, and use a password manager to manage it.

  3. Make sure your computer is not compromised. I would go one step further and change passwords for all of my online accounts just in case.

Now to address another pressing matter:

What this means to me is that the attackers most likely didn't clone the repositories (would be a logistical nightmare to do this for all the victims, anyway) (assumption #1)
(...)
and that the chances for them going over the source code looking for sensitive data, or of making the code public are low (...) (assumption #2)

It also means to me that is not a targeted attack but a random, bulk attack, carried out by a script (...) (assumption #3)

Assumptions #1 and #3 may or may not be true (I personally don't think it's a logistical nightmare at all to clone repos when your plan is to deface them for ransom - the attacker may well have a server dedicated for that task, configured through a VPN or the sort. And it might be that you were targeted). But they're not very crucial.

However, assumption #2 is one you can't afford to make right now.

If the code or repo history contained private information or any sort of trade secret, begin taking contingency steps immediately.

To quote part of their message:

If we dont receive your payment in the next 10 Days, we will make your code public or use them otherwise.

I'm afraid it's safe for you to assume they will do that whether you pay the ransom or not. Specially the "use them otherwise" bit.

  • 1
    right. i was speaking strictly about my case and what this means to me. absolutely, if one has sensitive data (like usernames and passwords) they should go and change them right away – Stefan Gabos May 6 at 6:22
  • 3
    For passwords / credentials for apps use .env files - there are PHP packages and other things to use them - secrets should never be in any Git history. – Daniel Ruf May 6 at 6:46
  • thanks! i didn't know about this! will look into it right away – Stefan Gabos May 6 at 11:46

protected by Community May 4 at 10:56

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.