Previous Step:Translate
Next Step: Load
If you've been running a closed wiki and you're sure you have no spam, just skip this page.
On the other hand, if half of your edits were made by spambots, it's time to dump that dreck permanently. Removing all the spam and anti-spam edits reduced my site's total size from 60MB to 20MB and from 2006 edits down to 1272.. My site was almost 1/2 spam if you count edits and, if you count by total volume of each revision, it was 2/3 spam! Spam edits are usually quite large, usually adding around 10K per edit. Defacements, on the other hand, tend to remove more than they add.
Find the Spam
First, make sure you're working with an exploded directory of revisions. If not, go back to the Translate step.
Now, how can we find spammy edits? A bunch of ways.
Grep for it
Just look for spam.
$ grep -r 'v[1il]aga?ra' *
Hunt it and delete it in one step:
$ grep -lZir pu55y destdir | xargs -0 git rm $ git commit -m "Remove lame pu55y spam."
Check the file sizes
You can see it in the file sizes. For example, the Car page is 4.3K. Spam edits cause it to grow dramatically, revert edits return the page to its original size:
-rw-r--r-- 1 bronson bronson 4.3K 2007-07-14 09:10 Car-20070714-161051 -rw-r--r-- 1 bronson bronson 13K 2007-08-17 07:15 Car-20070817-141555 -rw-r--r-- 1 bronson bronson 27K 2007-08-19 09:29 Car-20070819-162947 -rw-r--r-- 1 bronson bronson 39K 2007-08-21 09:25 Car-20070821-162517 -rw-r--r-- 1 bronson bronson 4.3K 2007-08-21 15:47 Car-20070821-224749 -rw-r--r-- 1 bronson bronson 16K 2007-08-23 16:06 Car-20070823-230623 -rw-r--r-- 1 bronson bronson 30K 2007-08-25 21:20 Car-20070826-042054 -rw-r--r-- 1 bronson bronson 4.3K 2007-08-26 17:04 Car-20070827-000450
All of these edits except for the first one can be removed. Without the spam edits, the revert edits turn into no-ops.
etc
Please edit this page to add techniques that you find useful.
I played with a few more techniques, like 'find -newer' to bulk delete every revision after a known last-good revision.
Typical Spam
Here are some of the spammy things I saw.
Tiny Defacings
As mentioned in the description of the iki-diff-next command, some defacings are tiny:
########## changes made by nodes/Git-20071104-065732 @@ -1,3 +1,4 @@ +ergetorelta This page contains some notes that I took while I was learning git. == Support ==
I also got hit with "varacda", "pastaro" and other nonsense words. Is this just tagging?
Delete a Lot, Add a Little
You need to also be wary of changesets that delete a lot and add a little.
-=== Install RadRails === - -There have been significant improvements since the last RadRails beta so we'll tell Aptana to install the most recent milestone. - -* Window -> Preferences -> Install/Update -* Paste this URL into the Policy URL field: http://update.aptana.com/update/rails/beta/3.2/policy.xml -* Click OK - -Now, close Eclipse's Welcome tab. You should see the Aptana Start Page (if not, Help -> Aptana Start Page). At the bottom of the Plugins view you should see "Aptana RadRails". Click on "install" and let Eclipse restart again. +fhexo mvpz tzvhxjusb jwxslzbm nepjyioz pcdmnt kxwrvqn
I missed this one in the first scan through the file. Except for the last line, it looks like a decent enough edit. If it wasn't accompanied by the comment "aeosxvzlf vszl", I probably would have missed it entirely.
Spammy Words
Here are some words that were heavily used in spam and hardly appeared at all on my site. Beware, NSFW!
egrep -ir "interracial|porn|shemale|sex|adderall|nudity| \
lviv|cock|adult-|russian|bankruptcy|personals|bondage| \
cock|pussy|whore|loacal|cognac|pharma|webcam|spycam| \
CoffeeCup" . | less
Also,for some reaon a lot of pages were randomly defaced with "1000" always appearing at the beginning of the line.
grep -r "^1000"
That showed one or two legitimate pages of course, but it also showed a lot of defacings too.
Spam Links
If you're worried about spam, it's probably worth looking at all instances of http:// on your site.
egrep -r "https?://" .
You can instantly recognize most links as good