[Script] Block bad referrers in vestaCP

One of the nastiest problems when it comes to webserver resource and bandwidth usage is represented by the bad referrers which associate with a series of artificial queries from various hostnames that spam the crap out of the webserver and statistical logs.

Because the accuracy of the statistical traffic reports is highly influenced by such queries, the folks at Piwik have built a list of domains engaged in referrer spam that they keep updated.

I came across this list looking for something else and thought about forking an existing Github Gist into a script that would block bad referrers on both Nginx and Apache2 under vestaCP.

Why both Nginx and Apache2? Well depending on the specific configuration used by every webmaster either Nginx or Apache2 could be used as webserver, or even both with Nginx as a reverse proxy.

So, since building config files for both won’t hurt in any way, but would be globally effective I made the script below:

#!/bin/bash
# Referrer Spam generator for vestaCP
# by Malin Cenusa
# (malin[at]blackhat.pm)
# Inspired from https://gist.github.com/pikshub/5f200cd6695b683a5e5d
# v0.1

DATE=`date '+%Y-%m-%d %H:%M:%S'`
# Pulling the rerferer spam blacklist maintained by Piwik (remove first if existing)
rm -rf spammers.txt referrer_spam-nginx.conf referrer_spam-apache.conf
wget https://raw.githubusercontent.com/piwik/referrer-spam-blacklist/master/spammers.txt
# Prepare the config files with prepended data / where needed

        echo "# blacklist generated on $DATE" >> referrer_spam-nginx.conf
        echo "# blacklist generated on $DATE" >> referrer_spam-apache.conf
        echo '<Directory "/home/*/web/*/public_html">' >> referrer_spam-apache.conf

# Sort them and build the denies
sort spammers.txt | uniq | sed 's/\./\\\\./g' | while read host;
do

# Nginx
        echo "if (\$http_referer ~ '$host') {return 403;}" >> referrer_spam-nginx.conf

# Apache
        echo "RewriteCond %{HTTP_REFERER} $host [NC,OR]" >> referrer_spam-apache.conf
done;
# We need to also add the deny at the end of the Apache config file and close the directory tag
        echo "RewriteRule ^.* - [F,L]" >> referrer_spam-apache.conf
        echo "" >> referrer_spam-apache.conf

# Let's put the updated files in the config directories and reload the services
cp -a ./referrer_spam-nginx.conf /etc/nginx/conf.d/ && service nginx reload
cp -a ./referrer_spam-apache.conf /etc/apache2/conf-enabled/ && service apache2 reload

What this script does is pull the updated list from the Piwik repo, generate config files for both Nginx and Apache2, move them in place and reload the webserver daemons.

The script has been tested earlier today and confirmed to be working, yet if you happen to find bugs or wish to bring improvements you can let me know through the comments.

P.S. You may run the script manually when needed or set it to run on a cron and update the configs like once per week.