Categories
Plesk Technology

Plesk 8.3 AWStats on Linux – Rebuilding Previous Month Statistics

Well it’s Jan 2008 and I decided to run the Plesk 8.3 update and try out the new patch for AWStats previous month history. To my surprise, (well not really) none of the previous month statistics were available, only the current month. So I set out to find a way to generate the history of statistics for each domain that was using AWStats.

The following steps are also useful if you change your AWStats configuration for one or more domains and you want to rebuild the statisitcs pages using the new settings. For example, if you enable the GeoIP plugin, you can rebuild your past statistics pages and now see them with the client country of origin.

Each domain will have a set of AWStats config files in the directory /usr/local/psa/etc/awstats These configuration files have a parameter that tells AWStats where to store it’s monthly data statistics database. For example, the http stats will go in DirData=”/var/www/vhosts/mydomain.com/statistics/webstat/current This folder turns out to be a symlink to the current year / month folder.


drwxr-xr-x  3 root root    2048 Jan  5 14:05 .
dr-xr-x---  7 root psaserv 1024 Jul 25 15:15 ..
drwxr-xr-x  2 root root    2048 Jan  6 03:49 2008-01
lrwxrwxrwx  1 root root       7 Jan  5 12:45 current -> 2008-01
-rwxr-xr-x  1 root root     457 Jan  6 03:49 index.html
-rwxr-xr-x  1 root root     955 Jan  6 03:49 nav.html

AWStats appears to ‘remember’ the last data parsed based on existing data files which are in the ‘current’ folder. Most likely you will only have one .txt file in this folder which represents the current month’s statistics data. The following rebuild script will remove any of these data files before rebuilding the full statistics history. After rebuilding this statistics data, you will notice a .txt file for each month inside of the ‘current’ folder. It is not quite clear yet if this is a good place to keep these files, however if needed the DirData parameter can be changed in each domain’s awstats config file.

Run this script with the domain name as the parameter
./awstats_rebuild.sh domain.com

Listing of awstats_rebuild.sh


#!/bin/sh
#
# Script to rebuild a domain's statistic data
#
# www.europheus.com
#

DOMAIN=$1
CONFIGDIR=/usr/local/psa/etc/awstats
AWSTATS=/var/www/cgi-bin/awstats/awstats.pl

# remove any current statistics data
rm -f /var/www/vhosts/${DOMAIN}/statistics/webstat/current/*.txt
rm -f /var/www/vhosts/${DOMAIN}/statistics/webstat-ssl/current/*.txt
rm -f /var/www/vhosts/${DOMAIN}/statistics/ftpstat/current/*.txt
rm -f /var/www/vhosts/${DOMAIN}/statistics/anon_ftpstat/current/*.txt

${AWSTATS} -update -config=${DOMAIN}-http -configdir=${CONFIGDIR} -LogFile=/var/www/vhosts/${DOMAIN}/statistics/logs/access_log.processed
${AWSTATS} -update -config=${DOMAIN}-https -configdir=${CONFIGDIR} -LogFile=/var/www/vhosts/${DOMAIN}/statistics/logs/access_ssl_log.processed

Now you should see a text file for each month in the DirData location. At this point, you can now generate the statistics HTML pages on a monthly basis for each domain. Here is a script that will help out somewhat, but it only pays attention to the HTTP stats, not the FTP stats. To produce the FTP stats, you should be able to clone the HTTP section, and change the -config parameter to the FTP conf file and update the outputdir.

Run this script with the month, year, and domain name as the parameters
./awstats_makemonth.sh 12 2007 domain.com

Listing of awstats_makemonth.sh


#!/bin/sh
#
# Script to create a domain's month statistics
#
# www.europheus.com
#

MONTH=$1
YEAR=$2
DOMAIN=$3
CONFIGDIR=/usr/local/psa/etc/awstats
OUTPUTDIR=/var/www/vhosts/${DOMAIN}/statistics/webstat/${YEAR}-${MONTH}
OUTPUTDIRSSL=/var/www/vhosts/${DOMAIN}/statistics/webstat-ssl/${YEAR}-${MONTH}
AWSTATS=/var/www/cgi-bin/awstats/awstats.pl

# create output dir if missing
if [ -d $OUTPUTDIR ] ; then
 echo "Updating existing http statistics "${DOMAIN} ${YEAR}-${MONTH}
else
 echo "Creating http statistics "${DOMAIN} ${YEAR}-${MONTH}
 mkdir ${OUTPUTDIR}
fi

# Create each http stats web page

${AWSTATS} -month=${MONTH} -year=${YEAR} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output > ${OUTPUTDIR}/awstats.${DOMAIN}-http.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=alldomains > ${OUTPUTDIR}/awstats.${DOMAIN}-http.alldomains.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=allhosts > ${OUTPUTDIR}/awstats.${DOMAIN}-http.allhosts.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=lasthosts > ${OUTPUTDIR}/awstats.${DOMAIN}-http.lasthosts.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=unknownip > ${OUTPUTDIR}/awstats.${DOMAIN}-http.unknownip.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=allrobots > ${OUTPUTDIR}/awstats.${DOMAIN}-http.allrobots.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=lastrobots > ${OUTPUTDIR}/awstats.${DOMAIN}-http.lastrobots.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=session > ${OUTPUTDIR}/awstats.${DOMAIN}-http.session.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=urldetail > ${OUTPUTDIR}/awstats.${DOMAIN}-http.urldetail.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=urlentry > ${OUTPUTDIR}/awstats.${DOMAIN}-http.urlentry.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=urlexit > ${OUTPUTDIR}/awstats.${DOMAIN}-http.urlexit.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=osdetail > ${OUTPUTDIR}/awstats.${DOMAIN}-http.osdetail.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=unknownos > ${OUTPUTDIR}/awstats.${DOMAIN}-http.unknownos.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=browserdetail > ${OUTPUTDIR}/awstats.${DOMAIN}-http.browserdetail.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=unknownbrowser > ${OUTPUTDIR}/awstats.${DOMAIN}-http.unknownbrowser.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=refererse > ${OUTPUTDIR}/awstats.${DOMAIN}-http.refererse.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=refererpages > ${OUTPUTDIR}/awstats.${DOMAIN}-http.refererpages.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=keyphrases > ${OUTPUTDIR}/awstats.${DOMAIN}-http.keyphrases.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=keywords > ${OUTPUTDIR}/awstats.${DOMAIN}-http.keywords.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=errors404 > ${OUTPUTDIR}/awstats.${DOMAIN}-http.errors404.html

# Create http index symlink
ln -s awstats.${DOMAIN}-http.html ${OUTPUTDIR}/index.html


# create output dir if missing
if [ -d $OUTPUTDIRSSL ] ; then
 echo "Updating existing https statistics "${DOMAIN} ${YEAR}-${MONTH}
else
 echo "Creating https statistics "${DOMAIN} ${YEAR}-${MONTH}
 mkdir ${OUTPUTDIRSSL}
fi

# Create each https stats web page

${AWSTATS} -month=${MONTH} -year=${YEAR} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=alldomains > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.alldomains.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=allhosts > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.allhosts.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=lasthosts > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.lasthosts.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=unknownip > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.unknownip.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=allrobots > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.allrobots.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=lastrobots > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.lastrobots.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=session > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.session.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=urldetail > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.urldetail.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=urlentry > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.urlentry.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=urlexit > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.urlexit.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=osdetail > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.osdetail.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=unknownos > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.unknownos.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=browserdetail > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.browserdetail.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=unknownbrowser > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.unknownbrowser.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=refererse > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.refererse.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=refererpages > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.refererpages.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=keyphrases > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.keyphrases.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=keywords > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.keywords.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=errors404 > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.errors404.html

# Create https index symlink
ln -s awstats.${DOMAIN}-https.html ${OUTPUTDIRSSL}/index.html

Finally run the Plesk statistics command to update the current statistics and to update the dropdown list in the HTML statistics page.

/usr/local/psa/admin/sbin/statistics –calculate-one –domain-name=mydomain.com

Or to run against all domains at once:

/usr/local/psa/admin/sbin/statistics

Finally to wrap these scripts up and update all of your AWStats enabled domains, try using the following script. You will need to modify the year loop and month loop to fit your needs. Also, take care and watch your CPU usage when running this script, AWStats can be a killer.

Run this script without any parameters
./awstats_updateall.sh

Listing of awstats_updateall.sh


#!/bin/sh
#
# Script to rebuild all awstat domains statistic data
#
# www.europheus.com
#

CONFIGDIR=/usr/local/psa/etc/awstats
VHOSTDIR=/var/www/vhosts

for domain in $( ls -1 $VHOSTDIR)
do
 # check if there is awstats config for this domain
 if [ -f "$CONFIGDIR/awstats.$domain-http.conf" ] ; then
         echo "Rebuilding statistics data for "$domain
         ./awstats_rebuild.sh $domain
         echo "Rebuilding HTML pages for "$domain
         for year in 2007
         do
           for month in 01 02 03 04 05 06 07 08 09 10 11 12
           do
                ./awstats_makemonth.sh $month $year $domain
           done
        done
        #update stats
        /usr/local/psa/admin/sbin/statistics --calculate-one --domain-name=$domain
  fi
done

Enjoy, and please post any corrections or enhancements in the comments section.

Script Download Link
Download .zip file containing scripts in this article

18 replies on “Plesk 8.3 AWStats on Linux – Rebuilding Previous Month Statistics”

I’ve had Plesk 8.3 installed since December 2007, currently the dropdown menu in awstats for all domains just reads 2008-02 for so,me reason the January static pages that were created were deleted when we moved into February, which I’m sure isn’t how it should work ??

Thanks for this very helpful set of scripts – perfectly working on Debian with Plesk 8.3.0. I just had to create and symlink before:
# ln -s /usr/lib/cgi-bin/awstats.pl /var/www/cgi-bin/awstats/awstats.pl
Could have changed the script, too of course.
Greetings,
Chris

Paul, yes the static pages should not delete after each month, and it sounds like its not working correctly. You may want to check out the Plesk forums for any suggestions. At least you can use the scripts in this article to rebuild your December stats.

Hey there, I used your awstats_makemonth.sh script and wrote a Perl script which completely rebuilds the awstats statistics. It automagically rebuilds each month / year you have stats for as well. Not sure if it’s helpful or not but here it is: http://nickforsale.com/fixstats/

I’ve tested it extensively on my own server (Plesk 8.3.0 / CentOS 5) and it hasn’t broken anything but caveat emptor of course.

Thanks very much, this did the job after a few modifications. I am a bit dubious about leaving the awstats.month.txt data files in the current folder since it is just a symlink to 2008-02. Does anyone know if Plesk will place each txt file in its month folder as it goes along? Im guessing I will have all the existing ones are going to remain there, then future months will be placed in the correct month folder.

Hi….what about webalizer,is there any suggestion??I have problem with one domain on my server, the webalizer worked untill last December, january & february – current not running

The script seems to work, however it only uses the access_log.processed. What about the access_log.processed.{1-4}. If weekly logrotation is enabled, then regenerating using only access_log.processed will only give you 1 week of data 🙂

Anyone ideas?

I am with the same problem than Gijsbert
I use log rotation
Any chance of having the script searchin for processed files also?

I don’t have an answer to the log rotation issue. Maybe someone can post their modifications?

I solve the rotation problem….

unpack the rotated logs and make one log file from all the rotated, using more 1.log >> 2.log

then i edit the awstats_rebuild.sh script to process the new log file (location)

Thanks for the scripts. They\\\’re great.

I am running Plesk 9.2.1 and there are some changes in the Awstats which envolves the script. So I modified the script to run with these new conditons:

1. You have to set an evironment variable to work with the configdir option. I added the following line:

export AWSTATS_ENABLE_CONFIG_DIR=YES

2. We had rotated and packed log files, so I added the following to import them (awstats_rebuild.sh):

UNPACKEDFILE=/tmp/unpacked_access_log_${DOMAIN}
gunzip -c $(for filename in `ls -rt /var/www/vhosts/${DOMAIN}/statistics/logs/access_log.*.gz`; do echo $filename; done) > ${UNPACKEDFILE}
${AWSTATS} -update -config=${DOMAIN}-http -configdir=${CONFIGDIR} -LogFile=${UNPACKEDFILE}
rm -f ${UNPACKEDFILE}

Hi There,

Thx for the great scripts. Really useful, however,when i ran this command :

/usr/local/psa/admin/sbin/statistics –calculate-one –domain-name=mydomain.com

after running the two first scripts (awstats_rebuild.sh and awstats_makemonth.sh ), the generated folder for the previous months (mm-yyyy) were deleted without prior notification. Sigh. How to update the current statistics and to update the dropdown list in the HTML statistics page without be worrying that the generation process above won\’t roll back again?

Leave a Reply to europheus Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.