Well it’s Jan 2008 and I decided to run the Plesk 8.3 update and try out the new patch for AWStats previous month history. To my surprise, (well not really) none of the previous month statistics were available, only the current month. So I set out to find a way to generate the history of statistics for each domain that was using AWStats.
The following steps are also useful if you change your AWStats configuration for one or more domains and you want to rebuild the statisitcs pages using the new settings. For example, if you enable the GeoIP plugin, you can rebuild your past statistics pages and now see them with the client country of origin.
Each domain will have a set of AWStats config files in the directory /usr/local/psa/etc/awstats These configuration files have a parameter that tells AWStats where to store it’s monthly data statistics database. For example, the http stats will go in DirData=”/var/www/vhosts/mydomain.com/statistics/webstat/current This folder turns out to be a symlink to the current year / month folder.
drwxr-xr-x 3 root root 2048 Jan 5 14:05 .
dr-xr-x--- 7 root psaserv 1024 Jul 25 15:15 ..
drwxr-xr-x 2 root root 2048 Jan 6 03:49 2008-01
lrwxrwxrwx 1 root root 7 Jan 5 12:45 current -> 2008-01
-rwxr-xr-x 1 root root 457 Jan 6 03:49 index.html
-rwxr-xr-x 1 root root 955 Jan 6 03:49 nav.html
AWStats appears to ‘remember’ the last data parsed based on existing data files which are in the ‘current’ folder. Most likely you will only have one .txt file in this folder which represents the current month’s statistics data. The following rebuild script will remove any of these data files before rebuilding the full statistics history. After rebuilding this statistics data, you will notice a .txt file for each month inside of the ‘current’ folder. It is not quite clear yet if this is a good place to keep these files, however if needed the DirData parameter can be changed in each domain’s awstats config file.
Run this script with the domain name as the parameter
./awstats_rebuild.sh domain.com
Listing of awstats_rebuild.sh
#!/bin/sh
#
# Script to rebuild a domain's statistic data
#
# www.europheus.com
#
DOMAIN=$1
CONFIGDIR=/usr/local/psa/etc/awstats
AWSTATS=/var/www/cgi-bin/awstats/awstats.pl
# remove any current statistics data
rm -f /var/www/vhosts/${DOMAIN}/statistics/webstat/current/*.txt
rm -f /var/www/vhosts/${DOMAIN}/statistics/webstat-ssl/current/*.txt
rm -f /var/www/vhosts/${DOMAIN}/statistics/ftpstat/current/*.txt
rm -f /var/www/vhosts/${DOMAIN}/statistics/anon_ftpstat/current/*.txt
${AWSTATS} -update -config=${DOMAIN}-http -configdir=${CONFIGDIR} -LogFile=/var/www/vhosts/${DOMAIN}/statistics/logs/access_log.processed
${AWSTATS} -update -config=${DOMAIN}-https -configdir=${CONFIGDIR} -LogFile=/var/www/vhosts/${DOMAIN}/statistics/logs/access_ssl_log.processed
Now you should see a text file for each month in the DirData location. At this point, you can now generate the statistics HTML pages on a monthly basis for each domain. Here is a script that will help out somewhat, but it only pays attention to the HTTP stats, not the FTP stats. To produce the FTP stats, you should be able to clone the HTTP section, and change the -config parameter to the FTP conf file and update the outputdir.
Run this script with the month, year, and domain name as the parameters
./awstats_makemonth.sh 12 2007 domain.com
Listing of awstats_makemonth.sh
#!/bin/sh
#
# Script to create a domain's month statistics
#
# www.europheus.com
#
MONTH=$1
YEAR=$2
DOMAIN=$3
CONFIGDIR=/usr/local/psa/etc/awstats
OUTPUTDIR=/var/www/vhosts/${DOMAIN}/statistics/webstat/${YEAR}-${MONTH}
OUTPUTDIRSSL=/var/www/vhosts/${DOMAIN}/statistics/webstat-ssl/${YEAR}-${MONTH}
AWSTATS=/var/www/cgi-bin/awstats/awstats.pl
# create output dir if missing
if [ -d $OUTPUTDIR ] ; then
echo "Updating existing http statistics "${DOMAIN} ${YEAR}-${MONTH}
else
echo "Creating http statistics "${DOMAIN} ${YEAR}-${MONTH}
mkdir ${OUTPUTDIR}
fi
# Create each http stats web page
${AWSTATS} -month=${MONTH} -year=${YEAR} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output > ${OUTPUTDIR}/awstats.${DOMAIN}-http.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=alldomains > ${OUTPUTDIR}/awstats.${DOMAIN}-http.alldomains.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=allhosts > ${OUTPUTDIR}/awstats.${DOMAIN}-http.allhosts.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=lasthosts > ${OUTPUTDIR}/awstats.${DOMAIN}-http.lasthosts.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=unknownip > ${OUTPUTDIR}/awstats.${DOMAIN}-http.unknownip.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=allrobots > ${OUTPUTDIR}/awstats.${DOMAIN}-http.allrobots.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=lastrobots > ${OUTPUTDIR}/awstats.${DOMAIN}-http.lastrobots.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=session > ${OUTPUTDIR}/awstats.${DOMAIN}-http.session.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=urldetail > ${OUTPUTDIR}/awstats.${DOMAIN}-http.urldetail.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=urlentry > ${OUTPUTDIR}/awstats.${DOMAIN}-http.urlentry.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=urlexit > ${OUTPUTDIR}/awstats.${DOMAIN}-http.urlexit.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=osdetail > ${OUTPUTDIR}/awstats.${DOMAIN}-http.osdetail.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=unknownos > ${OUTPUTDIR}/awstats.${DOMAIN}-http.unknownos.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=browserdetail > ${OUTPUTDIR}/awstats.${DOMAIN}-http.browserdetail.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=unknownbrowser > ${OUTPUTDIR}/awstats.${DOMAIN}-http.unknownbrowser.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=refererse > ${OUTPUTDIR}/awstats.${DOMAIN}-http.refererse.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=refererpages > ${OUTPUTDIR}/awstats.${DOMAIN}-http.refererpages.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=keyphrases > ${OUTPUTDIR}/awstats.${DOMAIN}-http.keyphrases.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=keywords > ${OUTPUTDIR}/awstats.${DOMAIN}-http.keywords.html
${AWSTATS} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output=errors404 > ${OUTPUTDIR}/awstats.${DOMAIN}-http.errors404.html
# Create http index symlink
ln -s awstats.${DOMAIN}-http.html ${OUTPUTDIR}/index.html
# create output dir if missing
if [ -d $OUTPUTDIRSSL ] ; then
echo "Updating existing https statistics "${DOMAIN} ${YEAR}-${MONTH}
else
echo "Creating https statistics "${DOMAIN} ${YEAR}-${MONTH}
mkdir ${OUTPUTDIRSSL}
fi
# Create each https stats web page
${AWSTATS} -month=${MONTH} -year=${YEAR} -config=${DOMAIN}-http -staticlinks -configdir=${CONFIGDIR} -output > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=alldomains > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.alldomains.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=allhosts > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.allhosts.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=lasthosts > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.lasthosts.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=unknownip > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.unknownip.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=allrobots > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.allrobots.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=lastrobots > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.lastrobots.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=session > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.session.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=urldetail > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.urldetail.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=urlentry > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.urlentry.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=urlexit > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.urlexit.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=osdetail > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.osdetail.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=unknownos > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.unknownos.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=browserdetail > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.browserdetail.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=unknownbrowser > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.unknownbrowser.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=refererse > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.refererse.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=refererpages > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.refererpages.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=keyphrases > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.keyphrases.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=keywords > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.keywords.html
${AWSTATS} -config=${DOMAIN}-https -staticlinks -configdir=${CONFIGDIR} -output=errors404 > ${OUTPUTDIRSSL}/awstats.${DOMAIN}-http.errors404.html
# Create https index symlink
ln -s awstats.${DOMAIN}-https.html ${OUTPUTDIRSSL}/index.html
Finally run the Plesk statistics command to update the current statistics and to update the dropdown list in the HTML statistics page.
/usr/local/psa/admin/sbin/statistics –calculate-one –domain-name=mydomain.com
Or to run against all domains at once:
/usr/local/psa/admin/sbin/statistics
Finally to wrap these scripts up and update all of your AWStats enabled domains, try using the following script. You will need to modify the year loop and month loop to fit your needs. Also, take care and watch your CPU usage when running this script, AWStats can be a killer.
Run this script without any parameters
./awstats_updateall.sh
Listing of awstats_updateall.sh
#!/bin/sh
#
# Script to rebuild all awstat domains statistic data
#
# www.europheus.com
#
CONFIGDIR=/usr/local/psa/etc/awstats
VHOSTDIR=/var/www/vhosts
for domain in $( ls -1 $VHOSTDIR)
do
# check if there is awstats config for this domain
if [ -f "$CONFIGDIR/awstats.$domain-http.conf" ] ; then
echo "Rebuilding statistics data for "$domain
./awstats_rebuild.sh $domain
echo "Rebuilding HTML pages for "$domain
for year in 2007
do
for month in 01 02 03 04 05 06 07 08 09 10 11 12
do
./awstats_makemonth.sh $month $year $domain
done
done
#update stats
/usr/local/psa/admin/sbin/statistics --calculate-one --domain-name=$domain
fi
done
Enjoy, and please post any corrections or enhancements in the comments section.
Script Download Link
Download .zip file containing scripts in this article
18 replies on “Plesk 8.3 AWStats on Linux – Rebuilding Previous Month Statistics”
Really very useful. Thanks.
But where are some errors because of wrong symbols at your script… such as “,’, etc. Please add also download version with text file.
Ok, just added the scripts download link at the end of the article. 🙂
I’ve had Plesk 8.3 installed since December 2007, currently the dropdown menu in awstats for all domains just reads 2008-02 for so,me reason the January static pages that were created were deleted when we moved into February, which I’m sure isn’t how it should work ??
Thanks for this very helpful set of scripts – perfectly working on Debian with Plesk 8.3.0. I just had to create and symlink before:
# ln -s /usr/lib/cgi-bin/awstats.pl /var/www/cgi-bin/awstats/awstats.plCould have changed the script, too of course.
Greetings,
Chris
Paul, yes the static pages should not delete after each month, and it sounds like its not working correctly. You may want to check out the Plesk forums for any suggestions. At least you can use the scripts in this article to rebuild your December stats.
Hey there, I used your awstats_makemonth.sh script and wrote a Perl script which completely rebuilds the awstats statistics. It automagically rebuilds each month / year you have stats for as well. Not sure if it’s helpful or not but here it is: http://nickforsale.com/fixstats/
I’ve tested it extensively on my own server (Plesk 8.3.0 / CentOS 5) and it hasn’t broken anything but caveat emptor of course.
Thanks very much, this did the job after a few modifications. I am a bit dubious about leaving the awstats.month.txt data files in the current folder since it is just a symlink to 2008-02. Does anyone know if Plesk will place each txt file in its month folder as it goes along? Im guessing I will have all the existing ones are going to remain there, then future months will be placed in the correct month folder.
Hi….what about webalizer,is there any suggestion??I have problem with one domain on my server, the webalizer worked untill last December, january & february – current not running
The script seems to work, however it only uses the access_log.processed. What about the access_log.processed.{1-4}. If weekly logrotation is enabled, then regenerating using only access_log.processed will only give you 1 week of data 🙂
Anyone ideas?
I am with the same problem than Gijsbert
I use log rotation
Any chance of having the script searchin for processed files also?
I don’t have an answer to the log rotation issue. Maybe someone can post their modifications?
Nice work! I’ll have to do a cross post on this one 😉
I solve the rotation problem….
unpack the rotated logs and make one log file from all the rotated, using more 1.log >> 2.log
then i edit the awstats_rebuild.sh script to process the new log file (location)
Thanks for the scripts. They\\\’re great.
I am running Plesk 9.2.1 and there are some changes in the Awstats which envolves the script. So I modified the script to run with these new conditons:
1. You have to set an evironment variable to work with the configdir option. I added the following line:
export AWSTATS_ENABLE_CONFIG_DIR=YES
2. We had rotated and packed log files, so I added the following to import them (awstats_rebuild.sh):
UNPACKEDFILE=/tmp/unpacked_access_log_${DOMAIN}
gunzip -c $(for filename in `ls -rt /var/www/vhosts/${DOMAIN}/statistics/logs/access_log.*.gz`; do echo $filename; done) > ${UNPACKEDFILE}
${AWSTATS} -update -config=${DOMAIN}-http -configdir=${CONFIGDIR} -LogFile=${UNPACKEDFILE}
rm -f ${UNPACKEDFILE}
Hi There,
Thx for the great scripts. Really useful, however,when i ran this command :
/usr/local/psa/admin/sbin/statistics –calculate-one –domain-name=mydomain.com
after running the two first scripts (awstats_rebuild.sh and awstats_makemonth.sh ), the generated folder for the previous months (mm-yyyy) were deleted without prior notification. Sigh. How to update the current statistics and to update the dropdown list in the HTML statistics page without be worrying that the generation process above won\’t roll back again?
To my surprise, (well not really) none of the previous month statistics were available, only the current month.
This made me laugh. Thanks.
Thans a lot!
Anyone with the windows IIS version of this fix?