Managing your Linux/Unix log files using logrotate

This How-To details the steps required to manage and rotate your server’s log files. A simple truth about Linux/Unix logs are that they are everywhere. Your kernel, program daemons, firewalls, etc, generate their respective log files. In fact, there are so many log files of various levels that sometimes, it can be a nightmare to maintain them. Hence, this guide is a simple step towards maintaining those log files to keep your system in check and in good health.

Log files are one of the most important files where almost all precious and sometimes unnecessary information are stored in regard to your server’s running state. For example, if your system’s security has been breached or compromised, it’s these log files which will come to your rescue to help you identity where or what went wrong.

In case if you don’t know, your Linux/Unix server is currently logging kernel and security logs in the file called /var/log/messages. Just do a simple ” tail -f /var/log/messages ” to get feel and see the actual current logs generated by various daemons running on your system.

Now if your server also has a Apache Web server or a Squid Proxy server running and you want to manage their respective logs in your own fashion, then the following information might help you out.

First of all, you will need the program called “logrotate”. Logrotate is very useful utility which can rotate log files and archive them in a location that you specify. We will be using “logrotate” in conjunction with “cron“.

In Linux/Unix, cron is a time-based scheduling service in Unix-like computer operating systems. It is available on almost all versions of Linux and Unix.

Having said that, logrotate should be installed in your Linux/Unix distribution but if is not, simply use your system package management system to install it.

For example, for Debian based system, all you need to do to install logrotate is:

apt-get install logrotate

For this guide, we will be rotating and managing the log files generated by Apache and Squid on a FreeBSD-6.x and a Linux Debian-4.1 box. However, it should be also work on other Linux distributions like RedHat, Slackware or SuSE since the fundamentals are the same of all Linux based distributions.

I also assume that your Apache logs are kept in /var/log/apache and your Squid logs are kept in /var/log/squid.

On a FreeBSD-6.x box:

(1.) Make and Install from ports:

cd /usr/ports/sysutils/logrotate

(2.) Configure and Compile

make install clean

If all goes well, we are done and logrotate is installed.

(3.) Create a new logrotate.conf file.

vi /usr/local/etc/logrotate.conf

# Added the following to rotate Apache and Squid logs

# see “man logrotate” for details
# rotate log files weekly
#weekly
daily

# keep 4 weeks worth of backlogs
rotate 7

# send errors to root
#errors root

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
compress

# RPM packages drop log rotation information into this directory
include /usr/local/etc/logrotate.d

/var/log/lastlog {
monthly
rotate 12
}

# system-specific logs may be configured here

(4.) Create a directory for specific logrotate files

mkdir -p /usr/local/etc/logrotate.d

(5.) First, create a logrotate file for Squid to rotate it’s access.log files for 90 days and cache.log for 7 days.

cd /usr/local/etc/logrotate.d/

vi /usr/local/etc/logrotate.d/squid

#Copy and paste the following

/var/log/squid/access.log {
daily
rotate 90
copytruncate
compress
notifempty
missingok
}
/var/log/squid/cache.log {
daily
rotate 7
copytruncate
compress
notifempty
missingok
}

(6.) Create the necessary directories and files for logrotate and test and debug logrotate

mkdir /var/lib/

touch /var/lib/logrotate.status

/usr/local/sbin/logrotate -d /usr/local/etc/logrotate.conf
/usr/local/sbin/logrotate -f /usr/local/etc/logrotate.conf

(7.) Next, we will rotate and manage Apache logs

vi /usr/local/etc/logrotate.d/apache

#Add the following to rotate and manage Apache access_log and error_log for 30 days.

#Note: If your Apache logs may be in a different directory, simply change the directory.

/var/log/apache/access_log {
daily
rotate 30
copytruncate
compress
notifempty
missingok
}
/var/log/apache/error_log {
daily
rotate 30
copytruncate
compress
notifempty
missingok
}

If all goes well, that’s it. Your Apache and Squid logs should be rotated.

The last thing is to add an entry into crontab and letting the cron daemon rotate your Apache and Squid logs automatically.

(8.) Automating logrotate using crontab

vi /etc/crontab

#Add the following to rotate your logs at 1 AM in the morning

#Logrotate
0 1 * * * root /usr/local/sbin/logrotate /usr/local/etc/logrotate.conf > /dev/null 2>&1

That’s it. Your Apache and Squid logs will be rotating without manual intervention!!

Using logrotate on a Debian-4.1 box

(1.) Install the logrotate program

apt-get install logrotate

(2.) Create the necessary directories and files

mkdir -p /var/lib/logrotate/

touch /var/lib/logrotate/status

mkdir -p /etc/logrotate.d/

(3.) Create a new logrotate.conf

vi /etc/logrotate.conf

#Copy and paste the following

# see “man logrotate” for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp, or btmp — we’ll rotate them here
/var/log/wtmp {
missingok
monthly
create 0664 root utmp
rotate 1
}

/var/log/btmp {
missingok
monthly
create 0664 root utmp
rotate 1
}

# system-specific logs may be configured here
(4.) Create the squid logrotate file to rotate and manage access.log for 90 days and cache.log for 7 days.

vi /etc/logrotate.d/squid

#Copy and paste the following

/var/log/squid/access.log {
daily
rotate 90
copytruncate
compress
notifempty
missingok
}
/var/log/squid/cache.log {
daily
rotate 7
copytruncate
compress
notifempty
missingok
}

(5.) Create the Apache logrotate file to rotate and manage access_log for 30 days and error_log for 30days.

vi /etc/logrotate.d/apache

#Copy and paste the following. Note: your apache log’s directory might be different. Simply change the path of your directory.

/var/log/apache/access_log {
daily
rotate 30
copytruncate
compress
notifempty
missingok
}
/var/log/apache/error_log {
daily
rotate 30
copytruncate
compress
notifempty
missingok
}
(6.) Test and debug your logrotate configuration for any errors

/usr/sbin/logrotate -d /etc/logrotate.conf

/usr/sbin/logrotate -f /etc/logrotate.conf

If all goes well, you are good to go.

(7.) Now all that is left is to automate the logrotate process from crontab

vi /etc/crontab

#Copy and paste the following

#Logrotate at 1 AM in the morning

0 01 * * * root /usr/sbin/logrotate /etc/logrotate.conf > /dev/null 2>&1

That’s it! The cron daemon will automatically rotate your Apache and Squid logs at 1 AM on a daily basis.

Happy Log rotating !!!