Mar 282008
 

Sponsored Link

Vlogger is a little piece of code borned to handle dealing with large amounts of virtualhost logs. It's bad news that apache can't do this on its own. Vlogger takes piped input from apache, splits it off to separate files based on the first field. It uses a file handle cache so it can't run out of file descriptors. It will also start a new logfile every night at midnight, and maintain a symlink to the most recent file. For security, it can drop privileges and do a chroot to the logs directory.

Install vlogger in debian etch

#aptitude install vlogger

This will complete the installation

First you need to make sure you have working apache server

Configuring vlogger

Now you have to change the LogFormat line (there are multiple LogFormat lines -- at least change the one that is named combined) in /etc/apache2/apache2.conf. We must add the string %v at the beginning of it

vi /etc/apache2/apache2.conf

#LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

save and exit the file

Then add the following CustomLog line to the same file (you can put it directly after the LogFormat line)

vi /etc/apache2/apache2.conf

CustomLog "| /usr/sbin/vlogger -s access.log /var/log/apache2″ combined

save and exit the file

NOTE :- Only CustomLog directive that we need in our whole Apache configuration. Please disable all other CustomLog directives, especially in your virtual host configurations

Now restart apache using the following command

#/etc/init.d/apache2 restart

Vlogger will now create subdirectories in the /var/log/apache2 directory, one per virtual host, and it will create access logs that contain the current date in the file name. It will also create a symlink called access.log that points to the current log file.

Let's assume we have two virtual hosts, www.domain1.com and www.domain2.com. Then this is how the /var/log/lighttpd directory will look like:

/var/log/apache2/

www.domain1.com/
02142008-access.log
02132008-access.log
02122008-access.log
access.log -> 02122008-access.log
www.domain2.com/
02142008-access.log
02132008-access.log
02122008-access.log
access.log -> 02122008-access.log

Vlogger man page

To use vlogger, you need to add a "%v" to the first part of your LogFormat:

LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Then call it from a customlog:

CustomLog "| /usr/sbin/vlogger -s access.log -u www-logs -g www-logs /var/log/apache2″ combined

Options are given in short format on the command line.

-a Do not autoflush files. This may improve performance but may break logfile analyzers that depend on full entries in the logs.

-e ErrorLog mode. In this mode, the host parsing is disabled, and the file is written out using the template under the specified LOGDIR.

-n Disables rotation. This option disables rotation altogether.

-f MAXFILES Maximum number of filehandles to keep open. Defaults to 100. Setting this value too high may result in the system running out of file descriptors. Setting it too low may affect performance.

-u UID Change user to UID when running as root.

-g GID Change group to GID when running as root.

-t TEMPLATE Filename template using Date::Format codes. Default is "%m%d%Y-access.log", or "%m%d%Y-error.log". When using the -r option, the default becomes "%m%d%Y-%T-access.log" or "%m%d%Y-%T-error.log".

-s SYMLINK Specifies the name of a symlink to the current file.

-r SIZE Rotate files when they reach SIZE. SIZE is given in bytes.

-d CONFIG Use the DBI usage tracker.

-h Displays help.

-v Prints version information.

Sponsored Link

 Posted by at 12:03 am

  3 Responses to “How to Split apache Logs With vlogger in debian etch”

  1. If you don’t use virtualhost and do the whole website to directory mapping with mod_rewrite, you have to use %V instead of %v in the line LogFormat…

    UseCanonicalName Off
    RewriteEngine on
    ###
    # Only a domain – eg. linux.org dyndns.org
    # /var/www/linux.org/ /var/www/dyndns.org/
    #
    RewriteCond %{SERVER_NAME} ^([^.]+)\.([^.]+)$
    RewriteRule ^(.*)$ /var/www/%1.%2$1

    ###
    # One host – eg. http://www.linux.org schmidi2.dyndns.org
    # /var/www/linux.org/www/ /var/www/dyndns.org/schmidi2/
    #
    RewriteCond %{SERVER_NAME} ^([^.]+)\.([^.]+)\.([^.]+)$
    RewriteRule ^(.*)$ /var/www/%2.%3/%1$1

    ###
    # Two hosts – eg. lists.admin.linux.org blog.schmidi2.dyndns.org
    # /var/www/linux.org/admin/lists/ /var/www/dyndns.org/schmidi2/blog/
    #
    RewriteCond %{SERVER_NAME} ^([^.]+)\.([^.]+)\.([^.]+)\.([^.]+)$
    RewriteRule ^(.*)$ /var/www/%3.%4/%2/%1$1

    ###
    # Else show an error page
    #

    # –> Not Found

    Use:

    ## vlogger
    LogFormat “%V %h %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\”” combined

  2. Tried the vlogger and always got this:
    [error] (2)No such file or directory: could not open transfer log file /etc/apache2/\xe2\x8e\xaa /usr/sbin/vlogger -s access.log -r 204800 -u www-logs -g www-logs /var/log/apache2.
    Unable to open logs
    Action ‘restart’ failed.

    Opps

  3. The error in the form of:

    [error] (2)No such file or directory: could not open transfer log file /etc/apache2/\xe2\x8e\xaa /usr/sbin/vlogger -s access.log -u www-logs -g www-logs /var/log/apache.
    Unable to open logs

    Is caused by using the example in the vlogger’s man file which contains a non-standard (at least for my system) pipe character:

    CustomLog “? /usr/sbin/vlogger -s access.log -u www-logs -g www-logs /var/log/apache” combined

    Apparently the “\xe2\x8e\xaa” is UTF-8 encoding for a character that looks like a pipe (i.e. “|”) but in fact is not seen as a pipe on my linux system (I’m using Ubuntu). (See http://www.utf8-chartable.de/unicode-utf8-table.pl?start=9088&number=128&names=-&utf8=string-literal)

    The following shows two pipe-like characters together (the first being the one from the man file, the second is a pipe I typed from my keyboard):

    CustomLog “?| /usr/sbin/vlogger -s access.log -u www-logs -g www-logs /var/log/apache” combined

    They should look look slightly different to you.

    Replace the man file’s pipe character in the CustomLog example with a pipe from your keyboard – and apache should restart without complaint.

 Leave a Reply

(required)

(required)

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>