firewall-wizards Digest, Vol 37, Issue 10

Send firewall-wizards mailing list submissions to
firewall-wizards@listserv.icsalabs.com

To subscribe or unsubscribe via the World Wide Web, visit
https://listserv.icsalabs.com/mailman/listinfo/firewall-wizards
or, via email, send a message with subject or body 'help' to
firewall-wizards-request@listserv.icsalabs.com

You can reach the person managing the list at
firewall-wizards-owner@listserv.icsalabs.com

When replying, please edit your Subject line so it is more specific
than "Re: Contents of firewall-wizards digest..."

Today's Topics:

1. Re: Handling large log files (Gy?ngy?si P?ter)

----------------------------------------------------------------------

Message: 1
Date: Mon, 11 May 2009 17:00:14 +0200
From: Gy?ngy?si P?ter <gyp@balabit.hu>
Subject: Re: [fw-wiz] Handling large log files
To: Firewall Wizards Security Mailing List
<firewall-wizards@listserv.icsalabs.com>
Message-ID: <4A083D7E.4030400@balabit.hu>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

(Disclaimer: I work for BalaBit, the company behind syslog-ng.)

Nate Hausrath wrote:
> Hello everyone,
>
> I have a central log server set up in our environment that would
> receive around 200-300 MB of messages per day from various devices
> (switches, routers, firewalls, etc). With this volume, logcheck was
> able to effectively parse the files and send out a nice email. Now,
> however, the volume has increased to around 3-5 GB per day and will
> continue growing as we add more systems. Unfortunately, the old
> logcheck solution now spends hours trying to parse the logs, and even
> if it finishes, it will generate an email that is too big to send.
>
The others have given lots of useful tips about log handling, but if
you're just having perfomance issues with logcheck, you should have a
look at the db-parser feature in the new syslog-ng 3.0.

The best places to find out more about it are these blog posts:

http://marci.blogs.balabit.com/2009/04/db-parser-high-speed-log-message-parser.html
http://marci.blogs.balabit.com/2009/04/intorduction-to-parser-in-syslog-ng-db.html
http://bazsi.blogs.balabit.com/2008/10/syslog-ng-message-parsing.html

It's able to handle (that means, classify based on log message contents,
filter based on this classification and store or forward) this kind of
traffic on commodity hardware. A ready-to-use pattern database converted
from logcheck's regexp list and for Cisco PIX messages can be downloaded
from the website and it's quite easy to write your own rules (the blog
posts mentioned above contain good examples).

Peter

------------------------------

_______________________________________________
firewall-wizards mailing list
firewall-wizards@listserv.icsalabs.com
https://listserv.icsalabs.com/mailman/listinfo/firewall-wizards

End of firewall-wizards Digest, Vol 37, Issue 10
************************************************