The community recently had a few discussions related to syslog collectors, which got me thinking about what messages to create alerts on.

In the past, I've browsed the history of log data from certain devices to identify messages that need alerts. That method works well for the more common messages, but what about log messages that haven't been seen before because they occur only in rare events? It seems the most important alerts would be for log messages I might not have seen before.

How do you guys handle this? Do you create generic alerts for all messages with a certain level or higher? Do you browse through a list of all possible messages to pick out what seems important?

IMO, this is one of the hardest parts of standing up and maturing an NMS. You have to see alerts to properly classify and place those alerts. But like you said some you might only see once or twice in the next 10 years and you must be able to see/react to them.

My approach is to work off of the devices default severity levels and go from there. I also have all "unknown" alerts kicked up to a severity where someone is still paged/emailed. At first this can be a bit of a pain and if its spamming too much then you will need to spend a few months or more closely watching whats coming in and classifying them accordingly. Once everything calms down then you can switch back.

_________________
http://blog.movingonesandzeros.net/