Please see Learning Rsyslog for the introduction and index to this series of blog posts about Rsyslog.

If you're learning Rsyslog, you'll see a lot of references to "Facility" and "Severity". These are both very useful and very limiting. Practical examples may help:

# /etc/rsyslog.conf

*.info;mail.none;authpriv.none;cron.none                /var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                  -/var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 :omusrmsg:*
uucp,news.crit                                          /var/log/spooler
local7.*                                                /var/log/boot.log

These are all default filter lines from a Fedora 32 system (Debian's defaults are very close, but not identical). And their meaning should be pretty clear: the second line means that everything that's got a "facility" of "authpriv" goes into the /var/log/secure file, and the first line indicates that all messages with a "severity" of "info" or higher go into /var/log/messages - except we're going to exclude facilities of mail, authpriv, and cron, which are directed to other files shortly after. That should be enough hints to read the whole thing.

Which is all fine and good: it shows basic filtering, and makes some sense of how your log files come into existence. But ... how did the log messages get marked with their facility and severity? Unfortunately, I'm at a point in my understanding of Rsyslog that means I'm better at asking these questions than answering them. I'll do my best, but these won't all be answered when I'm done. This blog entry started because I had one question: can I create my own facility names?

Most log messages that pass through Rsyslog are generated by one of the programs or subsystems that go with the names you're seeing: the kernel, cron jobs, mail. But what about Nginx or Apache? Well ... here's the thing. While Rsyslog is perfectly happy to handle your web server's log messages, under normal circumstances it doesn't. Nginx, Apache, HAProxy - they all log directly to your filesystem without ever communicating with Rsyslog in any way. It's not hard to make them do so (at least it doesn't appear to be for Nginx, and I assume the same is true of the others) - they can write to an Rsyslog-controlled socket instead of the default file, and Rsyslog handles the messages. But now you have a non-standard configuration, and you have to write new filters to tell Rsyslog where to send the new messages. And they DON'T (by default) have a facility attached to them, and may or may not have a severity. I thought it would be reasonable to create your own facility name, so I tried to do that with my Memory log (see Rsyslog: A New Log) when sending the log to my partially set up central logging server. I passed a "facility" value of "memfree." The receiving Rsyslog was supposed to file these under "%SYSLOGFACILITY-TEXT%.log", but saved it to "invld.log". It apparently thought my created facility name was invalid. Further research has shown that "[t]he syslog interface only allows a fixed set of facilities, defined by constants in /usr/include/sys/syslog.h. The only provision for custom facilities are local0 through local7." (Courtesy of StackExchange, see the link in the Bibliography).

"Defined by constants" may sound like numerical values: that's because they are. Each facility has a number, although for most purposes it's easier to deal with the name. (If you need the numbers, see the Tables section of this post).

Having a facility name attached to a log message makes it much easier to sort. Rsyslog provides extensive, powerful filtering - but to use it, you have to write filters. Writing memfree.* /var/log/remote/<host>/memfree.log is simple enough to verge on being intuitive. But Rsyslog's filters are complex and take a lot of time to learn. The simplest solution for for me is to create a consistent key: [ local0 = Nginx , local1 = Apache , local2 = Ruby , local3 = HAProxy ]. But for this to work across dozens or hundreds of machines feeding a centralized logging server, you have to stick with it like it's written in stone ... and hope you don't have more than eight services. Seven if you use Fedora, because if you look at the example rsyslog.conf above, "local7" is already in use.

I think I'm going to need filtering anyway: I'll write about it when I get to it.