Logging Made Easy

Berend De Schouwer

A number of programs will not log to syslog, only logging to a file they control, or to a custom log server. This post describes how to get one such program to write to syslog.

TL;DR

Setup a FIFO

[Unit]
Description=Journal Pipe
Documentation=man:systemd-journald.service(8) man:journald.conf(5)
DefaultDependencies=no
Before=sockets.target
IgnoreOnIsolate=yes

[Socket]
ListenFIFO=/run/systemd/journal/pipes/modsecurity
ReceiveBuffer=8M
Accept=no
Service=journal-pipe-modsecurity.service
SocketMode=0660
Timestamping=us
# Access control for the FIFO
SocketUser=some-user
SocketGroup=some-group

Setup a Service

[Unit]
Description=Journal Pipe for Modsecurity
After=network.target journal-pipe-modsecurity.socket
Requires=journal-pipe-modsecurity.socket

[Service]
Type=simple
StandardInput=fd:journal-pipe-modsecurity.socket
ExecStart=/usr/bin/systemd-cat --identifier=modsecurity
TimeoutStopSec=5
Group=some-group
PrivateTmp=yes
DynamicUser=yes
ProtectHome=yes

[Install]
WantedBy=default.target

Configure Modsecurity

SecAuditLogType Serial
SecAuditLog /run/systemd/journal/pipes/modsecurity

Problem Description

Some programs will only log to a file, or to a custom logger. There is no way to configure them to write to syslog, or a standard logger.

Good system administrators want to use syslog, or another standard logger. In this post, we’re going to get Modsecurity to log to syslog, which is an often requested feature.

Assumptions

We have an application, that can write logs, and we are interested in those logs.

That application insists on writing logs to a file, or to a custom log manager. A custom log manager means yet another application installed, and configured, and storage allocated.

When the application is configured to write logs to a file, the application must be configured to rotate the logs based on time or size, and must manage the log size.

What is Syslog?

Syslog is a Unix-y standardised logger. For this post, it’s only important that it’s standardised across the OS. In fact, we’re actually going to use journald, and not syslog.

Syslog accepts logs on a number of interfaces.

• A file interface at /dev/log
• A socket interface
• A UDP network interface

Syslog then writes the logs using a timestamp, a hostname, and an application and PID where possible.

Various syslog (and journal) servers exist that can

• Limit diskspace for logs
• Send logs to another drive, or another server, or a printer
• Manage logs before the filesystem is read-write, reducing lost logs
• Rotate logs without losing logs
• Throttle logs (unfortunately losing some) to prevent DoS attacks

What is Modsecurity?

Modsecurity is a web application firewall (WAF). It follows rules, and prevents or allows HTTP requests.

Logically, it sits between the web server and a web application. Usually the web server that runs Modsecurity acts as a proxy in front of a Java or PHP web application.

Think of it as an anti-virus or firewall for the web.

Why Do We Want Syslog?

Standardising the logs gives a number of benefits.

Viewing Multiple Logs, Ordered

It’s often useful to view multiple logs, ordered sequentially, to track bugs or security problems. This is doubly true for a program like Modsecurity, that hooks into a webserver — usually as a proxy — and an application server.

When something goes wrong, we can get the HTTP context from the webserver logs, the application context from the application server logs, and the security logs from the WAF, all ordered.

We also have timestamps in the same format.

Space Allocation

If the logs are all in a specific system, we can decide to allocate space on a specific drive, optimised for writes, separate from any database.

Legal Data Retention

We may need to keep certain logs for legal reasons.

Separate Secure Storage

Syslog servers can send their data to a separate server. In the best cases, to a server without an IP number. This means that any attackers cannot delete the logs.

That’s a major security help.

Compartementilisation

The syslog server does not have to run as the same user as the application. That means that when an attacker breaches security, and has access rights similar to the application, the attacker does not have permissions to delete the logs.

That’s also a major security help.

Separate Backups

If your application writes and manages your logs, chances are your backup system has to backup and restore those logs. It’s almost always better to manage the backups of logs, and the backups of the application data separate.

Pet Peeve

Applications that do their own logging is a pet peeve of mine. There are almost always bugs.

Logging is harder than you think, and mistakes are common.

For example, Tomcat — in it’s recommended configuration — will still claim diskspace of long-since deleted logs.