anhedonic

dynamic soap bubbles

May 26, 2025

Network-Block-Device Copy-on-Write fixes

Berend De Schouwer

Fixing two copy-on-write bugs in the NBD server.

What is What?

NBD is a network block device protocol. It has some overlap with iSCSI, and a little with NFS. The protocol is much simpler than either, and has one extra feature: copy-on-write.

Copy-on-write allows for sharing the same file to multiple machines, and only writing changes back to disk. This can save a lot of storage space in certain situations.

There are two bugs, and two one-line fixes presented here.

The Bugs

Sequential Read after Write

For the first case, it’s enough to read and write more than one sequential block. The second and subsequent blocks read will read into the wrong offset of the buffer, and copy invalid data to the client.

I use a 4096 block size in this example, but I’ve used others. I did that to match the filesystem, but for the test I don’t even need a filesystem.

The Test
export OFFSET=0
export COUNT=3 # anything >= 1
dd if=/dev/urandom of=testdata bs=4096 count=$COUNT # random data
dd if=testdata of=/dev/nbd0 bs=4096 seek=$OFFSET count=$COUNT
dd if=/dev/nbd0 of=compdata bs=4096 skip=$OFFSET count=$COUNT
sum testdata compdata

The data, testdata and compdata, will be different.

If the kernel does a partition check when /dev/nbd0 is mounted, this test will fail with COUNT=1 as well.

Sparse Write at the Wrong Offset

For the second case, with sparse_cow=true, we need to repeat the test with an offset > 0. expwrite() calls write() instead of pwrite().

The Test
export OFFSET=100
export COUNT=3 # anything >= 1
dd if=/dev/urandom of=testdata bs=4096 count=$COUNT # random data
dd if=testdata of=/dev/nbd0 bs=4096 seek=$OFFSET count=$COUNT
dd if=/dev/nbd0 of=compdata bs=4096 skip=$OFFSET count=$COUNT
sum testdata compdata

The first time it’s run, it will result in an Input/Output error.

The second time it’s run, it will work.

The Patches

diff --git a/nbd-orig/nbd-server.c b/nbd-patched/nbd-server.c
index 92fd141..18e5ddd 100644
--- a/nbd-orig/nbd-server.c
+++ b/nbd-patched/nbd-server.c
@@ -1582,6 +1582,7 @@ int expread(READ_CTX *ctx, CLIENT *client) {
                        if (pread(client->difffile, buf, rdlen, client->difmap[mapcnt]*DIFFPAGESIZE+offset) != rdlen) {
                                goto fail;
                        }
+                       ctx->current_offset += rdlen;
                        confirm_read(client, ctx, rdlen);
                } else { /* the block is not there */
                        if ((client->server->flags & F_WAIT) &&
(client->export == NULL)){
diff --git a/nbd-orig/nbd-server.c b/nbd-patched/nbd-server.c
index 92fd141..9a57ad5 100644
--- a/nbd-orig/nbd-server.c
+++ b/nbd-patched/nbd-server.c
@@ -1669,7 +1669,7 @@ int expwrite(off_t a, char *buf, size_t len,
CLIENT *client, int fua) {
                                if(ret < 0 ) goto fail;
                        }
                        memcpy(pagebuf+offset,buf,wrlen) ;
-                       if (write(client->difffile, pagebuf, DIFFPAGESIZE) != DIFFPAGESIZE)
+                       if (pwrite(client->difffile, pagebuf, DIFFPAGESIZE, client->difmap[mapcnt]*DIFFPAGESIZE) != DIFFPAGESIZE)
                                goto fail;
                }
                if (!(client->server->flags & F_COPYONWRITE))

May 25, 2025

Logging Made Easy

Berend De Schouwer

A number of programs will not log to syslog, only logging to a file they control, or to a custom log server. This post describes how to get one such program to write to syslog.

TL;DR

Setup a FIFO

/etc/systemd/system/journal-pipe-modsecurity.socket
[Unit]
Description=Journal Pipe
Documentation=man:systemd-journald.service(8) man:journald.conf(5)
DefaultDependencies=no
Before=sockets.target
IgnoreOnIsolate=yes

[Socket]
ListenFIFO=/run/systemd/journal/pipes/modsecurity
ReceiveBuffer=8M
Accept=no
Service=journal-pipe-modsecurity.service
SocketMode=0660
Timestamping=us
# Access control for the FIFO
SocketUser=some-user
SocketGroup=some-group

Setup a Service

/etc/systemd/system/journal-pipe-modsecurity.service
[Unit]
Description=Journal Pipe for Modsecurity
After=network.target journal-pipe-modsecurity.socket
Requires=journal-pipe-modsecurity.socket

[Service]
Type=simple
StandardInput=fd:journal-pipe-modsecurity.socket
ExecStart=/usr/bin/systemd-cat --identifier=modsecurity
TimeoutStopSec=5
Group=some-group
PrivateTmp=yes
DynamicUser=yes
ProtectHome=yes

[Install]
WantedBy=default.target

Configure Modsecurity

modsecurity.conf
SecAuditLogType Serial
SecAuditLog /run/systemd/journal/pipes/modsecurity

Problem Description

Some programs will only log to a file, or to a custom logger. There is no way to configure them to write to syslog, or a standard logger.

Good system administrators want to use syslog, or another standard logger. In this post, we’re going to get Modsecurity to log to syslog, which is an often requested feature.

Assumptions

We have an application, that can write logs, and we are interested in those logs.

That application insists on writing logs to a file, or to a custom log manager. A custom log manager means yet another application installed, and configured, and storage allocated.

When the application is configured to write logs to a file, the application must be configured to rotate the logs based on time or size, and must manage the log size.

What is Syslog?

Syslog is a Unix-y standardised logger. For this post, it’s only important that it’s standardised across the OS. In fact, we’re actually going to use journald, and not syslog.

Syslog accepts logs on a number of interfaces.

  • A file interface at /dev/log
  • A socket interface
  • A UDP network interface

Syslog then writes the logs using a timestamp, a hostname, and an application and PID where possible.

Various syslog (and journal) servers exist that can

  • Limit diskspace for logs
  • Send logs to another drive, or another server, or a printer
  • Manage logs before the filesystem is read-write, reducing lost logs
  • Rotate logs without losing logs
  • Throttle logs (unfortunately losing some) to prevent DoS attacks

What is Modsecurity?

Modsecurity is a web application firewall (WAF). It follows rules, and prevents or allows HTTP requests.

Logically, it sits between the web server and a web application. Usually the web server that runs Modsecurity acts as a proxy in front of a Java or PHP web application.

Think of it as an anti-virus or firewall for the web.

Why Do We Want Syslog?

Standardising the logs gives a number of benefits.

Viewing Multiple Logs, Ordered

It’s often useful to view multiple logs, ordered sequentially, to track bugs or security problems. This is doubly true for a program like Modsecurity, that hooks into a webserver — usually as a proxy — and an application server.

When something goes wrong, we can get the HTTP context from the webserver logs, the application context from the application server logs, and the security logs from the WAF, all ordered.

We also have timestamps in the same format.

Space Allocation

If the logs are all in a specific system, we can decide to allocate space on a specific drive, optimised for writes, separate from any database.

We may need to keep certain logs for legal reasons.

Separate Secure Storage

Syslog servers can send their data to a separate server. In the best cases, to a server without an IP number. This means that any attackers cannot delete the logs.

That’s a major security help.

Compartementilisation

The syslog server does not have to run as the same user as the application. That means that when an attacker breaches security, and has access rights similar to the application, the attacker does not have permissions to delete the logs.

That’s also a major security help.

Separate Backups

If your application writes and manages your logs, chances are your backup system has to backup and restore those logs. It’s almost always better to manage the backups of logs, and the backups of the application data separate.

Pet Peeve

Applications that do their own logging is a pet peeve of mine. There are almost always bugs.

Logging is harder than you think, and mistakes are common.

For example, Tomcat — in it’s recommended configuration — will still claim diskspace of long-since deleted logs.

Feb 22, 2024

why rust won’t let you borrow twice

Berend De Schouwer

What Question Am I Answering?

A question came up in a coding class, about why in Rust there’s a borrow, and there’s a an error when borrowing.

You may have seen it like (copied from the rust book):

error[E0499]: cannot borrow `s` as mutable more than once at a time
 --> src/main.rs:5:14
  |
4 |     let r1 = &mut s;
  |              ------ first mutable borrow occurs here
5 |     let r2 = &mut s;
  |              ^^^^^^ second mutable borrow occurs here
6 |
7 |     println!("{}, {}", r1, r2);
  |                        -- first borrow later used here

For more information about this error, try `rustc --explain E0499`.
error: could not compile `ownership` due to previous error

The Rust book does explain what borrow is, and the theory behind why it’s allowed and not allowed.

I’m showing the risks practical example.

How Am I Answering It?

With as little computer theory as possible.

  • No assembler
  • No diagrams
  • No multiple languages
  • short, simple examples
  • No theory until after the bug is shown

Caveat

The answer is in C. It’s in C because Rust won’t let me break borrow, not even in an unsafe {} block.

It’s all in plain C, though. No assembler, and no C++. As straightforward C as I can make it, specifically to allow for plain examples. As few as possible shortcuts are taken.

I’m not trying to write a C-programmer’s C code. I’m trying to write an example that shows the problem, for non-C programmers.

It should not be difficult to follow coming from Rust.

Inspiration

The inspiration comes from the C max() function, which is actually a macro.

That is great for explaining the differences between functions and macros, and it’s great for explaining pass-by-code instead of value or reference.

It’s also great for explaining surprises.

First Examples

First, we give example code for pass-by-value and pass-by-reference. Pass-by-reference is also called borrow in Rust. The examples are simple, and we do not yet discuss the difference.

For now, it’s simply about C syntax.

#include <stdio.h>

int double_by_value(int x) {
    x = x * 2;
    return x;
}

int double_by_reference(int *x) {
    *x = *x * 2;
    return *x;
}

int main() {
    int x;
    x = 5;
    printf("double_by_value(x) = %d\n", double_by_value(x));
    /* double_by_value(x) = 10 */
    x = 5;
    printf("double_by_reference(x) = %d\n", double_by_reference(&x));
    /* double_by_reference(x) = 10 */
    return 0;
}

The code is almost identical. You can see the syntax difference, but no functional difference yet.

The logic is the same, the input is the same, and the output is the same.

Extra variables

Let’s make it a tiny bit more challenging, and add a second variable.

#include <stdio.h>

int add_by_value(int x, int y) {
    x = x + y;
    return x;
}

int add_by_reference(int *x, int *y) {
    *x = *x + *y;
    return *x;
}

int main() {
    int x, y;

    x = y = 5;
    printf("add_by_value(x, y) = %d\n", add_by_value(x, y));
    /* add_by_value(x, y) = 10 */
    x = y = 5;
    printf("add_by_reference(x, y) = %d\n", add_by_reference(&x, &y));
    /* add_by_reference(x, y) = 10 */
    return 0;
}

The logic is the same, the input is the same, and the output is the same.

Put it together

Add the two and we get…

#include <stdio.h>

/* Previous example code here */

int add_and_double_by_value(int x, int y) {
    x = double_by_value(x);
    y = double_by_value(y);
    x = x + y;
    return x;
}

int add_and_double_by_reference(int *x, int *y) {
    *x = double_by_reference(x);
    *y = double_by_reference(y);
    *x = *x + *y;
    return *x;
}

int main() {
    int x, y;

    x = y = 5;
    printf("add_and_double_by_value(x, y) = %d\n", add_and_double_by_value(x, y));
    /* add_and_double_by_value(x, y) = 20 */
    x = y = 5;
    printf("add_and_double_by_reference(x, y) = %d\n", add_and_double_by_reference(&x, &y));
    /* add_and_double_by_reference(x, y) = 20 */
    return 0;
}

No surprised yet. Both examples print the same, correct answer.

The logic is the same, the input is the same, and the output is the same.

Surprise

Now let’s make it simpler…

#include <stdio.h>

/* Previous example code here */

int main() {
    int x;
    x = 5;
    printf("add_and_double_by_value(x, x) = %d\n", add_and_double_by_value(x, x));
    /* add_and_double_by_value(x, x) = 20 */
    x = 5;
    printf("add_and_double_by_reference(x, x) = %d\n", add_and_double_by_reference(&x, &x));
    /* add_and_double_by_reference(x, x) = 40 */
    return 0;
}

The logic is the same, the input is the same, but the output is different.

Why is it 40?

Double borrow

So what happened? Why did removing a variable, y, result in the “wrong” answer?

Pass by value copies the value, and passes the value, not the variable. The original is never modified.

Pass by reference (or borrow), passes a reference, not a copy. The original is modified.

Look at

int add_and_double_by_reference(int *x, int *y) {
    *x = double_by_reference(x);
    *y = double_by_reference(y);
    *x = *x + *y;
    return *x;
}

When x is doubled, at *x = double_by_reference(x);, and *x is also *y, y is also doubled. X is now 10, as expected, but y is also 10.

Then y is doubled. And since *y is _also *x, x is doubled again. Now both variables are 20.

Tada!

This is (one of the reasons) why Rust won’t let you borrow the same variable twice.

What do you do instead?

Borrow once, or .clone() /* copy */.

Bonus time

So why have pass by reference or borrow at all?

  1. It’s faster, when the data is large.
  2. It’s faster for streamed data.
  3. It’s not possible in assembler pass complicated variables by value. Manual copies must be made, and the language you use might not implement implicit copies.

Extra Points

For extra points, do the same with threads.