Fixing two copy-on-write bugs in the NBD server.
What is What?
NBD is a network block device protocol. It has some overlap with iSCSI,
and a little with NFS. The protocol is much simpler than either, and has
one extra feature: copy-on-write.
Copy-on-write allows for sharing the same file to multiple machines, and
only writing changes back to disk. This can save a lot of storage space
in certain situations.
There are two bugs, and two one-line fixes presented here.
The Bugs
Sequential Read after Write
For the first case, it’s enough to read and write more than one
sequential block. The second and subsequent blocks read will read into
the wrong offset of the buffer, and copy invalid data to the client.
I use a 4096 block size in this example, but I’ve used others. I did
that to match the filesystem, but for the test I don’t even need a filesystem.
The Testexport OFFSET=0
export COUNT=3 # anything >= 1
dd if=/dev/urandom of=testdata bs=4096 count=$COUNT # random data
dd if=testdata of=/dev/nbd0 bs=4096 seek=$OFFSET count=$COUNT
dd if=/dev/nbd0 of=compdata bs=4096 skip=$OFFSET count=$COUNT
sum testdata compdata
The data, testdata and compdata, will be different.
If the kernel does a partition check when /dev/nbd0 is mounted, this
test will fail with COUNT=1 as well.
Sparse Write at the Wrong Offset
For the second case, with sparse_cow=true, we need to repeat the test
with an offset > 0. expwrite() calls write() instead of pwrite().
The Testexport OFFSET=100
export COUNT=3 # anything >= 1
dd if=/dev/urandom of=testdata bs=4096 count=$COUNT # random data
dd if=testdata of=/dev/nbd0 bs=4096 seek=$OFFSET count=$COUNT
dd if=/dev/nbd0 of=compdata bs=4096 skip=$OFFSET count=$COUNT
sum testdata compdata
The first time it’s run, it will result in an Input/Output error.
The second time it’s run, it will work.
The Patches
diff --git a/nbd-orig/nbd-server.c b/nbd-patched/nbd-server.c
index 92fd141..18e5ddd 100644
--- a/nbd-orig/nbd-server.c
+++ b/nbd-patched/nbd-server.c
@@ -1582,6 +1582,7 @@ int expread(READ_CTX *ctx, CLIENT *client) {
if (pread(client->difffile, buf, rdlen, client->difmap[mapcnt]*DIFFPAGESIZE+offset) != rdlen) {
goto fail;
}
+ ctx->current_offset += rdlen;
confirm_read(client, ctx, rdlen);
} else { /* the block is not there */
if ((client->server->flags & F_WAIT) &&
(client->export == NULL)){
diff --git a/nbd-orig/nbd-server.c b/nbd-patched/nbd-server.c
index 92fd141..9a57ad5 100644
--- a/nbd-orig/nbd-server.c
+++ b/nbd-patched/nbd-server.c
@@ -1669,7 +1669,7 @@ int expwrite(off_t a, char *buf, size_t len,
CLIENT *client, int fua) {
if(ret < 0 ) goto fail;
}
memcpy(pagebuf+offset,buf,wrlen) ;
- if (write(client->difffile, pagebuf, DIFFPAGESIZE) != DIFFPAGESIZE)
+ if (pwrite(client->difffile, pagebuf, DIFFPAGESIZE, client->difmap[mapcnt]*DIFFPAGESIZE) != DIFFPAGESIZE)
goto fail;
}
if (!(client->server->flags & F_COPYONWRITE))
A number of programs will not log to syslog, only logging to a file
they control, or to a custom log server. This post describes how to
get one such program to write to syslog.
TL;DR
Setup a FIFO
/etc/systemd/system/journal-pipe-modsecurity.socket[Unit]
Description=Journal Pipe
Documentation=man:systemd-journald.service(8) man:journald.conf(5)
DefaultDependencies=no
Before=sockets.target
IgnoreOnIsolate=yes
[Socket]
ListenFIFO=/run/systemd/journal/pipes/modsecurity
ReceiveBuffer=8M
Accept=no
Service=journal-pipe-modsecurity.service
SocketMode=0660
Timestamping=us
# Access control for the FIFO
SocketUser=some-user
SocketGroup=some-group
Setup a Service
/etc/systemd/system/journal-pipe-modsecurity.service[Unit]
Description=Journal Pipe for Modsecurity
After=network.target journal-pipe-modsecurity.socket
Requires=journal-pipe-modsecurity.socket
[Service]
Type=simple
StandardInput=fd:journal-pipe-modsecurity.socket
ExecStart=/usr/bin/systemd-cat --identifier=modsecurity
TimeoutStopSec=5
Group=some-group
PrivateTmp=yes
DynamicUser=yes
ProtectHome=yes
[Install]
WantedBy=default.target
modsecurity.confSecAuditLogType Serial
SecAuditLog /run/systemd/journal/pipes/modsecurity
Problem Description
Some programs will only log to a file, or to a custom logger. There is
no way to configure them to write to syslog, or a standard logger.
Good system administrators want to use syslog, or another standard
logger. In this post, we’re going to get Modsecurity to log to syslog,
which is an often requested feature.
Assumptions
We have an application, that can write logs, and we are interested
in those logs.
That application insists on writing logs to a file, or to a custom log
manager. A custom log manager means yet another application installed,
and configured, and storage allocated.
When the application is configured to write logs to a file, the application
must be configured to rotate the logs based on time or size, and must
manage the log size.
What is Syslog?
Syslog is a Unix-y standardised logger. For this post, it’s only important
that it’s standardised across the OS. In fact, we’re actually going to
use journald, and not syslog.
Syslog accepts logs on a number of interfaces.
- A file interface at /dev/log
- A socket interface
- A UDP network interface
Syslog then writes the logs using a timestamp, a hostname, and an
application and PID where possible.
Various syslog (and journal) servers exist that can
- Limit diskspace for logs
- Send logs to another drive, or another server, or a printer
- Manage logs before the filesystem is read-write, reducing lost logs
- Rotate logs without losing logs
- Throttle logs (unfortunately losing some) to prevent DoS attacks
What is Modsecurity?
Modsecurity is a web application firewall (WAF). It follows rules, and
prevents or allows HTTP requests.
Logically, it sits between the web server and a web application. Usually
the web server that runs Modsecurity acts as a proxy in front of a Java
or PHP web application.
Think of it as an anti-virus or firewall for the web.
Why Do We Want Syslog?
Standardising the logs gives a number of benefits.
Viewing Multiple Logs, Ordered
It’s often useful to view multiple logs, ordered sequentially, to track
bugs or security problems. This is doubly true for a program like
Modsecurity, that hooks into a webserver — usually as a proxy — and
an application server.
When something goes wrong, we can get the HTTP context from the webserver
logs, the application context from the application server logs, and
the security logs from the WAF, all ordered.
We also have timestamps in the same format.
Space Allocation
If the logs are all in a specific system, we can decide to allocate space
on a specific drive, optimised for writes, separate from any database.
Legal Data Retention
We may need to keep certain logs for legal reasons.
Separate Secure Storage
Syslog servers can send their data to a separate server. In the best
cases, to a server without an IP number. This means that any attackers
cannot delete the logs.
That’s a major security help.
Compartementilisation
The syslog server does not have to run as the same user as the application.
That means that when an attacker breaches security, and has access rights
similar to the application, the attacker does not have permissions to
delete the logs.
That’s also a major security help.
Separate Backups
If your application writes and manages your logs, chances are your backup
system has to backup and restore those logs. It’s almost always better
to manage the backups of logs, and the backups of the application data separate.
Pet Peeve
Applications that do their own logging is a pet peeve of mine. There
are almost always bugs.
Logging is harder than you think, and mistakes are common.
For example, Tomcat — in it’s recommended configuration — will still
claim diskspace of long-since deleted logs.
What Question Am I Answering?
A question came up in a coding class, about why in Rust there’s a borrow,
and there’s a an error when borrowing.
You may have seen it like (copied from the rust book):
error[E0499]: cannot borrow `s` as mutable more than once at a time
--> src/main.rs:5:14
|
4 | let r1 = &mut s;
| ------ first mutable borrow occurs here
5 | let r2 = &mut s;
| ^^^^^^ second mutable borrow occurs here
6 |
7 | println!("{}, {}", r1, r2);
| -- first borrow later used here
For more information about this error, try `rustc --explain E0499`.
error: could not compile `ownership` due to previous error
The Rust book does explain what borrow is, and the theory behind
why it’s allowed and not allowed.
I’m showing the risks practical example.
How Am I Answering It?
With as little computer theory as possible.
- No assembler
- No diagrams
- No multiple languages
- short, simple examples
- No theory until after the bug is shown
Caveat
The answer is in C. It’s in C because Rust won’t let me break
borrow, not even in an unsafe {}
block.
It’s all in plain C, though. No assembler, and no C++. As
straightforward C as I can make it, specifically to allow for
plain examples. As few as possible shortcuts are taken.
I’m not trying to write a C-programmer’s C code. I’m trying to
write an example that shows the problem, for non-C programmers.
It should not be difficult to follow coming from Rust.
Inspiration
The inspiration comes from the C max()
function, which is actually
a macro.
That is great for explaining the differences between functions and macros,
and it’s great for explaining pass-by-code instead of value or reference.
It’s also great for explaining surprises.
First Examples
First, we give example code for pass-by-value and pass-by-reference.
Pass-by-reference is also called borrow in Rust. The examples are simple,
and we do not yet discuss the difference.
For now, it’s simply about C syntax.
#include <stdio.h>
int double_by_value(int x) {
x = x * 2;
return x;
}
int double_by_reference(int *x) {
*x = *x * 2;
return *x;
}
int main() {
int x;
x = 5;
printf("double_by_value(x) = %d\n", double_by_value(x));
/* double_by_value(x) = 10 */
x = 5;
printf("double_by_reference(x) = %d\n", double_by_reference(&x));
/* double_by_reference(x) = 10 */
return 0;
}
The code is almost identical. You can see the syntax difference, but no
functional difference yet.
The logic is the same, the input is the same, and the output is the same.
Let’s make it a tiny bit more challenging, and add a second variable.
#include <stdio.h>
int add_by_value(int x, int y) {
x = x + y;
return x;
}
int add_by_reference(int *x, int *y) {
*x = *x + *y;
return *x;
}
int main() {
int x, y;
x = y = 5;
printf("add_by_value(x, y) = %d\n", add_by_value(x, y));
/* add_by_value(x, y) = 10 */
x = y = 5;
printf("add_by_reference(x, y) = %d\n", add_by_reference(&x, &y));
/* add_by_reference(x, y) = 10 */
return 0;
}
The logic is the same, the input is the same, and the output is the same.
Put it together
Add the two and we get…
#include <stdio.h>
/* Previous example code here */
int add_and_double_by_value(int x, int y) {
x = double_by_value(x);
y = double_by_value(y);
x = x + y;
return x;
}
int add_and_double_by_reference(int *x, int *y) {
*x = double_by_reference(x);
*y = double_by_reference(y);
*x = *x + *y;
return *x;
}
int main() {
int x, y;
x = y = 5;
printf("add_and_double_by_value(x, y) = %d\n", add_and_double_by_value(x, y));
/* add_and_double_by_value(x, y) = 20 */
x = y = 5;
printf("add_and_double_by_reference(x, y) = %d\n", add_and_double_by_reference(&x, &y));
/* add_and_double_by_reference(x, y) = 20 */
return 0;
}
No surprised yet. Both examples print the same, correct answer.
The logic is the same, the input is the same, and the output is the same.
Surprise
Now let’s make it simpler…
#include <stdio.h>
/* Previous example code here */
int main() {
int x;
x = 5;
printf("add_and_double_by_value(x, x) = %d\n", add_and_double_by_value(x, x));
/* add_and_double_by_value(x, x) = 20 */
x = 5;
printf("add_and_double_by_reference(x, x) = %d\n", add_and_double_by_reference(&x, &x));
/* add_and_double_by_reference(x, x) = 40 */
return 0;
}
The logic is the same, the input is the same, but the output is different.
Why is it 40?
Double borrow
So what happened? Why did removing a variable, y, result in the “wrong” answer?
Pass by value copies the value, and passes the value, not the variable. The original is
never modified.
Pass by reference (or borrow), passes a reference, not a copy. The original is modified.
Look at
int add_and_double_by_reference(int *x, int *y) {
*x = double_by_reference(x);
*y = double_by_reference(y);
*x = *x + *y;
return *x;
}
When x is doubled, at *x = double_by_reference(x);
, and *x
is also *y
, y is
also doubled. X is now 10, as expected, but y is also 10.
Then y is doubled. And since *y
is _also *x
, x is doubled again. Now both variables
are 20.
Tada!
This is (one of the reasons) why Rust won’t let you borrow the same variable twice.
What do you do instead?
Borrow once, or .clone() /* copy */
.
Bonus time
So why have pass by reference or borrow at all?
- It’s faster, when the data is large.
- It’s faster for streamed data.
- It’s not possible in assembler pass complicated variables by value. Manual copies
must be made, and the language you use might not implement implicit copies.
For extra points, do the same with threads.