@jerry one thing I am missing from this list is compartmentalization.

Put stuff in VMs, put stuff in containers, put stuff on different servers that do not talk to each other, if at all possible. Make pull, not push, backups. Same for logs.

Your public SFTP (you are using SFTP, not FTP, right?) server really does not need to have write access to your LDAP machine.

Basically do everything you can to limit any potential infection as much as possible by the design of your infrastructure itself.

@rysiek @jerry pull-based backups could be interpreted as exfiltration too in a worst-case scenario

@pypper @jerry strong (public key SFTP, for example) authentication, plus limiting access to very specific IPs or subnets should take care of that.

And, if the bad guys are already on your server, they will find a way to exfiltrate, your push backups are not going to make that considerably easier.

But they *will* make it easier to actually *have backups*. The alternative means that anyone who gets access to the server can delete backups also...

@rysiek
I strongly disagree. Pull backup means you have a server that has access to all your servers. If that server is compromised, you are in a world of pain.
A push backup means you can setup a sanctuary (for instance with network diodes). If you fear about loosing your backup in case of a server compromise, rotate the dest dir, or set history to read-only with a cron or a post-push script.
@pypper @jerry

@x_cli @pypper @jerry that's also a good point. The right solution, obviously, is to have both. A push backup directly from the server onto some staging area (for the last month of dailies?), a pull backup (every week/month?) to a place not accessible by the original server.

Also, backups can be set up in a way that the backup server only has read-only access to the backed up server.

@rysiek @jerry @x_cli

in the case of pull backups, you could also have remote servers encrypt their backups and place them in a single remotely accessible directory, like a backup user's homedir who only has ssh access.

so a compromised backup server has read-only access to encrypted files on remote machines.

@pypper
That is a minor improvment, but you still have a server with the credentials for a connection to a local account on all of your servers. I am not sure this is a good idea.
@rysiek @jerry

@x_cli @pypper @jerry or you could have, on the server, a container just for that, with read-only access to only the backups directory.

That way even if somebody compromises the backups server, they only get (read-only) access to the backups directory on the server. They can't even *see* anything else on that server.

@rysiek
Yes. That's a possibility. And the transfer program could be harden with a MAC and seccomp to reduce the risk of kernel exploits (priv escalation to ring0) or container escape (who said libprocps?)
I have to admit that I am not sold though because I am risk averse, and that I will prefer the sanctuary approach which seems to me like a more robust architectural design. YMMV of course. 🙂
@pypper @jerry

@x_cli @pypper @jerry the question is what you worry more about: the backup server being taken over, or the production server.

Production servers need to be more public, with more services running, etc. Backup don't. In fact, if doing pull backups, the backup server doesn't ave to have *any* services running apart from cron! :)

But as always, everything depends on your threat model.

@rysiek
But that's the thing. I worry about the production servers being compromised by a rebound via the backup server, AND by the backup server being corrupted by the production servers. That's why I argue in favor of "push to sanctuary".
(A sanctuary being, just as a reminder, a location that can only be accesed through a network diode).
But maybe I worry too much 😂
@pypper @jerry

@x_cli @pypper @jerry no, no you don't. I'm there with you.

As a wise man said:

We're not crazy.
We're not paranoid.

We're just experienced.

(actual sign on my door)

@rysiek
I am afraid of having a (backup) server with a skeleton key to the whole infrastructure.
On the other hand, I would agree that the attack surface of a pull backup server might be made minimal/small (although, this is not so simple; SSH for administering the backup server, monitoring for the storage space, logs, etc.)
The exact same dillema exists for monitoring by the way.
@pypper @jerry

@rysiek you’re right. I wrote that list based on a currently frustrating situation I may or may not be dealing with.

@jerry ah. In case you are, in fact, dealing with a situation that may or may not be frustrating, good luck!

Sign in to participate in the conversation
Infosec Exchange

A Mastodon instance for info/cyber security-minded people.