Over the last year, I changed my personal storage and backup strategy. For a long time (almost decades) I was using a commercial RAID-5 NAS device. Over time, the NAS ran out of disk space and I added a server with a two-disk RAID-1. To protect my data from accidental deletion or from attacks with ransomware, I ran hand-made bash scripts to backup both NAS’ to external hard drives. The scripts are based on rsync, exploiting hard-links to create incremental backups. The external disks are connected only during taking backup.

This setup might sound better than it was. The commercial RAID-5 NAS offered samba shares, my server-based RAID-1 was mountable with SSHFS. So the system was quite heterogeneous. Furthermore, the user accounts and credentials of the two systems were independent. In order to take a new offline (off-site) backup, I needed to mount all shares from both devices. Creating a privileged used on the commercial NAS was easy, but creating a privileged (non-root) user on the server which is able to read all the user shares on the RAID is more difficult.

When I promoted the RAID-1 system on the server to a RAID-5 system by adding another disk, I decided to reevaluate my set up. This is a summary of the outcome.

Requirements

As a first step, let’s think about the requirements for a personal storage system.

  1. Encrypted traffic: Any traffic to and from the NAS should be encrypted. For a personal system at home, this is not of prime importance, however, it provides some security in cases a node in the home network is infected by a virus. Furthermore, this makes it easier to expose the NAS service to the outside world such that I can access my data when I’m not at home without setting up a VPN.

  2. Virtual shares: The requirement of virtual shares implies the requirement of a multi-user system. It should be possible that any number of users can be added to any number shared directories. This means the system can be configured such that every user has a private directory only accessible by themselves, but also that a group of users can get access to a shared directory. This is task can get quite involved with standard Linux file permissions. Furthermore, when a user mounts a share, that user should be the owner of the files.

  3. Server-side permissions: The system should not rely on client nodes to enforce user authentication and access authorization. Authentication and authorization must be handled by the server hosting the NAS. This safeguard is added for cases where a client node is infected by a virus.

  4. Secure against accidental data removal and ransomware: This point is different from the previous point: a user with legitimete access to a shared directory might accidentally delete data on the shared drive or malicious software might encrypt files on the shared drive. It is difficult for the to server detect this kind of behavior. The system should have measures against attacks by ransomware.

  5. Safe against disk failure and disasters: The system should be online in case of a single disk failure. Furthermore, all the data on the system should be recoverable in case of disastrous events, such as a fire.

  6. User-encrypted shares: The system should permit shares with encrypted data to which the encryption key is only available on client nodes. This means that the privcay of such data is guaranteed even in the case of a compromised NAS server.

The first three points are direct requirements on the protocol used for the NAS. The last three points are rather requirements for the workflow or the organizational structure around the NAS.

NAS Protocols

I think there are three well-tested protocols available in a LINUX environment: Server Message Block (SMB) or Common Internet File System (CIFS), Network File System (NFS) and Secure SHell File System (SSHFS). The last option might be a bit of a surprise to appear in this list, but as it turns out, SSHFS is well suited for the task at hand.

SMB or CIFS with the Samba implementation is the natural choice in heterogeneous networks because it is well supported by Linux and Windows. Since version 3.0 of the protocol, SMB supports end-to-end encryption. It is also possible to set up virtual shares. To be honest, I’ve never seen a samba server which exposed the full file system. As far as I know, all of the access control is handled by the server. This means the first three requirements would be satisfied by a samba-based realization.

In the Linux environment, the network file system is NFS. However, I’m not aware of a built-in feature, that allows easy configuration of virtual shares. Prior to NFS version 4, the access control was handled by client nodes. Old NFS servers only checked if a client node is allowed to mount the file system, however, permission checks of users on the client node accessing files are performed by that client node. Since version 4 of the protocol, it is possible to have individual users authenticating via a Kerberos server. In that case, the communication can be encrypted. This has the disadvantage, that one has to set up and maintain a Kerberos server (if not yet available). This doesn’t make NFS look good when compared to my requirements.

I have to admit, I’m not (yet) an expert on Samba, NFS, and Kerberos. Setting them up, involves the possibility of misconfiguration which could have disastrous consequences. After due consideration, I decided to realize my personal network attached storage using SSHFS. SSHFS is based on SSH which has encryption built in right from the start. For SMB and NFS, encryption or authorization was added quite recently.

So how does SSHFS compare? As I’ve already said, SSHFS is encrypted. Individual users authenticate with the server, the server takes the decision if an operation is permitted or not. However, as with NFS, there is no straight forward way to realize virtual shares. Does this mean I have to drop some of my requirements?

My Setup

In this section summarizes the configuration of my personal network storage which satisfies all of the above requirements. The commercial NAS was retired, so the system consists of the three-disk RAID-5 NAS on my server.

Encrypted traffic

Since my setup is based on SSH, all the traffic is encrypted. The OpenSSH server configuration contains the line

PasswordAuthentication no

such that all the clients need to register their public key with the system. The fingerprint of the server is well known, such that the authenticity of both endpoints of the connection is ensured.

Virtual Shares

The issue with virtual shares is two-fold. First there is the issue of permissions and file ownership. On the other hand, there is also the issue of blocking access to the root file system.

With SSHFS, the permissions would be identical to the file system permissions on the server. Assume we want to have a virtual shared directory to which two users have access. Both users should have full access to all files. Well, the first idea would be to create a Linux group for the shared folder, add all participants to the group and give full group permissions to the contents. This solution will fall short because we cannot prevent one of the users from removing the group access permissions of a file he owns.

The key to virtual shares is BINDFS. A special Linux user nas_user has exclusive access to the mounted RAID disk. The top-level directory of the NAS drive hosts directories which represent the virtual shares. To give users access to a share, we can use BINDFS to create a different view of the file system. For example, the users Frank and Tobi can get access to the shares share1 or share2 via fstab entries like

/mnt/array1/share1     /mnt/vhome/frank/share1   fuse.bindfs force-user=frank,force-group=frank,create-for-user=nas_user,create-for-group=nas_user 0 0 
/mnt/array1/share2     /mnt/vhome/frank/share2   fuse.bindfs force-user=frank,force-group=frank,create-for-user=nas_user,create-for-group=nas_user 0 0 
/mnt/array1/share1     /mnt/vhome/tobi/share1   fuse.bindfs force-user=tobi,force-group=tobi,create-for-user=nas_user,create-for-group=nas_user 0 0

The two users frank and tobi both have access to the virtual share share1. Both of them have a different view of the shared folder. Due to the mount option force-user, both see themselves as owners of all the files. If a user creates a new file, say in /mnt/vhome/frank/share1, the file is actually create at /mnt/array1/share owned by nas_user.

To solve the second part of the issue, the fact that people can ssh into the server and look around the root directory or run mount, we can instruct OpenSSH to prevent this. For NAS users, here added to the nas_clients Linux group, OpenSSH should force sftp and chroot into /mnt/vhome/%u.

Subsystem sftp internal-sftp

Match Group nas_clients
	ChrootDirectory /mnt/vhome/%u
	AllowTCPForwarding no
	X11Forwarding no
	ForceCommand internal-sftp

Please note, that the directory /mnt/vhome/%u must not writable and shares like etc or lib cannot exist in order to prevent Roaring Beast attacks.

In order to make the chroot work under CentOS 7 with SELinux in enforcing mode, you need to run

$ setsebool -P selinuxuser_use_ssh_chroot 1

Server-side permission

With SSHFS, permissions are checked on the server. This is an inherent feature of SSHFS. Therefore, this requirement is natively fulfilled.

Secure against accidental data removal and ransomware + safe against disk failure and disasters

These two goals can be achieved by the same organizational measures. I think the best way to prevent these cases, is to take frequent, incremental, offline and off-site backups. There have to be chronological (instead of simple mirrors) backups such that one can go back in history to restore data that was deleted or encrypted a long time ago without noticing. Incremental backup are a way to achieve this by saving disk space at the same time. For this purpose I started using borg.

The backup should include all files that are on the NAS–regardless of how created them an regardless of file permissions. For this, we can employ the virtual share infrastructure already in place. All we need is a new user, which has access to all shares. This very privileged maintenance user should be used only to create backups.

To prevent that the backup itself from being encrypted, the backup must be kept offline as much as possible. In order to restore data after a fire, there need to be backups (on tape for example) that are shipped off-site.

User-encrypted shares

External software, such as EncFS, stores encrypted files in one directory and provide a fuse-mounted directory with decrypted files. Any modification to files in the decrypted directory is propagated to the encrypted source directory. The source directory can be on a shared directory, the decrypted view can be mounted locally and thus fulfills the last requirement.

Summary

So far, I have had a very positive experience with my personal network attached storage based on SSHFS in combination with BINDFS to realize virtual shares.