# Looking for a shared encrypted filesystem

## Mr. Anderson

Hello,

I am looking for a solution to have an encrypted file system, lying on a central server. Data shall only be decrypted and encrypted on the client side. Furthermore, several clients shall be able to do I/O on the same files. Byte range locking shall be possible (support for fcntl is probably sufficient).

The bottleneck is the network connection. So, performance of the filesystem itself is not very important.

So far, I have thought about using encfs on afs, or maybe encfs on glusterfs. I have found a filesystem named cloudfs that is based on glusterfs, which also might do the job. However, I hope somebody here has experience and knowledge concerning all the existing filesystems and will recommend me something that is suited for my needs.

Thank you.

----------

## BradN

So basically, you expect this to work where data on the server is to be fully encrypted (ie, server is untrusted).  This puts to an end any idea of the server coordinating file level access control, or even directory update or any other metadata control.  The server will then basically act like an nbd server, just reading or writing blocks as directed by the clients.

So, I think at this point, you can ignore the encryption part and just focus on finding a filesystem that can can handle this access scheme (ie, it must do all the synchronization on the clients).

If you find a filesystem that works for this without encryption, encryption can probably be added in at the block level just by using the sector number as an IV - it won't be perfect (an adversary could likely say whether a given sector is the same or different as previous contents stored in that block), but is probably enough to do the job.  This may or may not be how one of the encrypted block devices works, but if it is, you would probably just make it use the nbd block device as its storage medium, and then have your filesystem mount the device exported by the encryption driver.  

Something like truecrypt wouldn't work here, I don't think.

Again, the key to allowing the encryption here is that for the encryption to operate, the encryption routines only care about 3 things to generate an encrypted block:  The unencrypted data contents, the encryption key, and the block number it will be stored at.  Actually the third is optional but not using it will lose a lot of security by allowing an adversary to compare data blocks against each other (ie, they could identify patterns like copied files, and count instances of different data contents, even though they wouldn't know the contents of the data without guessing).  

The point is, this kind of encryption would be compatible in general with any block based multiple-client filesystem.

More advanced encryption would need to be multiple access path aware, and I think you'll have enough difficulty finding a reliable filesystem for this purpose without even considering the slight data leaks the sector number based IV encryption has.

----------

## BradN

Looking up some of the things you mentioned...

Encfs operates locally on the client and transforms some other filesystem (in general) into an encrypted filesystem by encrypting files on a one by one basis.  This will likely leak the sizes and number of files, which may or may not be a problem.

Possible AFS limitation (from wikipedia):  "A consequence of the file locking strategy is that AFS does not support large shared databases or record updating within files shared between client systems. This was a deliberate design decision based on the perceived needs of the university computing environment. It leads, for example, to the use of a single file per message in the original email system for the Andrew Project, the Andrew Message System, rather than a single file per mailbox."

So, I see your strategy was to use a filesystem that does depend on the server coordinating locking, and then encrypting the files themselves as they're put onto it.  This may be a valid strategy if leaking the metadata info isn't a problem, ie you don't care if the server knows the kinds of things you might be doing, as long as it doesn't know the specific data involved.

----------

## dE_logics

Maybe you can store data on an encrypted partition (DM crypt), then pass the data to the clients using stuff like sshfs, ftps (this can be mounted with curlftp), sftp, NFS over ssh etc...

----------

## jormartr

Ecryptfs does the part of encrypting/decrypting on the client side, is understood by the system as a real file system (not fuse).

----------

## Mr. Anderson

Thank you for your answers.

 *Quote:*   

> So basically, you expect this to work where data on the server is to be fully encrypted (ie, server is untrusted). This puts to an end any idea of the server coordinating file level access control, or even directory update or any other metadata control. The server will then basically act like an nbd server, just reading or writing blocks as directed by the clients.

 

Exactly.

 *Quote:*   

> This may or may not be how one of the encrypted block devices works, but if it is, you would probably just make it use the nbd block device as its storage medium, and then have your filesystem mount the device exported by the encryption driver.

 

An nbd would be perfect, but I had some doubts using an nbd. As far as I understand, an nbd would have to be mounted by each client, and I supposed mounting a filesystem several times concurrently is a bad idea. But if there is a filesystem that allows this without corruption, I would be glad. However I have not found anything about this aspect. Maybe you can point me to a filesystem that can be mounted r/w concurrently?

 *Quote:*   

> Encfs operates locally on the client and transforms some other filesystem (in general) into an encrypted filesystem by encrypting files on a one by one basis. This will likely leak the sizes and number of files, which may or may not be a problem.

 

This would be an acceptable drawback as I am pretty sure that this information would not be of any use. Actually, all metadata may be leaked as long as the actual contents of the files are kept secret.

 *Quote:*   

> Possible AFS limitation (from wikipedia): "A consequence of the file locking strategy is that AFS does not support large shared databases or record updating within files shared between client systems. This was a deliberate design decision based on the perceived needs of the university computing environment. It leads, for example, to the use of a single file per message in the original email system for the Andrew Project, the Andrew Message System, rather than a single file per mailbox."

 

Thank you for pointing this out. This turns AFS unsuitable for my needs.

 *Quote:*   

> So, I see your strategy was to use a filesystem that does depend on the server coordinating locking, and then encrypting the files themselves as they're put onto it. This may be a valid strategy if leaking the metadata info isn't a problem, ie you don't care if the server knows the kinds of things you might be doing, as long as it doesn't know the specific data involved.

 

Exactly. However, if it is possible, I would prefer a solution exporting an nbd. But I do not know which filesystem I can use then.

 *Quote:*   

> Maybe you can store data on an encrypted partition (DM crypt), then pass the data to the clients using stuff like sshfs, ftps (this can be mounted with curlftp), sftp, NFS over ssh etc...

 

Sorry, this would not be enough. The data would be decrypted on the server side, so a trojan horse could read all content.

 *Quote:*   

> Ecryptfs does the part of encrypting/decrypting on the client side, is understood by the system as a real file system (not fuse).

 

Sounds good, but can I mount this concurrently r/w on several clients without data loss?

edit: Let's say, I want to use an nbd. What file systems could I use, that can be concurrently mounted r/w and encrpyt my actual data?

----------

## BradN

I'm not aware of anyone having designed a concurrent filesystem around only block level read/write access, with no other synchronization primitives.

I suspect it may be possible, but it might make performance rather suck, and might have limitations like a fixed number of clients, designating a "master client" or heirarchy of clients or things of that nature.

In fact... if you would designate a single "client" machine as hosting the filesystem itself, but using an external storage through encryption, then had the other clients connect through that client machine, it may accomplish what you want, at least if all the clients are in one location.

If the clients are spread out, it might not be desirable network wise.

Just an idea, anyway...

[side topic]:

To further extend the basic nbd idea, I think an nbd server that also does things like cmpxchg or conditional data exchange might be necessary to create a system of acceptable performance, as it's basically the same game as multiprocessing synchronization, but I'm no expert in this area at all.  Some of these primitives might not play well with encrypted storage.  One sync primitive that would still work is "replace with x if contents are exactly y", since same/different can still be compared for a single block in the encrypted state.  Basically a client would investigate a sync block, and make a change to it only if it hasn't been updated by something else during that timeframe (this would be how a lock is acquired).  Designing a filesystem around this sounds like a chore though.  The encryption layer would also have to pass through the separate sync mechanism(s), or the filesystem would have to integrate the encryption.

I think this is getting a bit too theoretical to be of near term practical use though.  You will probably have more luck with individual file encryption, but in that case, you may also need to be careful of how block updates are done within files (is a single file changed at once, or can blocks within a file be individually updated?).  This is important for databases.

Most encryption schemes require data to be updated in blocks of a minimum size, so this may limit usefulness in database type applications (if the database software limits itself strictly to a fixed size block, it may work).  Schemes that allow byte level updates are likely insecure (such can be designed with a fixed xor mask for a file, but if an adversary obtains more than one copy of the same encrypted file with different changes applied, the encryption breaks down rapidly if any assumptions can be made about the file).  For instance, encrypting english text in such a scheme can be broken very rapidly, sometimes only by obtaining 2 encrypted versions of a file.

----------

