I would like to set up a per-file encryption architecture where every file is encrypted with its own key, and each key is encrypted using a master key.
Is there any good articles to read regarding this? Or any advice?
The questions I have are basically:
My files are stored in an on-premises S3, I also have a relational database and I'm using Vault (Hashicorp)
This can be reduced to a "key management" problem, because everything else is basically a solved problem. So lets take care of the accessory questions first.
How to re-encrypt files if I do a key rotation
You don't reencrypt files. It's a CPU-heavy and IO-heavy operation, and you don't want to read and write terabytes of data each time a user changes its keys.
What you use is something called Envelope Encryption: You create a symmetrical key (Data Encryption Key, or DEK) and encrypt the file. You use another key (Key Encryption Key, or KEK) to encrypt the DEK and store the encrypted file and the encrypted DEK. When you rotate the KEK, you decrypt it first with the old KEK, encrypt with it the new KEK, and store. The file isn't touched.
How to rotate the keys
This is trivial. You issue new keys, decrypt and reencrypt all DEK, and it's done. The old key lost all function because it cannot access anything anymore, so there's no need to destroy it.
Now the complex questions:
Where to store each file key
Where to store the master key
And this is your problem. It involves more process and choices than technology, so there's no "best way" to implement.
If you are the one managing the keys, you have to make sure nobody but the user can access the key. This can be achieved using a key derivation algorithm to derive a key from the user's password, and using that key to encrypt the user's private key (the KEK). Then you can store it on a database. You could use the master key to encrypt those keys, allowing you to implement a "password recovery" process. That would create a big downside: if the master key leaks, every single account can be compromised and every single file decrypted.
If your data storage will employ End to End Encryption, the KEK must be stored on the client. You must protect the key client-side too, so encrypting it using a key derivation function is a good idea. The downside is that you cannot create a "password recovery" function because you don't have the password and cannot decrypt the KEK without it.
The master key storage depends on your security requirements and your budget. A Hardware Security Module (HSM) is a good starting point. It's a dedicated hardware for key storage, and its job is to not let the private key be accessed in plain. You must create a secure process to use the key in case of a password reset or other need.
