Search on encrypted files

[dandar]

Dear Jonas Winkler,

Thank you for keeping paperless-ng moving forward. Regarding search and encryption as stated by the Important Note:

Important Note
Document scanners are typically used to scan sensitive documents. Things like your social insurance number, tax records, invoices, etc. Everything is stored in the clear without encryption by default (it needs to be searchable, so if someone has ideas on how to do that on encrypted data, I'm all ears). This means that Paperless should never be run on an untrusted host. Instead, I recommend that if you do want to use it, run it locally on a server in your own home.

Is it possible to use an encrypted search index file/files? Encrypted by the user's password, random salt, or both?
A different index per user would be ok, I think?
The search index can be decrypted and stored in a locked-down/privileged portion of disk or memory upon user login or first search if the index is small enough, and destroyed on session end, disconnect, or logout?
Depending on the user's device specs, and the size of the index (which I imagine is small even for several hundred thousand files) the decrypted index can even be stored in the browser cache/memory of the user, which means decryption is done by the client device, likely by javascript?
Check search techniques used by Proton Mail or Tutanota, maybe?

I am not familiar with the internals of paperless-ng, yet it seems if the search is indexed-based and fast, we might be able to do something that makes sense without degrading performance.

[jonaswinkler]

Oh. That note is still in there from Daniel Quinn and I should probably make that clear. My bad.

Anyway, on encryption.

Paperless does in fact use a search index, and it does contains terms from your document in plain text. It does not contain the actual content of the documents, but it is certainly possible to reconstruct some content of your documents with that index.

Encrypting the search index is pretty difficult. Primarily due to the fact that many internal components in paperless need to access and modify not only the search index, but also other parts that store data (the database, the media directory, and the model that's responsible for the auto matching algorithm):

• You're able to drop documents into the consumption folder. When that happens, Paperless needs to update the index with the new document. This should work, even if not logged in. If the index is encrypted, paperless needs the key to decrypt it.
• When paperless adds documents from mails, the same as above applies.
• Paperless regularly updates the auto matching algorithm and needs access to clear text content for that.

All of the above require paperless to have the means to decrypt encrypted data. This would require storing the key somewhere on the server (That's what has been done in old paperless, and I don't think its a good solution), and that defeats the purpose of having the data encrypted in the first place. Encryption protects in case of a server breach, and having the key available on that server, well...

I'm also not an expert on security, so take that with a grain of salt.

Regarding your specific questions:

A different index per user would be ok, I think?

If I were to add some actual support for multiple users, yes. Or some other means to make sure that users don't find things they're not supposed to see.

The search index can be decrypted and stored in a locked-down/privileged portion of disk or memory upon user login or first search if the index is small enough, and destroyed on session end, disconnect, or logout?

See above.

the decrypted index can even be stored in the browser cache/memory of the user, which means decryption is done by the client device, likely by javascript?

This is exactly what I would do If I would design and implement a secure DMS. Have everything encrypted on the server, and the key for decryption is derived from a user password and never sent to the server.

However: These are no easy changes. In fact, we'd need to reconsider pretty much every aspect of paperless. How documents are added. How data is stored in the database. How searching works.

Therefore, it's very unlikely that I'll ever add anything related to encryption to paperless. This would be a new project.