Generic Storage interface for use on all client and on the server.

[sorotokin]

New interface for persistent storage to replace our current client and server interfaces. On the server side, new interface supports expiration, supports Postgresql and is much better specified and tested than the current interface. On the client side, important features are moving to sqlite-based storage and support for iOS. Unifying server and client storage also gives our provisioning code much better foundation (this is part of the code that we can choose to run either in the mobile client or in the wallet server).

New interface is non-blocking and it can be safely invoked from any thread (even from the main thread on the client). All blocking operations are run on the appropriate thread on all platforms.

This is not hooked to anything yet.

Supports ephemeral storage (no persistence), SQLite on both iOS and Android and server-side databases using jdbc.

Added unit tests to get fairly good coverage of the code and exercise all implementations in exactly the same way.

Also tested by implementing existing StorageEngine interface using new interface (using runBlocking to stitch it, as the old interface is blocking) and making sure that wallet app works correctly.

[koukarine]

Not a blocker, just a few thoughts:

1. Why should we care about handling that during "get" instead of "create"? The sooner the issue is detected the less an impact of such a user (programmer) error. Or there are certain use cases when this is a legit possibility? (i.e.: it's OK to have multiple tables created with the same name, but getTable must fail if used in such a scenario)?
2. If it appears that the situation with "several parts of the app defining same name table" is anticipated, perhaps, we could instead allow that to pass but use a unique tables identification for tables created instead of giving the user full freedom of naming them? That could serve both the app stability and ease of troubleshooting. As if "code part A" is using the same name as "code part B" you can't tell from this exception context which code part is at fault (and must be fixed).
3. What if instead of arbitrary naming, the user could receive the generated handle of that table during create request, instead of reusing some text string name to access it randomly (yes, that's convenient, but might be highly human-error-prone). It could be a simple String handle as well (for persisting or sharing), but 100% guaranteed unique upfront.

[sorotokin]

The most common case why the same name but different spec would be used is when someone accidentally names their table the same. It easier to do than it seems, especially since we want to be able to run our server-side code in the app (convenient for demos and debugging). And once such error happens it may not be caught right away. Think of a spec (which is a singleton), not the name to be the real table definition. It has to be defined and owned by some part of the code, but the table can be accessed anywhere the singleton is visible. We still need a unique name as this is persistent storage, right? How else would the same part of the code know that it is loading stuff from the same part of the storage?

[koukarine]

My point #1 exactly. I have suggested the option #3 which is to refrain from allowing manual naming, but generating and returning the unique handle instead. E.g the hash of the spec if it's truly unique (or +number if not). The user can save that handle for further use as needed, including within the corresponding spec singleton (it could be actually generated when the singleton is created and available for reading as well for flexibility of separate identification by that String handle only).
"If there is a room for the human error, that human is eventually coming and making it". (rephrased Murphy's Law) :)

[sorotokin]

Sorry, I still do not get it. We want to avoid collisions, but we need to be able to create persistent identifiers for object in storage.

The spec is not unique, of course, there are only two bits in it. Spec object is unique, but two specs can legitimally have exactly the same content. And how number is better than text?

The other thing is that it is nice to be able to find that table using SQL, so having stable table names is a good feature to have. Pulling the db file from the app and running sqlite3 command on it is a very useful technique.

[koukarine]

No problem. My point is that at the time of the table creation in the code we can easily take care of the name uniqueness ourselves, 100% preventing the problem you have outlined.

Returning the generated unique table name is only a simple solution for that. Appending a "Number" is another solution (user can supply the table name, but must use the amended name (e.g. (with a number appended) returned during the table creation.

If there is a need to access that table from a generic SQL client for troubleshooting, it's easy to Log.i() the returned table name during its creation and use it in the manual SQL session. While in case the name is the same (hardcoded string) but used in two places as allowed in the current impl, you will have trouble distinguishing one table from another.

[sorotokin]

I think the focus should be not on preventing, but detecting. We do not want accidentally have two tables named (or numbered) the same even if this is handled gracefully at runtime somehow - but I do not understand how.

Suppose we have two tables that should be distinct but defined in exactly the same way in our code. How do you propose we resolve this consistently every time the application restarts?

[koukarine]

For example, I could save the unique generated name received from the Storage API in my client process-bound persistent storage (belonging to the "create" caller, not to the Storage API; e.g. the storage key can be generated from the calling class name/package if there is a common caller of sorts anticipated), and read it from there on restart from that same caller. It actually does not matter for the API impl., as that persistence task depends solely on the client implementation then. When the client receives the object identifying the new table created by the API, it should be clear, that in order to access it again they have to save it somehow and propagate to following calls as needed similar to any RT data value. How exactly depends on the client/caller impl. (e.g. as a parameter or part of the bundle might suffice for an Android impl.).

Whereas if that's just a user defined name (string), there will be always a temptation to have an utility companion object holding that name statically.

It's like naming a constant visible only in the narrow scope of it's definition. On the OS level (API level impl. in our case) it does not matter how you name a constant it in your code, for the OS it has a unique address in its internal address space and it's identified by the name of that constant for any transactions within the original(client) definition scope.

[sorotokin]

I could save the unique generated name received from the Storage API in my client process-bound persistent storage (belonging to the "create" caller, not to the Storage API; e.g. the storage key can be generated from the calling class name/package if there is a common caller of sorts anticipated), and read it from there on restart from that same caller.

Sorry, I do not understand how you'd do this. You have two identical, but distinct Kotlin objects. One generates name1, the other generates name2. The process restarts. How you know which one is which?

The other point is that I am not trying to reinvent a well-established concept like table name! All I am trying to do is to make sure that if a table is defined in one part of the code, all usages of this table refer to that part of the code. If another table with the same name is defined, we get a runtime error. It would be even better to have a compile-time error, but I really need this to be an error. I do not want to invent a different way of naming a table merely to avoid this error.

[koukarine]

Let me explain shorter: Both processes would need to save the generated unique table ID value somehow for reuse (if/when needed). It's still a text string so it can be saved by any perm. storage means provided by the framework (but not by this Storage API). It looks like a clean separation of concerns of either party.

I'm not trying to reinvent SQL either :) But you have outlined the potential problem of your API impl. I'm suggesting the solution which is eliminating that problem (which is allowing user error when setting the internal name of the table equal to other table name) at its root (by physically separating the client impl. code and API impl. code. That way, as the problem can't happen anymore, there is no need to throw at RT, nor to notify at Code Parsing.

In fact, you could probably address it at the table creation flow. Native SQL would not allow creating a new table with the same name as the existing one. Or what I'm missing at the impl. level?

[sorotokin]

Both processes would need to save the generated unique table ID value somehow for reuse (if/when needed).

Let's say the object in file A.kt generated table named name1 and the object in file B.kt (identical in value to the one in A.kt) generated table named name2. What should they save in storage so that when the process is killed and restarted the object in A.kt knows that it should use name1 and the object in B.kt knows that it should use name2? Perhaps I am missing something really obvious, but I just do not see how this works.

[koukarine]

Just a question: why these expiration feature is so important? I feel it's something as controversial as the GC mechanism in Java. Yes it sounds simple and helpful, but uncontrollable memory leaks are "the plague of the century" 😄 . Why we can't control purging of the unneeded data manually instead of relying on a "dumb" timer? Or that's some specific security precaution / measure?

[sorotokin]

Expiration is not similar to GC at all in a sense that nothing really leaks. Expiration time is inherent to the object and observable in most cases. The only way to avoid having expiration is to actually set up a timer that would have to run at certain point in time and delete an entry - this would be expensive and fragile.

• OpenId4VCI protocol requires nonces and tokens created by the server which must be rejected after expiration.
• Similarly, DPoP requires clients to produce unique expiring nonces
• Some of these tokens (but also nonces) are actually convenient to carry session information, so we do not want to add too much entropy there so that uniqueness is purely probabilistic, as they become too long

Personally, I am not a big fan of nonces, I think they have way too many of them (as it requires state where once could easily go without), but these protocols are what they are.

[koukarine]

Thank you for the explanation. Definitely a useful feature then.

"Leaks" of my concern in this context could be leaks of the data storage information. E.g. if the app process is killed in the middle of the purge, on the app (or some other process) restart you might have some data wiped, but some still intact and used unexpectedly.

For the "manual" purge, there is no need to have a timer. The same "purge all what expired" can be called when the app is clearly in a safe state (like synchronously). While during async flows you just check the integrated expiration timer and pretend the data is already wiped. That would also improve async flows performance as there is no heavy DB activity could kick-in unexpectedly for auto-purge.

[sorotokin]

if the app process is killed in the middle of the purge, on the app (or some other process) restart you might have some data wiped, but some still intact and used unexpectedly.

I don't see how it is possible with my implementation, especially "used unexpectedly" part, which is the only thing that matters (If you are killed in the middle of the purge, some data is not going to be wiped in any design)

The same "purge all what expired" can be called when the app is clearly in a safe state (like synchronously).

I am still not sure what exactly you propose we do. There is no state of the app that is inherently safer than the other. And there is no evidence that we are going to have an issue in this spot. Why add any complexity there?

While during async flows you just check the integrated expiration timer and pretend the data is already wiped.

I am confused because this is exactly what I do.

That would also improve async flows performance as there is no heavy DB activity could kick-in unexpectedly for auto-purge.

While we should not do anything that is inherently non-performant, we are not trying to optimize things. If anything, we want our storage layer to be clean and boring. For very high volume of transactions, we'd have to do things very differently (e.g. we'd have better ways to shard the data, short lived-nonces probably would not go into storage at all, etc.), but then we'd probably have a few engineers working just on that. For this project this is just not something we want to focus on. We need something that is just adequate.

[koukarine]

Sure thing, I'm not arguing about your API implementation, it's good! I'm just making sure I understand the general intent of having that method and making notes how to use it right while we are at its inception and with the owner online :)
BTW. The "safe" purge place for an android app would be on the app start. F.W.I.H.W., I doubt it would accumulate a lot of data pending purge during typical user session.

[koukarine]

I see another potential issue with the "expiration feature" (but indeed it depends on the answer to my prior questions). The table we insert into might be considered as a complex state. If the expiration time assignment and tracking is not precisely coordinated that state can be easily broken as a whole if some of its definitions expired (and purged) sooner or later than others. Aside from an obvious human error potential it might lead to a partially broken state delivered asynchronously due to the purge latency (the chance is growing with the purge job volume growth).

[sorotokin]

I do not understand - could you give me a specific scenario as an example?

[koukarine]

Sure. E.g.: we have a flow A using table1, and flow B initiating the table1 purge. If they are independent and B is starting the purge when A is still using table1 there is a chance that B (purge) could be paused by he system to release some ticks to other threads. It could be flow A receiving that time slice. Which could lead to the situation when A is expecting to obtain N values from the table1, but instead getting only N-M(purged) records.

[sorotokin]

I don't think this is possible with SQL-based storage. And when we read, we always ignore expired records, so it should not matter if they are physically purged or not.

I guess I can add a statement that the guarantee that we have is that one is not going to see an expired record according the the clock time somewhere between that start and the end of the API call, but that's more or less self-evident?

[koukarine]

SQL-based, maybe not. But this is an API, so the impl. might be different on some other platform. Even with SQL there is a chance of async "get" call inserted between delete calls of purge() if they are implemented as individual rows deletions in a loop.

[koukarine]

Another exploratory question:
Is there a pressing need for the Update method? I understand it's the RDBMS standard. But from my experience it's only purpose is to reduce huge databases fragmentation growth speed and some data audit tasks. Do we need that? Otherwise delete + insert new is much closer to the "unmutable state" paradigm (paramount in async flows). Update would also cause complications for the automatic data expiration mechanism. E.g. should it be aware, that the expiration time can change on the fly? Or that time can't be updated? Or purge should wait for the update pending? Or instead, you can't update while purge is in progress, so it should throw? Or should it schedule that update if purge is in progress? E.t.c.

[sorotokin]

Frankly this is the area where I think we should not innovate. I do not think delete and insert is a good paradigm - at least if there are no transactions. There are storage mechanisms that have only inserts but you add a timestamp when you insert and old data is purged, but this is done on the lower level for various other reasons (like scalability, history, and replication) and on the API level you'd typically still see update.

To see where we use update on the server, you can see it by looking at usages of com.android.identity.flow.server.Storage.update (this is an interface we aim to replace). On the client a lot of StorageEngine.put calls are updates, as it follows Map semantics today.

[koukarine]

So we do need to support UPDATE for scalability and audit tasks? Like for compatibility with other platforms maybe and for manual troubleshooting? In a standard Android project I would avoid a simple Map mutability wherever possible.

[sorotokin]

We need to support UPDATE because higher-level code does expect support for mutable state.

[koukarine]

Are you referring to the JP Compose mutable state? That's a bit different thing. Mutable State there is used as a signaling mechanism. But I trust your expertise, if you see the Update method beneficial, let's keep it intact.

[sorotokin]

Oh, no, Compose state is a different thing altogether. What I am taking about is that we have code that basically wants to "save an updated state" in colloquial terms.

[koukarine]

Very interesting method. Could you explain the expected use case? In my experience, such a chunked listing might be crucial only for dependent objects (e.g. when searching for small sequentally placed groups of items needed for cross-dependent calculations.
Also, the need to remember the last key to continue listing might suggest a more trivial solution like getIterator(partitionId).use { iterator -> instead (hiding that continuation complexity within the iterator invocation and thus reducing the likelihood of a human error coding the proper chunk processing and setup loop).
Just an idea. Depends on the anticipated use case.

[sorotokin]

I am not really sure how important chunked iteration is, but it is very difficult to put after the fact (once full iteration starts to kill performance in some cases), e.g. look at GenericStorageEngine and see how you could add it.

Remember that this interface is a contract with the storage engine. We may add convenience methods (or even some layers) as extensions on top of it.

Another use case here is iterating from a given element for a limited number of items. This is more or less required for implementing UI like LazyColumn once you have hundreds of items, but we will need to add ability to iterate in both directions for that use case. Not sure if we have to add it right away, though, right now there is no pressing need for this, at least on the client.

[koukarine]

Gotcha, so that's more like a caching mechanism.