In my work developing custom web applications for small customers I’m finding that these web apps are more and more used from mobile devices, with sometimes flaky or non-existent Internet connections, the ideal solution to that problem is to build these applications as an offline-first webapp.
This means that ideally I would have a database in the browser that would at least give me enough query power to do CRUD like applications: getting a list (sorting and filtering), insert, update and delete, joins… (think for example Django Admin).
Then the databases would need to be synchronized with each other. I first looked into solutions like PouchDB to do that, but while researching[1] I was coming to the conclusion that it would be much simpler to describe all changes to the database as a stream of events of schema and data changes (as in event sourcing). This would solve the synchronization problem: clients would get all the events since the last synchronization and afterwards append the uncommitted locally generated events to the stream; and do some conflict detection in between (see CQRS). I’ve found some ready to use event stream servers that could be used out of the box (although of course one could implement it fairly easily in PureScript or Haskell some day).
So what’s left to do would be to define a DSL for schema and data. I was thinking of something fairly simple as the GraphQL schema language. That would be generic enough that a “backend” could be implemented in everything from IndexedDB to relational databases. As somebody who likes types I would want to keep a typed schema[1].
As you can see this becomes quite a project on itself, but I think that it would be manageable to build this in PureScript.
So here comes my first question: Am I making sense? Is something like this already done? Would you like to help on this project[3]?
[1] My inspiration came from ACID state (Haskell), Postgres WAL log, Martin Kleppmann, Greg Young, and I’ve found https://debezium.io/ in the meanwhile, and probably many others.
[2] Schema migrations could be backed by data migrations which are delivered with the code (JavaScript, so not as an event): migration :: Schema V1 -> Schema V2 etc.
[3] Free software and developed in the open of course!
Check out typedefs by statebox for a typed interface description language (IDL).
You are describing what is the general direction programming is headed, the step after (which is the one I’m interested in taking straight away) is to do away with the server and use dat/ipfs for persistence and making every client a server.
Although to be fair there is an inbetween step (ActivityPub) where servers act as event aggregators to simplify the system, ideally these are emergent and formed via compositionality.
FWIW, I’m using PouchDB quite successfully from PureScript. It deals well with the persistence, browser, and network stuff. The records in my database are tagged with a type and only handled by PureScript and JavaScript map&reduce functions. I managed to avoid heterogeneous queries for the most part, which means I get to abuse the default simple-json (de)serialization. This makes the code quite pleasant.
Naturally, my biggest issue is schema migrations. I try hard to make only backwards-compatible changes, but sometimes I need to write extra scripts to handle migrations. Fortunately, there’s not too many users and I can force them to sync, do the migration, and and then only run new code, so I don’t have to worry about old data living in some disconnected browser somewhere and coming back through sync months later. Handling the general case can probably be done, but is incredibly subtle in an eventually consistent, distributed world.
This is the library I wrote and use, if you’re interested: https://github.com/fehrenbach/purescript-pouchdb It contains fairly low level FFI wrappers around the PouchDB API as well as a more opinionated, higher-level interface. (I should probably tag a release at some point…)
@ilmu hank you for referencing typedefs. Very interesting! I will surely check this out.
Yes, I came across the unhosted.org website which seems to do exactly this. Interesting possible future for the Web.
@fehrenbach Thanks for sharing your experience with PouchDB. The thing that keeps me from going that route is that I need to go all-in on CouchDB, a tech which I don’t know yet, and which is not as simple as an event store or as familiar to me as PostgreSQL.
Is anyone aware of an application that uses a simple database query layer on top of IndexedDB? Something like nanoSQL maybe? Of course I could create bindings for that API.
Hi @daniel,
I’ve made a CGI script (yes, I know… old school) in Haskell which can store the events in files. It’s open source, but not documented at the moment (I will hopefully get to that soon):
I’ve also created a PureScript Halogen app that communcates with the CGI script. I will open source parts of this soon. I haven’t yet had the time to create the code to store the events on the client side (I will probably use IndexedDB), but that is something that I like to do. That would make it possible to use the app offline.