Weeknotes 2022/15 - Dexie, SQL.js, and the Limits of LocalStorage

I started a new Slipmat.io sub-project where I’m working with DJ music collections and playlists. This introduces some interesting problems when done with Web technologies as we’re mostly talking about displaying lists of thousands, tens of thousands, or even hundreds of thousands of rows of data in one view.

Rendering super long lists is not common nor generally a smart idea in the browser in general, but this spesific case is more like the Finder on macOS than a typical Web app. In addition of just rendering these lists I also bumped into the limits of localStorage when trying to cache things. I’ve never had to worry about QuotaExeededError exceptions with localStorage but when you’re trying to cache tens of thousands of lines of JSON, you’ll get there pretty fast.

I started experimenting with Sql.js which is SQLite compiled to JavaScript. It’s super easy to use with Vite but not very well suited for a typical Web app because you need to download and execute a ~1,4 Mb WebaAssembly binary to instantiate the database.

Another interesting piece of technology is Dexie, which is a wrapper for IndexedDB. Dexie is already a pretty mature library and plays well with modern reactive tools like Vue and React.

While localStorage quota is in the range of 2-10 megabytes depending on the browser, IndexedDB is much higher, typically at least 1 Gb for desktop browsers. (Maximum storage depends on the browser and available disk space.) For my purposes I can easily fit everything I need into IndexedDB. Now the problem narrows down to being able to render huge lists in a way that feels smooth and doesn’t eat too much memory.

JavaScript world has a few solutions for rendering huge or “infinite” lists. DOM Recycling has been around for a while and there are several libraries like vue-virtual-scroller that implement this. I’ve only started to scratch the surface here but I’m pretty sure that by the end of this challenge I’m much more familiar with the devtools performance tab that I am currently!