From 7955a3aa691f5bcf93ffd5fb2bd74eb36a7f2b79 Mon Sep 17 00:00:00 2001 From: Joe Ardent Date: Fri, 14 Jul 2023 18:10:48 -0700 Subject: [PATCH] checkpoint --- content/rnd/ulid_benchmarks/index.md | 235 +++++++++++++++++++++++++++ 1 file changed, 235 insertions(+) create mode 100644 content/rnd/ulid_benchmarks/index.md diff --git a/content/rnd/ulid_benchmarks/index.md b/content/rnd/ulid_benchmarks/index.md new file mode 100644 index 0000000..5f9f947 --- /dev/null +++ b/content/rnd/ulid_benchmarks/index.md @@ -0,0 +1,235 @@ ++++ +title = "A One-Part Serialized Mystery, Part 2: The Benchmarks" +slug = "one-part-serialized-mystery-part-2" +date = "2023-07-09" +[taxonomies] +tags = ["software", "rnd", "proclamation", "upscm", "rust", "macros"] ++++ + +# A one-part serial mystery post-hoc prequel + +I [wrote recently](/rnd/one-part-serialized-mystery) about switching the types of the primary keys in +the database for an [in-progress web app](https://gitlab.com/nebkor/ww) I'm building. At that time, +I'd not yet done any benchmarking, but had reason to believe that using [sortable primary +keys](https://github.com/ulid/spec) would yield some possibly-significant gains in performance, in +both time and space. I'd also read accounts of regret that databases had not used ULIDs (instead of +[UUIDs](https://en.wikipedia.org/wiki/Universally_unique_identifier#Version_4_(random))) from the +get-go, so I decided it couldn't hurt to switch to them before I had any actual data in my DB. + +And that was correct: it didn't hurt performance, but it also didn't help much either. I've spent a +bunch of time now doing comparative benchmarks between ULIDs and UUIDs, and as I explain below, the +anticipated space savings did not materialize, and the speed-up is merely augmenting what was +already more than fast enough into slightly more faster than that. Of course, of course, and as +always, the real treasure was the friends we made along the way etc., etc. So come along on a brief +journey of discovery! + +# Bottom Line Up Front + +With sqlite and my final table schema, the size difference and speed differences are negligible, + +TODO MOR STUFFF + + +However, with my initial database layout and import code, ULIDs resulted in about 5% less space and +took only about 2/3rds as much time as when using UUIDs (5.7 vs 9.8 seconds). The same space and +time results held whether or not [`without rowid`](https://www.sqlite.org/withoutrowid.html) was +specified on table creation, which was counter to expectation, though I now understand why; I'll +explain at the end. + +# It's a setup + +My benchmark is pretty simple: starting from an empty database, do the following things: + + 1. insert 10,000 randomly chosen movies (title and year of release, from between 1965 and 2023) into + the database + 1. create 1,000 random users[^random-users] + 1. for each user, randomly select around 100 movies from the 10,000 available and put them on their list of + things to watch + +Only that last part is significant, and is where I got my timing information from. + +The table that keeps track of what users want to watch was defined[^not-final-form] like this: + +``` sql +create table if not exists witch_watch ( + id blob not null primary key, + witch blob not null, -- "user" + watch blob not null, -- "thing to watch" + [...] + foreign key (witch) references witches (id) on delete cascade on update no action, + foreign key (watch) references watches (id) on delete cascade on update no action +); +[...] +create index if not exists ww_witch_dex on witch_watch (witch); +create index if not exists ww_watch_dex on witch_watch (watch); +``` + +The kind of queries I'm trying to optimize with those indices is "what movies does a certain user +want to watch?" and "what users want to watch a certain movie?". The IDs are 16-byte blobs; an +entire row in the table is less than 100 bytes. + +## A digression on SQLite and performance + +I've mentioned once or twice before that I'm using [SQLite](https://www.sqlite.org/index.html) for +this project. Any time I need a database, my first reach is for SQLite: + + * the database is a single file, along with a couple temp files that live alongside it, simplifying + management + * there's no network involved between the client and the database; a connection to the database is + a pointer to an object that lives in the same process as the host program; this means that read + queries return data back in just a [few + *microseconds*](https://www.youtube.com/watch?v=qPfAQY_RahA) + * it scales vertically extremely well; it can handle database sizes of many terabytes + * it's one of the most widely-installed pieces of software in the world; there's at least one + sqlite database on every smartphone, and there's a robust ecosystem of [useful + extensions](https://litestream.io/) and other bits of complimentary code freely available + +And, it's extremely performant. When using the [WAL journal mode](https://www.sqlite.org/wal.html) +and the [recommended durability setting](https://www.sqlite.org/pragma.html#pragma_synchronous) for +WAL mode, along with all other production-appropriate settings, I got almost 20,000 *writes* per +second[^nothing is that slow]. There were multiple concurrent writers, and each write was a transaction that inserted about +100 rows at a time. I had [retry +logic](https://gitlab.com/nebkor/ww/-/blob/4c44aa12b081c777c82192755ac85d1fe0f5bdca/src/bin/import_users.rs#L143-145) +in case a transaction failed due to the DB being locked by another writer, but that never happened: +each write was just too fast. + +# Over-indexing on sortability + +The reason I had hoped that ULIDs would help with keeping the sizes of the indexes down was the +possibility of using [clustered +indexes](https://www.sqlite.org/withoutrowid.html#benefits_of_without_rowid_tables). To paraphrase +that link: + +> In an ordinary SQLite table, the PRIMARY KEY is really just a UNIQUE index. The key used to look +> up records on disk is the rowid. [...]any other kind of PRIMARY KEYs, including "INT PRIMARY KEY" +> are just unique indexes in an ordinary rowid table. +> +> ... +> +> Consider querying this table to find the number of occurrences of the word "xsync".: +> SELECT cnt FROM wordcount WHERE word='xsync'; +> +> This query first has to search the index B-Tree looking for any entry that contains the matching +> value for "word". When an entry is found in the index, the rowid is extracted and used to search +> the main table. Then the "cnt" value is read out of the main table and returned. Hence, two +> separate binary searches are required to fulfill the request. +> +> A WITHOUT ROWID table uses a different data design for the equivalent table. [in those tables], +> there is only a single B-Tree... Because there is only a single B-Tree, the text of the "word" +> column is only stored once in the database. Furthermore, querying the "cnt" value for a specific +> "word" only involves a single binary search into the main B-Tree, since the "cnt" value can be +> retrieved directly from the record found by that first search and without the need to do a second +> binary search on the rowid. +> +> Thus, in some cases, a WITHOUT ROWID table can use about half the amount of disk space and can +> operate nearly twice as fast. Of course, in a real-world schema, there will typically be secondary +> indices and/or UNIQUE constraints, and the situation is more complicated. But even then, there can +> often be space and performance advantages to using WITHOUT ROWID on tables that have non-integer +> or composite PRIMARY KEYs. + +
sorry what was that about secondary indices i didn't quite catch that
+ +HALF the disk space *and* TWICE as fast?? Yes, sign me up, please! + +## Sorry, the best I can do is all the disk space + +There are some [guidelines](https://www.sqlite.org/withoutrowid.html#when_to_use_without_rowid) +about when to use `without rowid`: + +> The WITHOUT ROWID optimization is likely to be helpful for tables that have non-integer or +> composite (multi-column) PRIMARY KEYs and that do not store large strings or BLOBs. +> +> [...] +> +> WITHOUT ROWID tables work best when individual rows are not too large. A good rule-of-thumb is +> that the average size of a single row in a WITHOUT ROWID table should be less than about 1/20th +> the size of a database page. That means that rows should not contain more than ... about 200 bytes +> each for 4KiB page size. + +As I mentioned, each row in that table was less than 100 bytes, so comfortably within the given +heuristic. In order to test this out, all I had to do was change the table creation statement to: + +``` sql +create table if not exists witch_watch ( + id blob not null primary key, + witch blob not null, -- "user" + watch blob not null, -- "thing to watch" + [...] + foreign key (witch) references witches (id) on delete cascade on update no action, + foreign key (watch) references watches (id) on delete cascade on update no action +) without rowid; +``` + +So I did. + +Imagine my surprise when it took nearly 20% longer to run, and the total size on disk was nearly 5% +larger. Using random UUIDs was even slower, so there's still a relative speed win for ULIDs, but it +was still an overall loss to go without the rowid. Maybe it was time to think outside the box? + +## Schema pruning + +I had several goals with this whole benchmarking endeavor. One, of course, was to get data on ULIDs +vs. UUIDs in terms of performance, at the very least so that I could write about when I publicly +said I would. But another, and actually-more-important goal, was to optimize the design of my +database and software, especially as it came to size on disk (my most-potentially-scare computing +resource; network and CPU are not problems until you get *very* large, and you would have long ago +bottlenecked on disk size if you weren't careful). + +So it was Cool and Fine to take advantage of the new capabilities that ULIDs offered if those new +capabilities resulted in better resource use. Every table in my original, UUID-based schema had had +a `created_at` column, stored as a 64-bit signed offset from the [UNIX +epoch](https://en.wikipedia.org/wiki/Unix_time). Because ULIDs encode their creation time, I could +remove that column from every table that used ULIDs as their primary key. Doing so dropped the +overall DB size by 5-10% compared to UUID-based tables with a `created_at` column. + +But I also realized that for the `watch_quests` table, no explicit + +# At last, I've reached my final form + +In the course of writing this post, I had a minor epiphany, which is that the reason for the +regressed performance when using `without rowid` was that the secondary indices needed to point to +the entries in the table, using the primary key of the table as the target. So when there was a ULID +or UUID primary key, the indexes looked like, eg, this: + +``` text +16-byte blob -> 16-byte blob +``` +
left side is, eg, user id, and right side is id of a row in the quests table
+ + + +using implicit rowid with ULIDs: + +``` text +*** Indices of table WATCH_QUESTS ********************************************* + +Percentage of total database...................... 43.3% +Number of entries................................. 199296 +Average fanout.................................... 106.00 +``` + +``` text +$ cargo run --release --bin import_users -- -d ~/movies.db -u 2000 -m 200 +[...] +Added 398119 quests in 20.818506 seconds +``` +
20k writes/second, baby
+ +size on disk is 75% of previous size (13M vs 17M) + + + +---- + +[^random-users]: I did the classic "open `/usr/share/dict/words` and randomly select a couple things + to stick together" method of username generation, which results in gems like + "Hershey_motivations84" and "italicizes_creaminesss54". This is old-skool generative AI. + +[^not-final-form]: The original schema was defined some time ago, and it took me a while to get to + the point where I was actually writing code that used it. In the course of doing the benchmarks, + and even in the course of writing this post, I've made changes in response to things I learned + from the benchmarks and to things I realized by thinking more about it and reading more docs. + +[^nothing is that slow]: old job python 100 reqs/sec fall down + +[an_image]: /images/programmers_creed.jpg "some kinda image idunno"