287 lines
13 KiB
Markdown
287 lines
13 KiB
Markdown
+++
|
|
title = "Presenting Julids: another fine sundry, by Nebcorp Heavy Industries and Sundries"
|
|
slug = "presenting-julids"
|
|
date = "2023-07-29"
|
|
[taxonomies]
|
|
tags = ["software", "sundry", "proclamation", "sqlite", "rust", "ulid", "julid"]
|
|
+++
|
|
|
|
# Presenting Julids
|
|
Nebcorp Heavy Industries and Sundries, long the world leader in sundries, is proud to announce the
|
|
public launch of the official identifier type for all Nebcorp companies' assets and database
|
|
entries, [Julids](https://gitlab.com/nebkor/julid). Julids are globally unique sortable identifiers,
|
|
backwards-compatible with [ULIDs](https://github.com/ulid/spec), but better.
|
|
|
|
Inside your Rust program, simply add `julid-rs` to your project's `Cargo.toml` file, and use it
|
|
like:
|
|
|
|
``` rust
|
|
use julid::Julid;
|
|
|
|
fn main() {
|
|
let id = Julid::new();
|
|
dbg!(id.created_at(), id.as_string());
|
|
}
|
|
```
|
|
|
|
Such a program would output something like:
|
|
|
|
``` text
|
|
[main.rs:5] id.created_at() = 2023-07-29T20:21:50.009Z
|
|
[main.rs:5] id.as_string() = "01H6HN10SS00020YT344XMGA3C"
|
|
```
|
|
|
|
However, it can also be built as a [loadable extension](https://www.sqlite.org/loadext.html) for
|
|
SQLite, adding database functions for creating and querying Julids:
|
|
|
|
``` text
|
|
$ sqlite3
|
|
SQLite version 3.40.1 2022-12-28 14:03:47
|
|
Enter ".help" for usage hints.
|
|
Connected to a transient in-memory database.
|
|
Use ".open FILENAME" to reopen on a persistent database.
|
|
sqlite> .load ./libjulid
|
|
sqlite> select hex(julid_new());
|
|
018998768ACF000060B31DB175E0C5F9
|
|
sqlite> select julid_string(julid_new());
|
|
01H6C7D9CT00009TF3EXXJHX4Y
|
|
sqlite> select julid_seconds(julid_new());
|
|
1690480066.208
|
|
sqlite> select datetime(julid_timestamp(julid_new()), 'auto');
|
|
2023-07-27 17:47:50
|
|
sqlite> select julid_counter(julid_new());
|
|
0
|
|
```
|
|
|
|
## Julids vs ULIDs
|
|
|
|
Julids are a drop-in replacement for ULIDs; all Julids are valid ULIDs, but not all ULIDs are valid Julids.
|
|
|
|
Given their compatibility relationship, Julids and ULIDs must have quite a bit in common, and indeed
|
|
they do:
|
|
|
|
* they are 128-bits long
|
|
* they are lexicographically sortable
|
|
* they encode their creation time as the number of milliseconds since the [UNIX
|
|
epoch](https://en.wikipedia.org/wiki/Unix_time)
|
|
* their string representation is a 26-character [base-32
|
|
Crockford](https://en.wikipedia.org/wiki/Base32) encoding of their big-endian bytes
|
|
* IDs created within the same millisecond are still meant to sort in their order of creation
|
|
|
|
Julids and ULIDs have different ways to implement that last piece. If you look at the layout of bits
|
|
in a ULID, you see:
|
|
|
|
|
|
|
|
According to the ULID spec, for ULIDs created within the same millisecond, the least-significant bit
|
|
should be incremented for each new ID. Since that portion of the ULID is random, that means you may
|
|
not be able to increment it without spilling into the timestamp portion. Likewise, it's easy to
|
|
guess a new possibly-valid ULID simply by incrementing an already-known one. And finally, this means
|
|
that sorting will need to read all the way to the end of the ULID for IDs created in the same
|
|
millisecond.
|
|
|
|
To address these shortcomings, Julids (Joe's ULIDs) have the following structure:
|
|
|
|
|
|
|
|
As with ULIDs, the 48 most-significant bits encode the time of creation. Unlike ULIDs, the next 16
|
|
most-significant bits are not random: they're a monotonic counter for IDs created within the same
|
|
millisecond[^monotonic]. Since it's only 16 bits, it will saturate after 65,536 IDs
|
|
intra-millisecond creations, after which, IDs in that same millisecond will not have an intrinsic
|
|
total order (the random bits will still be different, so you shouldn't have collisions). My PC,
|
|
which is no slouch, can only generate about 20,000 per millisecond, so hopefully this is not an
|
|
issue! Because the random bits are always fresh, it's not possible to easily guess a valid Julid if
|
|
you already have one.
|
|
|
|
# How to use
|
|
|
|
As noted, the Julid crate can be used in two different ways: as a regular Rust library, declared
|
|
in your Rust project's `Cargo.toml` file (say, by running `cargo add julid-rs`), and used as shown
|
|
above. There's a rudimentary
|
|
[benchmark](https://gitlab.com/nebkor/julid/-/blob/main/examples/benchmark.rs) example in the repo,
|
|
which I'll talk more about below. But the primary use case for me was as a loadable SQLite
|
|
extension, as I [previously
|
|
wrote](/rnd/one-part-serialized-mystery-part-2/#next-steps-with-ids). Both are covered in the
|
|
[documentation](https://docs.rs/julid-rs/latest/julid/), but let's go over them here, starting with
|
|
the extension.
|
|
|
|
## Inside SQLite as a loadable extension
|
|
|
|
The extension, when loaded into SQLite, provides the following functions:
|
|
|
|
* `julid_new()`: create a new Julid and return it as a 16-byte
|
|
[blob](https://www.sqlite.org/datatype3.html#storage_classes_and_datatypes)
|
|
* `julid_seconds(julid)`: get the number seconds (as a 64-bit float) since the UNIX epoch that this
|
|
julid was created
|
|
* `julid_counter(julid)`: show the value of this julid's monotonic counter
|
|
* `julid_sortable(julid)`: return the 64-bit concatenation of the timestamp and counter
|
|
* `julid_string(julid)`: show the [base-32 Crockford](https://en.wikipedia.org/wiki/Base32)
|
|
encoding of this julid; the raw bytes won't be valid UTF-8, so use this or the built-in `hex()`
|
|
function to `select` a human-readable representation
|
|
|
|
### Building and loading
|
|
|
|
If you want to use it as a SQLite extension:
|
|
|
|
* clone the [repo](https://gitlab.com/nebkor/julid)
|
|
* build it with `cargo build --features plugin` (this builds the SQLite extension)
|
|
* copy the resulting `libjulid.[so|dylib|whatevs]` to some place where you can...
|
|
* load it into SQLite with `.load /path/to/libjulid` as shown at the top
|
|
* party
|
|
|
|
If you, like me, wish to use Julids as primary keys, just create your table like:
|
|
|
|
``` sql
|
|
create table users (
|
|
id blob not null primary key default (julid_new()),
|
|
...
|
|
);
|
|
```
|
|
|
|
and you've got a first-class ticket straight to Julid City, baby!
|
|
|
|
For a table created like:
|
|
|
|
``` sql
|
|
-- table of things to watch
|
|
create table if not exists watches (
|
|
id blob not null primary key default (julid_new()),
|
|
kind int not null, -- enum for movie or tv show or whatev
|
|
title text not null,
|
|
metadata_url text, -- possible url for imdb or other metadata-esque site to show the user
|
|
length int,
|
|
release_date int,
|
|
added_by blob not null, -- ID of the user that added it
|
|
last_updated int not null default (unixepoch()),
|
|
foreign key (added_by) references users (id)
|
|
);
|
|
```
|
|
|
|
and then [some
|
|
code](https://gitlab.com/nebkor/ww/-/blob/cc14c30fcfbd6cdaecd85d0ba629154d098b4be9/src/import_utils.rs#L92-126)
|
|
that inserted rows into that table like
|
|
|
|
``` sql
|
|
insert into watches (kind, title, length, release_date, added_by) values (?,?,?,?,?)
|
|
```
|
|
|
|
where the wildcards get bound in a loop with unique values and the Julid `id` field is
|
|
generated by the extension for each row, I get over 100,000 insertions/second.
|
|
|
|
## Inside a Rust program
|
|
|
|
Of course, you can also use it outside of a database; the `Julid` type is publicly exported. There's
|
|
a simple benchmark in the examples folder of the repo, the important parts of which look like:
|
|
|
|
``` rust
|
|
use julid::Julid;
|
|
|
|
fn main() {
|
|
[....]
|
|
let start = Instant::now();
|
|
for _ in 0..num {
|
|
v.push(Julid::new());
|
|
}
|
|
let end = Instant::now();
|
|
let dur = (end - start).as_micros();
|
|
|
|
for id in v.iter() {
|
|
eprintln!(
|
|
"{id}: created_at {}; counter: {}; sortable: {}",
|
|
id.created_at(),
|
|
id.counter(),
|
|
id.sortable()
|
|
);
|
|
}
|
|
println!("{num} Julids generated in {dur}us");
|
|
```
|
|
|
|
If you were to run it on a computer like mine[^my computer], you might see something like this:
|
|
|
|
``` text
|
|
$ cargo run --example=benchmark --release -- -n 30000 2> /dev/null
|
|
30000 Julids generated in 1240us
|
|
```
|
|
|
|
That's about 24,000 IDs/millisecond; 24 *MILLION* per second!
|
|
|
|
The default optional Cargo features include implementations of traits for getting Julids into and
|
|
out of SQLite via [SQLx](https://github.com/launchbadge/sqlx), and for generally
|
|
serializing/deserializing with [Serde](https://serde.rs/), via the `sqlx` and `serde` features,
|
|
respectively. One final default optional feature, `chrono`, uses the Chrono crate to return the
|
|
timestamp as a [`DateTime`](https://docs.rs/chrono/latest/chrono/struct.DateTime.html) by adding a
|
|
`created_at(&self)` method to `Julid`.
|
|
|
|
Something to note: don't enable the `plugin` feature in your Cargo.toml if you're using this crate
|
|
inside your Rust application, *especially* if you're also loading it as an extension in SQLite in
|
|
your application. You'll get a long and confusing runtime panic due to there being multiple
|
|
entrypoints defined with the same name.
|
|
|
|
# Why Julids?
|
|
|
|
The astute may have noticed that this is the third time I've written about globally unique
|
|
sortable IDs ([here is part one](/rnd/one-part-serialized-mystery), and [part two is
|
|
here](/rnd/one-part-serialized-mystery-part-2)). What's, uh... what's up with that?
|
|
|
|
![marge just thinks they're neat][marge ids]
|
|
<div class = "caption">we both just think they're neat</div>
|
|
|
|
Like Marge says, I just think they're neat! I'm not the only one; here are just some alternatives:
|
|
|
|
* Segment's [KSUID](https://segment.com/blog/a-brief-history-of-the-uuid/), released in 2017. This
|
|
was possibly my first exposure to this idea. They're 36 bits larger than UUIDs or ULIDs, but
|
|
otherwise very similar to ULIDs (and hence Julids)
|
|
* [ULIDs](https://github.com/ulid/spec), as previously discussed at length
|
|
* [UUIDv7](https://www.ietf.org/archive/id/draft-peabody-dispatch-new-uuid-format-01.html#name-uuidv7-layout-and-bit-order);
|
|
these are *very* similar to Julids; the primary difference is that the lower 62 bits are left up
|
|
to the implementation, rather than always containing pseudorandom bits as in Julids (which use
|
|
the lower 64 bits for that, instead of UUIDv7's 62)
|
|
* [Snowflake ID](https://en.wikipedia.org/wiki/Snowflake_ID), developed by Twitter in 2010; these
|
|
are 63-bit identifiers (so they fit in a signed 64-bit number), where the top 41 bits are a
|
|
millisecond timestamp, the next 10 bits are a machine identifier[^twitter machine count], and the
|
|
last 12 bits are for an intra-millisecond sequence counter (what Julid calls a "monotonic
|
|
counter"); unlike all the other IDs discussed, there are no random bits
|
|
|
|
and I'm sure the list can go on.
|
|
|
|
I wanted to use them in my SQLite-backed [web app](https://gitlab.com/nebkor/ww), in order to fix
|
|
some deficiencies in ULIDs and the way I was using them, as [I said
|
|
before](/rnd/one-part-serialized-mystery-part-2/#next-steps-with-ids):
|
|
|
|
> [...] it bothers me that ID generation is not done inside the database itself. Aside from being
|
|
> a generally bad idea, this lead to at least one frustrating debug session where I was inserting
|
|
> one ID but reporting back another. SQLite doesn't have native support for this, but it does have
|
|
> good native support for loading shared libraries as plugins in order to add functionality to it,
|
|
> and so my next step is to write one of those, and remove the ID generation logic from the
|
|
> application.
|
|
|
|
So, is this the last time I'll time I'll be writing at length about these things? It's hard to say for
|
|
sure, but signs point to "yes". I hope you've found them at least a little interesting!
|
|
|
|
# Thanks
|
|
|
|
This crate wouldn't have been possible without a lot of inspiration (and a little shameless
|
|
stealing) from the [ulid-rs](https://github.com/dylanhart/ulid-rs) crate. For the loadable
|
|
extension, the [sqlite-loadable-rs](https://github.com/asg017/sqlite-loadable-rs) crate made it
|
|
*extremely* easy to write; what I thought would take a couple days instead took a couple
|
|
hours. Thank you, authors of those crates! Feel free to steal from this project!
|
|
|
|
----
|
|
|
|
[^monotonic]: At least, they will still have a total order if they're all generated within the same
|
|
process in the same way; the code uses a [64-bit atomic
|
|
integer](https://gitlab.com/nebkor/julid/-/blob/2484d5156bde82a91dcc106410ed56ee0a5c1e07/src/julid.rs#L11-12)
|
|
to ensure that IDs generated within the same millisecond have incremented counters, but that
|
|
atomic counter is not global; calling `Julid::new()` in Rust and `select julid_new()` in SQLite
|
|
will not be aware of each others' counters. I just make sure to only generate them inside the
|
|
DB.
|
|
|
|
[^my computer]: According to the output of `lscpu`, my computer has an "AMD Ryzen 9 3900X 12-Core
|
|
Processor", running between 2.2 and 4.6 GHz. It's no slouch!
|
|
|
|
[^twitter machine count]: There are only ten bits for the machine ID, which means there are only
|
|
1,024 possible machine IDs; did twitter only have a thousand machines in production? Maybe only
|
|
a thousand at a time, so you could use the timestamp to look up what machine any given 10-bit ID
|
|
referred to?
|
|
|
|
[marge ids]: ./marge_thinks_theyre_neat.png "marge simpson holding a potato labeled 'globally unique sortable identifiers'"
|