uuid's fast-rng feature doesn't mesh with python's frequent use of fork

[reneegyllensvaan]

We've been debugging an issue today after introducing uuid-utils into our backend workers, which use Celery. Celery workers use fork(2) (at least in some configurations) to spin up worker processes.

The rand crate, which rust's uuid crate uses if you pass the fast-rng crate, doesn't seem to reseed the RNG on a fork. In Rust land, I suppose this normally isn't an issue because it seems pretty rare to use fork. In python, it's a lot more common, causing a situation where the rng isn't reseeded when forking out celery workers.

Enjoy some repros:

fn main() {
    // first, just make sure the rng is initialized in the parent process
    let _ = uuid::Uuid::new_v4();
    let tid = unsafe { libc::fork() };
    println!("{:?} from {tid}", uuid::Uuid::new_v4());
}

import sys
import uuid_utils
sys.modules["uuid"] = uuid_utils
import os

import uuid

a = uuid.uuid4()
print(f"a = {a}")
os.fork()
b = uuid.uuid4()
print(f"b = {b}")

I suppose this is technically not a bug, I'm just interested in your take on this. Ultimately it's kind of a culture issue that python forks, but as of now it's a tradeoff between the "just works"-ness of this as a library and generating uuids very quickly.

Versions

• uuid_utils 0.6.1
• cpython 3.12.0
• macos and linux

[aminalaee]

This looks very interesting, I was thinking maybe the uuid-rs crate should make use of ReseedingRng to make this work.
But apparently ThreadRng is using the ReseedingRng anyway.
From the docs:

When a process is forked on UNIX, the RNGs in both the parent and child processes will be reseeded just before the next call to BlockRngCore::generate, i.e. “soon”. For ChaCha and Hc128 this is a maximum of fifteen u32 values before reseeding.

[aminalaee]

rust-random/rand#1362

[reneegyllensvaan]

Oh yeah, that does seem to be the case - three uuids and then they diverge

import uuid_utils
import os

uuid_utils.uuid4()
os.fork()
for ix in range(5):
    b = uuid_utils.uuid4()
    print(f"{ix} = {b}")

$ python fork-test.py | sort | uniq
0 = a7718fa1-25f8-4596-a1e3-428ae6e1b910
1 = 9e9c7c74-4760-4bef-ac7c-3b8d73d5c26b
2 = b515f7c3-62b3-444e-bbd3-27b990cc61d9
3 = 7f20ef05-efd4-4eee-8842-45b7be31c014
3 = a88f6a90-9c1f-497c-a176-9c5db085525f
4 = 48715820-019a-4e87-b52d-99b1e7bd6fa1
4 = fe03e851-444e-420f-858d-6baed7ee91bf

strace confirms as much

$ PYTHONUNBUFFERED=1 strace python foo.py
...
write(1, "0 = 24260054-0e3e-42b7-9eac-cd91"..., 400 = 24260054-0e3e-42b7-9eac-cd91941a7d36) = 40
write(1, "\n", 1
)                       = 1
write(1, "1 = f4be0a56-62d5-447c-8c07-ed82"..., 401 = f4be0a56-62d5-447c-8c07-ed8250d5ba67) = 40
write(1, "\n", 1
)                       = 1
write(1, "2 = 94a2ef10-96e0-4e7b-a869-a943"..., 402 = 94a2ef10-96e0-4e7b-a869-a943806bf718) = 40
write(1, "\n", 1
)                       = 1
getrandom("\xe2\x9d\x97\x7f\xe7\x81\xf7\xa8\x5e\xc5\x1a\xf6\xa7\xe4\x23\xa2\x40\xae\xc2\xc3\x63\x2f\x72\xba\x0e\x0f\xad\xf2\xd5\x4e\x3b\x73", 32, 0) = 32
write(1, "3 = 65e9c130-d62d-4c20-aad4-9cc2"..., 403 = 65e9c130-d62d-4c20-aad4-9cc25e562d60) = 40
write(1, "\n", 1
)                       = 1
write(1, "4 = 89ed5b5a-ff8a-45a0-8ee6-fb67"..., 404 = 89ed5b5a-ff8a-45a0-8ee6-fb673a1e8288) = 40
write(1, "\n", 1
)                       = 1

[dhardy]

FYI rand's ReseedingRng has never advertised that it reseeds immediately on fork, only that it will reseed after some delay (clearing the current buffer). In the next version of rand even that delayed-reseed-on-fork will be removed (only the slow period reseeding remaining), but a helper function will be available to make explicit immediate reseeding easier: https://github.com/rust-random/rand/blob/master/src/rngs/thread.rs#L54