ml.utils.io

Defines I/O utility functions.

This module contains a global read lock that can be used to prevent multiple processes from doing disk reads at the same time. This is useful when reading from a shared filesystem, such as NFS, where concurrent reads can cause performance issues.

from ml.utils.io import read_lock, set_read_lock

set_read_lock(lock)

with read_lock:
    # Do some disk reads.

ml.utils.io.set_read_lock(lock: Lock) → None

ml.utils.io.prefetch_samples(iterable: Iterator[T], pre_load_n: int) → Iterator[T]