ml.utils.logging

Logging utilities.

This extends the basic Python logger to log across all ranks, and to colorize the logs to make them easier to parse.

class ml.utils.logging.RankFilter(*, rank: int | None = None)

Bases: Filter

Logging filter which filters out INFO logs on non-zero ranks.

Parameters:

rank – The current rank

filter(record: LogRecord) → bool

Determine if the specified record is to be logged.

Returns True if the record should be logged, or False otherwise. If deemed appropriate, the record may be modified in-place.

class ml.utils.logging.ColoredFormatter(*, prefix: str | None = None, rank: int | None = None, world_size: int | None = None, use_color: bool = True)

Bases: Formatter

Defines a custom formatter for displaying logs.

Initialize the formatter with specified format strings.

Initialize the formatter either with the specified format string, or a default as described above. Allow for specialized date formatting with the optional datefmt argument. If datefmt is omitted, you get an ISO8601-like (or RFC 3339-like) format.

Use a style parameter of ‘%’, ‘{’ or ‘$’ to specify that you want to use one of %-formatting, str.format() ({}) formatting or string.Template formatting in your format string.

Changed in version 3.2: Added the style parameter.

RESET_SEQ = '\x1b[0m'

COLOR_SEQ = '\x1b[1;%dm'

BOLD_SEQ = '\x1b[1m'

COLORS: dict[str, Literal['black', 'red', 'green', 'yellow', 'blue', 'magenta', 'cyan', 'white', 'grey', 'light-red', 'light-green', 'light-yellow', 'light-blue', 'light-magenta', 'light-cyan']] = {'CRITICAL': 'yellow', 'DEBUG': 'grey', 'DEBUGALL': 'grey', 'ERROR': 'red', 'FATAL': 'red', 'INFO': 'cyan', 'INFOALL': 'magenta', 'WARNING': 'yellow'}

format(record: LogRecord) → str

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

class ml.utils.logging.TqdmHandler(level=0)

Bases: Handler

Initializes the instance - basically setting the formatter to None and the filter list to empty.

emit(record: LogRecord) → None

Do whatever it takes to actually log the specified logging record.

This version is intended to be implemented by subclasses and so raises a NotImplementedError.

ml.utils.logging.configure_logging(*, prefix: str | None = None, rank: int | None = None, world_size: int | None = None, use_tqdm: bool = True) → None

Instantiates print logging, to either stdout or tqdm.

Parameters:

• prefix – An optional prefix to add to the logger

• rank – The current rank, or None if not using multiprocessing

• world_size – The total world size, or None if not using multiprocessing

• use_tqdm – Write using TQDM instead of sys.stdout

class ml.utils.logging.IntervalTicker(interval: float)

Bases: object

tick() → bool