ml.tasks.datasets.error_handling

Defines error handling wrappers for datasets.

The worst feeling in the world is when you’re training a model and it crashes after 10 hours of training. This module defines some error handling wrappers for datasets which will catch errors and log them (in batches).

ml.tasks.datasets.error_handling.get_loc(num_excs: int = 1) → str

class ml.tasks.datasets.error_handling.ErrorHandlingConfig(enabled: bool = True, maximum_exceptions: int = 10, backoff_after: int = 5, sleep_backoff: float = 0.1, sleep_backoff_power: float = 2.0, log_full_exception: bool = False, flush_exception_summary_every: int = 500, report_top_n_exception_types: int = 5, exception_location_traceback_depth: int = 3)

Bases: object

enabled: bool = True

maximum_exceptions: int = 10

backoff_after: int = 5

sleep_backoff: float = 0.1

sleep_backoff_power: float = 2.0

log_full_exception: bool = False

flush_exception_summary_every: int = 500

report_top_n_exception_types: int = 5

exception_location_traceback_depth: int = 3

class ml.tasks.datasets.error_handling.ExceptionSummary(config: ErrorHandlingConfig)

Bases: object

add_exception(exc: Exception, loc: str) → None

step() → None

summary() → str

flush() → None

class ml.tasks.datasets.error_handling.ErrorHandlingDataset(dataset: Dataset[BatchT], config: ErrorHandlingConfig)

Bases: Dataset[BatchT]

Defines a wrapper for safely handling errors.

class ml.tasks.datasets.error_handling.ErrorHandlingMapDataPipe(datapipe: MapDataPipe[BatchT], config: ErrorHandlingConfig)

Bases: ErrorHandlingDataset[BatchT], MapDataPipe[BatchT]

Defines a wrapper for safely handling errors.

class ml.tasks.datasets.error_handling.ErrorHandlingIterableDataset(dataset: IterableDataset[BatchT], config: ErrorHandlingConfig)

Bases: IterableDataset[BatchT]

Defines a wrapper for safely handling errors in iterable datasets.

class ml.tasks.datasets.error_handling.ErrorHandlingIterDataPipe(datapipe: IterDataPipe[BatchT], config: ErrorHandlingConfig)

Bases: ErrorHandlingIterableDataset[BatchT], IterDataPipe[BatchT]

Defines a wrapper for safely handling errors in iterable datapipe.

ml.tasks.datasets.error_handling.error_handling_dataset(dataset: DatasetT, config: ErrorHandlingConfig) → DatasetT

Returns a dataset which wraps the base dataset and handles errors.

Parameters:

• dataset – The dataset to handle errors for

• config – An associated config, describing which errors to handle

Returns:

The wrapped dataset, which catches some errors

Raises:

NotImplementedError – If the dataset type is not supported

ml.tasks.datasets.error_handling.test_exception_summary() → None