SourceDatasetConverter

class standard_e2e.caching.source_dataset_converter.SourceDatasetConverter(source_processor, input_path, split, num_workers=0, do_parallel_processing=True, arguments=None)

Bases: ABC

Base class orchestrating conversion from raw datasets to processed frames.

Parameters:

• source_processor (SourceDatasetProcessor)

• input_path (str)

• split (str)

• num_workers (int)

• do_parallel_processing (bool)

• arguments (Namespace | dict | None)

final convert()

Convert all frames then run any configured context aggregators.

Return type:

None

property dataset_name: str

Return the name of the dataset.

classmethod get_arg_parser()

Return an argument parser for the converter.

property max_workers: int | None

Optional cap on parallel-pool size; None means no cap.

Used by datasets where pool throughput plateaus or regresses past a certain worker count – typically because the processor carries large state (e.g. a prescanned HD-map cache) and Pool’s per-task dispatch overhead grows with worker count. Subclasses whose processors are small can leave this at None.

property multiprocessing_start_method: str

Start method for the worker pool.

Default "spawn" is the conservative choice: TensorFlow and OpenCV both keep global thread / mutex state that fork() inherits in a deadlock-prone way (typically before the first frame completes). Spawn pays a per-worker import cost (~5 s per worker, dominated by TensorFlow) but is the safe pattern for any worker that may run TF or cv2 work post-fork.

Subclasses whose worker hot path is fully TF-free (no tf.io.decode_image, no frame_utils.* calls, etc.) may override to "fork" to avoid the spawn import tax. This is a very large speedup on small / DEBUG runs and a meaningful one on full splits.