WaymoPerceptionDatasetConverter

class standard_e2e.caching.src_datasets.waymo_perception.WaymoPerceptionDatasetConverter(*args, **kwargs)

Bases: TFRecSourceDatasetConverter

Converter for the Waymo Perception dataset.

Pre-populates the processor’s per-segment HD-map cache at __init__ time. Waymo’s proto puts map_features only on the first frame of each segment, and per-instance state on the processor is pickled independently to every multiprocessing worker – so without the pre-scan, workers that don’t happen to receive frame 0 would see an empty cache. The pre-scan reads only the first record of each tfrecord (~10 ms per file) and runs in the parent process, so the cache is part of the pickled state every worker inherits. Cost on the training split (~800 segments) is ~8 s.

convert()

Convert all frames then run any configured context aggregators.

Return type:

None

property dataset_name: str

Return the name of the dataset.

classmethod get_arg_parser()

Return an argument parser for the converter.

property max_workers: int | None

Optional cap on parallel-pool size; None means no cap.

Used by datasets where pool throughput plateaus or regresses past a certain worker count – typically because the processor carries large state (e.g. a prescanned HD-map cache) and Pool’s per-task dispatch overhead grows with worker count. Subclasses whose processors are small can leave this at None.

property multiprocessing_start_method: str

Start method for the worker pool.

Default "spawn" is the conservative choice: TensorFlow and OpenCV both keep global thread / mutex state that fork() inherits in a deadlock-prone way (typically before the first frame completes). Spawn pays a per-worker import cost (~5 s per worker, dominated by TensorFlow) but is the safe pattern for any worker that may run TF or cv2 work post-fork.

Subclasses whose worker hot path is fully TF-free (no tf.io.decode_image, no frame_utils.* calls, etc.) may override to "fork" to avoid the spawn import tax. This is a very large speedup on small / DEBUG runs and a meaningful one on full splits.