SourceDatasetProcessor

class standard_e2e.caching.source_dataset_processor.SourceDatasetProcessor(common_output_path, split, index_data_generator=None, adapters=None, context_aggregators=None)

Bases: ABC

Abstract base class for processing source datasets.

Parameters:

• common_output_path (str)

• split (str)

• index_data_generator (IndexDataGenerator | None)

• adapters (list[AbstractAdapter] | None)

• context_aggregators (list[SegmentContextAggregator] | None)

property allowed_splits: list[str]

Return the list of allowed splits for the dataset.

property context_aggregators

abstract property dataset_name: str

Return the name of the dataset.

property inner_path: str

Return the inner path relative to the common output path.

needs_attr(attr)

Whether at least one registered adapter reads this StandardFrameData field. Used by per-dataset processors to skip expensive modality builds (cameras, lidar, hd_map, detections, …) when no adapter would consume them. True when attr is in the consumed-attrs union, plus a hard-coded special case: the identifier / index fields are always treated as needed since they are required for the cache + index regardless of adapter chain.

Return type:

bool

Parameters:

attr (StandardFrameDataField)

property output_path: str

Return the output path for the processed dataset.

final process_frame(raw_frame_data)

Return type:

tuple[TransformedFrameData, FrameIndexData]

Parameters:

raw_frame_data (Any)

final process_frame_and_save_data(raw_frame_data)

Process a single frame of raw data, save the processed frame data to disk, and return the corresponding FrameIndexData.

Return type:

FrameIndexData

Parameters:

raw_frame_data (Any)

property specific_output_path: str

Return the specific output path for the dataset.

property split: str

Return the dataset split.