NavsimDatasetProcessor

class standard_e2e.caching.src_datasets.navsim.NavsimDatasetProcessor(common_output_path, split, index_data_generator=None, adapters=None, context_aggregators=None, maps_root_path=None)

Bases: SourceDatasetProcessor

Processor for NAVSIM (OpenScene-v1.1 trainval).

A “frame” is one ego timestamp inside a scene pickle. Per-log state (the entire pickle) is loaded once and reused for every frame in that log; the cache key is the pickle path, so workers that hop between logs only reload at log boundaries.

HD-map handling: nuPlan map.gpkg files for the 4 OpenScene cities are loaded lazily via get_maps_api and cached per map_location string (so each worker holds at most 4 NuPlanMap instances). The maps root is resolved with the same precedence NAVSIM users expect:

1. explicit maps_root_path constructor arg;

2. NUPLAN_MAPS_ROOT environment variable (NAVSIM’s own convention, documented at navsim/docs/install.md);

3. <input_path>/maps derived from the converter’s input_path at frame-prep time (matches OpenScene-v1.1’s on-disk layout).

If none of the three resolves to an existing directory, HD-map extraction is skipped and a warning is logged once.

Parameters:

• common_output_path (str)

• split (str)

• index_data_generator (IndexDataGenerator | None)

• adapters (list[AbstractAdapter] | None)

• context_aggregators (list[SegmentContextAggregator] | None)

• maps_root_path (Optional[str])

DATASET_NAME = 'navsim'

property allowed_splits: list[str]

Return the list of allowed splits for the dataset.

property context_aggregators

property dataset_name: str

Return the name of the dataset.

property inner_path: str

Return the inner path relative to the common output path.

needs_attr(attr)

Whether at least one registered adapter reads this StandardFrameData field. Used by per-dataset processors to skip expensive modality builds (cameras, lidar, hd_map, detections, …) when no adapter would consume them. True when attr is in the consumed-attrs union, plus a hard-coded special case: the identifier / index fields are always treated as needed since they are required for the cache + index regardless of adapter chain.

Return type:

bool

Parameters:

attr (StandardFrameDataField)

property output_path: str

Return the output path for the processed dataset.

process_frame(raw_frame_data)

Return type:

tuple[TransformedFrameData, FrameIndexData]

Parameters:

raw_frame_data (Any)

process_frame_and_save_data(raw_frame_data)

Process a single frame of raw data, save the processed frame data to disk, and return the corresponding FrameIndexData.

Return type:

FrameIndexData

Parameters:

raw_frame_data (Any)

property specific_output_path: str

Return the specific output path for the dataset.

property split: str

Return the dataset split.