Author: Adarsh Nanjaiya Latha
Last updated: March 2026 (community contributions welcome!)

I've spent weeks hunting for a single go-to resource covering the most widely used datasets in autonomous perception for cars and drones—especially in cooperative/multi-agent settings. Unsurprisingly, no such one-stop list existed. So here it is: a curated catalogue of key datasets.

The list is not exhaustive (new datasets drop constantly), but it covers the foundational and cutting-edge ones. Please feel free to fork this post on GitHub and add others—or comment below with suggestions!

Categorization:
By number of platforms (single vs. multi-agent) and modalities (single-sensor vs. multi-sensor fusion).
Sub-notes on platforms: V2V (ground-ground), V2U (ground-air), U2U (air-air), V2I/V2X (vehicle-infra/everything).
[Real-world] or [Simulation] flag for each.

For sensor hardware details (e.g., exact LiDAR models, camera resolutions), check the original papers or project pages—I welcome PRs if anyone wants to expand that here.

1. Single Platform, Single Modality

Standard, non-cooperative datasets from one ego vehicle or drone using only one sensor type (typically camera-only or LiDAR-only). Great for baseline semantic segmentation, tracking, or point-cloud benchmarking.

• Cityscapes (2016): Dense pixel-level urban scene understanding from a single ground vehicle. Camera-only. [Real-world] Website
• SYNTHIA (2016): Large synthetic urban scenes for semantic segmentation. Pure camera data. [Simulation]
• Stanford Drone Dataset (SDD) (2016): 2D object tracking and trajectory prediction from a single overhead drone using video. Camera-only. [Real-world] Website
• LIBRE: LiDAR Benchmarking and Reference (2020): Single ground vehicle with 10–12 different LiDAR sensors (various manufacturers/models) for evaluating point-cloud performance across conditions (static, weather, dynamic traffic). LiDAR-only. [Real-world] arXiv

Additional notable ones:

• VisDrone (2018–ongoing): Drone-mounted camera for object detection/tracking in urban scenes. Camera-only. [Real-world]
• UAVDT (2018): Drone camera dataset focused on detection/tracking under varying altitudes/angles. Camera-only. [Real-world]

2. Single Platform, Multi-Modality

One ego vehicle or drone equipped with a heterogeneous sensor suite (e.g., cameras + LiDAR + radar/thermal). The classic "full-stack" single-agent perception benchmarks.

• KITTI (2012): Foundational benchmark for autonomous driving. Single vehicle with stereo cameras + LiDAR. [Real-world] Website
• nuScenes (2019) & Waymo Open Dataset (2019): Massive-scale single-vehicle datasets with 360° coverage. Cameras + LiDAR + radar (nuScenes includes radar). [Real-world] nuScenes | Waymo
• UAVScenes (2025): Multi-modal dataset from a single drone: RGB cameras + thermal imaging + LiDAR. Supports semantic segmentation, depth estimation, localization, and novel-view synthesis. [Real-world] GitHub | arXiv

3. Multi-Platform, Single Modality

Cooperative/collaborative datasets where multiple vehicles, drones, or infrastructure nodes share data, but only one sensor medium is used. Early steps toward swarm/U2U or pure-LiDAR V2V collaboration.

• CoPerception-UAV (2022): Drone-to-drone (U2U) swarm collaboration for bird's-eye-view semantic segmentation and detection. Pure camera imagery from 5 coordinated UAVs. [Simulation] (CARLA + AirSim) Project page
• UAV3D (2024): Large-scale UAV-to-UAV (U2U) benchmark with synchronized camera imagery. 1,000 scenes across diverse towns; supports 3D perception tasks. [Simulation] (CARLA + AirSim) NeurIPS 2024 paper
• V2VLoc (2025): Built for cooperative LiDAR localization and detection in GNSS-denied environments. Focuses on shared LiDAR point clouds between vehicles (V2V). [Simulation]

Additional:

• U2UData (2024): Large-scale swarm UAV (U2U) dataset with LiDAR focus (plus some depth/RGB but primarily single-modality benchmarks available). [Simulation] arXiv

4. Multi-Platform, Multi-Modality

The cutting edge of cooperative perception: vehicles, drones, and/or smart infrastructure collaborating with fused sensor streams (cameras + LiDAR + radar). These enable V2V, V2U, U2U, and V2I/V2X research at scale.

• V2U4Real (2025): First large-scale real-world dataset explicitly for Vehicle-to-UAV (V2U) cooperative perception. Ground vehicle + drone sharing synchronized multi-view LiDAR + RGB camera streams. [Real-world] arXiv
• AGC-Drive (2025): Massive real-world aerial-ground cooperative dataset (V2V + V2U). Two ground vehicles (5 cams + 1 LiDAR each) + one UAV (forward cam + LiDAR). Covers dynamic interactions like cut-ins/lane changes. [Real-world] GitHub | NeurIPS 2025
• DAIR-V2X (2022): First massive real-world Vehicle-to-Infrastructure (V2I) dataset. Synchronized camera + LiDAR from vehicles and roadside units. [Real-world] GitHub
• TUMTraf-V2X (2024): Cooperative 3D object detection, tracking, and maneuver analysis with multi-modal data from roadside units + vehicles (V2I/V2X). [Real-world]
• OPV2V (2022): Heavy baseline for multi-vehicle (V2V) cooperative perception. Camera + LiDAR streams. [Simulation] (CARLA) Website
• V2X-Sim (2022): Foundational simulated V2X benchmark (vehicles + roadside infrastructure). Synchronized camera + LiDAR. [Simulation] (CARLA + SUMO) Project page
• V2V4Real (2023) & V2X-Real (2024): Large-scale real-world V2V / V2X datasets tracking hundreds of km. Camera + LiDAR streams. [Real-world] V2V4Real
• V2X-Radar (2025) / V2X-R (simulation counterpart): Newest branch introducing weather-robust 4D Radar alongside cameras and LiDAR for V2X collaboration. Supports roadside + vehicle perception. [Real-world / Simulation] GitHub | Hugging Face | [NeurIPS 2025 Spotlight]

Additional cutting-edge ones (highly recommended):

• HoloVIC (2024): Real-world V2I with high-quality LiDAR + camera for 3D detection/tracking. [Real-world]
• Rcooper (2024): First large-scale real-world roadside (I2I/V2I) dataset addressing sensor heterogeneity. [Real-world]
• Griffin (2025): Aerial-ground cooperative detection and tracking (V2U). [Real-world]

This catalogue bridges single-agent classics (KITTI, nuScenes) with modern cooperative benchmarks (V2U4Real, AGC-Drive, V2X-Radar). Whether you're working on cars, drones, or hybrid air-ground swarms, these datasets cover the full spectrum.

Want to contribute? Found a new dataset? Reply below or open a PR on the GitHub version.
Have hardware/sensor details for any entry? Let's add them!
Need code for loading any of these? Many have official devkits (OpenCOOD, etc.).

Happy researching—let's keep pushing autonomous perception forward!