Author: Adarsh Nanjaiya Latha
Last updated: March 2026 (community contributions welcome!)

I've spent weeks hunting for a single go-to resource covering the most widely used datasets in autonomous perception for cars and drones, especially in cooperative/multi-agent settings. Unsurprisingly, no such one-stop list existed. So here it is: a curated catalogue of key datasets.

The list is not exhaustive (new datasets drop constantly), but it covers the foundational and cutting-edge ones. Please feel free to fork this post on GitHub and add others, or comment below with suggestions!

Categorization:
By number of platforms (single vs. multi-agent) and modalities (single-sensor vs. multi-sensor fusion).
Sub-notes on platforms: V2V (ground-ground), V2U (ground-air), U2U (air-air), V2I/V2X (vehicle-infra/everything).
[Real-world] or [Simulation] flag for each.

For sensor hardware details (e.g., exact LiDAR models, camera resolutions), check the original papers or project pages. I welcome PRs if anyone wants to expand that here.

1. Single Platform, Single Modality

Standard, non-cooperative datasets from one ego vehicle or drone using only one sensor type (typically camera-only or LiDAR-only). Great for baseline semantic segmentation, tracking, or point-cloud benchmarking.

• Cityscapes (2016): Dense pixel-level urban scene understanding from a single ground vehicle. Camera-only. [Real-world] Website
• SYNTHIA (2016): Large synthetic urban scenes for semantic segmentation. Pure camera data. [Simulation]
• Stanford Drone Dataset (SDD) (2016): 2D object tracking and trajectory prediction from a single overhead drone using video. Camera-only. [Real-world] Website
• LIBRE: LiDAR Benchmarking and Reference (2020): Single ground vehicle with 10–12 different LiDAR sensors (various manufacturers/models) for evaluating point-cloud performance across conditions (static, weather, dynamic traffic). LiDAR-only. [Real-world] arXiv

Additional notable ones:

• VisDrone (2018–ongoing): Drone-mounted camera for object detection/tracking in urban scenes. Camera-only. [Real-world]
• UAVDT (2018): Drone camera dataset focused on detection/tracking under varying altitudes/angles. Camera-only. [Real-world]

2. Single Platform, Multi-Modality

One ego vehicle or drone equipped with a heterogeneous sensor suite (e.g., cameras + LiDAR + radar/thermal). The classic "full-stack" single-agent perception benchmarks.

• KITTI (2012): Foundational benchmark for autonomous driving. Single vehicle with stereo cameras + LiDAR. [Real-world] Website
• nuScenes (2019) & Waymo Open Dataset (2019): Massive-scale single-vehicle datasets with 360° coverage. Cameras + LiDAR + radar (nuScenes includes radar). [Real-world] nuScenes | Waymo
• UAVScenes (2025): Multi-modal dataset from a single drone: RGB cameras + thermal imaging + LiDAR. Supports semantic segmentation, depth estimation, localization, and novel-view synthesis. [Real-world] GitHub | arXiv

3. Multi-Platform, Single Modality

Cooperative/collaborative datasets where multiple vehicles, drones, or infrastructure nodes share data, but only one sensor medium is used. Early steps toward swarm/U2U or pure-LiDAR V2V collaboration.

• CoPerception-UAV (2022): Drone-to-drone (U2U) swarm collaboration for bird's-eye-view semantic segmentation and detection. Pure camera imagery from 5 coordinated UAVs. [Simulation] (CARLA + AirSim) Project page
• UAV3D (2024): Large-scale UAV-to-UAV (U2U) benchmark with synchronized camera imagery. 1,000 scenes across diverse towns; supports 3D perception tasks. [Simulation] (CARLA + AirSim) NeurIPS 2024 paper
• V2VLoc (2025): Built for cooperative LiDAR localization and detection in GNSS-denied environments. Focuses on shared LiDAR point clouds between vehicles (V2V). [Simulation]

Additional:

• U2UData (2024): Large-scale swarm UAV (U2U) dataset with LiDAR focus (plus some depth/RGB but primarily single-modality benchmarks available). [Simulation] arXiv

4. Multi-Platform, Multi-Modality

The cutting edge of cooperative perception: vehicles, drones, and/or smart infrastructure collaborating with fused sensor streams (cameras + LiDAR + radar). These enable V2V, V2U, U2U, and V2I/V2X research at scale.

• V2U4Real (2025): First large-scale real-world dataset explicitly for Vehicle-to-UAV (V2U) cooperative perception. Ground vehicle + drone sharing synchronized multi-view LiDAR + RGB camera streams. [Real-world] arXiv
• AGC-Drive (2025): Massive real-world aerial-ground cooperative dataset (V2V + V2U). Two ground vehicles (5 cams + 1 LiDAR each) + one UAV (forward cam + LiDAR). Covers dynamic interactions like cut-ins/lane changes. [Real-world] GitHub | NeurIPS 2025
• DAIR-V2X (2022): First massive real-world Vehicle-to-Infrastructure (V2I) dataset. Synchronized camera + LiDAR from vehicles and roadside units. [Real-world] GitHub
• TUMTraf-V2X (2024): Cooperative 3D object detection, tracking, and maneuver analysis with multi-modal data from roadside units + vehicles (V2I/V2X). [Real-world]
• OPV2V (2022): Heavy baseline for multi-vehicle (V2V) cooperative perception. Camera + LiDAR streams. [Simulation] (CARLA) Website
• V2X-Sim (2022): Foundational simulated V2X benchmark (vehicles + roadside infrastructure). Synchronized camera + LiDAR. [Simulation] (CARLA + SUMO) Project page
• V2V4Real (2023) & V2X-Real (2024): Large-scale real-world V2V / V2X datasets tracking hundreds of km. Camera + LiDAR streams. [Real-world] V2V4Real
• V2X-Radar (2025) / V2X-R (simulation counterpart): Newest branch introducing weather-robust 4D Radar alongside cameras and LiDAR for V2X collaboration. Supports roadside + vehicle perception. [Real-world / Simulation] GitHub | Hugging Face | [NeurIPS 2025 Spotlight]

Additional cutting-edge ones (highly recommended):

• HoloVIC (2024): Real-world V2I with high-quality LiDAR + camera for 3D detection/tracking. [Real-world]
• Rcooper (2024): First large-scale real-world roadside (I2I/V2I) dataset addressing sensor heterogeneity. [Real-world]
• Griffin (2025): Aerial-ground cooperative detection and tracking (V2U). [Real-world]

This catalogue bridges single-agent classics (KITTI, nuScenes) with modern cooperative benchmarks (V2U4Real, AGC-Drive, V2X-Radar). Whether you're working on cars, drones, or hybrid air-ground swarms, these datasets cover the full spectrum.

Want to contribute? Found a new dataset? Reply below or open a PR on the GitHub version.
Have hardware/sensor details for any entry? Let's add them!
Need code for loading any of these? Many have official devkits (OpenCOOD, etc.).

Happy researching, and let's keep pushing autonomous perception forward!