EGO Data Acquisition Terminal Data Format Standard — Training-Ready Data Structure

📁 Directory Structure Example

20260520_173140_stack_bowls/
├── cam0.mp4               # front-left 1280x720 @ 30fps
├── cam1.mp4               # front-right 1280x720 @ 30fps
├── cam2.mp4               # rear-left 1280x720 @ 30fps
├── cam3.mp4               # rear-right 1280x720 @ 30fps
├── imu.bin                # 6-DoF + magnetometer ~200Hz
├── imu.csv                # IMU CSV format (optional)
├── intrinsics.json        # intrinsics of 4 cameras
├── extrinsics.json        # camera-to-IMU calibration
└── meta.json              # device/firmware/fps/resolution

📋 File Descriptions

cam0~3.mp4
4-channel global-shutter H.264 video, frame-level hardware sync, trigger phase offset <50 µs

imu.bin / imu.csv
Acceleration + angular rate + magnetometer, timestamped by the M0 coprocessor

intrinsics.json
Focal length / principal point / distortion coefficients, calibrated per camera

extrinsics.json
Camera→IMU extrinsic transformation matrix

meta.json
Device info / firmware version / lens ID / fps / resolution

⚡ Hardware Sync Timing Architecture

The M0 coprocessor drives the FSYNC sync signal, and three independent GPIO lines connect to the CMOS shutter trigger, the IMU interrupt and the magnetometer ready signal. All sensors latch on the same time base, with frame-to-IMU jitter <10 µs. The output data is directly compatible with the data loaders of mainstream training frameworks (OpenRobotLab, VLA, etc.).

🔄 EGO-DAQ ↔ LeRobot v2.1 Field Mapping

EGO-DAQ output data maps directly to the Hugging Face LeRobot v2.1 standard format fields:

EGO-DAQ Output	LeRobot v2.1 Field	Description
cam0.mp4	`observation.images.cam0`	30fps, H.264, hardware sync
cam1.mp4	`observation.images.cam1`	30fps, H.264, hardware sync
cam2.mp4	`observation.images.cam2`	30fps, H.264, hardware sync
cam3.mp4	`observation.images.cam3`	30fps, H.264, hardware sync
imu.bin	`observation.state`	10-DoF: acc(3)+gyro(3)+mag(3)+temp(1)
intrinsics.json	`camera_info`	fx/fy/cx/cy + distortion coefficients
extrinsics.json	`extrinsics`	Camera→IMU transformation matrix
meta.json	`info.json features`	Device/firmware/fps/resolution

Technical Details

Technical specifications behind the data capture and the data indexing standard

📹 Video Encoding Parameters

Codec	H.264 (avc1), HEVC/SVT-AV1 supported
Resolution	1280 × 720
Frame Rate	30 fps, 4 independent MIPI CSI
Sync Method	FSYNC frame-level hardware sync, trigger phase offset <50µs

🧬 IMU Specifications

Degrees of Freedom	6-DoF + 3-DoF magnetometer = 10-DoF
Output Rate	≈200 Hz
Timestamp Precision	Hardware latched by M0 coprocessor
Frame-to-IMU Jitter	<10 µs

📅 Time Indexing Standard

Following the LeRobot v2.1 indexing standard, EGO-DAQ adopts the following frame index conventions:

Field	Type	Description
episode_index	uint32	Recording index, incrementing from 0
frame_index	uint32	Frame index within an episode, starting from 0
index	uint64	Global frame ID (unique across episodes)
timestamp	float64	Time offset within the episode (seconds)
next.done	bool	Whether it is the last frame of an episode (optional)

🧬 VITRA Data Dimension Reference

Human hand state dimension definition (VITRA paper, Sec.3.2):

Human hand state: 122D = 51(right hand) + 10(MANO β) + 51(left hand) + 10(MANO β)
↳ where 51D = 3(translation) + 3(global rotation) + 45(15×3 joint angles)

EGO-DAQ output: 10-DoF = 3(acc) + 3(gyro) + 3(mag) + 1(temp)
↳ Dimension padding is handled by VITRA pad_state_human()

🔄 Time Base Calibration

All sensor data is latched on the same hardware time base (M0 coprocessor RTC clock). The PTS (presentation timestamp) of each video frame and the timestamps of IMU samples are driven by the same clock source, ensuring timestamp consistency. The calibration files (intrinsics.json / extrinsics.json) are calibrated before the device leaves the factory and are delivered with the terminal.

🐍 Python SDK: EGO-DAQ Raw → LeRobot Dataset

The following code converts the EGO-DAQ timestamped directory structure into a Hugging Face LeRobot v2.1 standard dataset:

# pip install lerobot huggingface_hub opencv-python

from lerobot import LeRobotDataset

from pathlib import Path

import json, numpy as np, cv2

def convert_episode(ego_dir: Path, dataset: LeRobotDataset):

  meta = json.loads((ego_dir / "meta.json").read_text())

  caps = [cv2.VideoCapture(str(ego_dir / f"cam{i}.mp4")) for i in range(4)]

  imu = np.fromfile(ego_dir / "imu.bin", dtype=np.float32).reshape(-1, 10)

  idx = 0

  while True:

    frames = [cap.read()[1] for cap in caps]

    if any(f is None for f in frames): break

    dataset.add_frame({

      f"observation.images.cam{i}": frames[i] for i in range(4),

      "observation.state": imu[idx],

      "action": np.zeros(10),

    })

    idx += 1

  dataset.save_episode()

dataset = LeRobotDataset.create(

  repo_id="gear-lab/ego-daq-demo", fps=30, root="./lerobot/",

  features={

    f"observation.images.cam{i}": {"dtype": "video", "shape": (3, 720, 1280)}

      for i in range(4),

    "observation.state": {"dtype": "float32", "shape": (10,)},

    "action": {"dtype": "float32", "shape": (10,)},

  },

)

for ep in sorted(Path("./recordings/").iterdir()):

  convert_episode(ep, dataset)

dataset.push_to_hub("gear-lab/ego-daq-demo")

Training-Ready Data Structure