Reconstruction: Landmark Triangulation from Motion Capture

The reconstruction pipeline transforms synchronized videos into 3D motion trajectories through two stages:

1. 2D landmark detection: a tracker processes each camera's video to identify anatomical landmarks (e.g., joint positions) in every frame
2. 3D triangulation: using the calibrated camera system, corresponding 2D observations from multiple cameras are triangulated into 3D world coordinates

The pipeline uses the camera intrinsics and extrinsics established during calibration to locate landmarks in physical space.

Available Trackers

ONNX Trackers

Caliscope uses ONNX pose estimation models for 2D landmark tracking. Several RTMPose Halpe26 models (tiny through xlarge, 26 body landmarks) ship as built-in model cards and can be downloaded in-app on first use. You can also load custom models exported from SLEAP, DeepLabCut, RTMPose, or other frameworks — see Custom ONNX Trackers for setup instructions.

The reconstruction pipeline is a convenience tool for verifying calibration quality and quick landmark export. For production reconstruction workflows, tools like anipose and Pose2Sim are better suited. Caliscope's aniposelib-compatible camera export makes it straightforward to calibrate here and hand off to those tools.

Workflow

1. Navigate to the Reconstruction tab
2. Select the recording you want to process from the list
3. Recordings are detected automatically from subfolders within recordings/ that contain synchronized videos
4. You may need to reload the workspace if recordings were added while the application was running
5. Choose a tracker from the dropdown menu
6. Click Process to begin landmark tracking and triangulation
7. Results appear in the 3D viewer when processing completes
8. Open the recording's output subfolder to access trajectory files

Output Files

After processing, output is saved to a subfolder named after the tracker within the recording directory (e.g., recordings/walking/ONNX_rtmpose_t_halpe26/).

File	Format	Description
xy_{tracker}.csv	Long CSV	2D tracked points per camera (sync_index, cam_id, frame_index, frame_time, point_id, img_loc_x, img_loc_y, obj_loc_x, obj_loc_y)
xyz_{tracker}.csv	Long CSV	Triangulated 3D points (sync_index, point_id, x_coord, y_coord, z_coord, frame_time)
xyz_{tracker}_labelled.csv	Wide CSV	Named columns (e.g., nose_x, nose_y, nose_z, left_shoulder_x, ...)
xyz_{tracker}.trc	TRC	OpenSim-compatible format for biomechanical modeling
camera_array.toml	TOML	Snapshot of the camera calibration used for this reconstruction

Coordinate Units

All 3D coordinates are in meters. The physical scale is determined by the calibration target dimensions you entered during extrinsic calibration. See Calibration Targets for details on how the scale chain propagates from board geometry to world coordinates.

Per-Recording Camera Snapshot

Each reconstruction saves a copy of camera_array.toml alongside its output files. This ensures that recalibrating your camera system does not invalidate previous reconstruction results. Each recording retains the exact calibration parameters used to produce it.

In longitudinal studies where camera positions may shift between sessions, this prevents the need to reprocess archived recordings.

Practical Recording Guidelines

Minimize Motion Blur

Motion blur substantially compromises landmark recognition. To reduce blur:

• Use higher frame rates (e.g., 60 fps or above for dynamic movements)
• Increase lighting to maintain exposure at faster shutter speeds
• Avoid slow shutter speeds that allow excessive motion during exposure

Lighting

• Ensure adequate, even lighting across the capture volume
• Avoid harsh shadows or high-contrast regions that can confuse trackers
• Diffuse lighting generally produces more consistent tracking than point sources