Skip to content

Custom ONNX Trackers

Caliscope can load custom ONNX pose estimation models for 2D landmark tracking. You can use models trained on your specific subjects (particular species, body regions, behavioral features) without modifying Caliscope's source code. A TOML "model card" file describes the model's input requirements and output format.

After installation, ONNX models appear alongside the built-in trackers in the Reconstruct tab's dropdown menu.

Requirements

Requires the [tracking] extra

ONNX inference requires pip install caliscope[tracking]. The [gui] extra includes it automatically.

On first launch, Caliscope seeds your models directory with model cards for the RTMPose Halpe26 family. The .onnx weight files are not shipped; they can be downloaded in-app or placed manually.

In-App Download

For built-in models that include a [source] section in their model card, Caliscope can download the weights directly:

  1. Open the Reconstruct tab
  2. Select an ONNX tracker from the dropdown
  3. If the weights are not yet present, a download button appears with the model's license information
  4. Click to download. The weights are fetched from the upstream source and placed in the models directory

You can also download weights manually and place the .onnx file in the models directory. The filename must match the model_path field in the model card.

Setup Steps

  1. Obtain or export an ONNX pose estimation model
  2. Many pose estimation frameworks support ONNX export (MMPose, SLEAP, DeepLabCut with model zoo)

  3. The model must output either SimCC vectors or heatmaps (see format details below)

  4. Locate your models directory

  5. Caliscope uses platform-specific data directories (see Models Directory below)
  6. The directory is created automatically on first run

  7. Place your .onnx model file anywhere accessible (does not need to be in the models directory)

  8. Create a model card TOML file

  9. Must be placed in the models directory
  10. Name it descriptively (e.g., rtmpose-halpe26.toml)

  11. See Model Card Reference below for format details

  12. Restart Caliscope

  13. At startup, Caliscope scans the models directory for .toml files

  14. Valid model cards are registered and appear in the tracker dropdown

  15. Verify the model appears

  16. Open Caliscope and navigate to the Reconstruct tab
  17. Your custom model should appear in the tracker selection dropdown

Models Directory

Caliscope uses platform-specific data directories following standard conventions:

Platform Models Directory
Linux ~/.local/share/caliscope/models/
macOS ~/Library/Application Support/caliscope/models/
Windows C:\Users\<user>\AppData\Local\caliscope\caliscope\models\

Place your .toml model card files in this directory. The .onnx model file can live anywhere; the model card points to it via the model_path field (use an absolute path).

To find your models directory:

  1. Launch Caliscope
  2. Check the log file location shown in the terminal output
  3. The models directory is in the same parent directory

Or from Python:

from caliscope import MODELS_DIR
print(MODELS_DIR)

Model Card Reference

A model card is a TOML file that describes your ONNX model's configuration. Here's a complete annotated example:

[model]
# Display name in the GUI (optional, defaults to the .onnx filename if omitted)
name = "RTMPose-t Halpe26"

# Absolute path to your ONNX model file (required)
model_path = "/home/user/models/rtmpose-t-halpe26.onnx"

# Output format: "simcc" or "heatmap" (required)
format = "simcc"

# Model input dimensions as [width, height] (required)
input_size = [192, 256]

# Minimum confidence to report a point (optional, default 0.3)
# Lower values include more detections but may increase false positives
confidence_threshold = 0.3

# Keypoint name-to-index mapping (required)
# Maps human-readable names to the model's output indices
[points]
nose = 0
left_eye = 1
right_eye = 2
left_ear = 3
right_ear = 4
left_shoulder = 5
right_shoulder = 6
left_elbow = 7
right_elbow = 8
left_wrist = 9
right_wrist = 10
left_hip = 11
right_hip = 12
left_knee = 13
right_knee = 14
left_ankle = 15
right_ankle = 16
head = 17
neck = 18
hip = 19
left_big_toe = 20
right_big_toe = 21
left_small_toe = 22
right_small_toe = 23
left_heel = 24
right_heel = 25

# Wireframe segments for 3D visualization (optional)
# Each segment connects two points with a colored line
[segments.shoulders]
color = "y"  # Matplotlib color string (single char or full name)
points = ["left_shoulder", "right_shoulder"]

[segments.left_arm]
color = "g"
points = ["left_shoulder", "left_elbow"]

[segments.left_forearm]
color = "g"
points = ["left_elbow", "left_wrist"]

[segments.right_arm]
color = "r"
points = ["right_shoulder", "right_elbow"]

[segments.right_forearm]
color = "r"
points = ["right_elbow", "right_wrist"]

[segments.pelvis]
color = "y"
points = ["left_hip", "right_hip"]

[segments.left_flank]
color = "y"
points = ["left_hip", "left_shoulder"]

[segments.right_flank]
color = "y"
points = ["right_hip", "right_shoulder"]

[segments.left_thigh]
color = "g"
points = ["left_hip", "left_knee"]

[segments.left_shank]
color = "g"
points = ["left_knee", "left_ankle"]

[segments.right_thigh]
color = "r"
points = ["right_hip", "right_knee"]

[segments.right_shank]
color = "r"
points = ["right_knee", "right_ankle"]

[segments.neck_segment]
color = "y"
points = ["neck", "head"]

Required Fields

Field Description
model.model_path Absolute path to the .onnx file
model.format Either "simcc" or "heatmap" (see format details below)
model.input_size [width, height] as the model expects (not your video resolution)
[points] Maps point name (string) to output index (integer)

Optional Fields

Field Default Description
model.name ONNX filename stem Display name in the GUI
model.confidence_threshold 0.3 Minimum confidence to report a point (0.0 to 1.0)
[segments.*] None Wireframe segment definitions for 3D visualization
[source] None Download metadata for in-app weight fetching (see below)

Wireframe Segments

Each segment definition requires: - color: Matplotlib color string (e.g., "r", "blue", "#FF5733") - points: 2-element list of point names (must exist in [points] section)

Segments are used by the 3D visualizer to draw connections between keypoints, making it easier to interpret motion trajectories.

Source Section (For In-App Download)

Model cards can include a [source] section that enables in-app downloading of weights. This is optional. Custom models without a [source] section work normally but require manual placement of the .onnx file.

[source]
url = "https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/onnx_sdk/rtmpose-t_simcc-body7_pt-body7-halpe26_700e-256x192-6020f8a6_20230605.zip"
extraction = "zip_end2end"
license = "Apache-2.0"
license_url = "https://github.com/open-mmlab/mmpose/blob/main/LICENSE"
file_size_mb = 13
sha256 = "de5fa6ef754e1b19a0f8199d53affef122813e30c580c48be87fcf86c4ec47a7"
Field Required Description
url Yes Direct download URL for the model weights
extraction Yes (when url present) How to handle the download: "zip_end2end" (extract .onnx from zip) or "direct" (URL points directly to .onnx)
license No SPDX license identifier shown to user before download
license_url No Link to the full license text
file_size_mb No Approximate download size shown in the UI
sha256 No SHA-256 hash for integrity verification after download

SimCC vs Heatmap Formats

Set model.format to match your model's output type:

  • "simcc": RTMPose and other models using the SimCC head. Two output tensors (X and Y distributions).
  • "heatmap": SLEAP exports and other heatmap-based models. One output tensor (2D heat image per keypoint).

If unsure, check your training framework's docs or inspect the model's output tensors: two outputs means SimCC, one means heatmap.

Built-in Model Cards

The following RTMPose Halpe26 models ship as built-in model cards. Weights are downloaded in-app on first use.

Model Format Input Size Keypoints Download Size Notes
RTMPose-t Halpe26 SimCC 192×256 26 13 MB Fastest, good for real-time
RTMPose-s Halpe26 SimCC 192×256 26 21 MB Balanced speed/accuracy
RTMPose-m Halpe26 SimCC 192×256 26 50 MB Higher accuracy
RTMPose-l Halpe26 SimCC 192×256 26 100 MB High accuracy
RTMPose-x Halpe26 SimCC 288×384 26 178 MB Highest accuracy, largest input

All models use the Halpe26 keypoint set (26 body landmarks including feet) and are licensed under Apache-2.0 by OpenMMLab.

Other RTMPose variants and SLEAP-exported models should work with appropriate model card configuration. If you successfully use a model not listed here, consider contributing the model card.

Troubleshooting

Symptom Cause Fix
Model not in dropdown TOML not in models dir, syntax error, or missing required field Check the terminal log on launch
Few/no detections Wrong input_size or threshold too high Match input_size to training config; try confidence_threshold = 0.1
Wrong keypoint positions [points] indices don't match model output ordering Check your framework's keypoint convention (COCO vs Halpe vs OpenPose differ)
Model file not found Relative path or typo Use absolute paths in model_path

Limitations

Current limitations

Single-person detection with CPU inference only. GPU acceleration and multi-person tracking are not yet implemented. Open an issue if you need these.