Apple Silicon Object-Detection for Frigate NVR

Overview

A self-hosted Frigate NVR (network video recorder) deployment extended with a custom-built object-detection client optimized for Apple Silicon. The studio’s contribution is an ONNX Runtime detector that offloads inference to the Mac’s Neural Engine via CoreML and plugs into Frigate over its ZMQ detector protocol.

Why It Exists

Frigate’s built-in detectors target Coral TPUs or x86 GPUs; running high-FPS detection on Apple Silicon hardware needed a purpose-built bridge. This project supplies that, letting a Mac act as a fast, low-power detection backend for a containerized Frigate instance watching RTSP cameras.

What We Built

Two parts. First, a containerized Frigate 0.17 deployment (docker-compose.yml, config/, RTSP feeds, storage volumes). Second, the real engineering, an apple-silicon-detector/: a Python ZMQ client (zmq_onnx_client.py, model_util.py) implementing Frigate’s REQ/REP detector protocol over IPC, running ONNX models through ONNX Runtime with the CoreML execution provider on the Neural Engine. It supports YOLOv9, RF-DETR, D-FINE, and custom ONNX models, auto-loading whatever model Frigate configures, with a Makefile install/run flow and a packaged macOS app (FrigateDetector.app, AppIcon, scripts) for a no-terminal launch. A separate Dockerfile.yolov9 builds and exports YOLOv9 weights to simplified ONNX for serving.

Technologies & Approach

Python with ONNX Runtime (CoreML provider), OpenCV (headless + contrib) for image handling, PyZMQ for the Frigate IPC protocol, and Pydantic for typed config. The model-build path uses uv and the upstream YOLOv9 exporter to produce ONNX artifacts. The detector is a drop-in replacement for Frigate’s built-in detectors, decoupled over ZMQ so it can run as a native app alongside the Dockerized NVR.

Outcome / Impact

Delivered hardware-accelerated, Apple-Silicon-native object detection for a real NVR deployment, demonstrating custom CV inference engineering, protocol-level integration with an existing OSS system, and an ONNX model build/export pipeline.

Capabilities Demonstrated

• Hardware-accelerated CV inference on Apple Silicon (CoreML / Neural Engine)
• Custom detector plugin implementing a ZMQ REQ/REP protocol
• Multi-model ONNX serving (YOLOv9, RF-DETR, D-FINE)
• ONNX model build/export pipeline
• Integration with a containerized NVR (Frigate) and RTSP camera streams