Where inference runs changes privacy, latency, reliability, cost, and the work required from operations teams. The choice is not ideological. It should follow the event, camera population, network, hardware lifecycle, and evidence-retention boundary.
Choose edge when locality is valuable
Edge inference supports low latency, reduced video transfer, offline operation, and local data boundaries. It also creates a fleet of devices that need provisioning, health monitoring, secure updates, compatibility management, and physical support.
Choose cloud when centralization is valuable
Cloud processing can simplify elastic compute, centralized model rollout, and aggregation across sites. Continuous video transfer may create bandwidth, privacy, and outage concerns. Measure complete transport and processing latency.
Use hybrids deliberately
A common design performs detection locally and sends structured events, thumbnails, or short clips for central review. Cloud systems can manage configuration, fleet health, analytics, and selected reprocessing without receiving every frame.
Compare total operating cost
Include device purchase, installation, spares, remote management, network egress, cloud compute, storage, support labor, model rollout, and replacement. Test costs against realistic frame rates and camera concurrency.
Choose placement per processing stage
Capture, filtering, inference, tracking, event storage, analytics, and model management do not need to live in one place. Edge inference can reduce bandwidth and reaction time; cloud services can support fleet-wide analysis and easier updates. A hybrid design often keeps raw video local while sending bounded events and diagnostics centrally.
Model the worst credible network condition, not only average bandwidth. Decide what the site does while disconnected, how much it buffers, how it reconciles after recovery, and which remote controls remain available. Connectivity behavior is part of the product contract.
Compare total operations
Edge cost includes hardware, installation, power, secure provisioning, updates, replacement, and field support. Cloud cost includes transfer, decoding, compute, storage, egress, and centralized operations. Estimate both at expected and peak stream counts with the real frame rate and retention policy.
Security and privacy depend on architecture and execution. Local processing can reduce raw-data movement, but unmanaged devices introduce physical and software risk. Cloud centralization simplifies some controls but expands transfer and concentration. Document threat boundaries, credentials, encryption, patching, and data lifecycle for either choice.
Executive decision record
The decision is where each capture, inference, storage, analytics, and management stage should run given latency, connectivity, privacy, cost, and support. Write that decision before selecting a model, vendor, framework, or implementation pattern. A written boundary keeps technical exploration connected to the operating outcome and makes it possible to explain why the organization advanced, revised, or stopped the work.
Approval should depend on network failure tests, stream-level cost models, target-device profiling, security boundaries, fleet procedures, and end-to-end service measurements. The evidence does not need to remove every uncertainty, but it should address the uncertainty capable of changing value, architecture, risk, or ownership. Record the baseline, assumptions, unresolved questions, and the person accepting the next stage.
Failure boundary and operating ownership
The central failure to guard against is choosing one placement ideology for the entire pipeline while ignoring disconnection, field maintenance, transfer, or data concentration. Treat that condition as a testable scenario. Define how the system detects it, what users experience, which action is prevented or reversed, and what evidence reaches the person responsible for recovery.
Long-term accountability sits with the product architecture team with explicit counterparts for site operations, cloud services, security, and data lifecycle. Supporting specialists can provide platforms, research, review, or delivery capacity, but they cannot substitute for an owner who controls policy and operating change. Name that owner before production and include the ownership path in release evidence and incident procedure.
A practical 90-day application plan
During the first 30 days, convert network failure tests, stream-level cost models, target-device profiling, security boundaries, fleet procedures, and end-to-end service measurements into a bounded evidence plan. Assign each artifact to a named contributor, identify the representative inputs required, and agree on the comparison baseline before implementation expands. The objective of this period is to expose the assumption most likely to invalidate the work while the cost of changing direction is still low.
During days 31 through 60, build or instrument the smallest complete workflow that can support the decision about where each capture, inference, storage, analytics, and management stage should run given latency, connectivity, privacy, cost, and support. Include the real data and authorization path where feasible, record exceptions, and review difficult cases with the people who own the underlying process. Resist adding breadth until the team can explain the measured behavior of this narrow slice.
During days 61 through 90, test the boundary represented by choosing one placement ideology for the entire pipeline while ignoring disconnection, field maintenance, transfer, or data concentration. Exercise degraded dependencies, ambiguous inputs, recovery, and handoff rather than demonstrating only successful cases. End the period with a written advance, revise, or stop decision that cites evidence, residual exposure, expected operating cost, and the next authority boundary.
The review should be accepted by the product architecture team with explicit counterparts for site operations, cloud services, security, and data lifecycle. That group should confirm not only that the system can work, but that ownership, support capacity, monitoring, and change control are credible. If those conditions are absent, the responsible outcome is another bounded learning stage rather than an unsupported production commitment.
Practical checklist
- Maximum event latency
- Connectivity and offline requirement
- Video transfer and retention boundary
- Per-site camera concurrency
- Hardware lifecycle and support capacity
- Update and rollback process
- Three-year operating cost model
Engagement scenario
A multi-site operator runs compact detection on local gateways and transmits only review events. A central service manages model versions and device health. Selected clips are uploaded under retention rules for evaluation and incident review.
Continue reading
- [Computer vision development guide](/computer-vision-development-guide)
- [Distilling models for the edge](/distilling-vision-models-for-the-edge)
Sources and further reading
- [ONNX Runtime: deploy on IoT and edge](https://onnxruntime.ai/docs/tutorials/iot-edge/)
- [ONNX Runtime execution providers](https://onnxruntime.ai/docs/execution-providers/)