Service / On-device
On-device & edge
Put a capable model on hardware you don't control, inside a memory budget that doesn't move.
The constraint is the product
On a server you can add another GPU. On a phone you get a fixed memory ceiling, a thermal envelope, a battery, and a user who notices both. The model that scored well in your evaluation may not fit, and the one that fits may not be good enough yet.
On-device work is the discipline of moving along that trade-off deliberately: how much quality you give up per gigabyte, which layers tolerate four bits, when to fall back to the cloud, and how to keep it all working across a fleet of hardware you will never hold.
Capabilities
What this covers.
Feasibility assessment
Whether your task fits on your target hardware at all, answered early with a measurement rather than late with an apology.
Quantisation for the edge
INT4, INT8 and mixed-precision, tuned per model and evaluated on your task — not a default preset applied uniformly and hoped over.
Runtime selection
llama.cpp, MLX, ONNX Runtime, ExecuTorch, LiteRT and vendor SDKs, chosen against your platform matrix and maintenance appetite.
Accelerator targeting
Apple Neural Engine, Qualcomm Hexagon, and the NPUs that ship in current silicon — including when the CPU is honestly the better answer.
Memory budgeting
Weights, KV cache and activations accounted for against the OS ceiling, with headroom for the rest of your app.
Thermal and battery
Sustained throughput under real thermal conditions, measured on device, because benchmarks run cold and users do not.
Hybrid architecture
On-device by default, cloud when the task earns it. Routing that is invisible to the user and honest about what left the device.
Model delivery
Shipping, versioning and updating weights without shipping a whole app build, and without a 4 GB first-run download.
Offline-first behaviour
Defined behaviour with no network, including what degrades, what queues, and what the user is told.
Deliverables
What you're left with.
- A feasibility report with measured numbers on your actual target devices
- A quantised model meeting an agreed quality bar, with the evaluation that proves it
- Integrated inference in your app, on your platforms, in your build
- A device-matrix benchmark your team can re-run against future models
- Model update and rollback mechanism
Stack
What we work in.
- llama.cpp
- MLX
- ExecuTorch
- ONNX Runtime
- LiteRT
- CoreML
- Qualcomm QNN
- GGUF
Tools follow the problem. If your team already runs something that works, we would rather extend it than replace it.
Questions
Asked often.
How small can a genuinely useful model be?
It depends entirely on the task. Narrow, well-scoped work — classification, extraction, structured rewriting — runs well in a few billion parameters at four bits. Open-ended reasoning does not, yet. We would rather narrow the task than oversell the model.
Is on-device actually cheaper?
Your marginal inference cost goes to zero, and your engineering cost goes up. It pays off at volume, when latency is non-negotiable, or when the data must not leave the device. At low volume, an API is usually the rational choice.
What about Android fragmentation?
It is the hard part, and it is handled with a tiered device matrix: a defined floor, defined behaviour below it, and a fallback path. Not by testing on a flagship and hoping.