We have all seen over the past few years that there is a standard approach to adopting and using generative AI. Send a prompt to somebody else's model, running on some external infrastructure, and pay for the tokens returned.
This approach removes the need to buy specialist hardware, operate model-serving infrastructure or maintain a fast-moving set of open-weight models. It also provides immediate access to systems that would be unrealistic for most organisations to host themselves.
However, that convenience creates dependencies which are easy to overlook during experimentation.
When we use these external tools, prompts can traverse external infrastructure and the provider can determine how much operational information is exposed. Model updates arrive on the provider's timetable and the versions we use can be retired. On a practical note, pricing and usage controls remain outside the customer's direct control.
All these issues become more significant as adoption scales. There are Copilots, coding assistants, retrieval systems and agents that can consume huge numbers of tokens. While once a model cost was negligible during a proof of concept, it can balloon when embedded across a business.
For government, defence and regulated organisations, there is another concern. Sensitive data may be difficult or inappropriate to send beyond a controlled environment. Even when cloud services meet the necessary compliance requirements, organisations may still want greater control over model versions, system behaviour, availability and evidence.
Meanwhile, local models have improved enormously over the last few years.
Small open-weight models can now perform useful summarisation, classification, extraction, coding and retrieval tasks. Quantisation has reduced their memory requirements. Mixture-of-experts architectures make it possible to hold large models while activating only part of the network for each token. Tooling around local serving, evaluation and fine-tuning has become markedly more accessible.
Local models are no longer a novelty, they can provide a highly effective, workload appropriate model next to the user and data. NVIDIA's DGX Spark device provided a platform for me to experiment. This is the first article in a series about what I found. I am using the device as a test bed for a broader question: does this new class of AI workstation materially change the options available?
What is a DGX Spark?
The Spark is a compact AI workstation built around NVIDIA's GB10 Grace Blackwell platform. Its most consequential feature is 128 GB of unified memory shared between CPU and GPU.
That memory capacity changes what can be run on a desktop. Models which cannot fit inside consumer GPUs can be loaded without splitting their weights.
There is a trade-off. The ability to hold a model does not guarantee that the model will run quickly. Inference performance remains sensitive to memory bandwidth, model architecture, quantisation, context length and serving software.
Why I bought one
Partly professional obligation, partly curiosity.
I spend much of my time helping UK public-sector and defence organisations think about AI adoption, assurance and sovereignty. Those conversations regularly encounter the same set of choices.
Hosted frontier models offer high capability and low initial friction. However, they also introduce external dependencies around data, models, infrastructure, commercial terms and operations.
On-premises systems can provide more direct control over data movement, model versions and service availability. This is a double edged sword, as they also transfer more engineering, security and operational responsibility to the organisation.
I should be clear, local execution does not make an AI system sovereign by itself. The device still depends on NVIDIA hardware, the CUDA software stack, upstream libraries and model developers. Model weights carry their own licence, provenance and supply-chain considerations. However, what local execution can provide is a meaningful increase in operational control. The operator chooses the model, fixes the version, observes the runtime, controls the data path and decides when changes are introduced.
This is consistent with the more pragmatic definition of sovereignty I use elsewhere: the ability to make and change decisions about the AI systems on which an organisation depends, without undue reliance on a single external actor.
Sovereignty is a spectrum of control choices rather than just a hosting location.
What can an AI workstation be used for?
Over the next couple of weeks I'll be testing how the AI workstation works with:
- Local inference
- Local retrieval and agentic systems
- Fine-tuning and model adaptation
- GPU-accelerated data science
This article will address the first on the list and some would say an obvious starting point.
Local inference
The device can serve a model directly to local applications. That might support an individual assistant, document classification, structured extraction, code generation or the language layer inside a wider system.
This is the workload tested in this article.
The experiment
I ran a structured inference benchmark across seven open-weight models, ranging from approximately one billion to 120 billion total parameters. The test set deliberately spans:
- Small, medium and very large models.
- Dense and Mixture-of-Experts (MoE) architectures.
- Different training approaches and model families.
The models were served through an Unsloth API. Requests were sent between docker containers. The test set included short prompts and longer inputs of approximately 4,000 and 20,000 tokens. Each test was repeated 10 times. The models were run using Q4 quantisations.
I recorded:
- time to first token;
- generation throughput;
- output-token count;
- request duration;
- reported compute power;
- GPU utilisation;
- temperature;
- behaviour under concurrent requests.
A methodological warning
This is a serving benchmark to provide insight, not a model-quality evaluation.
Note that tokens per second measures how quickly a model emits tokens. It does not indicate whether an answer is correct, concise, faithful or useful. A fast model producing poor answers is efficient but fundamentally wrong.
Reasoning-oriented models also tend to generate many more tokens than conventional instruction-tuned models. That can make throughput and energy-per-token figures look respectable while the cost of completing an actual task rises substantially.
The results below establish whether these configurations are operationally usable. They do not determine which model is best.
Finding 1: local is fast enough
I used 30 output tokens per second as a practical working reference for an interactive experience. It is not a formal standard, but below this level delays become increasingly visible during ordinary chat-style use.
All seven tested configurations exceeded that reference on the short-prompt benchmark.
Gemma 3 1B produced approximately 210 output tokens per second. Llama 3.2 3B produced around 101. Gemma 4 26B reached around 82.
Even the slowest configuration, Qwen3 8B, produced approximately 47 tokens per second.
The point is not that Gemma 3 1B is the best model. It clearly has a different capability from models hundreds or thousands of times its size.
The significance is that local models are no longer automatically disqualified because the user experience is too slow. Every configuration in this test generated output quickly enough to justify evaluating its accuracy and usefulness against a real workload. That does not mean a one billion parameter model can replace a frontier model across general reasoning, complex coding or ambiguous work.
Classification, extraction, routing, summarisation and other structured workflows may not need the largest available model. In these cases, using an oversized model can increase cost, latency, dependence and environmental impact without creating much improvement in the outcome.
The right comparison is whether the smaller model meets the performance criteria for a particular task. That requires a quality benchmark alongside this performance benchmark. I have not completed that comparison here, so I am deliberately not claiming that the fastest model produced the best (or even acceptable) answers.
Finding 2: local models handle more context than you think
Long inputs are demanding for local inference. The model must process the prompt before producing an answer, while the KV cache grows with context length.
Qwen 3.6 35B moved from approximately 67 tokens per second on short prompts to 65 at a 20,000-token input: a decline of about 4%.
Gemma 4 26B declined from around 82 to 73 tokens per second, roughly 10%.
Llama 3.2 3B showed a more material change, declining from approximately 101 to 70 tokens per second, around 31%.
The results depend on architecture and implementation. Longer inputs can increase the time before generation begins even where output speed remains stable. For document analysis and retrieval workloads, both prefill latency and decode speed need to be measured.
Finding 3: a workstation is not a shared inference service
The single-user results are encouraging. The concurrency results establish a clear boundary. For this test, I ran simultaneous requests against Qwen3 8B. Once a response began generating, per-request decode speed remained broadly stable but the queueing delay did not.
Median time to first token was approximately:
- 0.14 seconds for one request;
- 20 seconds for three concurrent requests;
- 41 seconds for five;
- 87 seconds for ten.
At the 95th percentile, ten simultaneous requests waited around 101 seconds before generation began. The device continued to generate tokens. Requests simply spent progressively longer waiting for access to the model.
A Spark can provide an excellent local experience for an individual practitioner. It may also support asynchronous workloads or a small number of carefully managed processes.
It is not a sensible shared inference service without another serving, scheduling and scaling layer. An organisation seeking a general internal assistant would need infrastructure designed for concurrency, resilience and operations.
Finding 4: the power used is remarkably modest
The big surprise was the power data. During sustained inference, NVIDIA's device telemetry generally reported approximately 45 to 58 watts across the tested models.
This is simplified slightly, it excludes power-supply losses, storage, networking, fans and other chassis components. I did not have an external plug meter for this experiment. The larger models did not simply draw proportionately more power. Much of the difference in energy per generated token came from how long the model needed to run.
Parameter count alone did not determine efficiency. Architecture, quantisation, implementation and generation behaviour all mattered.
The measurement is also per output token, not per completed task. Reasoning models often emit more tokens, so their energy per useful answer may be less favourable than the chart suggests. A proper sustainability benchmark needs a functional unit such as energy per correctly completed task, not simply energy per token.
Google has published the most comprehensive first-party methodology I found. It reports that the median Gemini Apps text prompt consumes 0.24 Wh and emits 0.03 gCO₂e. OpenAI has stated that an average ChatGPT query consumes approximately 0.34 Wh, though that figure was published without an equivalent technical methodology.
These figures cannot be compared directly with my local data.
Local inference can be operationally frugal, particularly when a small model is suitable for the task. More importantly, its consumption is observable.
I can associate each request with a model version, prompt length, duration, generated tokens and measured power. I can rerun the test when the software changes. I can decide whether an energy measurement should include the whole chassis, the local grid mix or embodied hardware.
With a hosted API, I usually receive token usage and a bill. Workload-level energy data is rarely available. Google's disclosure is a welcome exception, but it still describes a median prompt rather than the consumption of my particular request.
That observability has practical value. Energy can become an engineering measure alongside latency, quality, cost and safety rather than an estimate added after deployment.
It is also a small example of what greater operational control looks like in practice.
Conclusion
Six months ago, many of the assumptions I held about local AI would have pushed me towards the cloud for anything beyond experimentation. After several weeks of testing, that position feels increasingly outdated.
Local systems still don't outperform hyperscalers, nor is desktop hardware going to replace cloud infrastructure, but the real-life capability gap has narrowed dramatically.
Across a range of modern models, local inference was consistently fast enough to be useful. Large contexts remained practical. Models that would previously have required specialist infrastructure could be run on a device small enough to sit on a desk.
For organisations concerned with sovereignty, security, resilience, predictable costs or deployment into constrained environments, local AI is viable, and occasionally preferable.
The constraint has moved
A compact device drawing a modest amount of reported compute power can serve models ranging from one billion to 120 billion total parameters at usable generation speeds. The difficult questions are now:
Which model meets the actual quality threshold?
Which licence and provenance are acceptable?
How does the model behave on sensitive or adversarial inputs?
What context length is required?
What is the cost per completed task?
From my experimentation, local inference is operationally credible enough to deserve proper evaluation. For a meaningful class of workloads, renting intelligence from an external provider is no longer the only plausible design.
That does not make the cloud obsolete. Hosted frontier models will remain the sensible choice for many applications, particularly where maximum general capability, elasticity and managed operation matter. It does create credible choice though.
What I am testing next
The above work on local inference was the simplest and best entry test of the Spark.
I am now using it to build fully local agentic workflows, and investigating whether local RAG can retain enough retrieval and answer quality to be useful. I am also testing whether an agent can perform meaningful multi-step work without depending on an external model API.
In addition I am testing model fine-tuning through Unsloth. The question which has arisen is whether in this context adapting a smaller model creates enough task-specific improvement to justify the effort.
The final strand is GPU-accelerated data science: cuDF, cuML, RAPIDS and other workloads which take the analysis of these local models to the next level.
Keep a look out over the coming weeks for these other experiments.