PRIVACY BY DESIGN

Privacy by architecture, not by promise

A privacy statement describes intent. Architecture determines what can actually happen to data.

By Vijay Rengan, Founder & Chief AI Officer, EMLI

Enterprise AI changes the privacy conversation because it makes information easier to find, combine, summarize, and act on. It also creates a new data path: employees and applications send prompts, documents, retrieved passages, and business context to an inference engine. When that engine is operated by an external model provider, the organization must rely on contracts, service configurations, retention commitments, and the provider’s control environment to protect what leaves its boundary.

Those protections may be appropriate for some workloads. But they are not the only architecture available. For confidential, regulated, or high-volume enterprise knowledge, organizations can run open-weight or commercially licensed models on infrastructure they control. This changes both the privacy boundary and the economics of inference.

The issue is not the API. It is who controls the inference engine.

Most production AI systems use an API somewhere. A locally hosted model may still expose an internal API so approved applications can invoke it. The meaningful distinction is between sending enterprise data to a third-party, externally operated inference service and sending it to an inference endpoint running inside the organization’s workstation, data center, private cloud account, or isolated network.

With external model APIs, cost usually grows with tokens, requests, tools, service tiers, and repeated context. Data crosses an organizational boundary on every call. With self-hosted inference, the organization controls the model files, compute, network path, authentication, logs, retention, and shutdown policy. Cost shifts toward owned or reserved capacity. That can make spending more predictable when utilization is steady, although the organization also assumes responsibility for engineering, security, availability, model evaluation, and hardware efficiency.

So, what are your options?

Option One: A Local AI Workstation

A properly sized workstation can run a local language model for an individual professional or a trusted team. Documents, prompts, embeddings, and generated responses can remain on the device or within the local office network. This is often the simplest private starting point for legal analysis, research, product design, financial review, or other focused work involving sensitive material. Suitability is not determined by the size of the company, the office, or the total number of employees. It is determined by how many people need concurrent access, how intensively they use the system, and how much computation their workflows require.

What it offers: a clear privacy boundary and a largely fixed infrastructure cost. There is no per-token invoice each time an employee revises a summary or asks another question. It can also continue operating without an internet connection when the model and required data are already local.

The tradeoff: capacity, availability, and administration remain local responsibilities. A workstation is not automatically an enterprise platform. Disk encryption, operating-system security, identity, backups, patching, physical access, model provenance, and local logs require ongoing administration and monitoring. This does not mean constant manual intervention; many checks, updates, backups, and alerts can be automated. It does mean that a named person or service provider must own them and respond when automation identifies a problem. Performance is limited by available memory and accelerators, and a single machine may not provide the concurrency or resilience needed by a broader organization.

Option Two: Company-Controlled On-Premises Inference

A central server or private AI appliance can provide local-model inference to multiple departments while keeping traffic inside the corporate network. This approach supports shared governance: one approved model catalog, centralized authentication, source-system permissions, monitoring, retention rules, and cost allocation.

What it offers: for organizations with predictable usage, purchased capacity can turn an open-ended variable expense into a planned asset shared across users and workflows. The same infrastructure can serve repeated prompts without a new model-provider charge for every input and output token. The organization can select smaller models for routine work and reserve larger models for the requests that require them.

The tradeoff: the organization pays for the hardware whether it is busy or idle and must plan for both normal demand and peak concurrency. Too little capacity creates queues and slow responses; too much locks capital into underused accelerators. Hardware ages, model requirements change, and production availability may require redundant servers, power, cooling, monitoring, upgrades, backups, vulnerability management, and skilled operational support.

High-performance AI hardware may also exceed the electrical and cooling assumptions of a normal office. The GPU rating alone does not determine the required circuit: the complete system includes processors, memory, storage, fans, and power-supply overhead. Some data-center accelerators are rated at up to 700 watts, while GPU servers may use power supplies rated from 1,100 to 2,400 watts and may require 200–240-volt input to deliver their full capacity. Before installing this class of equipment, the organization should have a qualified professional assess the configured system, circuit capacity, receptacles, ventilation, heat removal, noise, and applicable electrical requirements.

Upgrading is possible when the chassis, power, cooling, slots, processor, and software stack support the next GPU, but it is rarely effortless or inexpensive. A costly server can become a stranded asset if larger models require more GPU memory, usage outgrows its concurrency, or a new accelerator cannot be supported by the existing platform. The organization is not necessarily stuck with unusable equipment—it may continue serving smaller models, retrieval, embeddings, or batch workloads—but resale value and flexibility can be limited. On-premises inference provides greater control, while placing facility, technology-lifecycle, resilience, and utilization risk directly on the company.

Option Three: Self-Hosted Models Inside a Dedicated AWS Environment

Organizations that want cloud flexibility without sending prompts to an external foundation-model API can deploy selected models on GPU-backed compute inside their own AWS environment. Access can be restricted through a customer-controlled virtual private cloud, private subnets, identity policies, encryption keys, security groups, and private service endpoints. There are two principal operating approaches.

Managed inference with SageMaker: the organization supplies or selects the model and deploys it to a SageMaker inference endpoint while AWS manages more of the hosting infrastructure. SageMaker offers real-time, asynchronous, batch, and serverless inference patterns for different workload shapes. Inference components and asynchronous endpoints can be configured to scale to zero in supported configurations, reducing active compute during unused periods. Scaling back from zero introduces a cold-start delay; for some real-time configurations, requests can fail while capacity is being provisioned, whereas asynchronous requests can wait in a queue.

Self-managed GPU compute: the organization can rent GPU-backed EC2 capacity and operate its own model server, containers, scaling logic, and internal inference endpoint. This provides more control over the serving software, model optimization, and lifecycle, but the organization assumes more engineering and operational responsibility. On-Demand EC2 compute is billed while the instance is running. Stopping the instance ends instance-usage billing, although attached storage and certain other resources can continue to incur charges.

AWS documents that SageMaker can be reached through VPC interface endpoints so invocation traffic remains within the AWS network. SageMaker also supports network isolation that prevents deployed inference containers from making outbound network calls. These capabilities allow an organization to create a private inference service while retaining control over the chosen model and its surrounding data path.

This is dedicated in the architectural sense: inference is deployed for the organization inside its controlled environment rather than sent to a shared external model endpoint. It should not be confused with a promise that every underlying cloud component is physically dedicated. Organizations requiring dedicated hardware, strict geographic residency, or sovereign operational controls must select and contract for those properties explicitly.

Capacity must follow the workflow: the required GPU type, GPU count, memory, and running time depend on the model, quantization, context size, computational intensity, response-time target, request volume, and concurrent usage. A small model serving a queued nightly workflow has a different footprint from a large model supporting many simultaneous interactive users.

The tradeoff: AWS converts model-provider token charges into metered infrastructure and operational costs. The organization needs explicit starting and stopping rules so unused GPU capacity does not run 24 hours a day without a business reason. Scheduled startup and shutdown may work for defined business hours; autoscaling may fit variable demand; scale-to-zero may suit interruptible or asynchronous work. Always-on capacity may still be justified when users require immediate responses, but that availability has a measurable price. Idle accelerators, oversized models, slow scale-down, unnecessary redundancy, storage, and data movement can erase the expected savings.

A sound design defines when capacity starts, what demand triggers scale-out, how long it remains idle before scale-in, whether it may reach zero, and what delay users can tolerate when the model reloads. The relevant comparison is total cost per successful business outcome, including compute, storage, operations, support, security, and reliability.

Option Four: Expand as You Grow

An organization does not have to begin with cloud infrastructure. It can start with a workstation, prove the workflow with real users, select and evaluate the local models, connect the required data, and establish its privacy controls before paying for broader capacity. This keeps the first investment contained and produces evidence about actual model performance, token paths, response times, and usage.

When the workstation becomes the constraint—because the organization needs faster inference, more concurrent users, remote access, longer operating hours, or greater resilience—the validated setup can move to AWS. If the local implementation uses portable model files, containers, configuration, retrieval components, prompts, and evaluation tests, much of the same application and model-serving stack can be deployed on SageMaker or GPU-backed EC2 capacity.

What remains the same: the selected model, business workflow, retrieval logic, prompt behavior, evaluation criteria, and core privacy rules can remain substantially consistent. The organization is scaling a working system rather than redesigning an unproven idea.

What changes: the workstation’s purchased GPU is replaced by metered cloud GPU time. AWS also introduces cloud-specific identity, networking, encryption, storage, monitoring, backup, and availability configuration. Portability should therefore be designed from the beginning; moving an unmanaged desktop installation “exactly” as-is is rarely a production architecture.

The cost tradeoff: cloud GPUs are not inexpensive. As a simple illustration, GPU capacity averaging $6.85 per running hour for a dedicated 8:00 a.m. to 8:00 p.m. schedule every day would cost approximately $30,000 per year in compute alone: $6.85 × 12 hours × 365 days. The actual rate varies materially by instance, region, operating system, purchase option, and architecture, and storage, networking, monitoring, and redundancy are additional. Running the same capacity around the clock would approximately double that compute figure.

The expansion decision should therefore have a measurable trigger. Move when workstation response time, concurrency, accessibility, or availability is preventing business value—not simply because cloud deployment appears more sophisticated. Once moved, schedule capacity around real operating hours, stop unused On-Demand instances, or use an appropriate scale-to-zero pattern so the GPU meter does not run without productive work.

Workflow and volume determine the right path

There is no single private deployment pattern for every organization. The correct choice depends on what the workflow does and how the workload behaves. A legal researcher summarizing confidential cases has different requirements from a customer-service platform supporting hundreds of concurrent users. A predictable document-processing queue has different economics from an application with sharp seasonal peaks.

Begin with the workflow. Identify the sensitivity of its data, the consequence of an incorrect answer, the response time users expect, the systems it must reach, and whether it can operate asynchronously. Then measure volume: active users, interactions per hour, tokens per interaction, model calls per task, concurrency, peak-to-average demand, and expected growth. These factors determine the model size, accelerator capacity, availability design, and operating cost.

Volume also changes the financial decision. External per-token inference can be attractive when usage is low or unpredictable because the organization pays only when requests occur. At sustained volume, repeated token charges may exceed the cost of operating dedicated capacity. Self-hosted infrastructure reverses that equation: idle capacity is expensive, but well-utilized capacity can deliver a more predictable cost per interaction. The crossover point must be measured from the organization’s actual workflows, not assumed from a model price sheet.

Use external inference selectively, not reflexively

External model APIs can provide rapid access to leading capabilities without acquiring infrastructure or operating models. They may be appropriate for public information, low-sensitivity workloads, temporary experiments, or specialized tasks whose value justifies the contractual and economic model. The architectural mistake is making an external API the default route for every prompt before classifying the data or measuring the workload.

A practical enterprise design can route requests by sensitivity and complexity. Public or approved low-risk work may use an external service. Confidential knowledge can use a model hosted inside the organization’s controlled environment. Highly restricted information can remain on a local workstation or isolated on-premises system. Model routing should follow policy rather than employee convenience.

Begin with purpose and flow

Document what information enters the capability, why it is needed, where it is processed, who can receive an answer, and how long each artifact remains. Include prompts, retrieved passages, embeddings, generated answers, feedback, logs, backups, and support records—not just the original source files. A useful map follows information from collection to deletion and names an accountable owner at every boundary.

Privacy requires more than keeping the model local

Self-hosting removes one major external dependency, but it does not make the complete system private by itself. A locally running model can still expose data through unrestricted retrieval, excessive logging, weak administration, unsecured backups, exported conversations, compromised endpoints, or employees who can access information beyond their role.

Collect and expose only what the use case needs. Preserve source-system permissions during retrieval. Separate high-sensitivity domains. Encrypt data in transit and at rest. Restrict administrative access. Use short retention where operationally possible, and make deletion reach indexes, caches, logs, and backups according to policy.

Make privacy and cost testable

Translate the architecture into acceptance criteria: restricted data never reaches an external inference endpoint; inference containers cannot make unauthorized outbound connections; source permissions are enforced before retrieval; sensitive fields are masked in lower environments; administrative access is time-bound; exports are logged; and retention jobs produce evidence.

Cost controls should be tested alongside privacy controls. Measure utilization, queue time, cost per accepted answer, energy or cloud compute, idle capacity, and the effect of smaller models. The objective is not to eliminate every external AI service. It is to ensure that the organization chooses when data crosses a boundary and when cost becomes variable.

Privacy by architecture means the safest path is built into the system. For many organizations, that path begins by placing inference close to the data—on a local workstation, on company-owned infrastructure, or within a private cloud environment running models the organization controls.

References: NIST Privacy Framework; NVIDIA H100 power specifications; Dell server power-supply specifications; AWS guidance for connecting to SageMaker AI within a VPC; AWS guidance for internet-free inference containers; AWS SageMaker inference options; AWS SageMaker scale-to-zero guidance; AWS EC2 On-Demand pricing; and AWS EC2 instance lifecycle and billing guidance.