Docs

Local AI models in a deployment

Models in a deployment keep prompts and responses inside the customer's cloud. A deployment there can call models in the customer's own Amazon Bedrock, Vertex AI, or Azure AI Foundry with one AI resource in alien.ts, or serve open-weight models on their GPUs.

This page is for a deployment in the customer's cloud. If your product runs in your own cloud, see Local AI models, which routes requests through the hosted AI Gateway.

alien.ts
import * as alien from "@alienplatform/core"

const llm = new alien.AI("llm").build()

const api = new alien.Worker("api")
  .code({ type: "source", src: "./api", toolchain: { type: "typescript" } })
  .link(llm)
  .permissions("execution")
  .build()

export default new alien.Stack("acme-data-plane")
  .platforms(["aws", "gcp", "azure"])
  .add(llm, "live")
  .add(api, "live")
  .permissions({
    profiles: {
      execution: { llm: ["ai/invoke"] },
    },
  })
  .build()

Architecture diagram. Left: Your side containing Your backend. Right: customer-1's cloud containing api (Worker); Amazon Bedrock.

The workload runs in the customer's cloud and calls the model service there with its own role. Prompts and responses never leave the account.

The workload calls the model service with its own cloud identity:

  • Prompts and responses stay in the customer's cloud.
  • There are no API keys to store or rotate.
  • The customer's data-handling terms with their cloud apply.

For open-weight models you serve yourself, run them in a GPU Container. See Serve open-weight models on GPUs.

Declare the AI resource

ai/invoke grants inference only: no model management and no other writes.

PlatformModel serviceWhat Alien sets up
AWSAmazon BedrockGrants the workload's role access. Nothing else is created.
Google CloudVertex AIEnables the Vertex AI API and grants the role
AzureAzure AI FoundryCreates a Foundry account and model deployments
Local (alien dev)Your provider keyReads OPENAI_API_KEY

Call a model

The workload talks to an embedded gateway that signs requests with its cloud identity:

api/index.ts
import { ai } from "@alienplatform/sdk"

const llm = ai("llm")

// Model IDs differ per cloud and account. Pick from what this deployment can call.
const [model] = await llm.getAvailableModels()

const completion = await llm.chat.completions.create({
  model: model.id,
  messages: [{ role: "user", content: "Summarize this support ticket." }],
})

getAvailableModels() returns only the models this deployment can call right now. Availability depends on the customer's cloud, region, and account.

Some models, such as Claude, need a one-time activation in the customer's cloud, for example accepting the provider's terms in Amazon Bedrock. Open-weight models hosted by the provider, such as GPT-OSS, Llama, Mistral, or Qwen on Amazon Bedrock, work without extra steps.

Zero data retention is a property of the model provider. On Amazon Bedrock, Vertex AI, and Azure AI Foundry, the customer's own agreement with their cloud applies. See AI for the full SDK, including Anthropic-format clients.

Serve open-weight models on GPUs

To run a model the provider doesn't host, or to pin exact weights, serve it yourself in a Container that requests GPUs. This example runs vLLM, which exposes an OpenAI-compatible API:

alien.ts
import * as alien from "@alienplatform/core"

// vLLM serves an open-weight model behind an OpenAI-compatible API on port 8000.
const inference = new alien.Container("inference")
  .code({ type: "image", image: "vllm/vllm-openai:v0.11.0" })
  .command(["vllm", "serve", "Qwen/Qwen3-8B", "--port", "8000"])
  .cpu(8)
  .memory("64Gi")
  .gpu({ type: "nvidia-a100", count: 1 })
  .port(8000)
  .permissions("inference")
  .build()

export default new alien.Stack("acme-inference")
  .platforms(["aws", "gcp", "azure"])
  .add(inference, "live")
  .permissions({ profiles: { inference: {} } })
  .build()

.gpu() takes a GPU type and a count from 1 to 8. Alien picks a matching GPU machine type in each cloud: p4d on AWS, a2 on Google Cloud, and NC or ND series on Azure.

The customer's account needs GPU quota in the chosen region.

To reach the server from other workloads, see Networking. The model weights download from the internet when the container starts, unless you bake them into the image.

Hosted AI Gateway

A hosted product that doesn't deploy code into the customer's cloud calls the customer's models through Alien's hosted AI Gateway instead. Prompts then pass through the gateway on the way to the customer's provider.

Choose the AI resource when prompts must stay inside the customer's cloud. See Local AI models.

Test it

Run the stack locally first. With alien dev, the AI resource reads OPENAI_API_KEY, so the same handler code runs on your machine:

export OPENAI_API_KEY=sk-...
alien dev

Then deploy to a cloud account you control, as your own test customer:

alien onboard "Test customer" --external-id org_test

Open the printed link and run initial setup. Then, in each cloud you offer:

  1. Confirm getAvailableModels() lists the models you need.
  2. Activate any that need it, such as accepting Claude's terms in Amazon Bedrock.
  3. Send a real request.

For GPU containers, confirm the account has GPU quota in the region before initial setup.

Next

Requests and models

How the AI Gateway picks the customer, model, provider, and protocol for each request, and how to read its logs and usage.

Overview

Give each customer a single-tenant deployment in its own AWS account, Google Cloud project, or Azure resource group, created and deployed by Alien in your cloud.

On this page

Declare the AI resourceCall a modelServe open-weight models on GPUsHosted AI GatewayTest itNext