RECOVERED — GPT-5.6 Pro dive: TPU Inference Economics Analysis

RECOVERED from ChatGPT history — dispatcher timer died before filing (commissioned 2026-07-15, Targeted 4). Recovered from the ChatGPT conversation history after the dispatching session failed. No new prompt was sent; this is the verbatim assistant report as it already existed in-account. The dispatching (pro-dive-fleet) session's in-session MCP request registry expired unread, but the ChatGPT conversation itself survived.

Citation anchors removed: citation anchors from the original interface were removed for publication; original preserved in the archive copy.

Provenance routing: any site change this dive justifies must enter through the update queue (research/update-queue.md), not via direct edits.

You are conducting a rigorous source-hunting deep dive for a quantitative model of frontier-LLM inference serving economics (cost per token, and margin at list prices). I need the strongest available PUBLIC anchors for Google TPU inference/serving economics — or a rigorous, well-documented finding that no such public anchor exists.

SCOPE:
- TPU v5e, v6e (Trillium), and v7 (Ironwood) used for LLM *inference/serving*, not training.
- Published inference throughput on frontier-class models: MLPerf Inference submissions (v4.x/v5.x rounds) on TPU; Google's own published serving claims (Gemini serving on TPU, JetStream / MaxText / vLLM-on-TPU benchmarks); and any third-party TPU inference benchmarks (via GKE, JetStream, SGLang, etc.). Prefer figures expressed as tokens/sec/chip (or per host) on a named model at a named precision.
- GCP TPU pricing: on-demand $/chip-hour and $/host-hour, 1-yr and 3-yr committed-use discounts, spot/preemptible rates, for v5e / v6e / v7 where published.
- Power and TCO comparisons vs NVIDIA H100/H200/GB200 (perf-per-watt, perf-per-dollar), from Google or credible third parties.
- Any disclosures about Gemini serving cost or TPU fleet economics.

GOAL: a candidate quantitative anchor for cost-per-token on TPU — ideally (tokens/sec/chip on a named model at a named precision) paired with ($/chip-hour) so a $/token can be derived. If no such public serving anchor exists (e.g. Google publishes peak FLOPS and rental pricing but no real-model serving throughput), say so explicitly and precisely — a rigorous negative result is exactly as valuable here. TPU is currently the single weakest-anchored platform in the model, so precision about what is and is not publicly disclosed matters more than breadth.

Structure your findings as dated claims. For each: the claim, primary source (URL + date), your assessment of source quality (vendor claim / third-party benchmark / production disclosure / analyst estimate), and whether it could serve as a quantitative anchor (tokens/sec/chip, $/hr, $/token). If no public anchor exists for a platform, say so explicitly as a finding — a rigorous negative result is wanted. End with a numbered list of your 5 most load-bearing sources.

Bottom line

A public cost-per-output-token anchor now exists for TPU v5e, v6e, and v7/Ironwood. The evidence is materially better than a model built only from peak FLOPS:

The strongest frontier-scale public anchor I found is Qwen3-Coder-480B-A35B-Instruct-FP8 on four Ironwood chips:

• 1,024 input / 8,192 output workload
• 2,075.47 aggregate output tokens/s, or 518.86 tokens/s/chip
• Current four-chip on-demand list price: $48/hour
• Derived accelerator-rental floor: $6.42 per million output tokens
• At the current three-year commitment price: $2.89 per million output tokens

That is a real named-model, named-precision, named-topology serving benchmark. It is nevertheless a maximum-throughput vendor reference run at infinite offered load, not a production cost disclosure or a latency-constrained serving result.

The rigorous negative finding is narrower but important:

There is no public basis for calculating Gemini gross margin from TPU economics. Public TPU rental prices are external customer list prices, not Google’s internal TPU fleet cost, and Google has not published a mapping from a named Gemini API SKU to TPU generation, topology, precision, output throughput, traffic mix, utilization, or supporting-model overhead.

Calculation convention

For a benchmark reporting aggregate output-token throughput, I use:

[ \text{Accelerator $ per 1M output tokens}

\frac{ N_{\text{chips}}\times $/{\text{chip-hour}} }{ \text{output tokens/sec}\times 3600 } \times 10^6 ]

This is an accelerator-rental floor, not fully loaded serving COGS. It excludes:

• Latency/SLO headroom and replica redundancy
• Idle time and utilization losses
• Host CPU, networking, storage, GKE, load balancing, and observability
• Compilation, model loading, failover, and maintenance
• Ancillary models, safety classifiers, routers, retrieval, and speculative-draft models
• Engineering, support, and datacenter overhead beyond what is embedded in the cloud rental price

Where a benchmark has a specified input length, its reported output-token rate implicitly includes the cost of processing those inputs. It should therefore be read as cost per output token for that particular input/output workload, not as a universal decode-only cost.

All calculated prices below use the current lowest listed US price for each generation as of July 15, 2026. Pairing a current price with an older benchmark is a current repricing of the benchmark, not necessarily the economics prevailing when it was published.

Current public GCP TPU pricing

Primary source: https://cloud.google.com/tpu/pricing — accessed 2026-07-15

Prices are USD per chip-hour. Google explicitly says the TPU price table is expressed per chip-hour even where billing interfaces may display VM-hours.

Spot/preemptible: Google’s current public TPU pricing page does not provide a stable numeric spot rate. It says Spot prices are dynamic and can change as frequently as once every 30 days. DWS Flex is a separately quoted queue-based rate and should not be relabeled as Spot. Therefore, a timeless numeric “TPU spot price” is not available from the public static price card.

Current topology documentation says:

• v5e has eight chips per host and common one-, four-, and eight-chip serving slices.
• v6e also has eight chips per host; v6e-4 is a half-host and v6e-8 is a full host optimized for inference.
• Current Ironwood standard topology documentation describes four-chip full-host VMs.

Best candidate quantitative anchors

The table uses output-token throughput, not total input-plus-output throughput.

The v5e figures come from Google’s 2024 JetStream and Hex-LLM publications; v6e figures come from Google’s 2025–2026 JetStream/vLLM work and an independent Gemma 4 paper; Ironwood figures come from Google’s 2026 AI Hypercomputer reference recipes.

Dated findings

1. July 15, 2026 — Public price legs exist for all three generations

Claim. GCP publishes stable on-demand, DWS Flex, DWS Calendar, one-year, and three-year per-chip-hour rates for v5e, v6e, and Ironwood. Ironwood is currently $12/chip-hour on demand; v6e is $2.70; v5e is $1.20. Three-year rates are $5.40, $1.22, and $0.54 respectively.

Primary source and date. https://cloud.google.com/tpu/pricing — accessed 2026-07-15

Source quality. Official commercial price card; strongest possible source for external list-price economics.

Quantitative-anchor status. Yes, for the $/hour leg. It does not disclose Google’s internal fleet cost. Spot remains a dynamic price rather than a stable published number.

2. April 10, 2024 — JetStream supplied a clean v5e INT8 throughput-and-cost anchor

Claim. Google reported JetStream on a v5e-8 slice with continuous batching and INT8 weights, activations, and KV cache. With maximum input and output lengths of 1,024 tokens, it reported:

• Llama 2 13B: 3,938 output tok/s
• Gemma 7B: 4,030 output tok/s
• Llama 2 7B: 4,783 output tok/s

The original publication also gave an explicit three-year-CUD estimate of approximately $0.30 per million output tokens for Gemma 7B and Llama 2 13B and $0.25 for Llama 2 7B.

Primary source and date. https://cloud.google.com/blog/products/compute/accelerating-ai-inference-with-google-cloud-tpus-and-gpus — 2024-04-10

Source quality. Vendor benchmark. The workload, slice size, continuous-batching mode, and quantization are unusually well specified. It is not independently reproduced and represents a maximum-throughput operating point.

Quantitative-anchor status. Yes. Repricing Llama 2 13B at today’s rates gives $0.677/M output tokens on demand and $0.305/M under the current three-year rate. This is probably the cleanest v5e anchor, although not a frontier-size model.

3. July 26, 2024 — Hex-LLM supplied the strongest public v5e 70B anchor

Claim. Google reported Hex-LLM on eight v5e chips using a ShareGPT variable-length workload. At its highest-throughput point:

• Gemma 7B reached 6,250 output tok/s.
• INT8 weight-quantized Llama 2 70B reached 1,510 output tok/s.

Google separately reported low-load per-output-token latencies, but the peak-throughput and minimum-latency figures are different points on the load curve and must not be combined into one serving configuration.

Primary source and date. https://cloud.google.com/blog/products/ai-machine-learning/hex-llm-on-tpus-in-vertex-ai-model-garden/ — 2024-07-26

Source quality. Vendor benchmark with a realistic variable-length dataset and a named 70B model. The source specifies INT8 weight quantization but does not fully specify activation and KV-cache precision.

Quantitative-anchor status. Yes. At current list pricing, 1,510 tok/s on a $9.60/hour v5e-8 slice implies:

• On demand: $1.766/M output tokens
• DWS Flex: $0.883/M
• Three-year: $0.795/M

This is the best simple public v5e anchor for a large dense model.

4. 2024–2025 MLPerf rounds — no frontier TPU LLM tokens/sec anchor

Claim. Google did submit a TPU v5e LLM result to MLPerf Inference v4.0, but it is not a strong cost-per-token anchor:

• System: four TPU v5e chips, one node, SAXML.
• Model: GPT-J 6B.
• Offline result: 10.1502 samples/s.
• Server result: 7.20 completed samples/s.
• The reference workload permits between 30 and 128 generated tokens and uses beam size four.

The variable generated length means samples/s cannot be converted exactly to output tokens/s from the summary result. GPT-J 6B also does not meet the requested frontier-class criterion.

Primary sources.

• https://github.com/mlcommons/inference_results_v4.0/blob/main/closed/Google/systems/tpu_v5e_x4.json
• https://github.com/mlcommons/inference_results_v4.0/blob/main/closed/Google/results/tpu_v5e_x4/gptj-99/offline/performance/run_1/mlperf_log_summary.txt
• https://github.com/mlcommons/inference_results_v4.0/blob/main/closed/Google/results/tpu_v5e_x4/gptj-99/server/performance/run_1/mlperf_log_summary.txt
• https://github.com/mlcommons/inference/blob/v4.0/language/gpt-j/backend.py

Source quality. Auditable MLCommons submission; very high benchmark integrity, but the reported unit and workload are unsuitable for exact $/output-token conversion.

Quantitative-anchor status. No exact $/token anchor. It can support a cost-per-sample calculation—about $0.000131 per offline sample at current on-demand v5e pricing—but not exact cost per generated token.

In the Google closed-submission directories inspected:

• v4.1’s TPU entries were for Stable Diffusion XL rather than an LLM.
• v5.0’s Trillium submission was also SDXL.
• v5.1’s Google results were H200/B200 systems, with no TPU LLM submission.

Therefore, MLPerf Inference v4.0 through v5.1 does not provide the desired frontier TPU LLM tokens/sec/chip anchor.

5. May 14, 2024 — Trillium power-efficiency claim is not a serving power anchor

Claim. At the Trillium announcement, Google claimed 4.7 times the peak compute per chip of v5e and “over 67%” greater energy efficiency. It also said Gemini 1.5 Flash, Imagen 3, and Gemma 2 were trained and served using TPUs, but did not attribute a named model’s production serving rate to a specified Trillium topology.

Primary source and date. https://cloud.google.com/blog/products/compute/introducing-trillium-6th-gen-tpus — 2024-05-14

Source quality. Vendor generation-level specification and production-use statement.

Quantitative-anchor status. No. It does not state the LLM, precision, tokens/s, watts, utilization, or latency corresponding to the energy-efficiency claim. It is a useful architectural prior, not a serving tokens/J measurement.

6. May 9, 2025 — JetStream supplied high-throughput v6e anchors

Claim. Google reported the following MaxText/JetStream output throughput, with maximum input and output lengths of 1,024:

• Llama 2 70B on eight Trillium chips: 6,389 output tok/s
• Mixtral 8×7B on four Trillium chips: 4,128 output tok/s

The article states that JetStream uses the same inference stack Google uses to serve Gemini.

Primary source and date. https://cloud.google.com/blog/products/compute/ai-hypercomputer-inference-updates-for-google-cloud-tpu-and-gpu — 2025-05-09

Source quality. Vendor benchmark with named models, chip counts, and workloads. The notable omission is precision/quantization. Load level, TTFT, and output latency are also not reported.

Quantitative-anchor status. Yes, with a precision caveat.

For Llama 2 70B:

• On-demand v6e-8: $0.939/M output tokens
• DWS Flex: $0.470/M
• Three-year: $0.424/M

This is the lowest public v6e 70B accelerator-floor estimate I found, but it should be modeled as an optimistic maximum-throughput case because the precision and latency point are undisclosed.

The same article reported 1,703 tok/s for Llama 3.1 405B and a threefold inference-per-dollar improvement over v5e. However, it did not disclose the Trillium chip count/topology or precision for that figure. Consequently, the 405B result cannot be converted into $/token and is not a valid frontier anchor despite being the most frontier-relevant model in the article.

7. April 16, 2026 — Google’s RPA paper supplies exact BF16 v6e rates, but as a software-progress figure

Claim. A Google-authored paper on Ragged Paged Attention reports up to 86% memory-bandwidth utilization in decode and 73% model-FLOP utilization in prefill on Ironwood. A fixed-workload software-progress figure also reports, for 1,800 input and 128 output tokens in BF16:

• Llama 3 8B on v6e-1: 11.6 requests/s as of August 14, 2025
• Llama 3 70B on v6e-8: 7.65 requests/s as of August 14, 2025

Because every request has 128 output tokens, these imply:

• 8B: 1,484.8 output tok/s on one chip
• 70B: 979.2 output tok/s on eight chips

Primary source and date. https://arxiv.org/abs/2604.15464 — submitted 2026-04-16

Source quality. Vendor-authored research preprint. Precision, model, topology, and fixed output length are specified; the chart is primarily a history of software improvements rather than a complete latency-qualified serving benchmark.

Quantitative-anchor status. Yes, as a conservative technical anchor. Current on-demand accelerator floors are approximately $0.505/M for the 8B run and $6.13/M for the 70B run. The large difference from the 2025 JetStream 70B figure illustrates how strongly TPU $/token depends on serving engine, quantization, batching, and benchmark workload.

The 86%/73% utilization results are not themselves cost anchors because they do not disclose end-to-end tokens/s for the corresponding Ironwood runs.

8. May 1, 2026 — An official vLLM tutorial supplied an exact W8A8 v6e anchor

Claim. Google’s vLLM-on-TPU tutorial used:

• Qwen3-32B
• Four v6e chips
• INT8 weights and activations, W8A8
• 1,000 random requests
• 1,800 input and 128 output tokens
• Asynchronous scheduling

It reported:

• 1,396.35 output tok/s
• 21,032.45 total input-plus-output tok/s
• Mean TTFT of 44,362 ms
• P99 TTFT of 88,584 ms

Primary source and date. https://cloud.google.com/blog/ja/products/infrastructure/lets-try-llm-inference-with-cloud-tpu-and-vllm — 2026-05-01

Source quality. Official, reproducible vendor tutorial with a named model, exact quantization, topology, workload, throughput, and latency metrics.

Quantitative-anchor status. Yes—but specifically for high-throughput/batch economics.

Using output tokens:

• On demand: $2.148/M
• DWS Flex, matching the tutorial’s procurement mode: $1.074/M
• One-year: $1.504/M
• Three-year: $0.971/M

Using the published total-token throughput instead would produce $0.143/M total processed tokens on demand, but that is a different denominator and should not be compared directly with API output-token prices.

The 44-second mean TTFT makes this unsuitable as a low-latency interactive serving anchor.

9. May–June 2026 — Best independent TPU-versus-H100 serving comparison

Claim. An independent preprint compared Gemma 4 31B on a v6e-8 system with the same model on two H100 GPUs. At comparable hourly rental prices, reported output throughput included:

The paper’s own cost table put the TPU at approximately $4.27/M output tokens for the shortest workload and $4.95/M at 4,096/512, versus $4.13/M and $8.44/M respectively for the H100 pair.

Primary source and date. https://arxiv.org/abs/2605.25645 — submitted 2026-05-25; version 3 dated 2026-06-05

Source quality. Independent third-party benchmark/preprint with scripts and multiple sequence lengths. It is not peer-reviewed or audited, and the runs used only 30 prompts per QPS point.

A significant comparability caveat is that the TPU used FP8 KV cache while the GPU path used BF16 KV cache, and the two paths used different software versions. Thus it is not a perfect hardware-only comparison.

Quantitative-anchor status. Yes. At the current official $21.60/hour v6e-8 rate:

• 1,403 tok/s: $4.28/M output tokens
• 1,206 tok/s: $4.98/M
• At the current three-year rate, those become approximately $1.93/M and $2.25/M

This is the strongest independent evidence I found for TPU serving performance per dollar versus NVIDIA. It compares against H100, not H200 or GB200.

10. April 9, 2025 — Ironwood launch claims alone were not a serving anchor

Claim. Google announced Ironwood as an inference-oriented TPU, with 2,307 BF16 TFLOPS/chip, 4,614 FP8 TFLOPS/chip, 192 GB HBM, and 7.38 TB/s HBM bandwidth. It claimed twice Trillium’s performance per watt.

Primary sources and dates.

• https://blog.google/innovation-and-ai/infrastructure-and-cloud/google-cloud/ironwood-tpu-age-of-inference/ — 2025-04-09, updated 2025-04-23
• https://docs.cloud.google.com/tpu/docs/tpu7x — last updated 2026-07-10

Source quality. Official vendor hardware specifications and peak-efficiency claim.

Quantitative-anchor status. No, by themselves. The performance-per-watt footnote is based on peak FP8 FLOPS divided by TDP, not measured end-to-end LLM output tokens per joule. Peak FLOPS, bandwidth, and rental price are insufficient to infer serving $/token without a real-model utilization and serving benchmark.

This was the correct negative conclusion at Ironwood’s launch. The result changed only when Google began publishing the 2026 reference-recipe benchmarks below.

11. April 24, 2026 — Ironwood Qwen3-32B per-chip benchmark

Claim. Google’s AI Hypercomputer recipe reports Qwen3-32B on Ironwood with:

• FP8 E4M3 weights and activations
• FP8 KV cache
• Tensor parallelism across the chip’s two TensorCores
• Infinite request rate, 320 prompts, concurrency 320
• Output throughput per chip:
  • 1K/1K: 6,556.72 tok/s
  • 1K/8K: 3,959.56 tok/s
  • 8K/1K: 1,382.18 tok/s

Primary source and date. https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/Qwen3-32B — throughput update merged 2026-04-24

The relevant commit explicitly updated the per-chip values to 6,556.72, 3,959.56, and 1,382.18 tok/s.

Source quality. Google-controlled vendor reference recipe using the open InferenceX client pinned to a specific commit. Highly reproducible, but not independent and clearly optimized for saturation throughput.

Quantitative-anchor status. Yes, on a normalized per-chip basis.

For the 1K/1K result:

• On demand: $0.508/M output tokens
• DWS Flex: $0.254/M
• Three-year: $0.229/M

For 1K/8K, the corresponding values are approximately $0.842/M, $0.421/M, and $0.379/M.

Material topology caveat. The recipe uses a tpu7x-standard-1t one-chip configuration, while current TPU7x documentation describes standard four-chip full-host VMs. This may reflect a preview or superseded machine type. The reported per-chip throughput remains a useful normalized engineering result, but a current customer may face a four-chip minimum bill. Unless four independent model replicas can fill that VM, the actual deployment cost can exceed the simple per-chip estimate by as much as fourfold.

12. May 4, 2026 — Ironwood frontier-scale Qwen3-Coder anchor

Claim. Google’s Ironwood vLLM recipe reports:

• Qwen3-Coder-480B-A35B-Instruct-FP8
• Four TPU7x chips, eight TensorCores
• Tensor parallelism eight
• FP8 KV cache
• Expert parallelism
• InferenceX at infinite offered load

Published rates:

Primary source and date. https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/Qwen3-Coder-480B-A35B — values updated 2026-05-04

The dated commit records both the aggregate and per-chip figures.

Source quality. Vendor reference benchmark using an open benchmark client. The model, checkpoint precision, topology, tensor/expert parallelism, input/output lengths, and concurrency are disclosed. No achieved TTFT, TPOT, or latency percentiles are supplied with the final table.

Quantitative-anchor status. Yes; this is the strongest public frontier-scale TPU cost anchor found.

For 1K input / 8K output:

• Four-chip on-demand cost: $48/hour
• Throughput: 2,075.47 output tok/s
• On-demand floor: $6.424/M output tokens
• DWS Flex floor: $3.212/M
• One-year/DWS Calendar floor: $4.497/M
• Three-year floor: $2.891/M

The fact that the 1K/8K workload slightly outperforms 1K/1K is plausible for a saturation benchmark because the longer decode phase amortizes prefill and scheduling overhead. It should not be interpreted as an interactive single-user rate.

13. Current by July 15, 2026 — corroborating Ironwood Qwen3.5 result

Claim. A second frontier-scale recipe reports Qwen3.5-397B-A17B-FP8 on four Ironwood chips:

• 1K/8K: 3,956.67 aggregate tok/s, or 989.17 tok/s/chip
• 8K/1K: 1,998.18 aggregate tok/s, or 499.55 tok/s/chip
• Maximum concurrency 128
• FP8 model checkpoint
• Four-chip, eight-TensorCore setup

Primary source and date. https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/Qwen3.5-397B — current recipe accessed 2026-07-15; benchmark image/build dated 2026-06-26

Source quality. Vendor reference recipe; reproducible but not independent or latency-qualified.

Quantitative-anchor status. Yes. The 1K/8K result implies approximately:

• On demand: $3.370/M output tokens
• DWS Flex: $1.685/M
• Three-year: $1.516/M

This corroborates that Ironwood’s public serving evidence is no longer limited to small dense models. However, the difference between this result and Qwen3-Coder also shows that active parameters, expert routing, architecture, kernels, and software maturity dominate simple parameter-count comparisons.

14. August 21, 2025 — strongest real-production Gemini disclosure, but not a cost/token anchor

Claim. Google’s production environmental-impact paper measured the median Gemini Apps text prompt in May 2025 at a total of 0.24 Wh, broken down as:

• Active accelerators: 0.14 Wh
• Host CPU and DRAM: 0.06 Wh
• Idle capacity allocation: 0.02 Wh
• Datacenter overhead: 0.02 Wh

The methodology covers the full production serving stack, including supporting models and machines. Google also reported a 33-fold reduction in energy per median prompt over the preceding year.

Primary source and date. https://arxiv.org/abs/2508.15734 — submitted 2025-08-21

Source quality. Primary production disclosure based on Google’s actual fleet, rather than a laboratory benchmark. This is the strongest public evidence about full-stack Gemini serving resource use.

Quantitative-anchor status. No $/token anchor. The paper does not disclose:

• Which Gemini model or mixture of models served the median prompt
• Input and output token lengths
• TPU versus GPU mix
• TPU generation or topology
• Tokens/s/chip
• Electricity price or hardware amortization
• Utilization and reservation economics in dollar terms

The 0.24 Wh figure can support an energy-per-request model. It cannot be converted rigorously into $/token or gross margin without inventing token counts, hardware assignment, and internal cost assumptions.

Power and TCO comparison verdict

What is public

Google publishes generation-to-generation peak/specification claims:

• Trillium: over 67% greater energy efficiency than v5e.
• Ironwood: approximately twice Trillium’s performance per watt.
• Ironwood’s cited comparison is peak FP8 FLOPS divided by TDP, not LLM serving tokens/J.

Google’s v5e and v6e spec pages also expose a relative “expected performance/TCO” index, with v5e shown below v6e, but the index is not tied to a named model, serving workload, price basis, latency target, or disclosed methodology. It is not usable as a quantitative serving-TCO anchor.

The independent Gemma 4 study provides a useful performance-per-rental-dollar comparison against two H100s, especially at long context, but it does not measure wall power.

Rigorous negative finding

I found no public, credible, apples-to-apples benchmark that simultaneously provides:

• The same named LLM
• The same model and KV-cache precision
• The same input/output distribution
• Comparable TTFT/TPOT or another latency SLO
• Measured wall or accelerator power
• TPU v5e, v6e, or v7 versus H200 or GB200

Therefore:

• There is a public TPU-versus-H100 performance/$ anchor.
• There is no comparable public TPU-versus-H200/GB200 measured tokens/J anchor.
• Google’s Ironwood and Trillium perf/W numbers should remain classified as vendor peak/spec claims, not serving measurements.
• Any modeled TPU versus GB200 energy advantage would presently be assumption-driven.

Gemini serving-cost and margin verdict

A valid model-level API margin calculation would require:

[ \text{Gross margin}

1- \frac{\text{fully loaded serving COGS per token}} {\text{API list revenue per token}} ]

Even the narrower accelerator-only approximation,

[ 1- \frac{\text{TPU rental floor per output token}} {\text{Gemini API output list price}}, ]

is not defensible because the numerator and denominator would refer to different things:

1. The public TPU benchmarks are predominantly Llama, Gemma, Qwen, and Mixtral self-hosting runs.
2. Google does not publicly map a named Gemini API SKU to a TPU generation, chip count, precision, serving engine, or measured output tokens/s.
3. GCP’s TPU rental price is the customer-facing retail price, not Google’s internal depreciation, power, networking, or datacenter cost.
4. Gemini production includes routers and supporting models that are absent from the open-model accelerator benchmarks.
5. API traffic has a latency and utilization profile that is not represented by request-rate=inf saturation tests.

Accordingly:

No rigorous public Gemini cost-per-token or list-price gross-margin anchor exists.

The production 0.24 Wh/prompt disclosure is valuable, and JetStream’s use in Gemini serving establishes technological relevance, but neither closes the missing token, model, topology, utilization, and dollar-cost fields.

Recommended modeling treatment

For a quantitative serving-economics model, I would represent TPU with three separate cases rather than one point estimate:

1. Saturation accelerator floor

Use the published output throughput and current per-chip rental rate directly. This captures the best case at effectively full utilization.

Best examples:

• v5e Llama 2 70B: $1.77/M on demand
• v6e Llama 2 70B: $0.94/M on demand, with a major precision caveat
• v7 Qwen3-Coder-480B-A35B: $6.42/M on demand
• v7 Qwen3.5-397B-A17B: $3.37/M on demand

2. Commitment-price accelerator floor

Use DWS Flex or a three-year commitment according to workload interruptibility and capacity certainty. Do not describe DWS as Spot.

Examples:

• Ironwood Qwen3-Coder: $3.21/M DWS, $2.89/M three-year
• v6e Qwen3-32B: $1.07/M DWS, $0.97/M three-year
• v5e Llama 2 70B: $0.88/M DWS, $0.79/M three-year

3. Production-adjusted scenario

Apply explicit, separately modeled multipliers for:

• Achieved utilization
• Latency/SLO headroom
• Redundancy and failover
• Host/infrastructure overhead
• Ancillary model overhead
• Effective price versus public cloud list price

For example, if a service realizes only 40% of saturation throughput after SLO headroom and utilization, the accelerator component is 2.5 times the benchmark floor before adding non-accelerator costs. This should be an explicit scenario assumption, not silently embedded in the TPU benchmark.

The public evidence supports the accelerator-floor cases. It does not identify a single empirically justified production-utilization multiplier for Gemini.

Five most load-bearing sources

1. Google Cloud TPU pricing — current per-chip on-demand, DWS, and committed-use price legs for v5e, v6e, and Ironwood. https://cloud.google.com/tpu/pricing — accessed 2026-07-15.

2. Ironwood Qwen3-Coder-480B-A35B vLLM recipe — best public frontier-scale TPU benchmark: named FP8 model, four-chip topology, explicit input/output workloads, and per-chip throughput. https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/Qwen3-Coder-480B-A35B — values updated 2026-05-04.

3. AI Hypercomputer inference updates for Trillium — Llama 2 70B and Mixtral throughput with explicit chip counts; also establishes JetStream’s relationship to Gemini serving. https://cloud.google.com/blog/products/compute/ai-hypercomputer-inference-updates-for-google-cloud-tpu-and-gpu — 2025-05-09.

4. Original JetStream v5e publication — unusually clean specification of v5e-8, INT8 weights/activations/KV cache, model, context limits, output throughput, and contemporary cost per million tokens. https://cloud.google.com/blog/products/compute/accelerating-ai-inference-with-google-cloud-tpus-and-gpus — 2024-04-10.

5. Independent Gemma 4 31B v6e-versus-H100 study — strongest non-Google throughput-per-dollar check, including sequence-length sensitivity and a direct H100 comparison. https://arxiv.org/abs/2605.25645 — submitted 2026-05-25, v3 2026-06-05.