BLOG
3 min read

DeepSeek v4.1 flash: Azure Foundry benchmark results

In a benchmark run by the Azure Foundry team, encrypted Blackbox delivers 3.02× Azure’s mean throughput and 75.1% lower mean query latency, with close quality scores.

The Azure Foundry team ran this benchmark of DeepSeek v4.1 flash (xhigh), comparing Azure with regular and encrypted inference on BLACKBOX AI. In the reported runs, Blackbox’s encrypted path delivered 245.88 tokens/s mean generation throughput and 30.43 seconds mean query latency, versus 81.47 tokens/s and 122.15 seconds on Azure.

3.02×
mean throughput vs. Azure
75.1%
lower mean query latency
0.379 s
encrypted mean TTFT

The performance figures compare DeepSeek v4.1 flash (xhigh) across Azure, regular Blackbox, and encrypted Blackbox. Orange consistently identifies the encrypted path. The results cover 230 requests on Azure, 179 on regular Blackbox, and 330 on encrypted Blackbox.

Three times the throughput

Regular Blackbox averaged 283.74 tokens/s. Encrypted Blackbox averaged 245.88 tokens/s, compared with 81.47 tokens/s on Azure: 3.48× and 3.02× Azure’s mean throughput, respectively.

Figure 1. More generated tokens per second

DeepSeek v4.1 flash (xhigh)

Higher is better
Figure 1. More generated tokens per second
EntryValue
Mean Azure81.47
Mean Blackbox regular283.74
Mean Blackbox encrypted245.88
AzureBlackbox regularBlackbox encrypted
Meantokens/s
81.47
Azure
283.74
Blackbox Regular
245.88
Blackbox Encrypted

Source: Azure Foundry team benchmark summary. Azure n=230; Blackbox regular n=179; encrypted n=330. All panels share the same zero-based scale.

Encrypted throughput was 13.3% below regular Blackbox. The comparison with Azure still shows a threefold increase in mean generation throughput.

TTFT at 0.379 seconds

Mean time to first token was 0.379 seconds on encrypted Blackbox, 0.399 seconds on regular Blackbox, and 10.71 seconds on Azure. The panels below compare the encrypted path with Azure at the mean, median, and P99.

Figure 2. TTFT

DeepSeek v4.1 flash (xhigh)

Lower is better
Figure 2. TTFT
EntryValue
Mean Azure10.71
Mean Blackbox encrypted0.379
P50 Azure7.437
P50 Blackbox encrypted0.378
P99 Azure51.385
P99 Blackbox encrypted0.440
AzureBlackbox encrypted
Meanseconds
10.71
Azure
0.379
Blackbox Encrypted
P50seconds
7.437
Azure
0.378
Blackbox Encrypted
P99seconds
51.385
Azure
0.440
Blackbox Encrypted

Source: Azure Foundry team benchmark summary. Azure n=230; Blackbox regular n=179; encrypted n=330. All panels share the same zero-based scale.

At P99, Azure’s TTFT was 51.385 seconds, versus 0.440 seconds on encrypted Blackbox. The bars use a common linear scale; the exact labels keep the sub-second values readable. All five reported statistics are retained in the full results below.

Shorter queries, including the tail

Mean query latency was 122.15 seconds on Azure, 26.82 seconds on regular Blackbox, and 30.43 seconds on encrypted Blackbox. The encrypted mean was 75.1% below Azure’s and 13.5% above regular Blackbox’s.

Figure 3. Lower latency from median to P99

DeepSeek v4.1 flash (xhigh)

Lower is better
Figure 3. Lower latency from median to P99
EntryValue
P50 Azure94.87
P50 Blackbox regular25.49
P50 Blackbox encrypted30.94
P95 Azure290.03
P95 Blackbox regular49.97
P95 Blackbox encrypted54.28
P99 Azure414.70
P99 Blackbox regular55.99
P99 Blackbox encrypted62.33
AzureBlackbox regularBlackbox encrypted
P50seconds
94.87
Azure
25.49
Blackbox Regular
30.94
Blackbox Encrypted
P95seconds
290.03
Azure
49.97
Blackbox Regular
54.28
Blackbox Encrypted
P99seconds
414.70
Azure
55.99
Blackbox Regular
62.33
Blackbox Encrypted

Source: Azure Foundry team benchmark summary. Azure n=230; Blackbox regular n=179; encrypted n=330. All panels share the same zero-based scale.

At P99, encrypted Blackbox completed queries in 62.33 seconds, compared with 414.70 seconds on Azure. Percentiles summarize separate runs; they do not pair individual requests.

Blackbox outperforms on throughput, speed and security with no quality loss

The speed gains come with close reported quality scores on the three shared accuracy datasets. Encrypted Blackbox is 2.4 percentage points above Azure on MuSR, 0.8 points below on tau2-telecom, and 0.2 points below on ChemBench. These results support comparable observed quality, rather than a claim that every score is unchanged.

Figure 4. Quality remains close across shared datasets

DeepSeek v4.1 flash (xhigh)

Higher is better
Figure 4. Quality remains close across shared datasets
EntryValue
MuSR Azure71.2%
MuSR Blackbox regular72.8%
MuSR Blackbox encrypted73.6%
tau2-telecom Azure98.2%
tau2-telecom Blackbox regular98.2%
tau2-telecom Blackbox encrypted97.4%
ChemBench Azure70.6%
ChemBench Blackbox regular70.7%
ChemBench Blackbox encrypted70.4%
AzureBlackbox regularBlackbox encrypted
MuSRaccuracy (%)
71.2%
Azure
72.8%
Blackbox Regular
73.6%
Blackbox Encrypted
tau2-telecomaccuracy (%)
98.2%
Azure
98.2%
Blackbox Regular
97.4%
Blackbox Encrypted
ChemBenchaccuracy (%)
70.6%
Azure
70.7%
Blackbox Regular
70.4%
Blackbox Encrypted

Source: supplied quality summaries. Azure uses Correct/Dataset. All three panels use a 0–100% scale; FrontierScience uses a different metric and is shown separately.

Each dataset has its own panel, with the same serving-path order and 0–100% scale. Against regular Blackbox, encrypted accuracy differs by +0.8 points on MuSR, −0.8 on tau2-telecom, and −0.3 on ChemBench.

FrontierScience: partial-credit scores

Blackbox reports a mean score including partial credit of 59.1% for regular inference and 57.7% for encrypted inference, with 74 and 71 fully correct answers out of 160 examples.

Figure 5. FrontierScience scores differ by 1.4 points

DeepSeek v4.1 flash (xhigh)

Higher is better
Figure 5. FrontierScience scores differ by 1.4 points
EntryValue
Blackbox regular59.1%
Blackbox encrypted57.7%
Blackbox regularBlackbox encrypted
Reported scoremean score (%)
59.1%
Blackbox Regular
57.7%
Blackbox Encrypted

Source: supplied Blackbox quality summary. 160 examples per path. Mean score includes partial credit; fully correct answers: regular 74, encrypted 71.

Azure reports 90 correct, 155 judged, and 160 dataset rows, alongside 57.7% Correct/Judged and 55.9% Correct/Dataset. Those percentages do not match simple division of the displayed counts (90/155 ≈ 58.1%; 90/160 ≈ 56.3%), and the summary does not establish equivalence with Blackbox’s partial-credit metric. Azure is therefore excluded from this figure.

Evaluation conditions

Encrypted Blackbox recorded zero errors across 3,309 examples; regular Blackbox recorded one on FrontierScience. Regular parallelism was 4 throughout. Encrypted parallelism was 1 on MuSR, 4 on tau2-telecom, 5 on FrontierScience, and 10 on ChemBench. Dataset names alone do not establish identical evaluation settings.

The cost of encryption in these runs

Encrypted Blackbox’s mean throughput was 13.3% lower than regular Blackbox’s, and mean query latency was 13.5% higher. Mean TTFT was 5.0% lower. These are observed differences between runs, not an isolated measurement of encryption overhead: request counts differ, and the summary does not document matched prompts, concurrency, hardware, cache state, or repeat trials.

Mean emitted output length was 34,522 characters on Azure, 34,435 on regular Blackbox, and 33,846 on encrypted Blackbox. Similar character counts provide length context; they do not establish equal token counts or equivalent answers.

Full performance results and calculation notes

The following tables reproduce the supplied performance summary for DeepSeek v4.1 flash (xhigh).

Serving pathRequests
Azure230
Blackbox regular179
Blackbox encrypted330

TTFT (s)

StatisticAzureBlackbox regularBlackbox encryptedEncryption cost
Mean10.710.3990.379-5.0%
P507.4370.3950.378-4.3%
P9022.9020.4580.404-11.8%
P9530.1970.4820.416-13.7%
P9951.3850.5440.44-19.1%

Query latency (s)

StatisticAzureBlackbox regularBlackbox encryptedEncryption cost
Mean122.1526.8230.4313.5%
P5094.8725.4930.9421.4%
P90254.6944.0249.312.0%
P95290.0349.9754.288.6%
P99414.755.9962.3311.3%

Generation throughput (tokens/s)

StatisticAzureBlackbox regularBlackbox encryptedEncryption cost
Mean81.47283.74245.8813.3%
P5072.48278.18242.612.8%
P90135.4319.75277.2113.3%
P95163.7346.12302.6112.6%
P99219.21378.94325.6514.1%

Total emitted characters (characters/response)

StatisticAzureBlackbox regularBlackbox encryptedEncryption cost
Mean34,52234,43533,846-1.7%
P5033,99534,69535,6822.8%
P9056,04655,04352,794-4.1%
P9560,03059,22358,458-1.3%
P9972,60469,34066,169-4.6%

The source calls its final column “Encryption cost.” Latency and emitted-character changes use 100 × (encrypted / regular − 1). Throughput uses 100 × (1 − encrypted / regular), so a positive number represents a throughput decrease. Percentages are reproduced as reported.

The most secure infrastructure from training to inference

Ready to serve your first token?

Tell us the workload and the controls it has to satisfy. We come back with a deployment plan and a per-token commit.