In short
- / points, Qwen / LlamaBoth models gave an explicit refusal less often in Hindi. We tested the same 404 harmful requests in both languages. Qwen’s rate fell from 51% to 19%. Llama’s fell from 89% to 47%.
- / points: Qwen soft refusals / Llama complianceIn Hindi, Qwen mostly sidesteps: replies that neither help nor clearly say no rise from to of harmful requests. Llama is more likely to comply outright: compliance rises from to .
- / Hindi gap, share of EnglishInside the models there is a signal that is higher for harmful requests than for harmless ones. In Hindi, the gap between the two is only about a third as large as in English. In Qwen, harmless Hindi requests score too high; in Llama, harmful Hindi requests score too low.
- / points of Hindi refusalSteering raised Hindi refusal on the same 404 held-out requests, by points for Qwen and for Llama.
Chat models are mainly trained to refuse harmful requests in English. Studies find that refusals weaken in other languages (Yong et al. 2023; Deng et al. 2024; Wang et al. 2024), including Indic ones (IndicSafe; IndicSafeEval; SurakshaEval). We ask what models do instead of refusing, and what changes inside them. This study adds three things: a side-by-side comparison of how refusals fail in two models, a look at their internal signals, and a steering intervention that raises Hindi refusals but also lifts English ones.
01How we measured it
We studied Hindi with two open 8B models, pre-registered each hypothesis test, and report the failures too. The results cover these two models, this request set and greedy decoding (always picking the most likely next token, a word or part of a word), not all chat models. “Refusal” means our judge labelled a reply as an explicit refusal; it does not measure harm to users.
Each harmful request was asked in Hindi and in English. Human annotators wrote most of them in Hindi (Aya Red-teaming; Aakanksha et al. 2024) with a human English translation, so we compare the languages on the same request. We screened every item for being a real request, a faithful translation, and one the model understood: items it misunderstood show the same gap. The held-out set is 404 harmful and 169 harmless request pairs, identical for both models.
Each reply gets one of four labels, checked in this order, so a reply that answers the request counts as complied even if it adds a warning.
| Label | What the reply does | In practice |
|---|---|---|
| Complied | Gives what was asked. | Answers the request, even if it adds a warning. |
| Explicit refusal | Says it will not help. | “I can’t help with that.” |
| Soft refusal | Neither helps nor says no. | Lectures, warns, or answers a safer question instead. |
| Broken | Not a usable reply. | Incoherent, stuck repeating itself, or in the wrong language. |
An AI judge assigned the labels, and a fluent Hindi speaker checked a sample, because AI judges can be less reliable across languages (Fu & Liu 2025). Agreement was good for explicit refusals (κ = 0.79, even blind) and weaker for the line between a soft refusal and a reply that warns but still answers (Appendix B). Sample sizes and intervals are in Appendix A.
02What the gap looks like
Each tab below is one held-out request, asked of both models in English and in Hindi, with the judge’s label. Requests are summarised, harmful details are redacted, and Hindi replies can be switched to the redacted original. These three show the common pattern; they are not a random sample. Across the harmful requests, Qwen refused explicitly in English but only softly in Hindi on , and Llama refused explicitly in English but complied in Hindi on .
03What replaces the refusal?
Explicit refusal drops sharply in both models. Where does it go?
Qwen’s lost refusals turn soft; Llama’s split between soft refusal and compliance
Replies to the same 404 harmful requests. Open circle: English. Filled circle: Hindi. The bar shows what happened when a model refused explicitly in English but not in Hindi.
Held-out test split, unsteered, greedy decoding. Labels from the LLM judge (Claude Sonnet 5.5), graded blind with every condition shuffled together. Qwen’s unsteered replies are the ones generated alongside its repair test (explicit refusal 50.5% → 18.6%); the earlier baseline run on the same items gave 51.2% → 19.1%. Intervals: Wilson 95% for shares; paired bootstrap (10,000 resamples) for changes. The Llama labels were checked against a fluent human (see “Can we trust the labels?”).
In Hindi, Qwen often warns, lectures or changes the subject instead of refusing clearly. Llama complies with harmful Hindi requests 1.9 times as often as Qwen. The split between soft refusal and compliance rests on a hard label boundary, so read it as indicative (Appendix B).
04Where do Hindi requests land inside the model?
Chat models carry a refusal direction in their internal activity, linked to whether they refuse (Arditi et al. 2024). We fitted it on English requests, used the layer where removing it cuts refusal most (layer 14 in Qwen, layer 10 in Llama), and asked where Hindi requests land. Each prompt gets a score on the direction: higher means the model’s internal activity points more toward refusing.
Refusal is carried in early-middle layers, and there both models still rank Hindi harmful requests above harmless ones
Top: refusal removed by ablating each layer’s direction. Bottom: how well the direction separates harmful from harmless requests.
Directions are differences of mean last-token activations between English harmful and harmless prompts, one per layer. “Removed” is (unablated harmful score − ablated score) / (harmful − harmless score), on the refusal score defined from first reply tokens; ablation was tested on the first 80% of layers, as registered. Qwen curves: tier A (1,174 pairs) with PolyGuard harmless prompts. Llama curves: the 404 harmful and 169 harmless test pairs (separation computed in an exploratory forward-pass run). AUROC is computed within each language. The caption’s AUROCs are recomputed for both models on the same 404 + 169 test pairs, so they differ slightly from Qwen’s tier-A curve. The Llama per-layer run is exploratory: it was added after the registered studies, to draw this figure, and tests no hypothesis. Its layer-10 values reproduce the registered Llama projections.
In both models the direction still ranks harmful Hindi requests above harmless ones (Figure 2), but the average score gap between them is only about a third as large as in English ( for Qwen, for Llama). The models shrink the gap from different sides (average scores; higher means more toward refusing):
| Qwen3-8B | Llama-3.1-8B | |
|---|---|---|
| Harmful Hindi prompts | Land where English ones do ( vs ). | Land far lower, at 28% of the English value ( vs ). |
| Harmless Hindi prompts | Land far higher than harmless English ones ( vs ). | Land a little higher ( vs ). |
| Interpretation (a hypothesis) | The refusal signal is there, so the weaker refusal probably arises later, in how it is worded. | The signal itself is weak, so many harmful Hindi prompts never trigger a refusal. |
These readings are hypotheses: they rest on two models and one kind of probe, and the score does not show that a model recognises harm (Zhao et al. 2025). Earlier work also finds shared refusal directions that are weaker in lower-resource languages (Wang et al. 2025; Aziz et al. 2026).
05Can a targeted push make the models refuse more in Hindi?
We turned these findings into two repairs. The aim: more explicit refusal of harmful Hindi requests, without refusing harmless ones or breaking replies.
05.1Gated steering
The repair is gated steering (Turner et al. 2023; Panickssery et al. 2024; Lee et al. 2025). A gate reads each prompt’s refusal signal; if the prompt looks harmful, a fixed direction is added to the model’s activity while it writes the reply. We chose settings on development requests by rules set in advance (Appendix D), then tested each once on the held-out requests. For Qwen the push is toward explicit refusal: we fitted a direction on English replies that separates explicit from soft refusals, and add it to the model’s activity. For Llama it strengthens the refusal direction, with the gate threshold set on Hindi prompts: an English threshold opened on only 1% of Llama’s harmful Hindi requests.
Where each repair reads and where it writes
Each box is a decoder layer. A gate reads the refusal score at one layer; if it passes a threshold, a fixed vector is added at a later layer.
05.2What the repairs achieve
Both repairs increased explicit refusals in Hindi on held-out requests (Figure 4). We also track English, because a push that made the model refuse more of everything would raise Hindi refusal without fixing anything specific to Hindi. English rose too, so Hindi caught up by much less than it gained. The repairs did cut compliance with harmful Hindi requests, from to for Llama and from to for Qwen.
Both repairs raise Hindi explicit refusal, and both lift English too
Explicit refusal of harmful requests before (open) and after (filled) steering, on held-out requests, with 95% intervals. Changes are paired by request.
Qwen: gate on the layer-14 refusal score, threshold at the median English development score; push 1.25× the explicit–soft vector at layer 25. Llama: gate on the layer-10 refusal score, threshold at the median Hindi development score; push 1.5× the Hindi–English shortfall along the refusal direction at layer 11. Both were selected on development data by frozen rules and run once on 404 + 169 held-out pairs. No repair raised harmless refusal or lowered Global-MMLU accuracy, but the gates rarely opened on benign inputs, so those checks are weak.
Why the repairs lift English too: the gate sees English harm at least as clearly
Each dot is one held-out request at its gate-layer refusal score. Requests right of the red line receive steering. Move the sliders to change the threshold.
Gate scores are the projections the gate actually read during the held-out run (Qwen layer 14, Llama layer 10), for the unsteered prompts: 404 harmful and 169 harmless pairs per language. Thresholds as registered: Qwen at the median English development harmful score, Llama at the median Hindi development harmful score.
Figure 5 shows why English rises too: the gate asks whether a prompt looks harmful, and English harmful prompts score at least as high as Hindi ones. Llama’s Hindi-set threshold lets through about half of harmful Hindi requests by design, and none of the 169 harmless ones.
Side effects. We ran three checks.
- General ability. On 300 Global-MMLU questions per language, accuracy did not change (Qwen in Hindi and in English; Llama in Hindi and in English).
- Harmless requests. Refusals stayed within our 3-point limit.
- A weak test. The gate never opened on Qwen’s MMLU questions, so those replies could not change. Forcing steering on every question moved Qwen’s accuracy from 51.3% to 52.3% in Hindi and from 64.0% to 63.7% in English, within noise.
05.3Request by request
The table counts, across all 404 held-out requests, what happened to the Hindi replies that were not explicit refusals.
| Held-out Hindi harmful requests | Qwen | Llama |
|---|
Fixed minus undone gives the headline gains: . When Llama’s push is applied it nearly always works; the gate, which cannot single out Hindi, limits the total (Figure 5). Here are the same three requests from section 02, before and after.
06Limits, and what’s next
- Scope. Two 8B models, one request set and greedy decoding, with one AI judge and one human grader. “Refusal” is a judged reply category, not harm, user outcomes or deployed safety.
- Steering. It changed replies here; we have not shown it improves multilingual safety more broadly. The harmless-request and ability checks are weak.
- Languages. Hindi is the only language studied in depth (Spanish control, Qwen only: Appendix C). The per-layer Llama curves come from an exploratory run added after the registered studies.
- What didn’t hold up (Appendix E). A sealed Bengali test failed its registered check, and a small set of neurons behind the English–Hindi difference moved English refusals, not Hindi ones.
- Next. Compare with simpler fixes such as a safety prompt or translating to English, which can rival steering (Wu et al. 2025), test on harmless requests that sound harmful (Balani & Panda 2026), build a gate selective for Hindi, and run a fresh confirmation set for Llama.
Methods and checks
Supporting detail for the numbers above. Open any section.
Appendix A: estimates and how sure we are
The headline numbers, all from the same 404 harmful requests. Each repair was tested once on them.
| Estimate | Qwen3-8B | Llama-3.1-8B |
|---|
How sure are we? These are best estimates from 404 requests, so they could shift if we had asked different requests. The margin of error on each difference is at most about ± points, so a smaller difference could be noise. Qwen’s catch-up in Hindi ( points) is within that margin and may be zero; Llama’s ( points) is not. We chose the repair settings on separate development requests and tested them once; held-out gains were about 80% of the best development gains.
Appendix B: can we trust the labels?
Many results depend on separating a soft refusal from a reply that warns but then answers. This is difficult. A fluent Hindi reader blindly graded 60 Llama replies. Initial agreement was κ = 0.64, below our registered threshold of 0.7. Most disagreements involved warnings followed by answers. The grader then reapplied the rubric: a warning followed by an answer counts as compliance. Agreement rose to 0.89. We report both results.
The headline results use explicit refusals, which had stronger agreement in the blind check (κ = 0.79). Qwen’s judge passed its validation check with κ = 0.82.
Appendix C: is this just Hindi? A Spanish control
Is the gap unique to Hindi? We ran a smaller control in Spanish, a high-resource language, on Qwen3-8B only: harmful and harmless Aya Red-teaming requests, each with an English version, labelled by an AI judge.
| English | Spanish | |
|---|---|---|
| Explicit refusal | ||
| Any refusal, explicit or soft | ||
| Complied | ||
| Harmless requests answered |
Spanish shows a gap too: explicit refusal falls by points ( requests refused only in English, only in Spanish), and harmless requests are still answered. That is about half the Hindi gap ( points for Qwen), consistent with a larger gap for lower-resource languages, but this run cannot establish that. Caveats: Qwen only; a different request set and a shorter reply limit than the Hindi runs, so the gaps are not directly comparable; AI-judge labels only (two judges agreed, κ = 0.88); and a provisional run made before the Hindi judge was validated and the model revision pinned.
Appendix D: how hard to push
We tried several steering strengths on development requests and kept the strongest one that stayed inside limits set in advance (broken replies within +5 points, harmless refusal within +3).
Stronger pushes buy more Hindi refusal until replies start to break
Each dot is one setting tested on development requests. It shows the change in Hindi refusals and broken replies. The shaded area marks settings that met our limits. Filled dots show the settings tested on held-out requests.
Qwen: 401 Hindi harmful and 171 harmless development requests, nine settings (strength 0.75, 1.0, 1.25 × gate at the 50th, 25th or 10th percentile of English development scores); the selection rule had no minimum gain, only broken output within +5 and harmless refusal within +3 (English was checked at test). Llama: 150 Hindi harmful and 60 harmless development requests, five pushes; a setting needed at least +15 points, broken output within +5, harmless refusal within +3, and English explicit refusal not below −3 with English broken output within +3.
Appendix E: the failed checks in detail
- Bengali test. We checked Qwen’s repair on a sealed Bengali test set. It raised explicit refusal by 23.6 points but also made 8.1 points more replies garbled, over our 5-point limit, so the registered verdict is a fail. A follow-up on new Bengali requests did not repeat the damage, which softens the result but does not erase it.
- “Hindi switch.” We looked for a small set of neurons behind the English–Hindi difference. 100 of 307,200 neurons carried three quarters of it. But changing them raised Qwen’s English refusals by 15.2 points and its Hindi refusals by only 1.2, so they are not a Hindi-specific lever.
Appendix F: how we tried to keep ourselves honest
- Rules first. Before each experiment we recorded what we would measure, how we would choose a setting and what would count as success; later changes went into dated addenda. The per-layer Llama curves in Figure 2 were added afterwards. They are descriptive and test no hypothesis.
- No harmful text. Requests are summarised, never quoted. The only reply text shown is the redacted excerpts in sections 02 and 05, with anything usable for harm blacked out.
References
- Aakanksha, Ahmadian, Ermis, Goldfarb-Tarrant, Kreutzer, Fadaee, Hooker (2024). The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm. Source of the Aya Red-teaming dataset. arXiv:2406.18682
- Arditi, Obeso, Syed, Paleka, Panickssery, Gurnee, Nanda (2024). Refusal in Language Models Is Mediated by a Single Direction. NeurIPS 2024. arXiv:2406.11717
- Aziz, Hanif, Koto (2026). Low-Resource Safety Failures Are Action Failures, Not Representation Failures. arXiv:2606.01196
- Balani & Panda (2026). Hard Negatives Reveal What Easy Negatives Hide: Cross-Lingual Harmfulness Representations Degrade with Resource Tier Under Hard Negatives. arXiv:2609.27758
- Banerjee et al. (2026). SurakshaEval: An Indic Safety Benchmark for Multilingual LLMs. arXiv:2608.07862
- Deng, Zhang, Pan, Bing (2024). Multilingual Jailbreak Challenges in Large Language Models. ICLR 2024. arXiv:2310.06474
- Fu & Liu (2025). How Reliable is Multilingual LLM-as-a-Judge? arXiv:2505.12201
- Lee et al. (2025). Programming Refusal with Conditional Activation Steering. ICLR 2025. arXiv:2409.05907
- Mondal, Mamta, Varshney, Cocarascu, Ekbal (2026). IndicSafeEval: Safety Robustness of Large Language Models under Multilingual Persuasive Jailbreak Attacks. Findings of EMNLP 2026. arXiv:2609.03781
- Panickssery et al. (2024). Steering Llama 2 via Contrastive Activation Addition. arXiv:2312.06681
- Pattnayak & Chowdhuri (2026). IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia. arXiv:2603.17915
- Turner et al. (2023). Steering Language Models With Activation Engineering. arXiv:2308.10248
- Wang, Tu, Chen, Yuan, Huang, Jiao, Lyu (2024). All Languages Matter: On the Multilingual Safety of Large Language Models. Findings of ACL 2024. arXiv:2310.00905
- Wang, Wang, Liu, Schütze, Plank (2025). Refusal Direction is Universal Across Safety-Aligned Languages. arXiv:2505.17306
- Wu et al. (2025). AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders. arXiv:2501.17148
- Yong, Menghini, Bach (2023). Low-Resource Languages Jailbreak GPT-4. NeurIPS 2023 SoLaR Workshop. arXiv:2310.02446
- Zhao, Huang, Wu, Bau, Shi (2025). LLMs Encode Harmfulness and Refusal Separately. arXiv:2507.11878