Mayank SaxenaWriting

Cross-lingual safety · research note

Why two open chat models refuse less in Hindi

We gave the same harmful requests in English and Hindi to two open chat models: Qwen3-8B and Llama-3.1-8B-Instruct. Both refused less often in Hindi. We studied what they did instead, what happened inside the models, and whether steering could increase refusals.

Mayank Saxena17 November 2026

In short

  1. / points, Qwen / Llama
    Both models gave an explicit refusal less often in Hindi. We tested the same 404 harmful requests in both languages. Qwen’s rate fell from 51% to 19%. Llama’s fell from 89% to 47%.
  2. / points: Qwen soft refusals / Llama compliance
    In Hindi, Qwen mostly sidesteps: replies that neither help nor clearly say no rise from to of harmful requests. Llama is more likely to comply outright: compliance rises from to .
  3. / Hindi gap, share of English
    Inside the models there is a signal that is higher for harmful requests than for harmless ones. In Hindi, the gap between the two is only about a third as large as in English. In Qwen, harmless Hindi requests score too high; in Llama, harmful Hindi requests score too low.
  4. / points of Hindi refusal
    Steering raised Hindi refusal on the same 404 held-out requests, by points for Qwen and for Llama.

Chat models are mainly trained to refuse harmful requests in English. Studies find that refusals weaken in other languages (Yong et al. 2023; Deng et al. 2024; Wang et al. 2024), including Indic ones (IndicSafe; IndicSafeEval; SurakshaEval). We ask what models do instead of refusing, and what changes inside them. This study adds three things: a side-by-side comparison of how refusals fail in two models, a look at their internal signals, and a steering intervention that raises Hindi refusals but also lifts English ones.

01How we measured it

We studied Hindi with two open 8B models, pre-registered each hypothesis test, and report the failures too. The results cover these two models, this request set and greedy decoding (always picking the most likely next token, a word or part of a word), not all chat models. “Refusal” means our judge labelled a reply as an explicit refusal; it does not measure harm to users.

Each harmful request was asked in Hindi and in English. Human annotators wrote most of them in Hindi (Aya Red-teaming; Aakanksha et al. 2024) with a human English translation, so we compare the languages on the same request. We screened every item for being a real request, a faithful translation, and one the model understood: items it misunderstood show the same gap. The held-out set is 404 harmful and 169 harmless request pairs, identical for both models.

Each reply gets one of four labels, checked in this order, so a reply that answers the request counts as complied even if it adds a warning.

LabelWhat the reply doesIn practice
CompliedGives what was asked.Answers the request, even if it adds a warning.
Explicit refusalSays it will not help.“I can’t help with that.”
Soft refusalNeither helps nor says no.Lectures, warns, or answers a safer question instead.
BrokenNot a usable reply.Incoherent, stuck repeating itself, or in the wrong language.

An AI judge assigned the labels, and a fluent Hindi speaker checked a sample, because AI judges can be less reliable across languages (Fu & Liu 2025). Agreement was good for explicit refusals (κ = 0.79, even blind) and weaker for the line between a soft refusal and a reply that warns but still answers (Appendix B). Sample sizes and intervals are in Appendix A.

02What the gap looks like

Each tab below is one held-out request, asked of both models in English and in Hindi, with the judge’s label. Requests are summarised, harmful details are redacted, and Hindi replies can be switched to the redacted original. These three show the common pattern; they are not a random sample. Across the harmful requests, Qwen refused explicitly in English but only softly in Hindi on , and Llama refused explicitly in English but complied in Hindi on .

03What replaces the refusal?

Explicit refusal drops sharply in both models. Where does it go?

Qwen’s lost refusals turn soft; Llama’s split between soft refusal and compliance

Replies to the same 404 harmful requests. Open circle: English. Filled circle: Hindi. The bar shows what happened when a model refused explicitly in English but not in Hindi.

Explicit refusalSoft refusalCompliedBroken
Figure 1: For Qwen3-8B, 115 of 148 lost explicit refusals became soft refusals. For Llama-3.1-8B, 88 of 172 became compliance and 75 soft refusals (similar shares, p = 0.35).

Held-out test split, unsteered, greedy decoding. Labels from the LLM judge (Claude Sonnet 5.5), graded blind with every condition shuffled together. Qwen’s unsteered replies are the ones generated alongside its repair test (explicit refusal 50.5% → 18.6%); the earlier baseline run on the same items gave 51.2% → 19.1%. Intervals: Wilson 95% for shares; paired bootstrap (10,000 resamples) for changes. The Llama labels were checked against a fluent human (see “Can we trust the labels?”).

In Hindi, Qwen often warns, lectures or changes the subject instead of refusing clearly. Llama complies with harmful Hindi requests 1.9 times as often as Qwen. The split between soft refusal and compliance rests on a hard label boundary, so read it as indicative (Appendix B).

04Where do Hindi requests land inside the model?

Chat models carry a refusal direction in their internal activity, linked to whether they refuse (Arditi et al. 2024). We fitted it on English requests, used the layer where removing it cuts refusal most (layer 14 in Qwen, layer 10 in Llama), and asked where Hindi requests land. Each prompt gets a score on the direction: higher means the model’s internal activity points more toward refusing.

Refusal is carried in early-middle layers, and there both models still rank Hindi harmful requests above harmless ones

Top: refusal removed by ablating each layer’s direction. Bottom: how well the direction separates harmful from harmless requests.

Refusal removedEnglishHindi
Figure 2: Removing the direction reduces Qwen’s refusal-score gap by at layer 14 (on the prompts used to pick the layer; 64% on held-out English prompts) and Llama’s by at layer 10 (held-out). At those layers, the direction still ranks harmful Hindi requests above harmless ones: AUROC, where 1.0 means a perfect ranking, is and (English and ).

Directions are differences of mean last-token activations between English harmful and harmless prompts, one per layer. “Removed” is (unablated harmful score − ablated score) / (harmful − harmless score), on the refusal score defined from first reply tokens; ablation was tested on the first 80% of layers, as registered. Qwen curves: tier A (1,174 pairs) with PolyGuard harmless prompts. Llama curves: the 404 harmful and 169 harmless test pairs (separation computed in an exploratory forward-pass run). AUROC is computed within each language. The caption’s AUROCs are recomputed for both models on the same 404 + 169 test pairs, so they differ slightly from Qwen’s tier-A curve. The Llama per-layer run is exploratory: it was added after the registered studies, to draw this figure, and tests no hypothesis. Its layer-10 values reproduce the registered Llama projections.

In both models the direction still ranks harmful Hindi requests above harmless ones (Figure 2), but the average score gap between them is only about a third as large as in English ( for Qwen, for Llama). The models shrink the gap from different sides (average scores; higher means more toward refusing):

Qwen3-8BLlama-3.1-8B
Harmful Hindi promptsLand where English ones do ( vs ).Land far lower, at 28% of the English value ( vs ).
Harmless Hindi promptsLand far higher than harmless English ones ( vs ).Land a little higher ( vs ).
Interpretation (a hypothesis)The refusal signal is there, so the weaker refusal probably arises later, in how it is worded.The signal itself is weak, so many harmful Hindi prompts never trigger a refusal.

These readings are hypotheses: they rest on two models and one kind of probe, and the score does not show that a model recognises harm (Zhao et al. 2025). Earlier work also finds shared refusal directions that are weaker in lower-resource languages (Wang et al. 2025; Aziz et al. 2026).

05Can a targeted push make the models refuse more in Hindi?

We turned these findings into two repairs. The aim: more explicit refusal of harmful Hindi requests, without refusing harmless ones or breaking replies.

05.1Gated steering

The repair is gated steering (Turner et al. 2023; Panickssery et al. 2024; Lee et al. 2025). A gate reads each prompt’s refusal signal; if the prompt looks harmful, a fixed direction is added to the model’s activity while it writes the reply. We chose settings on development requests by rules set in advance (Appendix D), then tested each once on the held-out requests. For Qwen the push is toward explicit refusal: we fitted a direction on English replies that separates explicit from soft refusals, and add it to the model’s activity. For Llama it strengthens the refusal direction, with the gate threshold set on Hindi prompts: an English threshold opened on only 1% of Llama’s harmful Hindi requests.

Where each repair reads and where it writes

Each box is a decoder layer. A gate reads the refusal score at one layer; if it passes a threshold, a fixed vector is added at a later layer.

Figure 3: Qwen reads at layer 14 and adds the style direction at layer 25. Llama reads at layer 10 and adds to the refusal direction at layer 11. Qwen’s threshold is set on English development requests, Llama’s on Hindi ones.

05.2What the repairs achieve

Both repairs increased explicit refusals in Hindi on held-out requests (Figure 4). We also track English, because a push that made the model refuse more of everything would raise Hindi refusal without fixing anything specific to Hindi. English rose too, so Hindi caught up by much less than it gained. The repairs did cut compliance with harmful Hindi requests, from to for Llama and from to for Qwen.

Both repairs raise Hindi explicit refusal, and both lift English too

Explicit refusal of harmful requests before (open) and after (filled) steering, on held-out requests, with 95% intervals. Changes are paired by request.

Figure 4: Qwen’s Hindi rate rose by points and its English rate by , so Hindi caught up by (interval includes zero). Llama’s Hindi rate rose by and its English rate by , closing the gap by , short of the registered +15 and +10 targets.

Qwen: gate on the layer-14 refusal score, threshold at the median English development score; push 1.25× the explicit–soft vector at layer 25. Llama: gate on the layer-10 refusal score, threshold at the median Hindi development score; push 1.5× the Hindi–English shortfall along the refusal direction at layer 11. Both were selected on development data by frozen rules and run once on 404 + 169 held-out pairs. No repair raised harmless refusal or lowered Global-MMLU accuracy, but the gates rarely opened on benign inputs, so those checks are weak.

Why the repairs lift English too: the gate sees English harm at least as clearly

Each dot is one held-out request at its gate-layer refusal score. Requests right of the red line receive steering. Move the sliders to change the threshold.

Figure 5: Qwen’s threshold opens for 47% of harmful requests in both languages. Llama’s opens for 47% of harmful Hindi requests but 92% of harmful English ones, so no threshold selects Hindi harm without also selecting English harm.

Gate scores are the projections the gate actually read during the held-out run (Qwen layer 14, Llama layer 10), for the unsteered prompts: 404 harmful and 169 harmless pairs per language. Thresholds as registered: Qwen at the median English development harmful score, Llama at the median Hindi development harmful score.

Figure 5 shows why English rises too: the gate asks whether a prompt looks harmful, and English harmful prompts score at least as high as Hindi ones. Llama’s Hindi-set threshold lets through about half of harmful Hindi requests by design, and none of the 169 harmless ones.

Side effects. We ran three checks.

05.3Request by request

The table counts, across all 404 held-out requests, what happened to the Hindi replies that were not explicit refusals.

Held-out Hindi harmful requestsQwenLlama

Fixed minus undone gives the headline gains: . When Llama’s push is applied it nearly always works; the gate, which cannot single out Hindi, limits the total (Figure 5). Here are the same three requests from section 02, before and after.

06Limits, and what’s next

Methods and checks

Supporting detail for the numbers above. Open any section.

Appendix A: estimates and how sure we are

The headline numbers, all from the same 404 harmful requests. Each repair was tested once on them.

EstimateQwen3-8BLlama-3.1-8B

How sure are we? These are best estimates from 404 requests, so they could shift if we had asked different requests. The margin of error on each difference is at most about ± points, so a smaller difference could be noise. Qwen’s catch-up in Hindi ( points) is within that margin and may be zero; Llama’s ( points) is not. We chose the repair settings on separate development requests and tested them once; held-out gains were about 80% of the best development gains.

Appendix B: can we trust the labels?

Many results depend on separating a soft refusal from a reply that warns but then answers. This is difficult. A fluent Hindi reader blindly graded 60 Llama replies. Initial agreement was κ = 0.64, below our registered threshold of 0.7. Most disagreements involved warnings followed by answers. The grader then reapplied the rubric: a warning followed by an answer counts as compliance. Agreement rose to 0.89. We report both results.

The headline results use explicit refusals, which had stronger agreement in the blind check (κ = 0.79). Qwen’s judge passed its validation check with κ = 0.82.

Appendix C: is this just Hindi? A Spanish control

Is the gap unique to Hindi? We ran a smaller control in Spanish, a high-resource language, on Qwen3-8B only: harmful and harmless Aya Red-teaming requests, each with an English version, labelled by an AI judge.

EnglishSpanish
Explicit refusal
Any refusal, explicit or soft
Complied
Harmless requests answered

Spanish shows a gap too: explicit refusal falls by points ( requests refused only in English, only in Spanish), and harmless requests are still answered. That is about half the Hindi gap ( points for Qwen), consistent with a larger gap for lower-resource languages, but this run cannot establish that. Caveats: Qwen only; a different request set and a shorter reply limit than the Hindi runs, so the gaps are not directly comparable; AI-judge labels only (two judges agreed, κ = 0.88); and a provisional run made before the Hindi judge was validated and the model revision pinned.

Appendix D: how hard to push

We tried several steering strengths on development requests and kept the strongest one that stayed inside limits set in advance (broken replies within +5 points, harmless refusal within +3).

Stronger pushes buy more Hindi refusal until replies start to break

Each dot is one setting tested on development requests. It shows the change in Hindi refusals and broken replies. The shaded area marks settings that met our limits. Filled dots show the settings tested on held-out requests.

Figure A1: For Qwen, looser gates added refusal but broke too many replies; the strongest acceptable setting was a 50th-percentile gate with a 1.25× push. For Llama, doubling the style push broke 23% of replies, while the refusal-direction pushes did not.

Qwen: 401 Hindi harmful and 171 harmless development requests, nine settings (strength 0.75, 1.0, 1.25 × gate at the 50th, 25th or 10th percentile of English development scores); the selection rule had no minimum gain, only broken output within +5 and harmless refusal within +3 (English was checked at test). Llama: 150 Hindi harmful and 60 harmless development requests, five pushes; a setting needed at least +15 points, broken output within +5, harmless refusal within +3, and English explicit refusal not below −3 with English broken output within +3.

Appendix E: the failed checks in detail
  • Bengali test. We checked Qwen’s repair on a sealed Bengali test set. It raised explicit refusal by 23.6 points but also made 8.1 points more replies garbled, over our 5-point limit, so the registered verdict is a fail. A follow-up on new Bengali requests did not repeat the damage, which softens the result but does not erase it.
  • “Hindi switch.” We looked for a small set of neurons behind the English–Hindi difference. 100 of 307,200 neurons carried three quarters of it. But changing them raised Qwen’s English refusals by 15.2 points and its Hindi refusals by only 1.2, so they are not a Hindi-specific lever.
Appendix F: how we tried to keep ourselves honest
  • Rules first. Before each experiment we recorded what we would measure, how we would choose a setting and what would count as success; later changes went into dated addenda. The per-layer Llama curves in Figure 2 were added afterwards. They are descriptive and test no hypothesis.
  • No harmful text. Requests are summarised, never quoted. The only reply text shown is the redacted excerpts in sections 02 and 05, with anything usable for harm blacked out.

References

  1. Aakanksha, Ahmadian, Ermis, Goldfarb-Tarrant, Kreutzer, Fadaee, Hooker (2024). The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm. Source of the Aya Red-teaming dataset. arXiv:2406.18682
  2. Arditi, Obeso, Syed, Paleka, Panickssery, Gurnee, Nanda (2024). Refusal in Language Models Is Mediated by a Single Direction. NeurIPS 2024. arXiv:2406.11717
  3. Aziz, Hanif, Koto (2026). Low-Resource Safety Failures Are Action Failures, Not Representation Failures. arXiv:2606.01196
  4. Balani & Panda (2026). Hard Negatives Reveal What Easy Negatives Hide: Cross-Lingual Harmfulness Representations Degrade with Resource Tier Under Hard Negatives. arXiv:2609.27758
  5. Banerjee et al. (2026). SurakshaEval: An Indic Safety Benchmark for Multilingual LLMs. arXiv:2608.07862
  6. Deng, Zhang, Pan, Bing (2024). Multilingual Jailbreak Challenges in Large Language Models. ICLR 2024. arXiv:2310.06474
  7. Fu & Liu (2025). How Reliable is Multilingual LLM-as-a-Judge? arXiv:2505.12201
  8. Lee et al. (2025). Programming Refusal with Conditional Activation Steering. ICLR 2025. arXiv:2409.05907
  9. Mondal, Mamta, Varshney, Cocarascu, Ekbal (2026). IndicSafeEval: Safety Robustness of Large Language Models under Multilingual Persuasive Jailbreak Attacks. Findings of EMNLP 2026. arXiv:2609.03781
  10. Panickssery et al. (2024). Steering Llama 2 via Contrastive Activation Addition. arXiv:2312.06681
  11. Pattnayak & Chowdhuri (2026). IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia. arXiv:2603.17915
  12. Turner et al. (2023). Steering Language Models With Activation Engineering. arXiv:2308.10248
  13. Wang, Tu, Chen, Yuan, Huang, Jiao, Lyu (2024). All Languages Matter: On the Multilingual Safety of Large Language Models. Findings of ACL 2024. arXiv:2310.00905
  14. Wang, Wang, Liu, Schütze, Plank (2025). Refusal Direction is Universal Across Safety-Aligned Languages. arXiv:2505.17306
  15. Wu et al. (2025). AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders. arXiv:2501.17148
  16. Yong, Menghini, Bach (2023). Low-Resource Languages Jailbreak GPT-4. NeurIPS 2023 SoLaR Workshop. arXiv:2310.02446
  17. Zhao, Huang, Wu, Bau, Shi (2025). LLMs Encode Harmfulness and Refusal Separately. arXiv:2507.11878