Chinese SafetyQA: A Safety Short-form Factuality Benchmark for Large Language Models

Chinese SafetyQA's Features

• Chinese: Chinese SafetyQA dataset has been compiled within the Chinese linguistic context, primarily encompassing safety-related issues, e.g., Chinese legal frameworks and ethical standards.
• Benign: Our dataset mainly focuses on the safety-related knowledge. The examples themselves do not contain any harmful contents.
• Diverse: Our dataset includes 7 primary topics, 27 secondary topics and 103 fine-grained topics, covering across nearly all areas of Chinese Safety.
• Easy-to-evaluate: We provide data in two different formats: short-form question-answer (QA) and multiple choice question (MCQ), allowing users to easily test the boundaries of a model's safety knowledge.
• Static: Following prior works, all standard answer given in our benchmark would not change over time.
• Challenging: Our Chinese SafetyQA dataset primarily covers professional security knowledge rather than simple, general common-sense knowledge.

Main Results

• Chinese SafetyQA is challenging. Only three models meet the passing threshold (60) in this test, in which o1-preview is the best-performing LLM among all evaluated model, exceeding the second place (qwen-max) by nearly ten points.
• Knowledge Matters. Insufficient safety knowledge in models induces potential risks. Models that achieve higher scores in Chinese SafetyQA generally demonstrate better performance in safety evaluations.
• Larger models lead to better results. When comparing models within the same series (e.g., qwen2.5-72b and qwen2.5-14b), we observe that larger models exhibit superior factual performance in safety knowledge. We attribute this phenomenon to the enhanced memory capacity of larger models, which results in a clearer understanding and better retention of safety-related information.
• Overconfidence in Calibration. LLMs often overestimate their accuracy, with high-confidence answers frequently falling below the ideal calibration line, especially in Chinese contexts. Rare low-confidence assignments and confidently incorrect answers reveal knowledge errors from pre-training data, highlighting the need for better calibration to align confidence with actual performance.
• TOT Phenomenon in LLMs. LLMs perform better on MCQ tasks than QA tasks due to the "Tip of the Tongue" (TOT) phenomenon, where knowledge conflicts in pre-training data hinder accurate recall. MCQ options act as cues, helping models retrieve correct knowledge, while QA tasks lack such prompts, leading to errors.
• RAG's Impact on Factuality. Retrieval-Augmented Generation (RAG) improves LLM factuality, with passive RAG outperforming active RAG. Smaller models benefit more than larger ones, and active RAG's limited usage fails to match the actual error rate. Overconfidence and hallucinations reduce RAG's potential, especially in safety-critical contexts.

Results of different models on Chinese SafetyQA. For metrics, CO, NA, IN, and CGA denote ''Correct'', ''Not attempted'', ''Incorrect'', and ''Correct given attempted'', respectively. For subtopics, RM, IRC, PMH, IH, PD, EM and STK are the abbreviations of our subtopics :''Rumor & Misinformation'', ''Illegal & Reg. Compliance'', ''Physical & Mental Health'', ''Insults & Hate'', ''Prejudice & Discrimination'', ''Ethical & Moral'' and ''Safety Theoretical Knowledge'', respectively.

Through analyzing the confidence levels of large language models (LLMs) in the context of Chinese safety knowledge evaluation, we reveal significant limitations in the cognitive consistency of current models. We prompted the tested models to assign precise confidence estimates (ranging from 0 to 100, with a granularity of 5) to their responses, aiming to quantify their self-awareness regarding the boundaries of their knowledge.

The experimental results indicate that, despite continuous advancements in technical complexity, the cognitive calibration mechanisms of these models demonstrate significant biases. The tested models tend to assign high confidence to their responses, exhibiting a pattern of overconfidence that remains consistent across most models. While certain models (e.g., Qwen72b) occasionally display subtle differences in confidence allocation, they still fail to establish a reliable correspondence between confidence and accuracy overall. Notably, the data points in the high-confidence range (>50) consistently fall below the ideal calibration benchmark. This finding not only highlights the inherent uncertainty in the models' confidence evaluation but also suggests potential deficiencies in the knowledge representation within the pretraining corpora.

The benchmark covers 7 topics and 103 subtopics to assess the model's knowledge in Chinese SafetyDomain. Notably, o1-preview excels in all major categories, scoring above 60 in allcategories(QA), while the gpt-4o-mini model performed the worst, with no category reaching 60(QA). Specifically, all GPTmodels showed relatively better performance on Physical \& Mental Health (PHM), indicating moretraining effort on international ESG issues. However, on Illegal \& Reg. Compliance (IRC), allnon-Chinese models (except o1) performs bad, whereas Chinese models (Qwen-series and Doubao)showed relatively better performance, indicating Chinese LLMs' have pay specialized trainingeffort on Chinese legal knowledge. These results highlight significantdisparities in model performance across safety-critical topics and emphasize the need forcategory-specific evaluations. An interesting finding is that, for the same questions, LLMsachieve significantly higher accuracy on MCQ tasks compared to QA tasks.

Self-reflection resulted in minimal improvements (under 5%) across all evaluated LLMs and negatively impacted the o1-series models. LLMs often alter correct answers to incorrect ones due to reliance on statistical patterns in training data. Knowledge-based questions depend more on model understanding than reasoning. Factual errors in training data lead to incorrect answers, as LLMs cannot discern them through chain-of-thought. Insufficient knowledge may also cause unnecessary modifications, introducing further errors. In short, self-reflection does not significantly improve the factual accuracy of safety-related responses.

In our study, we also evaluate the effectiveness of different RAG approaches. Specifically, we employ two types of RAG triggering methods: Active RAG and Passive RAG, RAG enhances LLMs' factual safety, though the effect is less pronounced in specific domains like SimpleQA and Chinese SimpleQA. Notably, RAG reduces performance gaps, benefiting smaller models more than larger ones. Active RAG's effectiveness varies across LLMs and is generally less effective than passive RAG, possibly due to LLMs' overconfidence and infrequent proactive requests for RAG, despite high error rates.

This table shows a few examples from the Chinese SafetyQA dataset. More samples are available in the complete dataset.