CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean
CoRR(2024)
摘要
Despite the rapid development of large language models (LLMs) for the Korean
language, there remains an obvious lack of benchmark datasets that test the
requisite Korean cultural and linguistic knowledge. Because many existing
Korean benchmark datasets are derived from the English counterparts through
translation, they often overlook the different cultural contexts. For the few
benchmark datasets that are sourced from Korean data capturing cultural
knowledge, only narrow tasks such as bias and hate speech detection are
offered. To address this gap, we introduce a benchmark of Cultural and
Linguistic Intelligence in Korean (CLIcK), a dataset comprising 1,995 QA pairs.
CLIcK sources its data from official Korean exams and textbooks, partitioning
the questions into eleven categories under the two main categories of language
and culture. For each instance in CLIcK, we provide fine-grained annotation of
which cultural and linguistic knowledge is required to answer the question
correctly. Using CLIcK, we test 13 language models to assess their performance.
Our evaluation uncovers insights into their performances across the categories,
as well as the diverse factors affecting their comprehension. CLIcK offers the
first large-scale comprehensive Korean-centric analysis of LLMs' proficiency in
Korean culture and language.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要