Can We Trust LLMs? Inter-model Reliability and Geopolitical Bias under Zero-Shot Classification Büyük Dil Modellerine Güvenebilir miyiz? Sifir-Örnekli Siniflandirmada Tutarlilik ve Jeopolitik Yanliliklar
34th Signal Processing and Communications Applications Conference, SIU 2026, İstanbul, Türkiye, 7 - 10 Temmuz 2026, (Tam Metin Bildiri)
- Yayın Türü: Bildiri / Tam Metin Bildiri
- Doi Numarası: 10.1109/siu71813.2026.11636798
- Basıldığı Şehir: İstanbul
- Basıldığı Ülke: Türkiye
- Anahtar Kelimeler: geopolitical bias, inter-model reliability, large language models, model divergence, zero-shot classification
- Çanakkale Onsekiz Mart Üniversitesi Adresli: Evet
Özet
This study analyzes the classification consistency of Large Language Models (LLMs) using geopolitical texts collected from social media. Data gathered under the headings of the United States, Russia, and China were classified using a zero-shot prompt with six different state-of-the-art LLMs ranging from 12 to 32 billion parameters. The level of inter-model agreement was measured in a multidimensional manner. The findings reveal that the inter-model split rate exceeded 70% across all topics and that consensus on specific topics, such as Russia, was statistically insignificant. Consequently, it was determined that inter-model inconsistencies are influenced by model size and architecture. Additionally, it was found that models originating from China and the U.S. also exhibited inconsistent labeling among themselves. In conclusion, researchers should not blindly trust LLM models in zero-shot classification tasks; properly designed validity and reliability tests must also be conducted.