Beyond Expert Judgment: An Explainable Framework for Truth Discovery, Weak Supervision, and Learning-Based Ranking in Open-Source Intelligence Risk Identification

Wei Meng

No 5642u_v1, SocArXiv from Center for Open Science

Abstract: In open-source intelligence (OSINT) research, traditional risk identification methods reliant on expert scoring face growing challenges due to their high subjectivity, cost, and lack of scalability. This study aims to propose and validate an algorithmic framework that transcends expert judgment. Centered on truth discovery, weakly supervised learning, and learning-based ranking, it enables automated, explainable risk identification within complex, multi-source heterogeneous data. The study first constructs a hierarchical-quota sampling system, acquiring and deduplicating data from four source categories: institutional authorities, official statements, mainstream and international reports, and visual materials. Subsequently, a truth discovery algorithm estimates source credibility to replace expert weighting. Weakly supervised labeling functions generate initial annotations, which are then aggregated by generative models to form robust labels. Finally, a learning ranking model dynamically prioritizes risk trajectories, with explainability ensured through Explainable AI techniques (e.g., SHAP, Grad-CAM). Results demonstrate that this framework reliably identifies risk signals across multiple time windows and control conditions. The classifier achieves PR-AUC improvements exceeding expert baselines, with average absolute error in inflection point localization maintained below 1 hour. It exhibits high consistency and robustness across cross-domain datasets. The study concludes that algorithmic expert-scoring replacement not only excels in accuracy and efficiency but also significantly outperforms traditional models in transparency and reproducibility, offering a systematic, scalable, and cutting-edge approach for OSINT risk research.

Date: 2025-09-14
New Economics Papers: this item is included in nep-cmp and nep-rmg
References: Add references at CitEc
Citations:

Downloads: (external link)
https://osf.io/download/68c4fe28a73c85c0c9e0ffae/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:socarx:5642u_v1

DOI: 10.31219/osf.io/5642u_v1

Access Statistics for this paper

More papers in SocArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().