An Improved Reference Paper Collection System Using Web Scraping with Three Enhancements

Tresna Maulana Fahrudin, Nobuo Funabiki (), Komang Candra Brata, Inzali Naing, Soe Thandar Aung, Amri Muhaimin and Dwi Arman Prasetya
Additional contact information
Tresna Maulana Fahrudin: Department of Information and Communication Systems, Okayama University, Okayama 700-8530, Japan
Nobuo Funabiki: Department of Information and Communication Systems, Okayama University, Okayama 700-8530, Japan
Komang Candra Brata: Department of Information and Communication Systems, Okayama University, Okayama 700-8530, Japan
Inzali Naing: Department of Information and Communication Systems, Okayama University, Okayama 700-8530, Japan
Soe Thandar Aung: Department of Information and Communication Systems, Okayama University, Okayama 700-8530, Japan
Amri Muhaimin: Department of Data Science, Universitas Pembangunan Nasional Veteran Jawa Timur, Surabaya 60294, Indonesia
Dwi Arman Prasetya: Department of Data Science, Universitas Pembangunan Nasional Veteran Jawa Timur, Surabaya 60294, Indonesia

Future Internet, 2025, vol. 17, issue 5, 1-28

Abstract: Nowadays, accessibility to academic papers has been significantly improved with electric publications on the internet, where open access has become common. At the same time, it has increased workloads in literature surveys for researchers who usually manually download PDF files and check their contents. To solve this drawback, we have proposed a reference paper collection system using a web scraping technology and natural language models. However, our previous system often finds a limited number of relevant reference papers after taking long time, since it relies on one paper search website and runs on a single thread at a multi-core CPU. In this paper, we present an improved reference paper collection system with three enhancements to solve them: (1) integrating the APIs from multiple paper search web sites, namely, the bulk search endpoint in the Semantic Scholar API, the article search endpoint in the DOAJ API, and the search and fetch endpoint in the PubMed API to retrieve article metadata, (2) running the program on multiple threads for multi-core CPU, and (3) implementing Dynamic URL Redirection , Regex-based URL Parsing , and HTML Scraping with URL Extraction for fast checking of PDF file accessibility, along with sentence embedding to assess relevance based on semantic similarity. For evaluations, we compare the number of obtained reference papers and the response time between the proposal, our previous work, and common literature search tools in five reference paper queries. The results show that the proposal increases the number of relevant reference papers by 64.38% and reduces the time by 59.78% on average compared to our previous work, while outperforming common literature search tools in reference papers. Thus, the effectiveness of the proposed system has been demonstrated in our experiments.

Keywords: reference paper collection; multiple API integration; PDF accessibility; open access; multiple threads (search for similar items in EconPapers)
JEL-codes: O3 (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/1999-5903/17/5/195/pdf (application/pdf)
https://www.mdpi.com/1999-5903/17/5/195/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jftint:v:17:y:2025:i:5:p:195-:d:1644623

Access Statistics for this article

Future Internet is currently edited by Ms. Grace You

More articles in Future Internet from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().