Design and Implementation of a Scalable Data Warehouse for Agricultural Big Data

Asterios Theofilou (), Stefanos A. Nastis, Michail Tsagris, Santiago Rodriguez-Perez and Konstadinos Mattas
Additional contact information
Asterios Theofilou: Department of Agricultural Economics, Aristotle University of Thessaloniki, 54124 Thessaloniki, Greece
Stefanos A. Nastis: Department of Agricultural Economics, Aristotle University of Thessaloniki, 54124 Thessaloniki, Greece
Michail Tsagris: Department of Economics, University of Crete, 74100 Rethymno, Greece
Santiago Rodriguez-Perez: Biotechnology Applications, IDENER, Early Ovington 24 Nave 8-9, 41300 Seville, Spain
Konstadinos Mattas: Department of Agricultural Economics, Aristotle University of Thessaloniki, 54124 Thessaloniki, Greece

Sustainability, 2025, vol. 17, issue 8, 1-19

Abstract: The rapid growth of agricultural data necessitates the development of storage systems that are scalable and efficient in storing, retrieving and analyzing very large datasets. The traditional relational database management systems (RDBMSs) struggle to keep up with large-scale analytical queries due to the volume and complexity inherent in those data. This study presents the design and implementation of a scalable data warehouse (DWH) system for agricultural big data. The proposed solution efficiently integrates data and optimizes data ingestion, transformation, and query performance, leveraging a distributed architecture based on HDFS, Apache Hive, and Apache Spark, deployed on dockerized Ubuntu Linux environments. This paper highlights the reasons why a DWH is irreplaceable for big data processing, without disputing the strengths of traditional databases in transactional use cases. By detailing the architectural choices and implementation strategy, this study provides a practical framework for deploying robust DWH solutions that are useful in supporting agricultural research, market predictions and policy decision-making.

Keywords: data warehouse; big data; agricultural data (search for similar items in EconPapers)
JEL-codes: O13 Q Q0 Q2 Q3 Q5 Q56 (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2071-1050/17/8/3727/pdf (application/pdf)
https://www.mdpi.com/2071-1050/17/8/3727/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jsusta:v:17:y:2025:i:8:p:3727-:d:1638701

Access Statistics for this article

Sustainability is currently edited by Ms. Alexandra Wu

More articles in Sustainability from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().