Order statistics in large arrays (OSILA): a simple randomised algorithm for a fast and efficient attainment of the order statistics in very large arrays

Andrea Cerasa ()
Additional contact information
Andrea Cerasa: European Commission, Joint Research Centre

Computational Statistics, 2024, vol. 39, issue 7, No 7, 3599-3624

Abstract: Abstract When dealing with large-scale applications, the availability of simple and efficient algorithms is essential. We focus on the algorithm for calculating the order statistics, i.e. for selecting the kth smallest element of an array X. Many statistical procedures rely on this basic operation, that is usually solved by sorting all the elements and selecting the one in position k. If the dimension of the array to sort is quite large, this simple operation can become excessively time consuming. For this purpose, we propose an original randomised algorithm that reduces the dimension of the selection problem by focusing only on a small subset of elements that contains the solution. Despite its random nature, it always returns the target value. Empirical results shows that, for arrays of dimensions running from $$10^5$$ 10 5 to $$10^8$$ 10 8 , our procedure resulted to be remarkably (up to almost 10 times) faster than the naïve procedure, independently of the programming environment and of the sorting algorithm, and with a relative advantage that tends to growth with the dimension of the array.

Keywords: Selection problem; Quantile; Statistical computing; Matlab; R; Big data (search for similar items in EconPapers)
Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s00180-023-01381-1 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:compst:v:39:y:2024:i:7:d:10.1007_s00180-023-01381-1

Ordering information: This journal article can be ordered from
http://www.springer.com/statistics/journal/180/PS2

DOI: 10.1007/s00180-023-01381-1

Access Statistics for this article

Computational Statistics is currently edited by Wataru Sakamoto, Ricardo Cao and Jürgen Symanzik

More articles in Computational Statistics from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().