Performance Comparison of Machine Learning Platforms

Asim Roy (), Shiban Qureshi (), Kartikeya Pande (), Divitha Nair (), Kartik Gairola (), Pooja Jain (), Suraj Singh (), Kirti Sharma (), Akshay Jagadale (), Yi-Yang Lin (), Shashank Sharma (), Ramya Gotety (), Yuexin Zhang (), Ji Tang (), Tejas Mehta (), Hemanth Sindhanuru (), Nonso Okafor (), Santak Das (), Chidambara N. Gopal (), Srinivasa B. Rudraraju () and Avinash V. Kakarlapudi ()
Additional contact information
Asim Roy: Department of Information Systems, Arizona State University, Tempe, Arizona 85287
Shiban Qureshi: Department of Information Systems, Arizona State University, Tempe, Arizona 85287
Kartikeya Pande: Department of Information Systems, Arizona State University, Tempe, Arizona 85287
Divitha Nair: Department of Information Systems, Arizona State University, Tempe, Arizona 85287
Kartik Gairola: Department of Information Systems, Arizona State University, Tempe, Arizona 85287
Pooja Jain: Department of Information Systems, Arizona State University, Tempe, Arizona 85287
Suraj Singh: Department of Information Systems, Arizona State University, Tempe, Arizona 85287
Kirti Sharma: Department of Information Systems, Arizona State University, Tempe, Arizona 85287
Akshay Jagadale: Department of Information Systems, Arizona State University, Tempe, Arizona 85287
Yi-Yang Lin: Department of Information Systems, Arizona State University, Tempe, Arizona 85287
Shashank Sharma: Department of Information Systems, Arizona State University, Tempe, Arizona 85287
Ramya Gotety: Department of Information Systems, Arizona State University, Tempe, Arizona 85287
Yuexin Zhang: Department of Information Systems, Arizona State University, Tempe, Arizona 85287
Ji Tang: Department of Information Systems, Arizona State University, Tempe, Arizona 85287
Tejas Mehta: Department of Information Systems, Arizona State University, Tempe, Arizona 85287
Hemanth Sindhanuru: Department of Information Systems, Arizona State University, Tempe, Arizona 85287
Nonso Okafor: Department of Information Systems, Arizona State University, Tempe, Arizona 85287
Santak Das: Department of Information Systems, Arizona State University, Tempe, Arizona 85287
Chidambara N. Gopal: Department of Information Systems, Arizona State University, Tempe, Arizona 85287
Srinivasa B. Rudraraju: Department of Information Systems, Arizona State University, Tempe, Arizona 85287
Avinash V. Kakarlapudi: Department of Information Systems, Arizona State University, Tempe, Arizona 85287

INFORMS Journal on Computing, 2019, vol. 31, issue 2, 207-225

Abstract: In this paper, we present a method for comparing and evaluating different collections of machine learning algorithms on the basis of a given performance measure (e.g., accuracy, area under the curve (AUC), F -score). Such a method can be used to compare standard machine learning platforms such as SAS, IBM SPSS, and Microsoft Azure ML. A recent trend in automation of machine learning is to exercise a collection of machine learning algorithms on a particular problem and then use the best performing algorithm. Thus, the proposed method can also be used to compare and evaluate different collections of algorithms for automation on a certain problem type and find the best collection. In the study reported here, we applied the method to compare six machine learning platforms – R, Python, SAS, IBM SPSS Modeler, Microsoft Azure ML, and Apache Spark ML. We compared the platforms on the basis of predictive performance on classification problems because a significant majority of the problems in machine learning are of that type. The general question that we addressed is the following: Are there platforms that are superior to others on some particular performance measure? For each platform, we used a collection of six classification algorithms from the following six families of algorithms – support vector machines, multilayer perceptrons, random forest (or variant), decision trees/gradient boosted trees, Naive Bayes/Bayesian networks, and logistic regression. We compared their performance on the basis of classification accuracy, F -score, and AUC. We used F -score and AUC measures to compare platforms on two-class problems only. For testing the platforms, we used a mix of data sets from (1) the University of California, Irvine (UCI) library, (2) the Kaggle competition library, and (3) high-dimensional gene expression problems. We performed some hyperparameter tuning on algorithms wherever possible.

Keywords: machine learning platforms; classification algorithms; comparison of algorithms; comparison of platforms (search for similar items in EconPapers)
Date: 2019
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (4)

Downloads: (external link)
https://doi.org/10.1287/ijoc.2018.0825 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:orijoc:v:31:y:2019:i:2:p:207-225

Access Statistics for this article

More articles in INFORMS Journal on Computing from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().