Assessing causal claims about complex engineered systems with quantitative data: internal, external, and construct validity

David A. Broniatowski and Conrad Tucker

Systems Engineering, 2017, vol. 20, issue 6, 483-496

Abstract: Engineers seek to design systems that will produce an intended change in the state of the world. How are we to know if a system will behave as intended? This article addresses ways that this question can be answered. Specifically, we focus on three types of research validity: (1) internal validity, or whether an observed association between two variables can be attributed to a causal link between them; (2) external validity, or whether a causal link generalizes across contexts; and (3) construct validity, or whether a specific set of metrics corresponds to what they are intended to measure. In each case, we discuss techniques that may be used to establish the corresponding type of validity: namely, quasi‐experimental design, replication, and establishment of convergent‐discriminant validity and reliability. These techniques typically require access to data, which has historically been limited for research on complex engineered systems. This is likely to change in the era of “big data.” Thus, we discuss the continued utility of these validity concepts in the face of advances in machine learning and big data as they pertain to complex engineered sociotechnical systems. Next, we discuss relationships between these validity concepts and other prominent approaches to evaluating research in the field. Finally, we propose a set of criteria by which one may evaluate research utilizing quantitative observation to test causal theory in the field of complex engineered systems.

Date: 2017
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
https://doi.org/10.1002/sys.21414

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:wly:syseng:v:20:y:2017:i:6:p:483-496

Access Statistics for this article

More articles in Systems Engineering from John Wiley & Sons
Bibliographic data for series maintained by Wiley Content Delivery ().