Estimating Network Effects Using Naturally Occurring Peer Notification Queue Counterfactuals

Craig Tutterow and Guillaume Saint-Jacques

Papers from arXiv.org

Abstract: Randomized experiments, or A/B tests are used to estimate the causal impact of a feature on the behavior of users by creating two parallel universes in which members are simultaneously assigned to treatment and control. However, in social network settings, members interact, such that the impact of a feature is not always contained within the treatment group. Researchers have developed a number of experimental designs to estimate network effects in social settings. Alternatively, naturally occurring exogenous variation, or 'natural experiments,' allow researchers to recover causal estimates of peer effects from observational data in the absence of experimental manipulation. Natural experiments trade off the engineering costs and some of the ethical concerns associated with network randomization with the search costs of finding situations with natural exogenous variation. To mitigate the search costs associated with discovering natural counterfactuals, we identify a common engineering requirement used to scale massive online systems, in which natural exogenous variation is likely to exist: notification queueing. We identify two natural experiments on the LinkedIn platform based on the order of notification queues to estimate the causal impact of a received message on the engagement of a recipient. We show that receiving a message from another member significantly increases a member's engagement, but that some popular observational specifications, such as fixed-effects estimators, overestimate this effect by as much as 2.7x. We then apply the estimated network effect coefficients to a large body of past experiments to quantify the extent to which it changes our interpretation of experimental results. The study points to the benefits of using messaging queues to discover naturally occurring counterfactuals for the estimation of causal effects without experimenter intervention.

Date: 2019-02
New Economics Papers: this item is included in nep-exp
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://arxiv.org/pdf/1902.07133 Latest version (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:arx:papers:1902.07133

Access Statistics for this paper

More papers in Papers from arXiv.org
Bibliographic data for series maintained by arXiv administrators ().