Determinants of firms' decisions to block AI crawlers: Strategic insights from the BEDC framework

Chiang-Yu Cheng

Technological Forecasting and Social Change, 2025, vol. 219, issue C

Abstract: The rise of AI crawlers has intensified firms' concerns over digital asset protection and competitive visibility. This study develops the BEDC Framework—encompassing Business Competition, Engineering Constraints, Domain Authority, and Commerce & Monetization—to conceptualize strategic determinants of AI crawler governance. An empirical analysis of 100 top-ranking global websites reveals significant variation in crawler restriction practices, highlighting the interplay between strategic imperatives and governance behaviors. The findings advance understanding of firm-centric technology governance in the context of automated data extraction, offering insights into how organizations strategically balance protection, accessibility, and monetization.

Keywords: AI Crawler Governance; Automated Data Extraction; digital asset protection; firm-centric technology governance (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0040162525003063
Full text for ScienceDirect subscribers only

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:tefoso:v:219:y:2025:i:c:s0040162525003063

DOI: 10.1016/j.techfore.2025.124275

Access Statistics for this article

Technological Forecasting and Social Change is currently edited by Fred Phillips

More articles in Technological Forecasting and Social Change from Elsevier
Bibliographic data for series maintained by Catherine Liu ().