Boba: Authoring and Visualizing Multiverse Analyses

Yang Liu

Alex Kale

Tim Althoff

[0]

Jeffrey Heer

[0]

IEEE Trans. Vis. Comput. Graph., pp. 1753-1763, 2021.

被引用：1|引用|浏览81|DOI:https://doi.org/10.1109/TVCG.2020.3028985

WOS EI

其它链接：arxiv.org|pubmed.ncbi.nlm.nih.gov|academic.microsoft.com|dblp.uni-trier.de

关键词：

domain-specific languagedirected acyclic graphmodel fitvisual analysis systemdatum analysis更多(7+)

微博一下：

This paper presents Boba, an integrated domain-specific language and visual analysis system for authoring and interpreting multiverse analyses

摘要：

Multiverse analysis is an approach to data analysis in which all "reasonable" analytic decisions are evaluated in parallel and interpreted collectively, in order to foster robustness and transparency. However, specifying a multiverse is demanding because analysts must manage myriad variants from a cross-product of analytic decisions, and ...更多

代码：

数据：

精读

下载 PDF 全文

PPT (上传PPT)

简介

The last decade saw widespread failure to replicate findings in published literature across multiple scientific fields [2, 6, 35, 41].
As the replication crisis emerged [1], scholars began to re-examine how data analysis practices might lead to spurious findings.
An important contributing factor is the flexibility in making analytic decisions [16,17,48].
Flexibility in making decisions might inflate false-positive rates when researchers explore multiple alternatives and selectively report desired outcomes [48], a practice known as p-hacking [34].
A crowdsourced study [47] shows that well-

重点内容

The last decade saw widespread failure to replicate findings in published literature across multiple scientific fields [2, 6, 35, 41]
An important contributing factor is the flexibility in making analytic decisions [16,17,48]
We enable users to take into account sampling uncertainty and model fit by comparing observed data with model predictions [14]
This paper presents Boba, an integrated domain-specific language (DSL) and visual analysis system for authoring and interpreting multiverse analyses
With the DSL, users annotate their analysis script to insert local variations, from which the compiler synthesizes executable script variants corresponding to all compatible analysis paths
We provide a command line tool for compiling the DSL specification, running the generated scripts, merging the outputs, and invoking the visual analysis system

方法

DESIGN REQUIREMENTS

The authors' overarching goal is to make it easier for researchers to conduct multiverse analyses.
As noted in prior work [12, 30], specifying a multiverse is tedious.
This is primarily because a multiverse is composed of many forking paths, yet non-linear program structures are not well supported in conventional tools [45].

结论

Through the process of designing, building, and using Boba, the authors gain insights into challenges that multiverse analysis poses for software designers and users.
One strategy is to represent analysis goals in higher-level abstractions, from which appropriate analysis methods might be synthesized [22].
Another is to guide less experienced users through key decision points and possible alternatives [30], starting from an initial script.This paper presents Boba, an integrated DSL and visual analysis system for authoring and interpreting multiverse analyses.
Boba is available as open source software at https://github.com/uwdata/boba

总结

Introduction:
The last decade saw widespread failure to replicate findings in published literature across multiple scientific fields [2, 6, 35, 41].
As the replication crisis emerged [1], scholars began to re-examine how data analysis practices might lead to spurious findings.
An important contributing factor is the flexibility in making analytic decisions [16,17,48].
Flexibility in making decisions might inflate false-positive rates when researchers explore multiple alternatives and selectively report desired outcomes [48], a practice known as p-hacking [34].
A crowdsourced study [47] shows that well-
Methods:
DESIGN REQUIREMENTS

The authors' overarching goal is to make it easier for researchers to conduct multiverse analyses.
As noted in prior work [12, 30], specifying a multiverse is tedious.
This is primarily because a multiverse is composed of many forking paths, yet non-linear program structures are not well supported in conventional tools [45].
Conclusion:
Through the process of designing, building, and using Boba, the authors gain insights into challenges that multiverse analysis poses for software designers and users.
One strategy is to represent analysis goals in higher-level abstractions, from which appropriate analysis methods might be synthesized [22].
Another is to guide less experienced users through key decision points and possible alternatives [30], starting from an initial script.This paper presents Boba, an integrated DSL and visual analysis system for authoring and interpreting multiverse analyses.
Boba is available as open source software at https://github.com/uwdata/boba

Boba: Authoring and Visualizing Multiverse Analyses

Introduction:

Methods:

Conclusion: