PAPERCLIP: Associating Astronomical Observations and Natural Language with Multi-Modal Models
CoRR(2024)
摘要
We present PAPERCLIP (Proposal Abstracts Provide an Effective Representation
for Contrastive Language-Image Pre-training), a method which associates
astronomical observations imaged by telescopes with natural language using a
neural network model. The model is fine-tuned from a pre-trained Contrastive
Language-Image Pre-training (CLIP) model using successful observing proposal
abstracts and corresponding downstream observations, with the abstracts
optionally summarized via guided generation using large language models (LLMs).
Using observations from the Hubble Space Telescope (HST) as an example, we show
that the fine-tuned model embodies a meaningful joint representation between
observations and natural language through tests targeting image retrieval
(i.e., finding the most relevant observations using natural language queries)
and description retrieval (i.e., querying for astrophysical object classes and
use cases most relevant to a given observation). Our study demonstrates the
potential for using generalist foundation models rather than task-specific
models for interacting with astronomical data by leveraging text as an
interface.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要