Astronomical data is rich in volume, information and facets. Although this offers multiple research perspectives, processing the data remains a challenge. Infrastructures for analyzing, inspecting, exploring and communicating with data are mandatory. To address this issue, we introduce Jasmine, the JAvaScript Multimodal INformation Explorer. Jasmine allows users to open different data viewer modals that show a specific data point from a set. The viewer currently supports image data, as well as point cloud objects. Users can decide on which information about the data point they like to have displayed. Point clouds are interactive and allow for zooming, tossing, and turning. Picking a data point is enabled by providing a structured view of the set, arranged by a key property. This arrangement is achieved by autoencoding.

High Performance Computing based simulations are crucial in Astrophysics and Cosmology, helping scientists investigate and understand complex astrophysical phenomena. Taking advantage of Exascale computing capabilities is essential for these efforts. However, the unprecedented architectural complexity of exascale systems impacts simulation codes. The SPACE Center of Excellence aims to re-engineer key astrophysical codes to adapt to these new computational challenges by adopting innovative programming paradigms and software solutions. Through co-design activities, SPACE brings together scientists, code developers, HPC experts, hardware manufacturers, and software developers. This collaboration enhances exascale astrophysics and cosmology applications, promoting the use of exascale and post-exascale computing capabilities. Additionally, SPACE addresses high-performance data analysis for the massive data outputs from exascale simulations, using machine learning and visualization tools. The project facilitates application deployment across platforms by focusing on code repositories and data sharing, integrating European astrophysical communities around exascale computing with standardized software and data protocols. In this paper, we present the SPACE Center of Excellence and the preliminary results achieved by the project.

Simulations are the best approximation to experimental laboratories in astrophysics and cosmology. However, the complexity, richness, and large size of their outputs severely limit the interpretability of their predictions. We describe a new, unbiased, and machine learning based approach to obtaining useful scientific insights from a broad range of simulations. The method can be used on today's largest simulations and will be essential to solve the extreme data exploration and analysis challenges posed by the Exascale era. Furthermore, this concept is so flexible, that it will also enable explorative access to observed data. Our concept is based on applying nonlinear dimensionality reduction to learn compact representations of the data in a low-dimensional space. The simulation data is projected onto this space for interactive inspection, visual interpretation, sample selection, and local analysis. We present a prototype using a rotational invariant hyperspherical variational convolutional autoencoder, utilizing a power distribution in the latent space, and trained on galaxies from IllustrisTNG simulation. Thereby, we obtain a natural Hubble tuning fork like similarity space that can be visualized interactively on the surface of a sphere by exploiting the power of HiPS tilings in Aladin Lite.

Modern wide field radio surveys typically detect millions of objects. Manual determination of the morphologies is impractical for such a large number of radio sources. Techniques based on machine learning are proving to be useful for classifying large numbers of objects. The self-organizing map (SOM) is an unsupervised machine learning algorithm that projects a many-dimensional dataset onto a two- or three-dimensional lattice of neurons. This dimensionality reduction allows the user to visualize common features of the data better and develop algorithms for classifying objects that are not otherwise possible with large datasets. To this aim, we use the PINK implementation of a SOM. PINK incorporates rotation and flipping invariance so that the SOM algorithm may be applied to astronomical images. In this cookbook we provide instructions for working with PINK, including preprocessing the input images, training the model, and offering lessons learned through experimentation. The problem of imbalanced classes can be improved by careful selection of the training sample and increasing the number of neurons in the SOM (chosen by the user). Because PINK is not scale-invariant, structure can be smeared in the neurons. This can also be improved by increasing the number of neurons in the SOM. We also introduce pyink, a Python package used to read and write PINK binary files, assist in common preprocessing operations, perform standard analyses, visualize the SOM and preprocessed images, and create image-based annotations using a graphical interface. A tutorial is also provided to guide the user through the entire process. We present an application of PINK to VLA Sky Survey (VLASS) images. We demonstrate that the PINK is generally able to group VLASS sources with similar morphology together. We use the results of PINK to estimate the probability that a given source in the VLASS QuickLook Catalogue is actually due to sidelobe contamination.

Nowadays, astronomers perform point spread function (PSF) fitting for most types of observational data. Interpolation of the PSF is often an intermediate step in such algorithms. In the case of the Multi-AO Imaging Camera for Deep Observations (MICADO) at the Extremely Large Telescope (ELT), PSF interpolation will play a crucial role in high-precision astrometry for stellar clusters and confirmation of the Intermediate-Mass Black Holes (IMBHs) presence. Significant PSF variations across the field of view invalidate the approach of deconvolution with a mean PSF or on-axis PSF. The ignoring of PSF variations can be especially unsatisfactory in the case of Single Conjugate Adaptive Optics (SCAO) observations, as these sophisticated and expensive systems are designed to achieve high resolution with ground-based telescopes by correcting for atmospheric turbulence in the direction of one reference star. In plenty of tasks, you face the question: How can I establish the quality of PSF fitting or interpolation? Our study aims to demonstrate the variety of PSF quality metrics, including the problem of revealing IMBHs in stellar clusters.

Context. We present a probabilistic cross-correlation approach to estimate time delays in the context of reverberation mapping (RM) of active galactic nuclei (AGN).Aims. We reformulate the traditional interpolated cross-correlation method as a statistically principled model that delivers a posterior distribution for the delay.Methods. The method employs Gaussian processes as a model for observed AGN light curves. We describe the mathematical formalism and demonstrate the new approach using both simulated light curves and available RM observations.Results. The proposed method delivers a posterior distribution for the delay that accounts for observational noise and the non-uniform sampling of the light curves. This feature allows us to fully quantify the uncertainty on the delay and propagate it to subsequent calculations of dependant physical quantities, such as black hole masses. The method delivers out-of-sample predictions, which enables us to subject it to model selection, and can calculate the joint posterior delay for more than two light curves.Conclusions. Because of the numerous advantages of our reformulation and the simplicity of its application, we anticipate that our method will find favour not only in the specialised community of RM, but also in all fields where cross-correlation analysis is performed. We provide the algorithms and examples of their application as part of our Julia GPCC package.

Photometric reverberation mapping can detect the radial extent of the accretion disc (AD) in Active Galactic Nuclei by measuring the time delays between light curves observed in different continuum bands. Quantifying the constraints on the efficiency and accuracy of the delay measurements is important for recovering the AD size-luminosity relation, and potentially using quasars as standard candles. We have explored the possibility of determining the AD size of quasars using next-generation Big Data surveys. We focus on the Legacy Survey of Space and Time (LSST) at the Vera C. Rubin Observatory, which will observe several thousand quasars with the Deep Drilling Fields and up to 10 million quasars for the main survey in six broadband filter during its 10-year operational lifetime. We have developed extensive simulations that take into account the characteristics of the LSST survey and the intrinsic properties of the quasars. The simulations are used to characterise the light curves from which AD sizes are determined using various algorithms. We find that the time delays can be recovered with an accuracy of 5 and 15% for light curves with a time sampling of 2 and 5 days, respectively. The results depend strongly on the redshift of the source and the relative contribution of the emission lines to the bandpasses. Assuming an optically thick and geometrically thin AD, the recovered time-delay spectrum is consistent with black hole masses derived with 30% uncertainty.

We provide a brief, and inevitably incomplete overview of the use of Machine Learning (ML) and other AI methods in astronomy, astrophysics, and cosmology. Astronomy entered the big data era with the first digital sky surveys in the early 1990s and the resulting Terascale data sets, which required automating of many data processing and analysis tasks, for example the star-galaxy separation, with billions of feature vectors in hundreds of dimensions. The exponential data growth continued, with the rise of synoptic sky surveys and the Time Domain Astronomy, with the resulting Petascale data streams and the need for a real-time processing, classification, and decision making. A broad variety of classification and clustering methods have been applied for these tasks, and this remains a very active area of research. Over the past decade we have seen an exponential growth of the astronomical literature involving a variety of ML/AI applications of an ever increasing complexity and sophistication. ML and AI are now a standard part of the astronomical toolkit. As the data complexity continues to increase, we anticipate further advances leading towards a collaborative human-AI discovery.

Ensemble weather predictions typically show systematic errors that have to be corrected via post-processing. Even state-of-the-art post-processing methods based on neural networks often solely rely on location-specific predictors that require an interpolation of the physical weather model's spatial forecast fields to the target locations. However, potentially useful predictability information contained in large-scale spatial structures within the input fields is potentially lost in this interpolation step. Therefore, we propose the use of convolutional autoencoders to learn compact representations of spatial input fields which can then be used to augment location-specific information as additional inputs to post-processing models. The benefits of including this spatial information is demonstrated in a case study of 2-m temperature forecasts at surface stations in Germany.

Context. We present a novel probabilistic flux variation gradient (PFVG) approach to separate the contributions of active galactic nuclei (AGN) and host galaxies in the context of photometric reverberation mapping (PRM) of AGN. Aims. We explored the ability of recovering the fractional contribution in a model-independent way using the entire set of light curves obtained through different filters and photometric apertures simultaneously. Methods. The method is based on the observed “bluer when brighter” phenomenon that is attributed to the superimposition of a two-component structure; the red host galaxy, which is constant in time, and the varying blue AGN. We describe the PFVG mathematical formalism and demonstrate its performance using simulated light curves and available PRM observations. Results. The new probabilistic approach is able to recover host-galaxy fluxes to within 1% precision as long as the light curves do not show a significant contribution from time delays. This represents a significant improvement with respect to previous applications of the traditional FVG method to PRM data. Conclusions. The proposed PFVG provides an efficient and accurate way to separate the AGN and host-galaxy luminosities in PRM monitoring data. The method will be especially helpful in the case of large upcoming photometric survey telescopes such as the public optical/near-infrared Legacy Survey of Space and Time (LSST) at the Vera C. Rubin Observatory. Finally, we have made the algorithms freely available as part of our Julia PFVG package.

The amount, size, and complexity of astronomical data-sets and databases are growing rapidly in the last decades, due to new technologies and dedicated survey telescopes. Besides dealing with poly-structured and complex data, sparse data has become a field of growing scientific interest. A specific field of Astroinformatics research is the estimation of redshifts of extra-galactic sources by using sparse photometric observations. Many techniques have been developed to produce those estimates with increasing precision. In recent years, models have been favored which instead of providing a point estimate only, are able to generate probabilistic density functions (PDFs) in order to characterize and quantify the uncertainties of their estimates. Crucial to the development of those models is a proper, mathematically principled way to evaluate and characterize their performances, based on scoring functions as well as on tools for assessing calibration. Still, in literature inappropriate methods are being used to express the quality of the estimates that are often not sufficient and can potentially generate misleading interpretations. In this work we summarize how to correctly evaluate errors and forecast quality when dealing with PDFs. We describe the use of the log-likelihood, the continuous ranked probability score (CRPS) and the probability integral transform (PIT) to characterize the calibration as well as the sharpness of predicted PDFs. We present what we achieved when using proper scoring rules to train deep neural networks as well as to evaluate the model estimates and how this work led from well calibrated redshift estimates to improvements in probabilistic weather forecasting. The presented work is an example of interdisciplinarity in data-science and illustrates how methods can help to bridge gaps between different fields of application.

Context. The Low Frequency Array (LOFAR) Two-metre Sky Survey (LoTSS) is a low-frequency radio continuum survey of the Northern sky at an unparalleled resolution and sensitivity.Aims. In order to fully exploit this huge dataset and those produced by the Square Kilometre Array in the next decade, automated methods in machine learning and data-mining will be increasingly essential both for morphological classifications and for identifying optical counterparts to the radio sources.Methods. Using self-organising maps (SOMs), a form of unsupervised machine learning, we created a dimensionality reduction of the radio morphologies for the similar to 25k extended radio continuum sources in the LoTSS first data release, which is only similar to 2 percent of the final LoTSS survey. We made use of PINK, a code which extends the SOM algorithm with rotation and flipping invariance, increasing its suitability and effectiveness for training on astronomical sources.Results. After training, the SOMs can be used for a wide range of science exploitation and we present an illustration of their potential by finding an arbitrary number of morphologically rare sources in our training data (424 square degrees) and subsequently in an area of the sky (similar to 5300 square degrees) outside the training data. Objects found in this way span a wide range of morphological and physical categories: extended jets of radio active galactic nuclei, diffuse cluster haloes and relics, and nearby spiral galaxies. Finally, to enable accessible, interactive, and intuitive data exploration, we showcase the LOFAR-PyBDSF Visualisation Tool, which allows users to explore the LoTSS dataset through the trained SOMs.

We develop a new analysis approach towards identifying related radio components and their corresponding infrared host galaxy based on unsupervised machine learning methods. By exploiting PINK, a self-organising map algorithm, we are able to associate radio and infrared sources without the a priori requirement of training labels. We present an example of this method using 894,415 images from the FIRST and WISE surveys centred towards positions described by the FIRST catalogue. We produce a set of catalogues that complement FIRST and describe 802,646 objects, including their radio components and their corresponding AllWISE infrared host galaxy. Using these data products we (i) demonstrate the ability to identify objects with rare and unique radio morphologies (e.g. 'X'-shaped galaxies, hybrid FR-I/FR-II morphologies), (ii) can identify the potentially resolved radio components that are associated with a single infrared host and (iii) introduce a "curliness" statistic to search for bent and disturbed radio morphologies, and (iv) extract a set of 17 giant radio galaxies between 700-1100 kpc. As we require no training labels, our method can be applied to any radio-continuum survey, provided a sufficiently representative SOM can be trained.

Machine learning has become a key tool to analyze and process complexly structured large datasets. This BoF will be different than the usual BoFs, with the aim of discussing a specific topic. Due to the large request in understanding and learning machine learning techniques at the previously organized ADASS, we will take the opportunity to introduce basic concepts of machine learning. Based on a few examples, different machine learning models will be introduced and their application will be shown. At the end of the BoF, participants will have a basic understanding of what machine learning is about. To enable the participants to further learn about machine learning and to allow for a broader overview, a list of good online-sources will be provided.

In many applications in astronomy, uncertainty quantification plays an important role. Probability density functions (PDFs) allow us to quantify the likelihood of results and therefore enable scientist to produce better analysis results. We present a Python package to generate PDFs for regression tasks. Besides providing several functionalities to generate such PDFs, we present a whole tool set for evaluating the quality and visualizing the performance of the generated PDFs. Photometric redshifts are an important measure of distance for various cosmological topics. As spectroscopic redshifts are only available for a very limited set of objects, statistical regression models are helpful to derive estimates based on photometric measurements. We use the example of generating the photometric redshift PDFs of quasars from SDSS(DR7) based on psf magnitudes to present the functionalities of ProbReg.

Future radio surveys will generate catalogs of tens of millions of radio sources, for which redshift estimates will be essential to achieve many of the science goals. However, spectroscopic data will be available for only a small fraction of these sources, and in most cases even the optical and infrared photometry will be of limited quality. Furthermore, radio sources tend to be at higher redshift than most optical sources (most radio surveys have a median redshift greater than 1) and so a significant fraction of radio sources hosts differ from those for which most photometric redshift templates are designed. We therefore need to develop new techniques for estimating the redshifts of radio sources. As a starting point in this process, we evaluate a number of machine-learning techniques for estimating redshift, together with a conventional template-fitting technique. We pay special attention to how the performance is affected by the incompleteness of the training sample and by sparseness of the parameter space or by limited availability of ancillary multiwavelength data. As expected, we find that the quality of the photometric-redshift degrades as the quality of the photometry decreases, but that even with the limited quality of photometry available for all-sky-surveys, useful redshift information is available for the majority of sources, particularly at low redshift. We find that a template-fitting technique performs best in the presence of high-quality and almost complete multi-band photometry, especially if radio sources that are also X-ray emitting are treated separately, using specific templates and priors. When we reduced the quality of photometry to match that available for the EMU all-sky radio survey, the quality of the template-fitting degraded and became comparable to some of the machine-learning methods. Machine learning techniques currently perform better at low redshift than at high redshift, because of incompleteness of the currently available training data at high redshifts.

With the advent of large scale-surveys the manual analysis and classification of individual radio source morphologies is rendered impossible as existing approaches do not scale. The analysis of complex morphological features in the spatial domain is a particularly important task. Here, we discuss the challenges of transferring crowdsourced labels obtained from the Radio Galaxy Zoo project and introduce a proper transfer mechanism via quantile random forest regression. By using parallelized rotation and flipping invariant Kohonen-maps, image cubes of Radio Galaxy Zoo selected galaxies formed from the Faint Images of the Radio Sky at Twenty-cm (FIRST) radio continuum and the Wide-field Infrared Survey Explorer (WISE) infrared all-sky surveys are first projected down to a two-dimensional embedding in an unsupervised way. This embedding can be seen as a discretized space of shapes with the coordinates reflecting morphological features as expressed by the automatically derived prototypes. We find that these prototypes have reconstructed physically meaningful processes across two channel images at radio and infrared wavelengths in an unsupervised manner. In the second step, images are compared with those prototypes to create a heat map, which is the morphological fingerprint of each object and the basis for transferring the user generated labels. These heat maps have reduced the feature space by a factor of 248, and are able to be used as the basis for subsequent machine-learning (ML) methods. Using an ensemble of decision trees we achieve upwards of 85.7% and 80.7% accuracy when predicting the number of components and peaks in an image, respectively, using these heat maps. We also question the currently used discrete classification schema and introduce a continuous scale that better reflects the uncertainty in transition between two classes, caused by sensitivity and resolution limits.

We present the results of a two year optical continuum photometric reverberation mapping campaign carried out on the nucleus of the Seyfert-1 galaxy Mrk509. Specially designed narrow-band filters were used in order to mitigate the line and pseudo-continuum contamination of the signal from the broad line region, while allowing for high-accuracy flux-calibration over a large field of view. We obtained light curves with a sub-day time sampling and typical flux uncertainties of 1%. The high photometric precision allowed us to measure inter-band continuum time delays of up to ∼2 days across the optical range. The time delays are consistent with the relation τ∝λ4/3 predicted for an optically thick and geometrically thin accretion disk model. The size of the disk is, however, a factor of 1.8 larger than predictions based on the standard thin-disk theory. We argue that, for the particular case of Mrk509, a larger black hole mass due to the unknown geometry scaling factor can reconcile the difference between the observations and theory.