The 2023 Soccer Prediction Challenge invited the machine learning community to develop innovative methods to predict the outcomes of 736 future soccer matches. The Challenge included two tasks. Task 1 was to forecast the exact match score, i.e., the number of goals scored by each team. Task 2 was to predict the match outcome as probability vector over the three possible result categories: victory of the home team, draw, and victory of the away team. Here, we present a new data- and knowledge-driven framework for building machine learning models from readily available data to predict soccer match outcomes. A key component of this framework is an innovative approach to modeling interdependent time series data of competing entities. Using this framework, we developed various predictive models based on k-nearest neighbors, artificial neural networks, naive Bayes, and ordinal forests, which we applied to the two tasks of the 2023 Soccer Prediction Challenge. Among all submissions to the Challenge, our machine learning models based on k-nearest neighbors and neural networks achieved top performances. Our main insights from the Challenge are that relatively simple learning algorithms perform remarkably well compared to more complex algorithms, and that the key to successful predictions lies in how well soccer domain knowledge can be incorporated in the modeling process.

Deep learning in bioinformatics and biomedicine Daniel Berrar, Daniel Berrar Data Science Laboratory, Department of Information and Communications Engineering, Tokyo Institute of Technology, Ookayama, Tokyo 152-8550, Japan Corresponding author: Daniel Berrar, Data Science Laboratory, Department of Information and Communications Engineering, Tokyo Institute of Technology, Ookayama, Tokyo 152-8550, Japan. E-mail: daniel.berrar@ict.e.titech.ac.jp Search for other works by this author on: Oxford Academic PubMed Google Scholar Werner Dubitzky Werner Dubitzky Freelance Data Scientist, Meitingen, Germany Search for other works by this author on: Oxford Academic PubMed Google Scholar Briefings in Bioinformatics, Volume 22, Issue 2, March 2021, Pages 1513–1514, https://doi.org/10.1093/bib/bbab087 Published: 10 March 2021

The task of the 2017 Soccer Prediction Challenge was to use machine learning to predict the outcome of future soccer matches based on a data set describing the match outcomes of 216,743 past soccer matches. One of the goals of the Challenge was to gauge where the limits of predictability lie with this type of commonly available data. Another goal was to pose a real-world machine learning challenge with a fixed time line, involving the prediction of real future events. Here, we present two novel ideas for integrating soccer domain knowledge into the modeling process. Based on these ideas, we developed two new feature engineering methods for match outcome prediction, which we denote as recency feature extraction and rating feature learning. Using these methods, we constructed two learning sets from the Challenge data. The top-ranking model of the 2017 Soccer Prediction Challenge was our k-nearest neighbor model trained on the rating feature learning set. In further experiments, we could slightly improve on this performance with an ensemble of extreme gradient boosted trees (XGBoost). Our study suggests that a key factor in soccer match outcome prediction lies in the successful incorporation of domain knowledge into the machine learning modeling process.

Significance testing has become a mainstay in machine learning, with the p value being firmly embedded in the current research practice. Significance tests are widely believed to lend scientific rigor to the interpretation of empirical findings; however, their problems have received only scant attention in the machine learning literature so far. Here, we investigate one particular problem, the Jeffreys–Lindley paradox. This paradox describes a statistical conundrum: the p value can be close to zero, convincing us that there is overwhelming evidence against the null hypothesis. At the same time, however, the posterior probability of the null hypothesis being true can be close to 1, convincing us of the exact opposite. In experiments with synthetic data sets and a subsequent thought experiment, we demonstrate that this paradox can have severe repercussions for the comparison of multiple classifiers over multiple benchmark data sets. Our main result suggests that significance tests should not be used in such comparative studies. We caution that the reliance on significance tests might lead to a situation that is similar to the reproducibility crisis in other fields of science. We offer for debate four avenues that might alleviate the looming crisis.

How well can machine learning predict the outcome of a soccer game, given the most commonly and freely available match data? To help answer this question and to facilitate machine learning research in soccer, we have developed the Open International Soccer Database. Version v1.0 of the Database contains essential information from 216,743 league soccer matches from 52 leagues in 35 countries. The earliest entries in the Database are from the year 2000, which is when football leagues generally adopted the “three points for a win” rule. To demonstrate the use of the Database for machine learning research, we organized the 2017 Soccer Prediction Challenge. One of the goals of the Challenge was to estimate where the limits of predictability lie, given the type of match data contained in the Database. Another goal of the Challenge was to pose a real-world machine learning problem with a fixed time line and a genuine prediction task: to develop a predictive model from the Database and then to predict the outcome of the 206 future soccer matches taking place from 31 March 2017 to the end of the regular season. The Open International Soccer Database is released as an open science project, providing a valuable resource for soccer analysts and a unique benchmark for advanced machine learning methods. Here, we describe the Database and the 2017 Soccer Prediction Challenge and its results.

In a recent crowdsourcing project, 29 teams analyzed the same data set to address the following question: “Are football (soccer) referees more likely to give red cards to players with dark skin tone than to players with light skin tone?” The major finding was that the results of the individual teams varied widely, from no effect to highly significant correlations between skin color and the rate of red cards, which some teams interpreted as indicative of a referee bias. We analyzed the same data using a Poisson log-linear regression model and obtained an odds ratio of 1.34 (95%-CI, 1.13–1.59), which means that players with a darker skin tone have in fact a slightly higher odds of receiving a red card. This result is in agreement with the median odds ratio of 1.31 from all 29 teams. We then extended the original study by investigating the likelihood of receiving yellow cards. If a referee bias was in fact present, it would be plausible to see a similar association. However, players with darker skin tone were significantly less likely to receive a yellow card, with an odds ratio of 0.94 (95%-CI, 0.91–0.97). The risk of receiving a card is most strongly affected by a player’s position, and there are significantly more players with darker skin tone at center back and defensive midfield where receiving red cards is generally more likely. Taken together, our results do not support the hypothesis of a referee bias. Our most important finding, however, is that the perceived diversity of results from the crowdsourcing teams is due to placing too much emphasis on dichotomous decisions (significant vs. nonsignificant). When we focus on point estimates and their reasonable bounds, the individual substudies predominantly reinforce each other. We argue that data scientists should put less emphasis on statistical significance and instead focus more on the careful interpretation of confidence intervals or alternative methods for measuring the effect size and its precision.

Null hypothesis significance testing has become a mainstay in machine learning, with the p-value being firmly embedded in the current research practice. Significance testing is widely believed to lend scientific rigor to the interpretation of empirical findings; however, its serious problems have received scant attention in the machine learning literature so far. Here, we investigate one particular problem: the Jeffreys-Lindley paradox. This paradox describes a statistical conundrum where the frequentist and Bayesian interpretation are diametrically opposed. In four experiments using synthetic data sets and a subsequent thought experiment, we demonstrate that this paradox has severe, real consequences for the current research practice. We caution that this practice might lead to a situation that is similar to the current reproducibility crisis in other fields of science. We offer for debate four avenues that might avert the looming crisis.

Modeling and simulation of gene-regulatory networks (GRNs) has become an important aspect of modern systems biology investigations into mechanisms underlying gene regulation. A key task in this area is the automated inference or reverse-engineering of dynamic mechanistic GRN models from gene expression time-course data. Besides a lack of suitable data (in particular multi-condition data from the same system), one of the key challenges of this task is the computational complexity involved. The more genes in the GRN system and the more parameters a GRN model has, the higher the computational load. The computational challenge is likely to increase substantially in the near future when we tackle larger GRN systems. The goal of this study was to develop a distributed computing framework and system for reverse-engineering of GRN models. We present the resulting software called MultiGrain/MAPPER. This software is based on a new architecture and tools supporting multiscale computing in a distributed computing environment. A key feature of MultiGrain/MAPPER is the realization of GRN reverse-engineering based on the underlying distributed computing framework and multi-swarm particle swarm optimization. We demonstrate some of the features of MultiGrain/MAPPER and evaluate its performance using both real and artificial gene expression data.

Systems medicine is the application of systems biology concepts, methods, and tools to medical research and practice. It aims to integrate data and knowledge from different disciplines into biomedical models and simulations for the understanding, prevention, cure, and management of complex diseases. Complex diseases arise from the interactions among disease-influencing factors across multiple levels of biological organization from the environment to molecules. To tackle the enormous challenges posed by complex diseases, we need a modeling and simulation framework capable of capturing and integrating information originating from multiple spatiotemporal and organizational scales. Multiscale modeling and simulation in systems medicine is an emerging methodology and discipline that has already demonstrated its potential in becoming this framework. The aim of this chapter is to present some of the main concepts, requirements, and challenges of multiscale modeling and simulation in systems medicine.

Modelling and simulation of gene-regulatory networks (GRNs) has become an important aspect of modern systems biology investigations. An important and unsolved problem in this area is the automated inference (reverse-engineering) of dynamic mechanistic GRN models from gene-expression time-course data. The conventional single-stage algorithm determines the values of all model parameters simultaneously, whereas recent two-stage algorithms can potentially improve the performance (accuracy) of single-stage approaches. The objective of this study is to compare the performance of the conventional single-stage and a novel version of the modern two-stage algorithm. We based this study on our implementation of a multi-swarm particle swarm optimisation process. A particular focus of this study is placed on the comparison of the computational performance of the single-stage vs. two-stage algorithm. Our results suggest that the 2-stage approach outperforms the single-stage methods by far in terms of model inference speed without loss of accuracy.

Modeling and simulation of gene-regulatory networks (GRNs) has become an important aspect of modern systems biology investigations into mechanisms underlying gene regulation. A key challenge in this area is the automated inference (reverse-engineering) of dynamic, mechanistic GRN models from gene expression time-course data. Common mathematical formalisms for representing such models capture two aspects simultaneously within a single parameter: (1) Whether or not a gene is regulated, and if so, the type of regulator (activator or repressor), and (2) the strength of influence of the regulator (if any) on the target or effector gene. To accommodate both roles, "generous" boundaries or limits for possible values of this parameter are commonly allowed in the reverse-engineering process. This approach has several important drawbacks. First, in the absence of good guidelines, there is no consensus on what limits are reasonable. Second, because the limits may vary greatly among different reverse-engineering experiments, the concrete values obtained for the models may differ considerably, and thus it is difficult to compare models. Third, if high values are chosen as limits, the search space of the model inference process becomes very large, adding unnecessary computational load to the already complex reverse-engineering process. In this study, we demonstrate that restricting the limits to the [-1, +1] interval is sufficient to represent the essential features of GRN systems and offers a reduction of the search space without loss of quality in the resulting models. To show this, we have carried out reverse-engineering studies on data generated from artificial and experimentally determined from real GRN systems.

Modeling and simulation of gene-regulatory networks (GRNs) has become an important aspect of modern computational biology investigations into gene regulation. A key challenge in this area is the automated inference (reverse-engineering) of dynamic, mechanistic GRN models from time-course gene expression data. Common mathematical formalisms used to represent such models capture both the relative weight or strength of a regulator gene and the type of the regulator (activator, repressor) with a single model parameter. The goal of this study is to quantify the role this parameter plays in terms of the computational performance of the reverse-engineering process and the predictive power of the inferred GRN models. We carried out three sets of computational experiments on a GRN system consisting of 22 genes. While more comprehensive studies of this kind are ultimately required, this computational study demonstrates that models with similar training (reverse-engineering) error that have been inferred under varying degrees of a priori known topology information, exhibit considerably different predictive performance. This study was performed with a newly developed multiscale modeling and simulation tool called MultiGrain/MAPPER.

Modeling and simulation of gene-regulatory networks (GRNs) has become an important aspect of modern systems biology investigations into mechanisms underlying gene regulation. An important and unsolved problem in this area is the automated inference (reverse-engineering) of dynamic, mechanistic GRN models from time-course gene expression data. The conventional one-stage model inference algorithm determines the values of all model parameters simultaneously. Recently, two-stage algorithms have been proposed to improve the accuracy of the inferred models and the efficiency of the reverse-engineering process. The main objective of this study is to compare the performance of the conventional one-stage and the modern two-stage algorithm, with emphasis on the computational complexity. We explored data generated from artificial and real GRN systems under different experimental conditions and regulatory structure constraints. Our results suggest that the 2-stage approach outperforms the one-stage methods by far in terms of model inference speed without a loss of accuracy.