Gene regulatory network (GRN) models are widely used to investigate gene-expression dynamics in biological systems. evolutionary GRN model proposed by Kaneko (2007), which investigated the relationship between robustness to nongenetic phenotypic noise and robustness to mutation. We explicitly reconstruct the simulation protocol by specifying computational details omitted from the original study and reproduce its main qualitative findings, including the noise-dependent suppression of low-fitness individuals, the relationship between isogenic phenotypic variance and genetic variance, and the increased robustness of evolved networks under certain expression noise. We further analyze the robustness of evolved populations to variations in initial conditions. While the representative-network analysis shows quantitative differences from the original study, the population-level analysis supports the qualitative conclusion that evolution under higher phenotypic noise leads to broader basins of attraction. We provide a reusable implementation and publicly release the source code to support reproducibility and facilitate further studies on robustness in dynamical GRN models.

We consider sparse signal reconstruction via minimization of the smoothly clipped absolute deviation (SCAD) penalty, and develop one-step replica-symmetry-breaking (1RSB) extensions of approximate message passing (AMP), termed 1RSB-AMP. Starting from the 1RSB formulation of belief propagation, we derive explicit update rules of 1RSB-AMP together with the corresponding state evolution (1RSB-SE) equations. A detailed comparison shows that 1RSB-AMP and 1RSB-SE agree remarkably well at the macroscopic level, even in parameter regions where replica-symmetric (RS) AMP, termed RS-AMP, diverges and where the 1RSB description itself is not expected to be thermodynamically exact. Fixed-point analysis of 1RSB-SE reveals a phase diagram consisting of success, failure, and diverging phases, as in the RS case. However, the diverging-region boundary now depends on the Parisi parameter due to the 1RSB ansatz, and we propose a new criterion – minimizing the size of the diverging region – rather than the conventional zero-complexity condition, to determine its value. Combining this criterion with the nonconvexity-control (NCC) protocol proposed in a previous RS study improves the algorithmic limit of perfect reconstruction compared with RS-AMP. Numerical solutions of 1RSB-SE and experiments with 1RSB-AMP confirm that this improved limit is achieved in practice, though the gain is modest and remains slightly inferior to the Bayes-optimal threshold. We also report the behavior of thermodynamic quantities – overlaps, free entropy, complexity, and the non-self-averaging susceptibility – that characterize the 1RSB phase in this problem.

This paper examines the quantization methods used in large-scale data analysis models and their hyperparameter choices. The recent surge in data analysis scale has significantly increased computational resource requirements. To address this, quantizing model weights has become a prevalent practice in data analysis applications such as deep learning. Quantization is particularly vital for deploying large models on devices with limited computational resources. However, the selection of quantization hyperparameters, like the number of bits and value range for weight quantization, remains an underexplored area. In this study, we employ the typical case analysis from statistical physics, specifically the replica method, to explore the impact of hyperparameters on the quantization of simple learning models. Our analysis yields three key findings: (i) an unstable hyperparameter phase, known as replica symmetry breaking, occurs with a small number of bits and a large quantization width; (ii) there is an optimal quantization width that minimizes error; and (iii) quantization delays the onset of overparameterization, which mitigate overfitting as indicated by the double descent phenomenon. We also discover that non-uniform quantization can enhance stability. Additionally, we develop an approximate message-passing algorithm to validate our theoretical results.

We discuss the prediction accuracy of assumed statistical models in terms of prediction errors for the generalized linear model and penalized maximum likelihood methods. We derive the forms of estimators for the prediction errors, such as $C_p$ criterion, information criteria, and leave-one-out cross validation (LOOCV) error, using the generalized approximate message passing (GAMP) algorithm and replica method. These estimators coincide with each other when the number of model parameters is sufficiently small; however, there is a discrepancy between them in particular in the parameter region where the number of model parameters is larger than the data dimension. In this paper, we review the prediction errors and corresponding estimators, and discuss their differences. In the framework of GAMP, we show that the information criteria can be expressed by using the variance of the estimates. Further, we demonstrate how to approach LOOCV error from the information criteria by utilizing the expression provided by GAMP.

Biological systems must be robust for stable functioning against perturbations, but robustness alone is insufficient. The ability to switch between appropriate states (phenotypes) in response to different conditions is essential for biological functions, as observed in allosteric enzymes and motor proteins. How are robustness and plasticity simultaneously acquired through evolution? In an attempt to answer this question, we examine the evolution of genotypes that realize plastic switching between two endpoint phenotypes upon external inputs as well as stationary expressions of phenotypes. Here, we introduce a statistical physics model consisting of spins, with active sites and regulatory sites, which are distinct from each other. In our model, we represent the phenotype and genotype as spin configurations and spin-spin interactions, respectively. The fitness for selection is given by the spin configuration, whose behavior is governed by the genotypes. Specifically, the fitness for selection is given so that it takes a higher value as more active sites take two requested spin configurations depending on the states of the regulatory sites. The remaining spins do not directly affect the fitness, but they interact with other spins. We numerically evolve the matrices of spin-spin interactions (genotypes) by changing them with mutations and selection of those with higher fitness. Our numerical simulations show that characteristic genotypes with higher fitness evolve slightly above the phase transition temperature between replica-symmetric and replica-symmetry-breaking phase in spin-glass theory. These genotypes shape the two spin configurations separately depending on the regulation. Each phenotype is primarily represented by the first or second eigenmode of the genotypes. Smooth switching between the two phenotypes is achieved by following a one-dimensional quarter-circle that connects them. Upon changes in regulations, spin configurations are attracted to this path, which allows for robust and plastic switching between the two phenotypes. The statistical physics analysis based on the two eigenmodes shows that the free energy landscape has a valley along the one-dimensional quarter-circle switching path. Robust attraction to the path is achieved through the evolution of the interactions within nonactive and nonregulatory spin sites, which themselves do not contribute to fitness. Our findings indicate that the compatibility between robustness and plasticity is acquired by the evolution of low dimensionality in the phenotype space, which will be relevant to the understanding of the robust function of protein as well as material design.

We study the inference problem in the noisy group testing to identify defective items from the perspective of the decision theory. We introduce Bayesian inference and consider the Bayesian optimal setting in which the true generative process of the test results is known. We demonstrate the adequacy of the posterior marginal probability in the Bayesian optimal setting as a diagnostic variable based on the area under the curve (AUC). Using the posterior marginal probability, we derive the general expression of the optimal cutoff value that yields the minimum expected risk function. Furthermore, we evaluate the performance of the Bayesian group testing without knowing the true states of the items: defective or non-defective. By introducing an analytical method from statistical physics, we derive the receiver operating characteristics curve, and quantify the corresponding AUC under the Bayesian optimal setting. The obtained analytical results precisely describes the actual performance of the belief propagation algorithm defined for single samples when the number of items is sufficiently large.

We consider compressed sensing formulated as a minimization problem of nonconvex sparse penalties, Smoothly Clipped Absolute deviation (SCAD) and Minimax Concave Penalty (MCP). The nonconvexity of these penalties is controlled by nonconvexity parameters, and L1 penalty is contained as a limit with respect to these parameters. The analytically derived reconstruction limit overcomes that of L1 and the algorithmic limit in the Bayes-optimal setting, when the nonconvexity parameters have suitable values. However, for small nonconvexity parameters, where the reconstruction of the relatively dense signals is theoretically guaranteed, the corresponding approximate message passing (AMP) cannot achieve perfect reconstruction. We identify that the shrinks in the basin of attraction to the perfect reconstruction causes the discrepancy between the AMP and corresponding theory using state evolution. A part of the discrepancy is resolved by introducing the control of the nonconvexity parameters to guide the AMP trajectory to the basin of the attraction.

For identifying infected patients in a population, group testing is an effective method to reduce the number of tests and correct test errors. In group testing, tests are performed on pools of specimens collected from patients, where the number of pools is lower than that of patients. The performance of group testing considerably depends on the design of pools and algorithms that are used for inferring the infected patients from the test outcomes. In this paper, an adaptive design method of pools based on the predictive distribution is proposed in the framework of Bayesian inference. The proposed method, executed using a belief propagation algorithm, results in more accurate identification of the infected patients compared with the group testing performed on random pools determined in advance.

The evolution of high-dimensional phenotypes is investigated using a statistical physics model consisting of interacting spins, in which phenotypes, genotypes, and environments are represented by spin configurations, interaction matrices, and external fields, respectively. We found that phenotypic changes upon diverse environmental change and genetic variation are highly correlated across all spins, consistent with recent experimental observations of biological systems. The dimension reduction in phenotypic changes is shown to be a result of the evolution of the robustness to thermal noise, achieved at the replica symmetric phase.

Group testing is a method of identifying infected patients by performing tests on a pool of specimens collected from patients. For the case in which the test returns a false result with finite probability, we propose Bayesian inference and a corresponding belief propagation (BP) algorithm to identify the infected patients from the results of tests performed on the pool. We show that the true-positive rate is improved by taking into account the credible interval of a point estimate of each patient. Further, the prevalence and the error probability in the test are estimated by combining an expectation-maximization method with the BP algorithm. As another approach, we introduce a hierarchical Bayes model to identify the infected patients and estimate the prevalence. By comparing these methods, we formulate a guide for practical usage.

We investigate the signal reconstruction performance of sparse linear regression in the presence of noise when piecewise continuous nonconvex penalties are used. Among such penalties, we focus on the smoothly clipped absolute deviation (SCAD) penalty. The contributions of this study are three-fold: we first present a theoretical analysis of a typical reconstruction performance, using the replica method, under the assumption that each component of the design matrix is given as an independent and identically distributed (i.i.d.) Gaussian variable. This clarifies the superiority of the SCAD estimator compared with in a wide parameter range, although the nonconvex nature of the penalty tends to lead to solution multiplicity in certain regions. This multiplicity is shown to be connected to replica symmetry breaking in the spin-glass theory, and associated phase diagrams are given. We also show that the global minimum of the mean square error between the estimator and the true signal is located in the replica symmetric phase. Second, we develop an approximate formula efficiently computing the cross-validation error without actually conducting the cross-validation, which is also applicable to the non-i.i.d. design matrices. It is shown that this formula is only applicable to the unique solution region and tends to be unstable in the multiple solution region. We implement instability detection procedures, which allows the approximate formula to stand alone and resultantly enables us to draw phase diagrams for any specific dataset. Third, we propose an annealing procedure, called nonconvexity annealing, to obtain the solution path efficiently. Numerical simulations are conducted on simulated datasets to examine these results to verify the consistency of the theoretical results and the efficiency of the approximate formula and nonconvexity annealing. The characteristic behaviour of the annealed solution in the multiple solution region is addressed. Another numerical experiment on a real-world dataset of Type Ia supernovae is conducted; its results are consistent with those of earlier studies using the formulation. A MATLAB package of numerical codes implementing the estimation of the solution path using the annealing with respect to in conjunction with the approximate CV formula and the instability detection routine is distributed in Obuchi (2019 https://github.com/T-Obuchi/SLRpackage_AcceleratedCV_matlab).

We analyse a linear regression problem with nonconvex regularization called smoothly clipped absolute deviation (SCAD) under an overcomplete Gaussian basis for Gaussian random data. We propose an approximate message passing (AMP) algorithm considering nonconvex regularization, namely SCAD-AMP, and analytically show that the stability condition corresponds to the de Almeida–Thouless condition in spin glass literature. Through asymptotic analysis, we show the correspondence between the density evolution of SCAD-AMP and the replica symmetric (RS) solution. Numerical experiments confirm that for a sufficiently large system size, SCAD-AMP achieves the optimal performance predicted by the replica method. Through replica analysis, a phase transition between replica symmetric and replica symmetry breaking (RSB) region is found in the parameter space of SCAD. The appearance of the RS region for a nonconvex penalty is a significant advantage that indicates the region of smooth landscape of the optimization problem. Furthermore, we analytically show that the statistical representation performance of the SCAD penalty is better than that of ℓ1-based methods, and the minimum representation error under RS assumption is obtained at the edge of the RS/RSB phase. The correspondence between the convergence of the existing coordinate descent algorithm and RS/RSB transition is also indicated.

We analyze the matrix factorization problem. Given a noisy measurement of a product of two matrices, the problem is to estimate back the original matrices. It arises in many applications, such as dictionary learning, blind matrix calibration, sparse principal component analysis, blind source separation, low rank matrix completion, robust principal component analysis, or factor analysis. It is also important in machine learning: unsupervised representation learning can often be studied through matrix factorization. We use the tools of statistical mechanics-the cavity and replica methods-to analyze the achievability and computational tractability of the inference problems in the setting of Bayes-optimal inference, which amounts to assuming that the two matrices have random-independent elements generated from some known distribution, and this information is available to the inference algorithm. In this setting, we compute the minimal mean-squared-error achievable, in principle, in any computational time, and the error that can be achieved by an efficient approximate message passing algorithm. The computation is based on the asymptotic state-evolution analysis of the algorithm. The performance that our analysis predicts, both in terms of the achieved mean-squared-error and in terms of sample complexity, is extremely promising and motivating for a further development of the algorithm.

We develop a method to evaluate the generalized degrees of freedom (GDF) for linear regression with sparse regularization. The GDF is a key factor in model selection, and thus its evaluation is useful in many modelling applications. An analytical expression for the GDF is derived using the replica method in the large-system-size limit with random Gaussian predictors. The resulting formula has a universal form that is independent of the type of regularization, providing us with a simple interpretation. Within the framework of replica symmetric (RS) analysis, GDF has a physical meaning as the effective fraction of non-zero components. The validity of our method in the RS phase is supported by the consistency of our results with previous mathematical results. The analytical results in the RS phase are calculated numerically using the belief propagation algorithm.

We develop a method for evaluating restricted isometry constants (RICs). This evaluation is reduced to the identification of the zero-points of entropy density which is defined for submatrices that are composed of columns selected from a given measurement matrix. Using the replica method developed in statistical mechanics, we assess RICs for Gaussian random matrices under the replica symmetric (RS) assumption. In order to numerically validate the adequacy of our analysis, we employ the exchange Monte Carlo (EMC) method, which has been empirically demonstrated to achieve much higher numerical accuracy than naive Monte Carlo methods. The EMC method suggests that our theoretical estimation of an RIC corresponds to an upper bound that is tighter than in preceding studies. Physical consideration indicates that our assessment of the RIC could be improved by taking into account the replica symmetry breaking.

Asynchronous dynamics given by the master equation in the Sherrington–Kirkpatrick (SK) spin-glass model is studied based on dynamical replica theory (DRT) with an extension to take into account the autocorrelation function. The dynamical behaviour of the system is approximately described by dynamical equations of the macroscopic quantities: magnetization, energy contributed by randomness and the autocorrelation function. The dynamical equations under the replica symmetry assumption are derived by introducing the subshell equipartitioning assumption and exploiting the replica method. The obtained dynamical equations are compared with Monte Carlo simulations, and it is demonstrated that the proposed formula describes well the time evolution of the autocorrelation function in some parameter regions. The study offers a reasonable description of the autocorrelation function in the SK spin-glass system.

We consider a learning problem of identifying a dictionary matrix D ∈ RM×N from a sample set of M dimensional vectors Y ∈ RM×P = N-1/2DX ∈ RM×P, where X ∈ RN×p is a sparse matrix in which the density of non-zero entries is 0 <; ρ <; 1. In particular, we focus on the minimum sample size Pc (sample complexity) necessary for perfectly identifying D of the optimal learning scheme when D and X are independently generated from certain distributions. By using the replica method of statistical mechanics, we show that Pc ~ O(N) holds as long as α = M/N > ρ is satisfied in the limit of N → ∞. Our analysis also implies that the posterior distribution given Y is condensed only at the correct dictionary D when the compression rate α is greater than a certain critical value αM(p). This suggests that belief propagation may allow us to learn D with a low computational complexity using O(N) samples.