We study the problem of optimally sending a real-valued source through multiple uses of a channel with feedback. First, we state a set of conditions that are sufficient for an encoder to achieve maximal mutual information between the source and all the channel outputs. This set of conditions are also necessary when the channel is input-identifiable, a condition widely satisfied by common channel models. More notably, we further study the information maximization-distortion minimization problem, where the mutual information between the source and all channel outputs still needs to be maximized, while at each step, the MMSE of estimating the source from the channel outputs so far also needs to be minimized. We derive a solution to this problem for discrete channels with certain symmetries, e.g. k-ary symmetric or k-ary erasure channels. We show that for such channels the famous posterior matching scheme, while not necessary for information maximization alone, is sufficient and essentially necessary for achieving both information maximization and distortion minimization. This work also provides a new perspective of regularizing distortion-minimizing feedback communication through information maximization, which enables us to find the optimal solution that otherwise would be intractable.

The traditional statistical inference is static, in the sense that the estimate of the quantity of interest does not affect the future evolution of the quantity. In some sequential estimation problems however, the future values of the quantity to be estimated depend on the estimate of its current value. This type of estimation problems has been formulated as the dynamic inference problem. In this work, we formulate the Bayesian learning problem for dynamic inference, where the unknown quantity-generation model is assumed to be randomly drawn according to a random model parameter. We derive the optimal Bayesian learning rules, both offline and online, to minimize the inference loss. Moreover, learning for dynamic inference can serve as a meta problem, such that all familiar machine learning problems, including supervised learning, imitation learning and reinforcement learning, can be cast as its special cases or variants. Gaining a good understanding of this unifying meta problem thus sheds light on a broad spectrum of machine learning problems as well.

We analyze the best achievable performance of Bayesian learning under generative models by defining and upper-bounding the minimum excess risk (MER): the gap between the minimum expected loss attainable by learning from data and the minimum expected loss that could be achieved if the model realization were known. The definition of MER provides a principled way to define different notions of uncertainties in Bayesian learning, including the aleatoric uncertainty and the minimum epistemic uncertainty. Two methods for deriving upper bounds for the MER are presented. The first method, generally suitable for Bayesian learning with a parametric generative model, upper-bounds the MER by the conditional mutual information between the model parameters and the quantity being predicted given the observed data. It allows us to quantify the rate at which the MER decays to zero as more data becomes available. Under realizable models, this method also relates the MER to the richness of the generative function class, notably the VC dimension in binary classification. The second method, particularly suitable for Bayesian learning with a parametric predictive model, relates the MER to the minimum estimation error of the model parameters from data via various continuity arguments. We also extend the definition and analysis of MER to the setting with multiple model families and the setting with nonparametric models. Along the discussions we draw some comparisons between the MER in Bayesian learning and the excess risk in frequentist learning.

Traditional statistical estimation, or statistical inference in general, is static, in the sense that the estimate of the quantity of interest does not change the future evolution of the quantity. In some sequential estimation problems however, we encounter the situation where the future values of the quantity to be estimated depend on the estimate of its current value. Examples include stock price prediction by big investors, interactive product recommendation, and behavior prediction in multi-agent systems. We may call such problems as dynamic inference. In this work, a formulation of this problem under a Bayesian probabilistic framework is given, and the optimal estimation strategy is derived as the solution to minimize the overall inference loss. How the optimal estimation strategy works is illustrated through two examples, stock trend prediction and vehicle behavior prediction. When the underlying models for dynamic inference are unknown, we can consider the problem of learning for dynamic inference. This learning problem can potentially unify several familiar machine learning problems, including supervised learning, imitation learning, and reinforcement learning.

An anytime decoding algorithm for tree codes using Monte-Carlo tree search is proposed. The meaning of anytime decoding here is twofold: 1) the decoding algorithm is an anytime algorithm, whose decoding performance improves as more computational resource, measured by decoding time, is allowed, and 2) the proposed decoding algorithm can approximate the maximum-likelihood sequence decoding of tree codes, which has the anytime reliability when the code is properly designed. The above anytime properties are demonstrated through experiments. The proposed method may be extended to the decoding of convolutional codes and block codes by Monte-Carlo trellis search, to enable smooth complexity-performance trade-offs in these decoding tasks. Some other extensions and possible improvements are also discussed.

We consider the problem of estimating the unknown location of a spatial signal defined on a circular interval from noisy measurements. Lower bounds are derived for the minimax risk of the localization error defined by the circular distance. The lower bounds reveal the fundamental dependence of the localization error on various problem parameters, including the shape of the signal, the noise level, the length of the interval, the number of sensors, and the number of measurement trials. Le Cam's method and Fano's method are used for the derivation. All lower bounds are non-asymptotic, and different lower bounds are tight in different problem parameters. We also derive a Bayesian Cramér-Rao lower bound for linear source localization, which helps us understand the tightness of the lower bounds for the circular source in some asymptotic situations as well.

We study the continuity property of the generalized entropy as a function of the underlying probability distribution, defined with an action space and a loss function, and use this property to answer the basic questions in statistical learning theory: the excess risk analyses for various learning methods. We first derive upper and lower bounds for the entropy difference of two distributions in terms of several commonly used f-divergences, the Wasserstein distance, a distance that depends on the action space and the loss function, and the Bregman divergence generated by the entropy, which also induces bounds in terms of the Euclidean distance between the two distributions. Examples are given along with the discussion of each general result, comparisons are made with the existing entropy difference bounds, and new mutual information upper bounds are derived based on the new results. We then apply the entropy difference bounds to the theory of statistical learning. It is shown that the excess risks in the two popular learning paradigms, the frequentist learning and the Bayesian learning, both can be studied with the continuity property of different forms of the generalized entropy. The analysis is then extended to the continuity of generalized conditional entropy. The extension provides performance bounds for Bayes decision making with mismatched distributions. It also leads to excess risk bounds for a third paradigm of learning, where the decision rule is optimally designed under the projection of the empirical distribution to a predefined family of distributions. We thus establish a unified method of excess risk analysis for the three major paradigms of statistical learning, through the continuity of generalized entropy.

We derive information-theoretic converses (i.e., lower bounds) for the minimum time required by any algorithm for distributed function computation over a network of point-to-point channels with finite capacity, where each node of the network initially has a random observation and aims to compute a common function of all observations to a given accuracy with a given confidence by exchanging messages with its neighbors. We obtain the lower bounds on computation time by examining the conditional mutual information between the actual function value and its estimate at an arbitrary node, given the observations in an arbitrary subset of nodes containing that node. The main contributions include the following. First, a lower bound on the conditional mutual information via so-called small ball probabilities, which captures the dependence of the computation time on the joint distribution of the observations at the nodes, the structure of the function, and the accuracy requirement. For linear functions, the small ball probability can be expressed by Lévy concentration functions of sums of independent random variables, for which tight estimates are available that lead to strict improvements over existing lower bounds on computation time. Second, an upper bound on the conditional mutual information via strong data processing inequalities, which complements and strengthens existing cutset-capacity upper bounds. Finally, a multi-cutset analysis that quantifies the loss (dissipation) of the information needed for computation as it flows across a succession of cutsets in the network. This analysis is based on reducing a general network to a line network with bidirectional links and self-links, and the results highlight the dependence of the computation time on the diameter of the network, a fundamental parameter that is missing from most of the existing lower bounds on computation time.

We derive upper bounds on the generalization error of a learning algorithm in terms of the mutual information between its input and output. The bounds provide an information-theoretic understanding of generalization in learning problems, and give theoretical guidelines for striking the right balance between data fit and generalization by controlling the input-output mutual information. We propose a number of methods for this purpose, among which are algorithms that regularize the ERM algorithm with relative entropy or with random noise. Our work extends and leads to nontrivial improvements on the recent results of Russo and Zou.

We derive lower bounds on the Bayes risk in decentralized estimation, where the estimator does not have direct access to the random samples generated conditionally on the random parameter of interest, but only to the data received from local processors that observe the samples. The received data are subject to communication constraints due to the quantization and the noisy communication channels from the processors to the estimator. We first derive general lower bounds on the Bayes risk using information-theoretic quantities, such as mutual information, information density, small ball probability, and differential entropy. We then apply these lower bounds to the decentralized case, using strong data processing inequalities to quantify the contraction of information due to communication constraints. We treat the cases of a single processor and of multiple processors, where the samples observed by different processors may be conditionally dependent given the parameter, for noninteractive and interactive communication protocols. Our results recover and improve recent lower bounds on the Bayes risk and the minimax risk for certain decentralized estimation problems, where previously only conditionally independent sample sets and noiseless channels have been considered. Moreover, our results provide a general way to quantify the degradation of estimation performance caused by distributing resources to multiple processors, which is only discussed for specific examples in existing works.

Analog-to-digital converter (ADC)-based multi-Gb/s serial link receivers have gained increasing attention in the back-plane community due to the desire for higher I/O throughput, ease of design portability, and flexibility. However, the power dissipation in such receivers is dominated by the ADC. ADCs in serial links employ signal-to-noise-and-distortion ratio (SNDR) and effective-number-of-bit (ENOB) as performance metrics as these are the standard for generic ADC design. This paper studies the use of information-based metrics such as bit-error-rate (BER) to design a BER-optimal ADC (BOA) for serial links. Channel parameters such as the m-clustering value and the threshold non-uniformity metric h(t) are introduced and employed to quantify the BER improvement achieved by a BOA over a conventional uniform ADC (CUA) in a receiver. Analytical expressions for BER improvement are derived and validated through simulations. A prototype BOA is designed, fabricated and tested in a 1.2 V, 90 nm LP CMOS process to verify the results of this study. BOA's variable-threshold and variable-resolution configurations are implemented via an 8-bit single-core, multiple-output passive digital-to-analog converter (DAC), which incurs an additional power overhead of < 0.1% (approximately 50 mu W). Measurement results show examples in which the BER achieved by the 3-bit BOA receiver is lower by a factor of 10(9) and 10(10), as compared to the 4-bit and 3-bit CUA receivers, respectively, at a data rate of 4-Gb/s and a transmitted signal amplitude of 180 mVppd.

Machine learning algorithms can be viewed as stochastic transformations that map training data to hypotheses. Following Bousquet and Elisseeff, we say that such an algorithm is stable if its output does not depend too much on any individual training example. Since stability is closely connected to generalization capabilities of learning algorithms, it is of theoretical and practical interest to obtain sharp quantitative estimates on the generalization bias of machine learning algorithms in terms of their stability properties. We propose several information-theoretic measures of algorithmic stability and use them to upper-bound the generalization bias of learning algorithms. Our framework is complementary to the information-theoretic methodology developed recently by Russo and Zou.

We consider the problem of distributed estimation, where local processors observe independent samples conditioned on a common random parameter of interest, map the observations to a finite number of bits, and send these bits to a remote estimator over independent noisy channels. We derive converse results for this problem, such as lower bounds on Bayes risk. The main technical tools include a lower bound on the Bayes risk via mutual information and small ball probability, as well as strong data processing inequalities for the relative entropy. Our results can recover and improve some existing results on distributed estimation with noiseless channels, and also capture the effect of noisy channels on the estimation performance.

High speed multi-input multi-output (MIMO) communication systems suffer from inter-channel and inter-symbol-interference (ICI and ISI). The vector Viterbi algorithm (VVA) is a maximum likelihood sequence detection (MLSD) algorithm for MIMO frequency selective channels. MLSD algorithms are desired because they minimize the probability of sequence detection error. However, they suffer from very high computational complexity. In this work, we show how a sphere decoding like algorithm can be used to reduce the complexity of VVA while preserving its optimality. For a 2 × 2 MIMO system with 16-QAM signal constellation, our algorithm cuts VVA's complexity by 50% at an SNR of 10 dB and by 60% at an SNR of 15 dB.

The design of a communications system is typically most effective only when each of its components can be accurately represented by a discrete, symbolic behavioural abstraction. Such abstractions, in addition to providing valuable design intuition, also enable highly efficient and scalable system-level simulation. However, given a SPICE-level description for a subsystem such as a latch, it is a challenge to come up with a discrete, symbol-level abstraction that accurately captures its continuous-time dynamics. Indeed, the manual construction of such an abstraction requires deep knowledge and understanding of the operation of the module in question; moreover, it is very time-consuming, tedious, error-prone and not easily scalable to larger designs.In recent work [1], we adapted methods from computational learning theory to develop an automated technique, DAE2FSM, that produces binary finite state machine (FSM) abstractions of non-linear analog/mixed-signal (AMS) circuits. In the present paper, we demonstrate the application of the DAE2FSM technique to automatically derive FSM abstractions for a mixed-signal communications circuit component, namely a current mode latch (CML) designed in IBM's 90nm LP process technology. We show that the FSMs learned by DAE2FSM not only capture the essence of the latch's behaviour during normal conditions, but also faithfully mimic its behaviour under adverse operating conditions (e. g., under lowered supply voltages). Moreover, in addition to a stand-alone CML, we also generate FSMs for cascades of two and three latches (such topologies are used in the design of power-efficient, bit-error optimised analog-to-digital converters). In spite of the inherent non-linearity of such systems, and in spite of the pronounced "analog-ness" of the waveforms in question, our FSM abstractions are able to produce discrete-time symbol sequences that closely match the data points obtained by sampling from continuous-time SPICE simulations.

We recently explored the concept of using BER-optimal ADCs for high-speed links. In this paper, we study the benefits of BER-optimal ADCs in terms of power savings and relaxation of component specifications in a 90 nm 1.2V CMOS process. These analyses are based on component models for a flash ADC that capture bandwidth limitation of pre-amplifiers and metastability of latches. We show that in the presence of these ADC non-idealities, a 3-bit BER-optimal ADC can provide a 3 dB ADC shaping gain over a 4-bit conventional ADC. The one bit reduction offers power savings of 75% in the VGA and 50% in the ADC. Further, the 3dB ADC shaping gain can be traded-off for a 50% reduction of transmit driver power, a 75% reduction of the pre-amplifier bandwidth, or a saving of one latch stage that leads to a 20% additional power reduction in the ADC.

Through the investigation on frame header (FH) structures of DTMB system, i.e. the digital terrestrial television broadcasting standard of China, spectrum sensing methods based on PN cross-correlation (PNCC) are proposed for all FH options. Miss-detection probability, decision threshold and sensing time requirement of the proposed methods are analyzed in detail. Comparisons on spectrum sensing performance, robustness to time dispersive channel and carrier frequency offset are made between the proposed methods and the existing methods based on PN autocorrelation (PNAC). Both theoretical analyses and simulation results demonstrate the considerable advantage of the proposed PNCC methods over the existing PNAC methods.

It is well known that the cyclic property of transmitted signal can be used to achieve joint frame and carrier frequency synchronization, while the uncompensated sampling frequency offset (SFO) would still be a problem. Based on the observation that the impact of SFO on the received cyclic block and its replica is equivalent to a relative time shift, a joint synchronization scheme solely based on cyclic property is proposed to achieve frame, sampling frequency and carrier frequency synchronization simultaneously. A two-Dimensional Sliding-Window Auto-Correlator (2D-SWAC) is presented to implement the joint synchronization with low complexity. The proposed scheme is suitable for any transmission system with similar cyclic property. Meanwhile, it is robust to large sampling and carrier frequency offset and strong channel frequency selectivity, as demonstrated by the analysis and simulations.

The estimation range and estimation accuracy are contradictory in conventional correlation based frequency estimators due to phase ambiguity. A novel frequency estimator is proposed to unwrap the correlation phase sequence through a differential-modulus-accumulation process. By utilizing the whole unwrapped phase sequence, the proposed estimator is able to achieve maximal estimation range with CRB approaching estimation accuracy and low SNR threshold simultaneously. Both analysis and simulation demonstrate the effectiveness of the proposed frequency estimator.