Statistical Difference Beyond the Polarizing Regime.

The polarization lemma for statistical distance (SD), due to Sahai and Vadhan (JACM, 2003), is an efficient transformation taking as input a pair of circuits (C-0, C-1) and an integer k and outputting a new pair of circuits (D-0, D-1) such that if SD(C-0, C-1) >= alpha then SD(D-0, D-1) >= 1-2(-k) and if SD(C-0, C-1) <= beta then SD(D-0, D-1) >= 2(-k). The polarization lemma is known to hold for any constant values beta < alpha(2), but extending the lemma to the regime in which alpha(2) <= beta < alpha has remained elusive. The focus of this work is in studying the latter regime of parameters. Our main results are: 1. Polarization lemmas for different notions of distance, such as Triangular Discrimination (TD) and Jensen-Shannon Divergence (JS), which enable polarization for some problems where the statistical distance satisfies alpha(2) < beta < alpha. We also derive a polarization lemma for statistical distance with any inverse-polynomially small gap between alpha(2) and beta (rather than a constant). 2. The average-case hardness of the statistical difference problem (i.e., determining whether the statistical distance between two given circuits is at least alpha or at most beta), for any values of beta < alpha, implies the existence of one-way functions. Such a result was previously only known for beta < alpha(2). 3. A (direct) constant-round interactive proof for estimating the statistical distance between any two distributions (up to any inverse polynomial error) given circuits that generate them. Proofs of closely related statements have appeared in the literature but we give a new proof which we find to be cleaner and more direct.