Disordered Systems and Neural Networks (cond-mat.dis-nn)

  • PDF
    Genetic regulatory circuits universally cope with different sources of noise that limit their ability to coordinate input and output signals. In many cases, optimal regulatory performance can be thought to correspond to configurations of variables and parameters that maximize the mutual information between inputs and outputs. Such optima have been well characterized in several biologically relevant cases over the past decade. Here we use methods of statistical field theory to calculate the statistics of the maximal mutual information (the `capacity') achievable by tuning the input variable only in an ensemble of regulatory motifs, such that a single modulator controls N targets. Assuming (i) sufficiently large N, (ii) quenched random kinetic parameters, and (iii) small noise affecting the input-output channels, we can accurately reproduce numerical simulations both for the mean capacity and for the whole distribution. Our results provide insight into the inherent variability in effectiveness occurring in regulatory systems with heterogeneous kinetic parameters.
  • PDF
    A number of recent papers have provided evidence that practical design questions about neural networks may be tackled theoretically by studying the behavior of random networks. However, until now the tools available for analyzing random neural networks have been relatively ad-hoc. In this work, we show that the distribution of pre-activations in random neural networks can be exactly mapped onto lattice models in statistical physics. We argue that several previous investigations of stochastic networks actually studied a particular factorial approximation to the full lattice model. For random linear networks and random rectified linear networks we show that the corresponding lattice models in the wide network limit may be systematically approximated by a Gaussian distribution with covariance between the layers of the network. In each case, the approximate distribution can be diagonalized by Fourier transformation. We show that this approximation accurately describes the results of numerical simulations of wide random neural networks. Finally, we demonstrate that in each case the large scale behavior of the random networks can be approximated by an effective field theory.
  • PDF
    Perceptual manifolds arise when a neural population responds to an ensemble of sensory signals associated with different physical features (e.g., orientation, pose, scale, location, and intensity) of the same perceptual object. Object recognition and discrimination requires classifying the manifolds in a manner that is insensitive to variability within a manifold. How neuronal systems give rise to invariant object classification and recognition is a fundamental problem in brain theory as well as in machine learning. Here we study the ability of a readout network to classify objects from their perceptual manifold representations. We develop a statistical mechanical theory for the linear classification of manifolds with arbitrary geometry revealing a remarkable relation to the mathematics of conic decomposition. Novel geometrical measures of manifold radius and manifold dimension are introduced which can explain the classification capacity for manifolds of various geometries. The general theory is demonstrated on a number of representative manifolds, including L2 ellipsoids prototypical of strictly convex manifolds, L1 balls representing polytopes consisting of finite sample points, and orientation manifolds which arise from neurons tuned to respond to a continuous angle variable, such as object orientation. The effects of label sparsity on the classification capacity of manifolds are elucidated, revealing a scaling relation between label sparsity and manifold radius. Theoretical predictions are corroborated by numerical simulations using recently developed algorithms to compute maximum margin solutions for manifold dichotomies. Our theory and its extensions provide a powerful and rich framework for applying statistical mechanics of linear classification to data arising from neuronal responses to object stimuli, as well as to artificial deep networks trained for object recognition tasks.
  • PDF
    We study the impact of quenched disorder (random exchange couplings or site dilution) on easy-plane pyrochlore antiferromagnets. In the clean system, a magnetically ordered state is selected from a classically degenerate manifold via an order-by-disorder mechanism. In the presence of randomness, however, different states can be locally selected depending on details of the disorder configuration. Using a combination of analytical considerations and classical Monte-Carlo simulations, we argue that any long-range-ordered magnetic state is destroyed beyond a critical level of randomness where the system breaks into magnetic domains due to random exchange anisotropies, becoming, therefore, a glass of spin clusters, in accordance with the available experimental data.

Recent comments

Travis Scholten Oct 02 2015 03:25 UTC

Apologies for the delayed reply.

No worries with regards to the code - when it does get released, would you mind pinging me? You can find me on [GitHub](https://github.com/Travis-S).

Nicola Pancotti Sep 23 2015 07:58 UTC

Hi Travis

Yes, that code is related to the work we did and that is my repo. However it is quite outdated. I used that repo for sharing the code with my collaborators. Now we are working for providing a human friendly version, commented and possibly optimized. If you would like to have a working

Travis Scholten Sep 21 2015 17:08 UTC

Has anyone found some source code for the SGD referenced in this paper? I came across a [GitHub repository](https://github.com/nicaiola/thesisproject) from Nicola Pancotti (at least, I think that is his username, and the code seems to fit with the kind of work described in the paper!). I am not sure

Salvatore Mandrà Nov 21 2014 13:50 UTC

The manuscript has been widely revised to focus the reader's attention on the proposed method and its application in presence of local disorder.


Salvatore, Gian Giacomo and Alán

Salvatore Mandrà Aug 01 2014 19:11 UTC

Thanks Dr. Hastings for your comment. It is true that the transverse field Ising model does not satisfy the requirements to apply our method with an exponential reduction. Indeed, the opposite would be quite impressive since the random Ising model is a NP-Hard problem and we ourselves would be prett

Matt Hastings Aug 01 2014 16:43 UTC

The "quite general" conditions seem not to include the transverse field Ising model, the subject of most of the intensive numerical work previously. Incidentally, the terms "Lanczos" and "Krylov subspace" might be helpful.