Subject: source separation From: Lonce LaMar Wyse <lwyse(at)ISS.NUS.SG> Date: Thu, 21 Sep 1995 18:02:55 +0800This just in from the connectionists mailing list - - lonce ------------------------------------------------------------------------ Blind Signal Processing is an emerging area in adaptive signal processing and neural networks. It was originated in France in the late 80's . Below please find an advanced program of a special invited session devoted to blind separation of sources and their applications at 1995 INTERNATIONAL SYMPOSIUM ON NONLINEAR THEORY AND ITS APPLICATIONS , NOLTA'95 in Las Vegas. Any comments will be highly appreciated, especially association of this approach to brain information processing and image and speech enhancement, filtering and noise reduction. Andrzej Cichocki, Head of Laboratory for Artificial Brain Systems, Frontier Research Program RIKEN, Institute of Physical and Chemical Research, Hirosawa 2-1, Saitama 351-01, WAKO-Schi, JAPAN E-mail: cia(at)kamo.riken.go.jp, FAX (+81) 048 462 4633. URL: http://zoo.riken.go.jp/bip.html =========================================================================== NOLTA'95, 1995 INTERNATIONAL SYMPOSIUM ON NONLINEAR THEORY AND ITS APPLICATIONS Caesars Palace, LAS VEGAS Dec. 10 -14, 1995 Program for Special Invited Session on "BLIND SEPARATION OF SOURCES -Brain Information Processing" Organizer and chair Dr. A. Cichocki Frontier Research Program RIKEN, Institute of Physical and Chemical Research, Hirosawa 2-1, Saitama 351-01, WAKO-Schi, JAPAN Advanced Program: 1. Prof.Christian JUTTEN , Laboratory TIRF, INPG, Grenoble, FRANCE, "Separation of Sources: Blind or Unsupervised? " Abstract: Basically, separation of sources are referred as BLIND methods. However, adaptive algorithms for source separation only emphasize on UNSUPERVISED aspects of the learning. In this talk, we propose a selected review of recent works to show how A PRIORI KNOWLEDGES on the sources or on the mixtures can simplify the algorithms and improve performance. 2. Prof. Jean-Francois CARDOSO , Ecole Nationale Superieure des Telecommunications, Telecom Paris, FRANCE "The Invariant Approach to Source Separation" Abstract: The `invariant approach' to source separation is based on the recognition that the unknown parameter in a source mixture is the mixing matrix, hence it belongs to a multiplicative group. In this contribution, we show that this simple fact can be exploited to build source separation algorithms behaving uniformly well in the mixing matrix. This is achieved if two sufficient conditions are met. + First, contrast functions (or estimating equations) used to identify the mixture should be designed in such a way that source separation is achieved when they are optimized (or solved) **without constraints** (such as normalization, etc). Examples of such contrast functions will be given, some of them being simple variants of classic contrast functions. This requirement is sufficient to guarantee uniform performance of the resulting batch algorithms. + Second, in the case of adaptive algorithm, uniform performance has a more extensive meaning: not only the residual error but also the convergence are important. Again, the multiplicative nature of the parameter calls for a special form of the learning rule, namely it suggests a `multiplicative update'. This approach results in adaptive source separation algorithms and enjoying uniform performance: convergence speed, residual error, stable points, etc... do not depend on the mixing matrix. In addition these algorithms show a very simple (and parallelizable) structure. The paper includes analytical results based on asymptotic performance analysis that quantify the behavior of both batch and adaptive equivariant source separators. In particular, these results allow to determine, given the source distribution, the optimal nonlinearities to be used in the learning rule. 3. Dr. Jie ZHU, Prof. Xi-Ren CAO, and prof. Ruey-Wen LIU, The Hong Kong University of Science and Technology Kowloon, Hong Kong, The University of Notre Dame, Notre Dame, NI 46556, U.S.A. "Blind Source Separation Based on Output Independence - Theory and Implementation" Abstract: The paper presents some recent results on the theory and implementation techniques of blind source separation. the approach is based on independence property of the outputs of a filter. In the theory part, we identify and study two major issues in the blind source separation problem: separability and separation principles. We show that separability is an intrinsic property of the measured signals and can be described by the concept of $m$-row decomposability introduced in this paper, and that the separation principles can be developed by using the structure characterization theory of random variables. In particular, we show that these principles can be derived concisely and intuitively by applying the Darmois-Skitovich theorem, which is well-known in statistical inference theory and psychology. In the implementation part, we show that if at most one of the source signals has a zero third (or fourth) order cumulant, then these signals can be separated by a filter whose parameters can be determined by a system of nonlinear equations using only third (or fourth) order cumulants of the measured signals. This results covers some previous results as special cases. 4. Dr. Jie HUANG, Prof. Noboru OHNISHI and Dr. Noboru Sugie ; Bio-Mimetic Control Research Center , The Institute of Physical and Chemical Research (RIKEN), Nagoya, JAPAN "Sound Separation Based on Perceptual Grouping of Sound Segments" Abstract : We would like to propose a sound separation method, which combines spatial cues (source direction) and structural cues (continuity and harmony). Sound separation is important in various scientific fields. There are mainly two different approaches to achieve this goal. One is based on blind estimation of inverse transfer functions from multiple sources to multiple receivers (microphones). The other is based on grouping sound segments in time-frequency domain. Our approach is based on the sound segments grouping. However, we use multiple microphones to obtain the spatial information. This approach is strongly inspired by the precedence effect and the cocktail party effect of human auditory system. The precedence effect suggests to us the way of coping with echoes in reverberant environment. The cocktail party effect suggests the use of spatial cues for sound separation. Psychological factors of auditory stream integration and segregation, such as continuity and harmony, are used as structural cues. It is realized by a continuity enhancement filter and a harmonic histogram to supplement the spatial segments grouping. The use of this method with real human speeches recorded in an anechoic chamber and a normal room was demonstrated. The experiments have shown that the method was effective to separate sounds in reverberant environments. 5. Dr. Kiyotoshi MATSUOKA and Dr. Mitsuru KAWAMOTO, Department of Control Engineering, Kyushu Institute of Technology, 1-1,Tobata, Kitakyushu, 804 Japan "Blind Signal Separation Based on a Mutual Information Criterion" Abstract: This paper deals with the problem of the so-called blind separation of sources. The problem is to recover a set of source signals from their linear mixtures observed by the same number of sensors, in the absence of any particular information about the transfer function that couples the sources and the sensors. The only a priori knowledge is, basically, the fact that the source signals are statistically mutually independent. Such a task arises in noise canceling of sound signals, image enhancement, medical measurement, etc. If the observed signals are stationary, Gaussian, white ones, then blind separation is essentially impossible. Conversely, blind separation can be realized by exploiting some information on nonstationary, non-Gaussian, or nonwhite characteristics of the observed signals, if any. Most of the conventional methods stipulate that the source signals are non-Gaussian, and use some high-order moments or cumulants. However, it is sometimes difficult to accurately estimate non-Gaussian statistics because random signals in practice are usually not so far from Gaussian. In this paper we propose an approach that utilizes only second-order moments of the observed signals. We consider two cases: (i) the source signals are nonstationary; (ii) the source signals have some temporal correlations, i.e., they are nonwhite signals. To realize signal separation we consider a recovering filter which takes in the observed signals as input and provides an estimate of the source signals as output. The parameters of the filter are determined such that all the outputs of the filter be mutually statistically independent. As a criterion of the statistical independence we adopt the well-known mutual information between the outputs. The adaptation rule for the filter's parameters is derived from the steepest descent minimization of the criterion function. A remarkable feature of our approach is that it is able to treat time-convolutive mixtures of a general number of (stationary) source signals. In contrast, most of the conventional studies on blind separation only consider the static mixing of the source signals. Namely, any delay in the mixing process is not taken into account. So, those methods are useless for many of the important applications of blind separation, e.g., separation of sound signals. Although there are some studies that deal with convolutive mixtures involving some delay, all of them consider only the case of two sources (2 x 2 channels) and do not seem extendible to the case of more than two sources. In our approach, also for convolutive mixtures, the adaptation rule is easily obtained by defining the information criterion in the frequency domain. 6. Dr. Eric MOREAU, and Prof. Odile MACCHI Laboratoire des Signaux et Systems, CNRS-ESE, FRANCE "Adaptive Unsupervised Separation of Discrete Sources" ABSTRACT: We consider the unsupervised source separation problem where observations are captured at the output of an unknown linear mixture of random signals called sources. The sources are assumed discrete, zero-mean and statistically independent. In this paper we consider the problem with a prewhitening stage. The a priori knowledge that sources are discrete with known level, is used in order to improve performances. A novel contrast which combines two parts, is proved. The first part forces statistical independence of the outputs while the second one forces the outputs to have the known distribution. Then a stochastic gradient adaptive algorithm is proposed. Its performance is illustrated thanks to computer simulations that clearly show that the novel contrast achieves much better performance. 7. Dr. Adel BELOUCHRANI, and Prof. Jean-Francois CARDOSO , Telecom Paris, CNRS URA 820, GdR TdSI 46 rue Barrault, 75634 Paris Cedex 13, FRANCE "Maximum Likelihood Source Separation by the Expectation-Maximization Technique: Deterministic and Stochastic Implementation" Abstract: This paper deals with the source separation problem which consists in the separation of a mixture of independent sources without a priori knowledge about the mixing matrix. When the source distributions are known in advance, this problem can be solved via the maximum likelihood (ML) approach by maximizing the data likelihood function using (i) the Expectation-Maximization (EM) algorithm and (ii) a stochastic version of it, the SEM, wich is efficiently implemented by resorting to Metropolis sampler. Two important features of our algorithm are that (a) the covariance of the additive noise can be estimated as a regular parameter, (b) in the case of discrete sources, it is possible to separate more sources than sensors. The effectiveness of this method is illustrated by numerical simulations. 8. Prof. L. TONG and Dr. X. CHEN, University of Connecticut, USA. "Blind Separation of Dynamically Mixed Multiple Sources and its Applications in CDMA Systems" Abstract: In this paper, we consider the problem of separating dynamically mixed multiple sources. Specifically, we address the problem of recovering the sources of an multiple-input multiple-output system. Two issues will be addressed: (i) Source Blind Separability; (ii) Blind Signal Separation Algorithms. Applications of the proposed approach to code-division multiple-access schemes in wireless communication are presented. 9. Prof. Shun-ichi AMARI, Prof. Andrzej CICHOCKI and Dr. Howard Hua YANG, Frontier Research Program RIKEN (Institute of Physical and Chemical Research), Wako-shi, JAPAN "Multi-layer Neural Networks with Local Learning Rules for Blind Separation of Sources" Abstract: In this paper we will propose multi-layer neural network models (feedforward and recurrent) with novel, local, adaptive, unsupervised learning rules which enable not only to separate on-line independent sources but also determine the number of active sources. In other words, we assume that the number of sources and their waveforms are completely unknown. Moreover, the separation problem can be very ill-conditioned and/or badly scaled. In fact the performance of the learning algorithm is independent of scaling factors and a condition number of the mixing matrix. Universal (flexible) computer simulation program will be presented which enable comparison of validity and performance of various recently developed adaptive on-line learning algorithms. 10.. Dr.L. De Lathauwer and Dr.P. Comon; E.E. Dept. - ESAT - SISTA, K.U.Leuven, BELGIUM, CNRS - I3S, Sophia Antipolis, Valbonne, FRANCE "Higher - Order Power Method" Abstract The scientific boom in the field of higher-order statistics involves an increasing need for numerical tools in multi-linear algebra: higher-order moments and cumulants of multivariate stochastic processes are higher-order tensors. We consider the problem of generalizing the computation of the best rank-R approximation of a given matrix to the computation of the best rank-(R1,R2,...,RN) approximation of an Nth-order tensor. We mainly focus on the best rank-1 approximation of third-order tensors. It is shown that this problem leads in a very natural way to a higher-order equivalent of the well-known power method for the computation of the eigendecomposition of matrices. It can be proved that each power iteration step decreases the least-squares error between the initial tensor and the lower-rank estimate. In the tensor case several stationary points might exist, each with a different domain of attraction. Surprisingly, the power iteration for a super-symmetric tensor can produce intermediate results that are unsymmetric. Imposing symmetry on the algorithm does not necessarily improve the convergence speed; the symmetric power iteration can even fail to converge. In the matrix case truncation of the singular value decomposition (SVD) yields the best rank-R approximation; in the tensor case it can only be proved that truncation of the higher-order singular value decomposition (HOSVD) yields a fairly good approximation. All our simulations show that the HOSVD-guess belongs to the attraction region corresponding to the optimal fit. ---------------------------------------------------------------------------