Date: Mon 8 Aug 1988 00:39-EDT From: AIList Moderator Nick Papadakis Reply-To: AIList@mc.lcs.mit.edu Us-Mail: MIT LCS, 545 Tech Square, Rm# NE43-504, Cambridge MA 02139 Phone: (617) 253-6524 Subject: AIList Digest V8 #41 To: AIList@mc.lcs.mit.edu Status: R AIList Digest Monday, 8 Aug 1988 Volume 8 : Issue 41 Queries: Sigmoid transfer function (3 responses) refs. for stochastic relaxation (1 response) ---------------------------------------------------------------------- Date: 4 Aug 88 20:28:13 GMT From: amdahl!pacbell!hoptoad!dasys1!cucard!aecom!krishna@ames.arpa (Krishna Ambati) Subject: Sigmoid transfer function I am looking for a "black box" circuit that has the product transfer function: Output voltage = 0.5 ( 1 + tanh ( Input voltage / "Gain" ) ) = 1 / ( 1 + exp ( -2 * Input voltage / "Gain" ) ) When plotted, this function looks like an elongated S When IV = - infinity, OV = 0 When IV = + infinity, OV = 1 When IV = 0 , OV = 0.5 By the way, this question arose in connection with a neural network problem. Thanks. Krishna Ambati krishna@aecom.uucp ------------------------------ Date: 6 Aug 88 16:58:05 GMT From: glacier!jbn@labrea.stanford.edu (John B. Nagle) Subject: Re: Sigmoid transfer function Recognize that the transfer function in a neural network threshold unit doesn't really have to be a sigmoid function. It just has to look roughly like one. The behavior of the net is not all that sensitive to the exact form of that function. It has to be continuous and monotonic, reasonably smooth, and rise rapidly in the middle of the working range. The trigonometric form of the transfer function is really just a notational convenience. It would be a worthwhile exercise to come up with some other forms of transfer function with roughly the same graph, but better matched to hardware implementation. How do real neurons do it? John Nagle ------------------------------ Date: 7 Aug 88 00:04:26 GMT From: ankleand@athena.mit.edu (Andy Karanicolas) Subject: Re: Sigmoid transfer function In article <1945@aecom.YU.EDU> krishna@aecom.YU.EDU (Krishna Ambati) writes: > >I am looking for a "black box" circuit that has the product transfer >function: > >Output voltage = 0.5 ( 1 + tanh ( Input voltage / "Gain" ) ) > > = 1 / ( 1 + exp ( -2 * Input voltage / "Gain" ) ) > >When plotted, this function looks like an elongated S > >When IV = - infinity, OV = 0 >When IV = + infinity, OV = 1 >When IV = 0 , OV = 0.5 > >By the way, this question arose in connection with a neural network >problem. > >Thanks. > >Krishna Ambati >krishna@aecom.uucp The function you are looking for is not too difficult to synthesize from a basic analog circuit builing block; namely, a differential amplifier. The accuracy of the circuit will depend on the matching of components, among other things. The differential amplifier is discussed in many textbooks concerned with analog circuits (analog integrated circuits especially). You can try: Electronic Principles, Grey and Searle, Wiley 1969. Bipolar and MOS Analog IC Design, Grebene, Wiley 1984. Design and Analysis of Analog IC's, Gray and Meyer, Wiley 1984. Unfortunately, drawing circuits on a text editor is a pain; I'll attempt it if these or other books are not available or helpful. Andy Karanicolas Microsystems Technology Laboratory ankleand@caf.mit.edu ------------------------------ Date: 7 Aug 88 19:55:49 GMT From: icsia!munro@ucbvax.berkeley.edu (Paul Munro) Subject: Re: Sigmoid transfer function In article <17615@glacier.STANFORD.EDU> jbn@glacier.UUCP (John B. Nagle) writes: [JN]Recognize that the transfer function in a neural network threshold unit [JN]doesn't really have to be a sigmoid function. It just has to look roughly [JN]like one. The behavior of the net is not all that sensitive to the [JN]exact form of that function. It has to be continuous and monotonic, [JN]reasonably smooth, and rise rapidly in the middle of the working range. [JN]The trigonometric form of the transfer function is really just a notational [JN]convenience. [JN] [JN] It would be a worthwhile exercise to come up with some other forms [JN]of transfer function with roughly the same graph, but better matched to [JN]hardware implementation. How do real neurons do it? [JN] [JN] John Nagle Try this one : f(x) = x / (1 + |x|) It is continuous and differentiable: f'(x) = 1 / (1 + |x|) ** 2 = ( 1 - |f|) ** 2 . - Paul Munro ------------------------------ Date: 5 Aug 88 18:38:42 GMT From: tness7!tness1!nuchat!moray!uhnix1!sun2.cs.uh.edu!rmdubash@bellco re.bellcore.com Subject: refs. for stochastic relaxation I am currently working on stochastic relaxation and relaxation algorithms for finely grained parallel architectures. In particular, I am studying their implementation on neural and connectionist models, with emphasis on inherent fault tolerance property of such implementations. I will be grateful if any of you can provide me with pointers, references etc. on this ( or related ) topics. Thanks. _______________________________________________________________________________ Rumi Dubash, Computer Science, Univ. of Houston, Internet : rmdubash@sun2.cs.uh.edu U.S.Mail : R.M.Dubash, Computer Science Dept., Univ. of Houston, ------------------------------ Date: 6 Aug 88 18:02:44 GMT From: brand.usc.edu!manj@oberon.usc.edu (B. S. Manjunath) Subject: Re: refs. for stochastic relaxation In article <824@uhnix1.uh.edu> rmdubash@sun2.cs.uh.edu () writes: >I am currently working on stochastic relaxation and relaxation algorithms for >finely grained parallel architectures. In particular, I am studying their >implementation on neural and connectionist models, with emphasis on inherent >fault tolerance property of such implementations. > >I will be grateful if any of you can provide me with pointers, references etc. >on this ( or related ) topics. >Rumi Dubash, Computer Science, Univ. of Houston, Geman and Geman (1984) is an excellent paper to start with. It also contains lot of refernces. The paper mainly deals with Markov Random Fields and applications to image processing. S.Geman and D.Geman,"Stochastic relaxation, Gibbs distributions and the bayesian restoration of images", IEEE trans. on pattern analysis and machine intelligence", PAMI-6,Nov 1984, pp. 721-742. Another reference that I feel might be useful is Marroquin,J.L. Ph. D Thesis "Probabilistic solution of Inverse problems", M.I.T. 1985. bs manjunath. ------------------------------ End of AIList Digest ********************