Date: Fri 12 Aug 1988 15:30-EDT From: AIList Moderator Nick Papadakis Reply-To: AIList@mc.lcs.mit.edu Us-Mail: MIT LCS, 545 Tech Square, Rm# NE43-504, Cambridge MA 02139 Phone: (617) 253-6524 Subject: AIList Digest V8 #46 To: AIList@mc.lcs.mit.edu Status: R AIList Digest Saturday, 13 Aug 1988 Volume 8 : Issue 46 Query Responses: Feigenbaum's citation Sigmoid transfer function ---------------------------------------------------------------------- Date: 11 Aug 88 06:07:38 GMT From: mcvax!inria!crin!napoli@uunet.uu.net (Amedeo NAPOLI) Subject: Feigenbaum's citation Is there anybody to tell me the title of the book in which E. Feigenbaum says: ``AI focused its attention most exclusively on the development of clever inference methods. But the power of its systems does not reside in the inference methods; almost any inference method will do. The power resides in the knowledge'' Many thanx in advance, -- --- Amedeo Napoli @ CRIN / Centre de Recherche en Informatique de Nancy EMAIL : napoli@crin.crin.fr - POST : BP 239, 54506 VANDOEUVRE CEDEX, France ------------------------------ Date: 11 Aug 88 16:29:22 GMT From: glacier!jbn@labrea.stanford.edu (John B. Nagle) Subject: Re: Feigenbaum's citation I heard him say things very similar to that around Stanford in 1983. In the early days of expert systems, that was a common remark. It reflects a turf battle with the logicians that was taking place at the time, theorem- proving having been a dominant paradigm in AI in the late 1970s and early 1980s. It's not clear that such a remark has relevance today. The optimistic hope that dumping lots of rules into a dumb inference engine would produce something profound has faded. Experience with that approach has produced more understanding of what can and cannot be accomplished in that way. More work is taking place on the underlying machinery again. But now, there is the realization that the machinery exists to process the knowledge base, not to implement some interesting logical function. In retrospect, both camps (and there were camps, at Stanford, in separate buildings) were taking extreme positions, neither of which turned out to be entirely satisfactory. Work today lies somewhere between those poles. Plans are underway, amusingly, to get both groups at Stanford under one roof again in a new building. John Nagle ------------------------------ Date: 11 Aug 88 17:17:26 GMT From: pasteur!agate!garnet.berkeley.edu!ked@ames.arpa (Earl H. Kinmonth) Subject: Re: Feigenbaum's citation As I remember, Feigenbaum achieved notoriety for his (probably largely ghosted book) on the Japanese "Fifth Generation Project." Did anything ever come out of the Fifth Generation project other than big lecture fees for Feigenbaum to go around warning about the Japanese peril? Is he really a pompous twit (the impression given by the book) or is that due to the scatter-brained ghost writer? ------------------------------ Date: 11 Aug 88 23:35:44 GMT From: prost.llnl.gov!daven@lll-winken.llnl.gov (David Nelson) Subject: Re: Feigenbaum's citation In article <17626@glacier.STANFORD.EDU> jbn@glacier.UUCP (John B. Nagle) writes: > [stuff about Feigenbaum's remark omitted] > It's not clear that such a remark has relevance today. The optimistic >hope that dumping lots of rules into a dumb inference engine would produce >something profound has faded. .... To be replaced by the optimistic hope that dumping lots of examples into a dumb neural net will produce something profound :-) daven daven (Dave Nelson) arpa: daven @ lll-crg.llnl.gov uucp: ...{seismo,mordor,sun,lll-lcc}!lll-crg!daven ------------------------------ Date: 8 Aug 88 03:00:35 GMT From: glacier!jbn@labrea.stanford.edu (John B. Nagle) Subject: Re: Sigmoid transfer function In article <25516@ucbvax.BERKELEY.EDU> munro@icsia.UUCP (Paul Munro) writes: > >Try this one : f(x) = x / (1 + |x|) > The graph looks OK, although some scaling is needed to make it comparable to the sigmoid. Someone should try it in one of the popular neural net simulators and see how the results change. John Nagle ------------------------------ Date: 10 Aug 88 17:46:01 GMT From: amdahl!pyramid!prls!philabs!aecom!krishna@ames.arpa (Krishna Ambati) Subject: Re: Sigmoid transfer function In a previous article, John B. Nagle writes: > In article <25516@ucbvax.BERKELEY.EDU> munro@icsia.UUCP (Paul Munro) writes: > > > >Try this one : f(x) = x / (1 + |x|) > > > > The graph looks OK, although some scaling is needed to make it > comparable > to the sigmoid. Someone should try it in one of the popular neural net > simulators and see how the results change. > > John Nagle I did try it out in a simulation for The Traveling Salesman using the Hopfield Tank model. Unfortunately, it yields pretty poor results. Probably because it does not rise quickly in the middle region, and furthermore, its convergence to 1 (after scaling) is pretty slow. I would be happy to hear more positive results. Krishna Ambati krishna@aecom.uucp ------------------------------ Date: 11 Aug 88 16:14:21 GMT From: glacier!jbn@labrea.stanford.edu (John B. Nagle) Subject: Re: Sigmoid transfer function In article <1960@aecom.YU.EDU> krishna@aecom.YU.EDU (Krishna Ambati) reports that f(x) = x / (1 + |x|) is a poor transfer function for neural net units, not rising steeply enough near the transition point. This seems reasonable. What we may need is something that looks like a step function fed through a low-pass filter. The idea is to come up with a function that works but can be computed with less hardware (analog or digital) than the sigmoid. Try again? John Nagle ------------------------------ Date: 11 Aug 88 20:09:22 GMT From: phri!cooper!gene@nyu.edu (Gene (the Spook) ) Subject: Re: Sigmoid transfer function in article <17615@glacier.STANFORD.EDU>, John B. Nagle says: > Xref: cooper sci.electronics:2943 comp.ai:1519 comp.ai.neural-nets:158 > > > Recognize that the transfer function in a neural network threshold unit > doesn't really have to be a sigmoid function. It just has to look roughly > like one. The behavior of the net is not all that sensitive to the > exact form of that function. It has to be continuous and monotonic, > reasonably smooth, and rise rapidly in the middle of the working range. > The trigonometric form of the transfer function is really just a notational > convenience. > > It would be a worthwhile exercise to come up with some other forms > of transfer function with roughly the same graph, but better matched to > hardware implementation. How do real neurons do it? Oooh, yeah! Why not make a differential amplifier out of two transistors or so? Just look up how ECL gates are constructed to get the basic design. If you want, you can take a basic op amp and "compare" an input reference of the median voltage, then scale up/down and level shift if necessary. I'm assuming that you mostly care about just the three input levels you mentioned. Try this: Vi = -oo Vo = 0.0 Vi = 0.0 Vo = 0.5 Vi = +oo Vo = 1.0 so take an op amp and run it off of, say, +-10V. With a gain of around +10, an absolute value of around 1V will saturate the output at the respective supply rail. For all practical purposes, you'll have a linear output within +-1.0V, with maximum output being the supply voltages. To increase the linear "spread", lower the gain; to decrease it, increase the gain. So fine, that'll get you a +-10V output. Now use two resistors to make a voltage divider. A 1k and 9k will give you a /10 divider, now giving you a +-1.0V output, for example. Use a trimmer if you want to and get the right voltage swing for your purposes. In your case, a /20 divider will get you a +-0.5V swing. Use a second op amp as a level shifter, set the shift at 0.5V, and voila! Now you have a 0.0 to 1.0 voltage swing! If that's acceptable for your purposes, fine. If you want to "soften" the corners, use a pair of inverse-parallel diodes which will start to saturate as you get near their corner- or knee-voltage (V-sub-gamma). In short, just play around with whatever comes to mind, and see if it suits your purpose. Have fun! Spookfully yours, Gene ...!cmcl2!cooper!gene ------------------------------ Date: 11 Aug 88 21:30:52 GMT From: ankleand@athena.mit.edu (Andy Karanicolas) Subject: Re: Sigmoid transfer function (long) THE VIEWS AND OPINIONS HERE ARE NOT NECESSARILY THOSE OF M.I.T. Here is a schematic of a circuit that should perform the "sigmoid" transfer function talked about. The op-amp could be replaced with current mirrors to perform a subtraction but this circuit is easier (to draw!). I'm sure there are plenty of other (better, simpler) ways to accomplish the task. Maintaining voltage as the analog variable adds to circuit complexity. * PLEASE, NO FLAMES; THIS IS JUST A SUGGESTION * NOTE: The 'X' indicates a connection where ambiguity with a crossing may exist. The 'N' on the transistors indicates emitter for NPN device. ___________________VC1 | | / / R0 \ \ R0 / / R1 \ \ ___/\/\/\______ | | R1 | | | X _______________/\/\/\_____X___|\ --VC2 | | | V2 |- \ _______X____ X _________|________________/\/\/\_____ ___|+ / | | V1 | R1 X |/ --(-VC2) | Q1 | | Q2 | A2 | |/ \| / / ----| |---- R1 \ RP2 \___VOUT | |\N RP0 N/| | / / | |___/\/\___| | \ \ | | | | VREF | | | | RP1 | | | | | GND____/\/\/\/\/\____VC1 GND | | | | | | | |___________|__________________ ______ __/\/\/\__GND | | --> IX | | X R4 | R2 ___/\/\___X | | | | R2 | | \| | |/ | ____/\/\____X__|\--VC2 | Q3 |----X----| Q4 X |- \ ___| N/| |\N | ____|+ / | | VIN GND |/--(-VC2) | | A1 / / \ R3 \ R3 / / \ \ |_____________X____(-VC2) The transfer function of this circuit is: VOUT ~= B2 * { B1 * VC1 + IX * R0 * TANH[ VIN / VTH ] } where VTH = kT/q and is about 25mV at room temp. The constants B2 and B1 are less than unity and are set by potentiometers RP2 and RP1 respectively. Circuit description: Q1 and Q2 form a differential amplifier that provides the tanh function. The potentiometer RP0 helps to equalize transistor mismatches. RP0 should be as small as possible to maintain the tanh function of this amplifier. Choosing a large RP0 will cut down the gain at the midpoint of the 'S'; the tanh function gets 'linearized' and the above transfer equation becomes invalid. The op-amp A1 provides an invereted version of the input voltage; together with the input itself, the input to the diff. amp is a differential mode signal (within component tolerances) equal to 2 * VIN. The input should be from a low impedance source or an input buffer will be needed. The op-amp A2 is configured in a differencing mode. The transfer function of this amplifier is: VOUT = V1 - V2 + VREF. The adjustable reference VREF adjusts the constant B1 and the attenuation pot. on the ouput of A2 adjusts the constant B2. Q3 and Q4 form a current source IX. It can be replaced by a simple resistor but the current source should help maintain the tanh function for large input signals. IX is set by R3, R4 and VC2. One design example: VC1 = 5V VC2 = 15V (typical supply is +5, +15, -15) set IX * R0 = 0.5 set B1 * VC1 = 0.5 (VREF = 0.5) use RP0 = 25 ohms set IX = 0.5mA so that R0 = 1K to cut down loading effects, set R1 = 47K (arbitrarily) set RP1 to 1K (much smaller than R1) set RP2 to 1K for IX = 0.5mA, set R3 = 100 ohms; R4 then is about (15 - .65)/.5mA set R4 to 27K the choice of R2 is not critical; use 1K Q1-4 can be standard 2N2222 or 2N3904 NPN's A1 and A2 can be LM301s (or, UGHH!!, 741's even..) Have fun and good luck! Andy Karanicolas Microsystems Technology Laboratory ankleand@caf.mit.edu ------------------------------ End of AIList Digest ********************