Preface
Dedication


Chapter 1—Introduction to Neural Networks
Neural Processing
Neural Network
Output of a Neuron
Cash Register Game
Weights
Training
Feedback
Supervised or Unsupervised Learning
Noise
Memory
Capsule of History
Neural Network Construction
Sample Applications
Qualifying for a Mortgage
Cooperation and Competition
Example—A Feed-Forward Network
Example—A Hopfield Network
Hamming Distance
Asynchronous Update
Binary and Bipolar Inputs
Bias
Another Example for the Hopfield Network
Summary

Chapter 2—C++ and Object Orientation
Introduction to C++
Encapsulation
Data Hiding
Constructors and Destructors as Special Functions of C++
Dynamic Memory Allocation
Overloading
Polymorphism and Polymorphic Functions
Overloading Operators
Inheritance
Derived Classes
Reuse of Code
C++ Compilers
Writing C++ Programs
Summary

Chapter 3—A Look at Fuzzy Logic
Crisp or Fuzzy Logic?
Fuzzy Sets
Fuzzy Set Operations
Union of Fuzzy Sets
Intersection and Complement of Two Fuzzy Sets
Applications of Fuzzy Logic
Examples of Fuzzy Logic
Commercial Applications
Fuzziness in Neural Networks
Code for the Fuzzifier
Fuzzy Control Systems
Fuzziness in Neural Networks
Neural-Trained Fuzzy Systems
Summary

Chapter 4—Constructing a Neural Network
First Example for C++ Implementation
Classes in C++ Implementation
C++ Program for a Hopfield Network
Header File for C++ Program for Hopfield Network
Notes on the Header File Hop.h
Source Code for the Hopfield Network
Comments on the C++ Program for Hopfield Network
Output from the C++ Program for Hopfield Network
Further Comments on the Program and Its Output
A New Weight Matrix to Recall More Patterns
Weight Determination
Binary to Bipolar Mapping
Pattern’s Contribution to Weight
Autoassociative Network
Orthogonal Bit Patterns
Network Nodes and Input Patterns
Second Example for C++ Implementation
C++ Implementation of Perceptron Network
Header File
Implementation of Functions
Source Code for Perceptron Network
Comments on Your C++ Program
Input/Output for percept.cpp
Network Modeling
Tic-Tac-Toe Anyone?
Stability and Plasticity
Stability for a Neural Network
Plasticity for a Neural Network
Short-Term Memory and Long-Term Memory
Summary

Chapter 5—A Survey of Neural Network Models
Neural Network Models
Layers in a Neural Network
Single-Layer Network
XOR Function and the Perceptron
Linear Separability
A Second Look at the XOR Function: Multilayer Perceptron
Example of the Cube Revisited
Strategy
Details
Performance of the Perceptron
Other Two-layer Networks
Many Layer Networks
Connections Between Layers
Instar and Outstar
Weights on Connections
Initialization of Weights
A Small Example
Initializing Weights for Autoassociative Networks
Weight Initialization for Heteroassociative Networks
On Center, Off Surround
Inputs
Outputs
The Threshold Function
The Sigmoid Function
The Step Function
The Ramp Function
Linear Function
Applications
Some Neural Network Models
Adaline and Madaline
Backpropagation
Figure for Backpropagation Network
Bidirectional Associative Memory
Temporal Associative Memory
Brain-State-in-a-Box
Counterpropagation
Neocognitron
Adaptive Resonance Theory
Summary

Chapter 6—Learning and Training
Objective of Learning
Learning and Training
Hebb’s Rule
Delta Rule
Supervised Learning
Generalized Delta Rule
Statistical Training and Simulated Annealing
Radial Basis-Function Networks
Unsupervised Networks
Self-Organization
Learning Vector Quantizer
Associative Memory Models and One-Shot Learning
Learning and Resonance
Learning and Stability
Training and Convergence
Lyapunov Function
Other Training Issues
Adaptation
Generalization Ability
Summary

Chapter 7—Backpropagation
Feedforward Backpropagation Network
Mapping
Layout
Training
Illustration: Adjustment of Weights of Connections from a Neuron in the Hidden Layer
Illustration: Adjustment of Weights of Connections from a Neuron in the Input Layer
Adjustments to Threshold Values or Biases
Another Example of Backpropagation Calculations
Notation and Equations
Notation
Equations
C++ Implementation of a Backpropagation Simulator
A Brief Tour of How to Use the Simulator
C++ Classes and Class Hierarchy
Summary

Chapter 8—BAM: Bidirectional Associative Memory
Introduction
Inputs and Outputs
Weights and Training
Example
Recall of Vectors
Continuation of Example
Special Case—Complements
C++ Implementation
Program Details and Flow
Program Example for BAM
Header File
Source File
Program Output
Additional Issues
Unipolar Binary Bidirectional Associative Memory
Summary

Chapter 9—FAM: Fuzzy Associative Memory
Introduction
Association
FAM Neural Network
Encoding
Example of Encoding
Recall
C++ Implementation
Program details
Header File
Source File
Output
Summary

Chapter 10—Adaptive Resonance Theory (ART)
Introduction
The Network for ART1
A Simplified Diagram of Network Layout
Processing in ART1
Special Features of the ART1 Model
Notation for ART1 Calculations
Algorithm for ART1 Calculations
Initialization of Parameters
Equations for ART1 Computations
Other Models
C++ Implementation
A Header File for the C++ Program for the ART1 Model Network
A Source File for C++ Program for an ART1 Model Network
Program Output
Summary

Chapter 11—The Kohonen Self-Organizing Map
Introduction
Competitive Learning
Normalization of a Vector
Lateral Inhibition
The Mexican Hat Function
Training Law for the Kohonen Map
Significance of the Training Law
The Neighborhood Size and Alpha
C++ Code for Implementing a Kohonen Map
The Kohonen Network
Modeling Lateral Inhibition and Excitation
Classes to be Used
Revisiting the Layer Class
A New Layer Class for a Kohonen Layer
Implementation of the Kohonen Layer and Kohonen Network
Flow of the Program and the main() Function
Flow of the Program
Results from Running the Kohonen Program
A Simple First Example
Orthogonal Input Vectors Example
Variations and Applications of Kohonen Networks
Using a Conscience
LVQ: Learning Vector Quantizer
Counterpropagation Network
Application to Speech Recognition
Summary

Chapter 12—Application to Pattern Recognition
Using the Kohonen Feature Map
An Example Problem: Character Recognition
C++ Code Development
Changes to the Kohonen Program
Testing the Program
Generalization versus Memorization
Adding Characters
Other Experiments to Try
Summary

Chapter 13—Backpropagation II
Enhancing the Simulator
Another Example of Using Backpropagation
Adding the Momentum Term
Code Changes
Adding Noise During Training
One Other Change—Starting Training from a Saved Weight File
Trying the Noise and Momentum Features
Variations of the Backpropagation Algorithm
Applications
Summary

Chapter 14—Application to Financial Forecasting
Introduction
Who Trades with Neural Networks?
Developing a Forecasting Model
The Target and the Timeframe
Domain Expertise
Gather the Data
Pre processing the Data for the Network
Reduce Dimensionality
Eliminate Correlated Inputs Where Possible
Design a Network Architecture
The Train/Test/Redesign Loop
Forecasting the S&P 500
Choosing the Right Outputs and Objective
Choosing the Right Inputs
Choosing a Network Architecture
Preprocessing Data
A View of the Raw Data
Highlight Features in the Data
Normalizing the Range
The Target
Storing Data in Different Files
Training and Testing
Using the Simulator to Calculate Error
Only the Beginning
What’s Next?
Technical Analysis and Neural Network Preprocessing
Moving Averages
Momentum and Rate of Change
Relative Strength Index
Percentage R
Herrick Payoff Index
MACD
“Stochastics”
On-Balance Volume
Accumulation-Distribution
What Others Have Reported
Can a Three-Year-Old Trade Commodities?
Forecasting Treasury Bill and Treasury Note Yields
Neural Nets versus Box-Jenkins Time-Series Forecasting
Neural Nets versus Regression Analysis
Hierarchical Neural Network
The Walk-Forward Methodology of Market Prediction
Dual Confirmation Trading System
A Turning Point Predictor
The S&P 500 and Sunspot Predictions
A Critique of Neural Network Time-Series Forecasting for Trading
Resource Guide for Neural Networks and Fuzzy Logic in Finance
Magazines
Books
Book Vendors
Consultants
Historical Financial Data Vendors
Preprocessing Tools for Neural Network Development
Genetic Algorithms Tool Vendors
Fuzzy Logic Tool Vendors
Neural Network Development Tool Vendors
Summary

Chapter 15—Application to Nonlinear Optimization
Introduction
Neural Networks for Optimization Problems
Traveling Salesperson Problem
The TSP in a Nutshell
Solution via Neural Network
Example of a Traveling Salesperson Problem for Hand Calculation
Neural Network for Traveling Salesperson Problem
Network Choice and Layout
Inputs
Activations, Outputs, and Their Updating
Performance of the Hopfield Network
C++ Implementation of the Hopfield Network for the Traveling Salesperson Problem
Source File for Hopfield Network for Traveling Salesperson Problem
Output from Your C++ Program for the Traveling Salesperson Problem
Other Approaches to Solve the Traveling Salesperson Problem
Optimizing a Stock Portfolio
Tabu Neural Network
Summary

Chapter 16—Applications of Fuzzy Logic
Introduction
A Fuzzy Universe of Applications
Section I: A Look at Fuzzy Databases and Quantification
Databases and Queries
Relations in Databases
Fuzzy Scenarios
Fuzzy Sets Revisited
Fuzzy Relations
Matrix Representation of a Fuzzy Relation
Properties of Fuzzy Relations
Similarity Relations
Resemblance Relations
Fuzzy Partial Order
Fuzzy Queries
Extending Database Models
Example
Possibility Distributions
Example
Queries
Fuzzy Events, Means and Variances
Example: XYZ Company Takeover Price
Probability of a Fuzzy Event
Fuzzy Mean and Fuzzy Variance
Conditional Probability of a Fuzzy Event
Conditional Fuzzy Mean and Fuzzy Variance
Linear Regression a la Possibilities
Fuzzy Numbers
Triangular Fuzzy Number
Linear Possibility Regression Model
Section II: Fuzzy Control
Designing a Fuzzy Logic Controller
Step One: Defining Inputs and Outputs for the FLC
Step Two: Fuzzify the Inputs
Step Three: Set Up Fuzzy Membership Functions for the Output(s)
Step Four: Create a Fuzzy Rule Base
Step Five: Defuzzify the Outputs
Advantages and Disadvantages of Fuzzy Logic Controllers
Summary

Chapter 17—Further Applications
Introduction
Computer Virus Detector
Mobile Robot Navigation
A Classifier
A Two-Stage Network for Radar Pattern Classification
Crisp and Fuzzy Neural Networks for Handwritten Character Recognition
Noise Removal with a Discrete Hopfield Network
Object Identification by Shape
Detecting Skin Cancer
EEG Diagnosis
Time Series Prediction with Recurrent and Nonrecurrent Networks
Security Alarms
Circuit Board Faults
Warranty Claims
Writing Style Recognition
Commercial Optical Character Recognition
ART-EMAP and Object Recognition
Summary
References
Appendix A
Appendix B
Glossary
Index