SlideShare a Scribd company logo
1 of 15
Download to read offline
Machine Learning
Neural Networks
Portland Data Science Group
Created by Andrew Ferlitsch
Community Outreach Officer
August, 2017
Initial History
• Neural Networks have been around a long time.
• 1943 - Warren McCulloch, a neurophysiologist and Walter Pitts,
a mathematician, published a paper on how neurons might work.
They modeled a simple neural network with electrical circuits.
• 1949 - The Organization of Behavior, by Donald Hebb reinforced
the concept of neurons.
• 1950s - Nathanial Rochester from the IBM research laboratories
led the first effort to simulate a neural network.
• 1959 - Bernard Widrow and Marcian Hoff of Stanford developed
the first real neural network – MADALINE.
• 1969 - Marvin Minsky and Seymour Papert's 1969 book
Perceptrons, kicked off the dissolutionment period where little
research continued until 1981.
i.e., demonstrated the Perceptron could not model an XOR operation.
Neuron
Neural Networks consist of Neurons
X1
Inputs
W1
W2
W3
X2
X3
Neuron
Inputs from
the features
(independent
variables) in
the dataset.
Weight (importance)
on how each feature
contributes to the output.
Output
Value
The model
(predictor)
The prediction
Can be:
Real value
Probability
Binary
Categorical
Neuron – Categorical Output
Neural Networks consist of Neurons
X1
Inputs
W1
W2
W3
X2
X3
Neuron
Y1
Y2
Y3
Outputs
Categorical
Outputs
(e.g., Apple,
Pear, Banana).
Neuron outputs only
a single value.
Output nodes Y1, Y2 and
Y3 each weight the output
from the neuron and make
a separate calculation for
their final output.
Neuron - Details
Neural Networks consist of Neurons
X1
Inputs
W1
W2
W3
X2
X3
Neuron
Output
Value
Normalize (0..1) or Standardize the inputs (feature scaling)
so no input dominates another.
𝑖=0
𝑛
𝑤𝑖 ∗ 𝑥𝑖Ø( )
Summation of the weighted inputs
Activation function
Backward propagation to
adjust (learn) the weights
(e.g., Gradient Descent).
The higher the weight,
the more it contributes
to the outcome
(prediction).
Activation Functions
• Most Common
• Threshold – Either a zero or one is outputted (binary).
Ø(x) =
• Sigmoid – A Curve that converges exponentially towards 0 for
x < 0 and 1 for x > 0.
{ 1 if x ≥ 0
0 if x < 0 }
Convergence to zero
Convergence to one
Also referred to as
a squashing function,
Squashing the output
between 0 and 1.
Popularly used in
output nodes for
probability prediction.
Activation Functions
• Most Common
• Hyperbolic Tangent – converges to -1 for x < 0 and 1 for x > 0.
Ø(x) =
𝟏 − 𝒆−𝟐𝒙
𝟏+ 𝒆−𝟐𝒙
• Rectifier – 0 if x <= 0, otherwise x
Ø(x) =
Ø(x) = max(0,x)
{ 0 if x ≤ 0
x if x > 0 } Popularly used in
hidden layers for
outputting to the next
layer.
Also referred to as
a squashing function,
Squashing the output
between -1 and 1.
Alternate representation.
Fully Connected Neural Network (FCNN)
• Full Connected Neural Network consists of:
• Input Layer – inputs from the data (samples).
• Output Layer – the predictions.
• Hidden Layer(s) – Between the input and output layers,
where the learning occurs.
• All nodes are connected to every other node in the next layer.
• Activation Functions – where outputs are binary, squashed, or
rectified.
• Forward Feeding and Backward Propagation - for learning the
weights.
Fully Connected Neural Network (FCNN)
X1
X2
Xn
Input Layer
Hidden Layer
ŷ
Output Layer
Simple FCNN:
- One Hidden Layer
- One Output Node
Rectifier Activation Function (ReLU)
Sigmoid Activation Function
If below zero, then
Output no signal.
Squash into a probability.
Acronym
Deep Neural Network (FCNN)
X1
X2
Xn
Input Layer
Hidden Layers
ŷ
Output Layer
It’s a Deep Neural Network
if it has more than one hidden
layer – That’s It!
Hidden Nodes are Specialized Learners
Age
Income
18-25
(low
income
)
ŷ Spending
Each Node in the Hidden Network Specializes
W1-1
W2-1
Learns weights to best predict when age is young and
income is low (i.e., they spend their parent’s money).
Outputs high signal
Outputs low or no signal
< 25
< 1000
Sample
The more hidden nodes, the more specialized learners
Cost Function
Age
Income
ŷ Spending - ŷ
Calculate Cost (Loss) During Training
W1-1
W2-1
< 25
< 1000
y (label)
Data
y
Predicted
And actual.
C =
𝟏
𝟐
𝒚 − ŷ 𝟐
One of the most commonly used
cost functions for neural networks.
Feed Forward - Training
Feed Forward Training Loop
Training
Data
Data
Data
Data
Data
Feed a single
row of data at
a time.
Repeat
Neural Network
C =
𝟏
𝟐
𝒚 − ŷ 𝟐
Calculate the cost (loss).
Converge
?
Can’t minimize the cost
function anymore.
Adjust Weights
Make small adjustments to
weights in the neural network.
Summation
∑ C =
𝟏
𝟐
𝒚 − ŷ 𝟐
No
Run the training set again
through the neural network.
Each run is called an Epoch.
Yes
StopTrained Neural Network
Multiple Output Nodes - Softmax
• Squashes a set of input values into 0 and 1 (probabilities), all
adding up to 1.
Softmax
z1
z2
z3
zk
f(z1) ∈ R{ 0, 1 }
f(z2) ∈ R{ 0, 1 }
f(z3) ∈ R{ 0, 1 }
f(zk) ∈ R{ 0, 1 }
Output Layer
Hidden Layer
x1
x2
x3
Input Layer
Features
Predicted
output
(real) values
Classification
probabilities, e.g.,
90% apple
6% pear
3% orange
1% banana
Each output node specializes
on a different classification.
Final Note – Training vs. Prediction
• Once we have trained the neural network, we do not have to
repeat the training steps when using the model for prediction.
• No repeating of Epochs, Gradient Descent and Backward Propagation.
• The model will run much faster than during training.

More Related Content

What's hot

Introduction to Neural networks (under graduate course) Lecture 6 of 9
Introduction to Neural networks (under graduate course) Lecture 6 of 9Introduction to Neural networks (under graduate course) Lecture 6 of 9
Introduction to Neural networks (under graduate course) Lecture 6 of 9Randa Elanwar
 
Artificial Neural Network
Artificial Neural NetworkArtificial Neural Network
Artificial Neural NetworkKnoldus Inc.
 
Introduction to Neural networks (under graduate course) Lecture 7 of 9
Introduction to Neural networks (under graduate course) Lecture 7 of 9Introduction to Neural networks (under graduate course) Lecture 7 of 9
Introduction to Neural networks (under graduate course) Lecture 7 of 9Randa Elanwar
 
Perceptron (neural network)
Perceptron (neural network)Perceptron (neural network)
Perceptron (neural network)EdutechLearners
 
lecture07.ppt
lecture07.pptlecture07.ppt
lecture07.pptbutest
 
Soft Computering Technics - Unit2
Soft Computering Technics - Unit2Soft Computering Technics - Unit2
Soft Computering Technics - Unit2sravanthi computers
 
Introduction to Neural networks (under graduate course) Lecture 4 of 9
Introduction to Neural networks (under graduate course) Lecture 4 of 9Introduction to Neural networks (under graduate course) Lecture 4 of 9
Introduction to Neural networks (under graduate course) Lecture 4 of 9Randa Elanwar
 
Introduction to Neural networks (under graduate course) Lecture 5 of 9
Introduction to Neural networks (under graduate course) Lecture 5 of 9Introduction to Neural networks (under graduate course) Lecture 5 of 9
Introduction to Neural networks (under graduate course) Lecture 5 of 9Randa Elanwar
 
2.5 backpropagation
2.5 backpropagation2.5 backpropagation
2.5 backpropagationKrish_ver2
 
Deep Feed Forward Neural Networks and Regularization
Deep Feed Forward Neural Networks and RegularizationDeep Feed Forward Neural Networks and Regularization
Deep Feed Forward Neural Networks and RegularizationYan Xu
 
The Perceptron and its Learning Rule
The Perceptron and its Learning RuleThe Perceptron and its Learning Rule
The Perceptron and its Learning RuleNoor Ul Hudda Memon
 
Activation function
Activation functionActivation function
Activation functionAstha Jain
 
Artificial Neural Network
Artificial Neural NetworkArtificial Neural Network
Artificial Neural NetworkDessy Amirudin
 
Neural networks
Neural networksNeural networks
Neural networksSlideshare
 
Counter propagation Network
Counter propagation NetworkCounter propagation Network
Counter propagation NetworkAkshay Dhole
 

What's hot (20)

Introduction to Neural networks (under graduate course) Lecture 6 of 9
Introduction to Neural networks (under graduate course) Lecture 6 of 9Introduction to Neural networks (under graduate course) Lecture 6 of 9
Introduction to Neural networks (under graduate course) Lecture 6 of 9
 
Artificial Neural Network
Artificial Neural NetworkArtificial Neural Network
Artificial Neural Network
 
Introduction to Neural networks (under graduate course) Lecture 7 of 9
Introduction to Neural networks (under graduate course) Lecture 7 of 9Introduction to Neural networks (under graduate course) Lecture 7 of 9
Introduction to Neural networks (under graduate course) Lecture 7 of 9
 
04 Multi-layer Feedforward Networks
04 Multi-layer Feedforward Networks04 Multi-layer Feedforward Networks
04 Multi-layer Feedforward Networks
 
Perceptron (neural network)
Perceptron (neural network)Perceptron (neural network)
Perceptron (neural network)
 
lecture07.ppt
lecture07.pptlecture07.ppt
lecture07.ppt
 
Soft Computering Technics - Unit2
Soft Computering Technics - Unit2Soft Computering Technics - Unit2
Soft Computering Technics - Unit2
 
03 Single layer Perception Classifier
03 Single layer Perception Classifier03 Single layer Perception Classifier
03 Single layer Perception Classifier
 
Introduction to Neural networks (under graduate course) Lecture 4 of 9
Introduction to Neural networks (under graduate course) Lecture 4 of 9Introduction to Neural networks (under graduate course) Lecture 4 of 9
Introduction to Neural networks (under graduate course) Lecture 4 of 9
 
Introduction to Neural networks (under graduate course) Lecture 5 of 9
Introduction to Neural networks (under graduate course) Lecture 5 of 9Introduction to Neural networks (under graduate course) Lecture 5 of 9
Introduction to Neural networks (under graduate course) Lecture 5 of 9
 
2.5 backpropagation
2.5 backpropagation2.5 backpropagation
2.5 backpropagation
 
Neural
NeuralNeural
Neural
 
Deep Feed Forward Neural Networks and Regularization
Deep Feed Forward Neural Networks and RegularizationDeep Feed Forward Neural Networks and Regularization
Deep Feed Forward Neural Networks and Regularization
 
The Perceptron and its Learning Rule
The Perceptron and its Learning RuleThe Perceptron and its Learning Rule
The Perceptron and its Learning Rule
 
Activation function
Activation functionActivation function
Activation function
 
Neural Networks
Neural NetworksNeural Networks
Neural Networks
 
Artificial Neural Network
Artificial Neural NetworkArtificial Neural Network
Artificial Neural Network
 
Neural networks
Neural networksNeural networks
Neural networks
 
Counter propagation Network
Counter propagation NetworkCounter propagation Network
Counter propagation Network
 
Unit 1
Unit 1Unit 1
Unit 1
 

Similar to Machine Learning - Neural Networks - Perceptron

Artificial Neural Network
Artificial Neural NetworkArtificial Neural Network
Artificial Neural NetworkPrakash K
 
JAISTサマースクール2016「脳を知るための理論」講義04 Neural Networks and Neuroscience
JAISTサマースクール2016「脳を知るための理論」講義04 Neural Networks and Neuroscience JAISTサマースクール2016「脳を知るための理論」講義04 Neural Networks and Neuroscience
JAISTサマースクール2016「脳を知るための理論」講義04 Neural Networks and Neuroscience hirokazutanaka
 
Artificial Neural Network
Artificial Neural NetworkArtificial Neural Network
Artificial Neural NetworkRenas Rekany
 
Artificial Neural Network_VCW (1).pptx
Artificial Neural Network_VCW (1).pptxArtificial Neural Network_VCW (1).pptx
Artificial Neural Network_VCW (1).pptxpratik610182
 
SOFT COMPUTERING TECHNICS -Unit 1
SOFT COMPUTERING TECHNICS -Unit 1SOFT COMPUTERING TECHNICS -Unit 1
SOFT COMPUTERING TECHNICS -Unit 1sravanthi computers
 
Neural network
Neural networkNeural network
Neural networkDeepikaT13
 
Artificial Neural Network (draft)
Artificial Neural Network (draft)Artificial Neural Network (draft)
Artificial Neural Network (draft)James Boulie
 
ACUMENS ON NEURAL NET AKG 20 7 23.pptx
ACUMENS ON NEURAL NET AKG 20 7 23.pptxACUMENS ON NEURAL NET AKG 20 7 23.pptx
ACUMENS ON NEURAL NET AKG 20 7 23.pptxgnans Kgnanshek
 
Neural networks of artificial intelligence
Neural networks of artificial  intelligenceNeural networks of artificial  intelligence
Neural networks of artificial intelligencealldesign
 
2011 0480.neural-networks
2011 0480.neural-networks2011 0480.neural-networks
2011 0480.neural-networksParneet Kaur
 
ANNs have been widely used in various domains for: Pattern recognition Funct...
ANNs have been widely used in various domains for: Pattern recognition  Funct...ANNs have been widely used in various domains for: Pattern recognition  Funct...
ANNs have been widely used in various domains for: Pattern recognition Funct...vijaym148
 
Neural Networks Ver1
Neural  Networks  Ver1Neural  Networks  Ver1
Neural Networks Ver1ncct
 

Similar to Machine Learning - Neural Networks - Perceptron (20)

Artificial Neural Network
Artificial Neural NetworkArtificial Neural Network
Artificial Neural Network
 
10-Perceptron.pdf
10-Perceptron.pdf10-Perceptron.pdf
10-Perceptron.pdf
 
JAISTサマースクール2016「脳を知るための理論」講義04 Neural Networks and Neuroscience
JAISTサマースクール2016「脳を知るための理論」講義04 Neural Networks and Neuroscience JAISTサマースクール2016「脳を知るための理論」講義04 Neural Networks and Neuroscience
JAISTサマースクール2016「脳を知るための理論」講義04 Neural Networks and Neuroscience
 
Neural network
Neural networkNeural network
Neural network
 
Artificial Neural Network
Artificial Neural NetworkArtificial Neural Network
Artificial Neural Network
 
Neural Network.pptx
Neural Network.pptxNeural Network.pptx
Neural Network.pptx
 
Perceptron
PerceptronPerceptron
Perceptron
 
Artificial Neural Network_VCW (1).pptx
Artificial Neural Network_VCW (1).pptxArtificial Neural Network_VCW (1).pptx
Artificial Neural Network_VCW (1).pptx
 
SOFT COMPUTERING TECHNICS -Unit 1
SOFT COMPUTERING TECHNICS -Unit 1SOFT COMPUTERING TECHNICS -Unit 1
SOFT COMPUTERING TECHNICS -Unit 1
 
tutorial.ppt
tutorial.ppttutorial.ppt
tutorial.ppt
 
ai7.ppt
ai7.pptai7.ppt
ai7.ppt
 
ai7.ppt
ai7.pptai7.ppt
ai7.ppt
 
ANN.ppt
ANN.pptANN.ppt
ANN.ppt
 
Neural network
Neural networkNeural network
Neural network
 
Artificial Neural Network (draft)
Artificial Neural Network (draft)Artificial Neural Network (draft)
Artificial Neural Network (draft)
 
ACUMENS ON NEURAL NET AKG 20 7 23.pptx
ACUMENS ON NEURAL NET AKG 20 7 23.pptxACUMENS ON NEURAL NET AKG 20 7 23.pptx
ACUMENS ON NEURAL NET AKG 20 7 23.pptx
 
Neural networks of artificial intelligence
Neural networks of artificial  intelligenceNeural networks of artificial  intelligence
Neural networks of artificial intelligence
 
2011 0480.neural-networks
2011 0480.neural-networks2011 0480.neural-networks
2011 0480.neural-networks
 
ANNs have been widely used in various domains for: Pattern recognition Funct...
ANNs have been widely used in various domains for: Pattern recognition  Funct...ANNs have been widely used in various domains for: Pattern recognition  Funct...
ANNs have been widely used in various domains for: Pattern recognition Funct...
 
Neural Networks Ver1
Neural  Networks  Ver1Neural  Networks  Ver1
Neural Networks Ver1
 

More from Andrew Ferlitsch

Pareto Principle Applied to QA
Pareto Principle Applied to QAPareto Principle Applied to QA
Pareto Principle Applied to QAAndrew Ferlitsch
 
Whiteboarding Coding Challenges in Python
Whiteboarding Coding Challenges in PythonWhiteboarding Coding Challenges in Python
Whiteboarding Coding Challenges in PythonAndrew Ferlitsch
 
Object Oriented Programming Principles
Object Oriented Programming PrinciplesObject Oriented Programming Principles
Object Oriented Programming PrinciplesAndrew Ferlitsch
 
Python - Installing and Using Python and Jupyter Notepad
Python - Installing and Using Python and Jupyter NotepadPython - Installing and Using Python and Jupyter Notepad
Python - Installing and Using Python and Jupyter NotepadAndrew Ferlitsch
 
Natural Language Processing - Groupings (Associations) Generation
Natural Language Processing - Groupings (Associations) GenerationNatural Language Processing - Groupings (Associations) Generation
Natural Language Processing - Groupings (Associations) GenerationAndrew Ferlitsch
 
Natural Language Provessing - Handling Narrarive Fields in Datasets for Class...
Natural Language Provessing - Handling Narrarive Fields in Datasets for Class...Natural Language Provessing - Handling Narrarive Fields in Datasets for Class...
Natural Language Provessing - Handling Narrarive Fields in Datasets for Class...Andrew Ferlitsch
 
Machine Learning - Introduction to Recurrent Neural Networks
Machine Learning - Introduction to Recurrent Neural NetworksMachine Learning - Introduction to Recurrent Neural Networks
Machine Learning - Introduction to Recurrent Neural NetworksAndrew Ferlitsch
 
Machine Learning - Introduction to Convolutional Neural Networks
Machine Learning - Introduction to Convolutional Neural NetworksMachine Learning - Introduction to Convolutional Neural Networks
Machine Learning - Introduction to Convolutional Neural NetworksAndrew Ferlitsch
 
Machine Learning - Introduction to Neural Networks
Machine Learning - Introduction to Neural NetworksMachine Learning - Introduction to Neural Networks
Machine Learning - Introduction to Neural NetworksAndrew Ferlitsch
 
Python - Numpy/Pandas/Matplot Machine Learning Libraries
Python - Numpy/Pandas/Matplot Machine Learning LibrariesPython - Numpy/Pandas/Matplot Machine Learning Libraries
Python - Numpy/Pandas/Matplot Machine Learning LibrariesAndrew Ferlitsch
 
Machine Learning - Accuracy and Confusion Matrix
Machine Learning - Accuracy and Confusion MatrixMachine Learning - Accuracy and Confusion Matrix
Machine Learning - Accuracy and Confusion MatrixAndrew Ferlitsch
 
Machine Learning - Ensemble Methods
Machine Learning - Ensemble MethodsMachine Learning - Ensemble Methods
Machine Learning - Ensemble MethodsAndrew Ferlitsch
 
ML - Multiple Linear Regression
ML - Multiple Linear RegressionML - Multiple Linear Regression
ML - Multiple Linear RegressionAndrew Ferlitsch
 
ML - Simple Linear Regression
ML - Simple Linear RegressionML - Simple Linear Regression
ML - Simple Linear RegressionAndrew Ferlitsch
 
Machine Learning - Dummy Variable Conversion
Machine Learning - Dummy Variable ConversionMachine Learning - Dummy Variable Conversion
Machine Learning - Dummy Variable ConversionAndrew Ferlitsch
 
Machine Learning - Splitting Datasets
Machine Learning - Splitting DatasetsMachine Learning - Splitting Datasets
Machine Learning - Splitting DatasetsAndrew Ferlitsch
 
Machine Learning - Dataset Preparation
Machine Learning - Dataset PreparationMachine Learning - Dataset Preparation
Machine Learning - Dataset PreparationAndrew Ferlitsch
 
Machine Learning - Introduction to Tensorflow
Machine Learning - Introduction to TensorflowMachine Learning - Introduction to Tensorflow
Machine Learning - Introduction to TensorflowAndrew Ferlitsch
 

More from Andrew Ferlitsch (20)

AI - Intelligent Agents
AI - Intelligent AgentsAI - Intelligent Agents
AI - Intelligent Agents
 
Pareto Principle Applied to QA
Pareto Principle Applied to QAPareto Principle Applied to QA
Pareto Principle Applied to QA
 
Whiteboarding Coding Challenges in Python
Whiteboarding Coding Challenges in PythonWhiteboarding Coding Challenges in Python
Whiteboarding Coding Challenges in Python
 
Object Oriented Programming Principles
Object Oriented Programming PrinciplesObject Oriented Programming Principles
Object Oriented Programming Principles
 
Python - OOP Programming
Python - OOP ProgrammingPython - OOP Programming
Python - OOP Programming
 
Python - Installing and Using Python and Jupyter Notepad
Python - Installing and Using Python and Jupyter NotepadPython - Installing and Using Python and Jupyter Notepad
Python - Installing and Using Python and Jupyter Notepad
 
Natural Language Processing - Groupings (Associations) Generation
Natural Language Processing - Groupings (Associations) GenerationNatural Language Processing - Groupings (Associations) Generation
Natural Language Processing - Groupings (Associations) Generation
 
Natural Language Provessing - Handling Narrarive Fields in Datasets for Class...
Natural Language Provessing - Handling Narrarive Fields in Datasets for Class...Natural Language Provessing - Handling Narrarive Fields in Datasets for Class...
Natural Language Provessing - Handling Narrarive Fields in Datasets for Class...
 
Machine Learning - Introduction to Recurrent Neural Networks
Machine Learning - Introduction to Recurrent Neural NetworksMachine Learning - Introduction to Recurrent Neural Networks
Machine Learning - Introduction to Recurrent Neural Networks
 
Machine Learning - Introduction to Convolutional Neural Networks
Machine Learning - Introduction to Convolutional Neural NetworksMachine Learning - Introduction to Convolutional Neural Networks
Machine Learning - Introduction to Convolutional Neural Networks
 
Machine Learning - Introduction to Neural Networks
Machine Learning - Introduction to Neural NetworksMachine Learning - Introduction to Neural Networks
Machine Learning - Introduction to Neural Networks
 
Python - Numpy/Pandas/Matplot Machine Learning Libraries
Python - Numpy/Pandas/Matplot Machine Learning LibrariesPython - Numpy/Pandas/Matplot Machine Learning Libraries
Python - Numpy/Pandas/Matplot Machine Learning Libraries
 
Machine Learning - Accuracy and Confusion Matrix
Machine Learning - Accuracy and Confusion MatrixMachine Learning - Accuracy and Confusion Matrix
Machine Learning - Accuracy and Confusion Matrix
 
Machine Learning - Ensemble Methods
Machine Learning - Ensemble MethodsMachine Learning - Ensemble Methods
Machine Learning - Ensemble Methods
 
ML - Multiple Linear Regression
ML - Multiple Linear RegressionML - Multiple Linear Regression
ML - Multiple Linear Regression
 
ML - Simple Linear Regression
ML - Simple Linear RegressionML - Simple Linear Regression
ML - Simple Linear Regression
 
Machine Learning - Dummy Variable Conversion
Machine Learning - Dummy Variable ConversionMachine Learning - Dummy Variable Conversion
Machine Learning - Dummy Variable Conversion
 
Machine Learning - Splitting Datasets
Machine Learning - Splitting DatasetsMachine Learning - Splitting Datasets
Machine Learning - Splitting Datasets
 
Machine Learning - Dataset Preparation
Machine Learning - Dataset PreparationMachine Learning - Dataset Preparation
Machine Learning - Dataset Preparation
 
Machine Learning - Introduction to Tensorflow
Machine Learning - Introduction to TensorflowMachine Learning - Introduction to Tensorflow
Machine Learning - Introduction to Tensorflow
 

Recently uploaded

Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 

Recently uploaded (20)

Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 

Machine Learning - Neural Networks - Perceptron

  • 1. Machine Learning Neural Networks Portland Data Science Group Created by Andrew Ferlitsch Community Outreach Officer August, 2017
  • 2. Initial History • Neural Networks have been around a long time. • 1943 - Warren McCulloch, a neurophysiologist and Walter Pitts, a mathematician, published a paper on how neurons might work. They modeled a simple neural network with electrical circuits. • 1949 - The Organization of Behavior, by Donald Hebb reinforced the concept of neurons. • 1950s - Nathanial Rochester from the IBM research laboratories led the first effort to simulate a neural network. • 1959 - Bernard Widrow and Marcian Hoff of Stanford developed the first real neural network – MADALINE. • 1969 - Marvin Minsky and Seymour Papert's 1969 book Perceptrons, kicked off the dissolutionment period where little research continued until 1981. i.e., demonstrated the Perceptron could not model an XOR operation.
  • 3. Neuron Neural Networks consist of Neurons X1 Inputs W1 W2 W3 X2 X3 Neuron Inputs from the features (independent variables) in the dataset. Weight (importance) on how each feature contributes to the output. Output Value The model (predictor) The prediction Can be: Real value Probability Binary Categorical
  • 4. Neuron – Categorical Output Neural Networks consist of Neurons X1 Inputs W1 W2 W3 X2 X3 Neuron Y1 Y2 Y3 Outputs Categorical Outputs (e.g., Apple, Pear, Banana). Neuron outputs only a single value. Output nodes Y1, Y2 and Y3 each weight the output from the neuron and make a separate calculation for their final output.
  • 5. Neuron - Details Neural Networks consist of Neurons X1 Inputs W1 W2 W3 X2 X3 Neuron Output Value Normalize (0..1) or Standardize the inputs (feature scaling) so no input dominates another. 𝑖=0 𝑛 𝑤𝑖 ∗ 𝑥𝑖Ø( ) Summation of the weighted inputs Activation function Backward propagation to adjust (learn) the weights (e.g., Gradient Descent). The higher the weight, the more it contributes to the outcome (prediction).
  • 6. Activation Functions • Most Common • Threshold – Either a zero or one is outputted (binary). Ø(x) = • Sigmoid – A Curve that converges exponentially towards 0 for x < 0 and 1 for x > 0. { 1 if x ≥ 0 0 if x < 0 } Convergence to zero Convergence to one Also referred to as a squashing function, Squashing the output between 0 and 1. Popularly used in output nodes for probability prediction.
  • 7. Activation Functions • Most Common • Hyperbolic Tangent – converges to -1 for x < 0 and 1 for x > 0. Ø(x) = 𝟏 − 𝒆−𝟐𝒙 𝟏+ 𝒆−𝟐𝒙 • Rectifier – 0 if x <= 0, otherwise x Ø(x) = Ø(x) = max(0,x) { 0 if x ≤ 0 x if x > 0 } Popularly used in hidden layers for outputting to the next layer. Also referred to as a squashing function, Squashing the output between -1 and 1. Alternate representation.
  • 8. Fully Connected Neural Network (FCNN) • Full Connected Neural Network consists of: • Input Layer – inputs from the data (samples). • Output Layer – the predictions. • Hidden Layer(s) – Between the input and output layers, where the learning occurs. • All nodes are connected to every other node in the next layer. • Activation Functions – where outputs are binary, squashed, or rectified. • Forward Feeding and Backward Propagation - for learning the weights.
  • 9. Fully Connected Neural Network (FCNN) X1 X2 Xn Input Layer Hidden Layer ŷ Output Layer Simple FCNN: - One Hidden Layer - One Output Node Rectifier Activation Function (ReLU) Sigmoid Activation Function If below zero, then Output no signal. Squash into a probability. Acronym
  • 10. Deep Neural Network (FCNN) X1 X2 Xn Input Layer Hidden Layers ŷ Output Layer It’s a Deep Neural Network if it has more than one hidden layer – That’s It!
  • 11. Hidden Nodes are Specialized Learners Age Income 18-25 (low income ) ŷ Spending Each Node in the Hidden Network Specializes W1-1 W2-1 Learns weights to best predict when age is young and income is low (i.e., they spend their parent’s money). Outputs high signal Outputs low or no signal < 25 < 1000 Sample The more hidden nodes, the more specialized learners
  • 12. Cost Function Age Income ŷ Spending - ŷ Calculate Cost (Loss) During Training W1-1 W2-1 < 25 < 1000 y (label) Data y Predicted And actual. C = 𝟏 𝟐 𝒚 − ŷ 𝟐 One of the most commonly used cost functions for neural networks.
  • 13. Feed Forward - Training Feed Forward Training Loop Training Data Data Data Data Data Feed a single row of data at a time. Repeat Neural Network C = 𝟏 𝟐 𝒚 − ŷ 𝟐 Calculate the cost (loss). Converge ? Can’t minimize the cost function anymore. Adjust Weights Make small adjustments to weights in the neural network. Summation ∑ C = 𝟏 𝟐 𝒚 − ŷ 𝟐 No Run the training set again through the neural network. Each run is called an Epoch. Yes StopTrained Neural Network
  • 14. Multiple Output Nodes - Softmax • Squashes a set of input values into 0 and 1 (probabilities), all adding up to 1. Softmax z1 z2 z3 zk f(z1) ∈ R{ 0, 1 } f(z2) ∈ R{ 0, 1 } f(z3) ∈ R{ 0, 1 } f(zk) ∈ R{ 0, 1 } Output Layer Hidden Layer x1 x2 x3 Input Layer Features Predicted output (real) values Classification probabilities, e.g., 90% apple 6% pear 3% orange 1% banana Each output node specializes on a different classification.
  • 15. Final Note – Training vs. Prediction • Once we have trained the neural network, we do not have to repeat the training steps when using the model for prediction. • No repeating of Epochs, Gradient Descent and Backward Propagation. • The model will run much faster than during training.