NPFL124 Introduction to Natural Language Processing
Deep Learning for Natural Language
Processing
Jindřich Helcl
April 14, 2020
Charles University
Faculty of Mathematics and Physics
Institute of Formal and Applied Linguistics
unless otherwise stated
Outline
Neural Networks Basics
Representing Words
Representing Sequences
Recurrent Networks
Convolutional Networks
Self-attentive Networks
Classification and Labeling
Generating Sequences
Pre-training Representations
Word2Vec
ELMo
BERT
Deep Learning for Natural Language Processing
1/90
Deep Learning in NLP
• NLP tasks learn end-to-end using deep learning — the number-one approach in current
research
Deep Learning for Natural Language Processing
2/90
Deep Learning in NLP
• NLP tasks learn end-to-end using deep learning — the number-one approach in current
research
• State of the art in POS tagging, parsing, named-entity recognition, machine translation,
…
Deep Learning for Natural Language Processing
2/90
Deep Learning in NLP
• NLP tasks learn end-to-end using deep learning — the number-one approach in current
research
• State of the art in POS tagging, parsing, named-entity recognition, machine translation,
…
• Good news: training without almost any linguistic insight
Deep Learning for Natural Language Processing
2/90
Deep Learning in NLP
• NLP tasks learn end-to-end using deep learning — the number-one approach in current
research
• State of the art in POS tagging, parsing, named-entity recognition, machine translation,
…
• Good news: training without almost any linguistic insight
• Bad news: requires enormous amount of training data and really big computational
power
Deep Learning for Natural Language Processing
2/90
What is deep learning?
• Buzzword for machine learning using neural networks with many layers using
back-propagation
Deep Learning for Natural Language Processing
3/90
What is deep learning?
• Buzzword for machine learning using neural networks with many layers using
back-propagation
• Learning of a real-valued function with millions of parameters that solves a particular
problem
Deep Learning for Natural Language Processing
3/90