Deep learning is primarily a study of multi-layered neural networks, spanning over a great range of model architectures. We cover the basic following content in theory and in practice. We adapt the content, especially in the last lectures, based on the latest advances:
Introduction to deep learning. A brief introduction of deep learning. History, recent trends.
Deep feedforward network. Neural networks are complex cascades of simple building blocks. We learn the basic idea that creates complexity out of simplicity.
Optimisation. Since neural networks are notoriously difficult to train, we will deepen our understanding of how to optimise them both theoretically and practically.
Deep Learning Regularisation. A central problem in machine learning is how to make an algorithm that will perform well not just on the training data, but also on new inputs. Many strategies used in machine learning are explicitly designed to reduce the test error, possibly at the expense of increased training error. These strategies are known collectively as regularisation.
Convolutional neural networks (CNN). The driving force behind the popularity of deep learning. CNNs have their main application in image recognition, object detection, automatic text translation, and speech recognition. We learn the basics as well as the latest models that are used by all academia and the tech giants.
Modern convolutional neural networks. These models include AlexNet, the first large-scale, modern convolutional network that wins the champion on a large-scale vision challenge; the VGG network (VGGnet), which is even deeper and makes use of a number of repeating blocks of elements; the GoogLeNet, which makes use of networks with parallel concatenations (GoogLeNet); residual networks (ResNet) which are currently the most popular go-to architecture today, and densely connected networks (DenseNet), which are expensive to compute but have set some recent benchmarks.
Transformers. The transformer model is a seq2seq model which uses attention in the encoder as well as the decoder, thus eliminating the need for RNNs, as we explain below. Transformers have been used for many (conditional) sequence generation tasks, such as machine translation, constituency parsing, music generation, protein sequence generation, abstractive text summarisation, image generation (treating the image as a rasterized 1d sequence), etc.
Graph neural networks. Traditionally, neural networks are applied to data with a well-specified and orderly structure, like images or text. With graph neural networks we learn how to harvest the power of deep learning on graph data, such as graphs extracted from social media, molecular graphs, and beyond.
Generative learning. One of the most impressive outputs of Deep Learning is algorithms that paint very realistic but fake data, such images of faces that never existed. Here, we will learn about different generations of generative learning algorithms, along with their connections and open problems.
Self-supervised Learning. Learning from raw data without annotations provides benefits as these methods can be applies to extremely large-scale data and offer other benefits. Starting from NLP, the methods in computer vision have tremendously matured and are diffusing into multiple applications.
Multi-modal Learning. Learning from multiple modalities has shown extremely promising results, especially for naturally paired data such as audio-video and human-constructed paired data such as images and their captions. We will learn about current approaches in this domain.
Open questions in deep learning. Deep learning continues to advance, but many questions remain open. We will discuss open questions ranging from continual learning to adversarial attack, as well as the open research questions regarding the geometric foundations of deep networks.