Module Directory

Machine learning has become a prominent tool in data analytics. One major category of it: the reinforcement learning, has been widely used in industry to maximise the notion of cumulative reward. This module is concerned with the conceptual background of reinforcement learning, i.e. Markov decision process (MDP) and dynamic programming.

Reinforcement learning, has been covered under Dynamic Programming for decades, designed on the divide-and-conquer basis, provide the structure to all the methods that have been developed in recent years. Describing the problem status by stages, states and transition matrices, but allowing decisions in the whole process.

The aims of this module are:

• To teach the basic mathematical concepts behind dynamic programming and reinforcement learning.


• To link the solution approach with statistics, computing, simulation, calculus and standard optimization techniques.

By the end of this module, students will be expected to have:

1. Conceptual understanding of the “divide and conquer” idea behind dynamic programming and be able to construct deterministic and stochastic dynamic programming models.


2. Detailed and systematic knowledge of key elements of a dynamic programming system and be able to write Bellman’s equations.


3. Understand the basic techniques of backward induction and be able to write pseudo codes to implement on small scale problems.


4. An ability to identify suitable approaches/software packages to solve some typical applications.

To use dynamic programming we require to have full knowledge of the dynamics of the problem (for example, knowing the probability that an autonomous car will not crash if it turns right at any point in time). These assumptions rarely hold in real-life problems, and to tackle those issues we will explore other methods that do not need such information, and just consider agents that learn by experience (trial & error).

We will also learn about modern reinforcement learning approaches and typical applications, that are designed for large scale problems and have shown excellent results in recent years. A very good example is the algorithm developed by DeepMind based on Deep Reinforcement Learning and Monte Carlo Tree Search that beat professional Go players.

The theory behind these problems and all the required concepts will be discussed in the Lectures, and complemented with ad-hoc experiments in the Laboratory Sessions.

Indicative syllabus

• Basics of sequential decision process: stage, state, action, objective, policy, etc.


• Deterministic dynamic Programming and Bellman's Equation.


• Stochastic: Markov Decision Process (MDP).


• Standard solution approach – backward induction and its computational challenge.


• Approximation approaches – aggregation, regression, neural networks, etc.


• Reinforcement learning techniques: Monte Carlo Methods, Q-learning, Temporal difference methods.


• Applications: Knapsack problem, Multi-armed Bandit, Revenue Management, Vehicle Routing etc.

Teaching in the School will be delivered using a range of face to face lectures, classes and lab sessions as appropriate for each module. Modules may also include online only sessions where it is advantageous, for example for pedagogical reasons, to do so.