Abstract: 本文是本系列的第一篇，介绍本系列的主要内容
Keywords: Information Theory，信息论，Inference，推理，Learning Algorithms，学习算法

# 信息论、推理与学习算法介绍

## 机器学习，信息论

• Bayesian Data Modelling
• Monte Carlo Methods
• Variational Methods
• Clustering ALgorithms
• Neural Networks

60年代一个领域 —— 控制理论（cybernetics）在信息论，计算机科学，和神经科学等学科中非常火爆，这些科学家们都在研究一个相同的问题，那时候信息论和机器学习还是属于同一类。大脑是一个压缩信息，进行沟通的系统，而在数据压缩（data conpression）和纠错码上（error-correcting code）表现最好的（state-of-the-art）的算法上使用的工具，在机器学习中也会使用。

## 学习地图

### Preface

1 Introduction to Information Theory
2 Probability, Entropy, and Inference
3 More about Inference

### I Data Compression

4 The Source Coding Theorem
5 Symbol Codes
6 Stream Codes
7 Codes for Integers

### II Noisy-Channel Coding

8 Dependent Random Variables
9 Communication over a Noisy Channel
10 The Noisy-Channel Coding Theorem
11 Error-Correcting Codes and Real Channels

### III Further Topics in Information Theory

12 Hash Codes: Codes for Ecient Information Retrieval
13 Binary Codes
14 Very Good Linear Codes Exist
15 Further Exercises on Information Theory
16 Message Passing
17 Communication over Constrained Noiseless Channels
18 Crosswords and Codebreaking
19 Why have Sex? Information Acquisition and Evolution

### IV Probabilities and Inference

20 An Example Inference Task: Clustering
21 Exact Inference by Complete Enumeration
22 Maximum Likelihood and Clustering
23 Useful Probability Distributions
24 Exact Marginalization
25 Exact Marginalization in Trellises
26 Exact Marginalization in Graphs
27 Laplace’s Method
28 Model Comparison and Occam’s Razor
29 Monte Carlo Methods
30 Ecient Monte Carlo Methods
31 Ising Models
32 Exact Monte Carlo Sampling
33 Variational Methods
34 Independent Component Analysis and Latent Variable Modelling
35 Random Inference Topics
36 Decision Theory
37 Bayesian Inference and Sampling Theory

### V Neural networks

38 Introduction to Neural Networks
39 The Single Neuron as a Classi er
40 Capacity of a Single Neuron
41 Learning as Inference
42 Hop eld Networks
43 Boltzmann Machines
44 Supervised Learning in Multilayer Networks
45 Gaussian Processes
46 Deconvolution

### VI Sparse Graph Codes

47 Low-Density Parity-Check Codes
48 Convolutional Codes and Turbo Codes
49 Repeat{Accumulate Codes
50 Digital Fountain Codes

### 地图

github: https://github.com/Tony-Tan/MachineLearningMath 上有高清大图

0%