Saarland University, Machine Learning Group, Fak. MI - Mathematik und Informatik, Campus E1 1, 66123 Saarbrücken, Germany

Machine Learning Group
Department of Mathematics and Computer Science - Saarland University

# TEACHING

## MACHINE LEARNING

Wintersemester 2013/2014

### Result of Re-Exam and Total Result:Result

You can check your exam on Thursday, 10.04., 10.00-12.00 in room 222.2.

### Organization of the re-exam on Tuesday, 01.04., 14.00-17.00.

• Location: Lecture hall 2 in E1 3
• Please bring your student identity card - otherwise you are not allowed to the exam !
• Please bring paper for the exam.
• Be there at 13.50 in order to check in which lecture hall you are and so that we can start on time.
• It is a closed book exam - no notes, books or pocket calculators or any other devices except pen and paper are allowed.
• Mobile phones, tablets, laptops and other electronic devices have to be turned off.

### LECTURE MATERIAL

Lecture notes: PDF . It is not recommended to print them as these notes will updated over the semester.

The practical exercises will be in Matlab.

### SLIDES AND EXCERCISES

 18.10. - Introduction Exercise 0 Solution 0 23.10. - Recap: Probability Theory Exercise 1 Solution 1 25.10. - Recap: Probability (ctd) + Decision Theory 30.10. - Bayesian Decision Theory Exercise 2 Solution 2 Matlab Decision Boundary Demo 06.11 - Empirical risk minimization Exercise 3 Solution 3 08.11. - Linear Regression 13.11. - Introduction to Optimization Exercise 4 Solution 4 Data for Problem 10 20.11. - Optimization (cont) Exercise 5 Solution 5 Data for Problem 11 22.11. - Linear Classification 27.11. - Linear Classification (ctd) Exercise 6 Solution 6 Data for Problem 13 29.11. - Kernel Methods 04.12. - Lecture canceled Exercise 7 Solution 7 Data 06.12. - Kernel Methods (cont.) 11.12. - Kernel Methods II Exercise 8 Solution 8 Data 13.12. - Evaluation, ROC-Curve, AUC 18.12. - Tests, Model selection Exercise 9 (Bonus) Solution 9 20.12. - Tests, Model selection (cont.) 08.01. - Feature selection I Exercise 10 Solution 10 Data 10.01. - Boosting 15.01. - Decision Trees, Neural Networks, Nearest Neighbor methods Exercise 11 Solution 11 Data 17.01. - Semi-supervised Learning 22.01. - SSL (continued) Exercise 12 Solution 12 24.01. - Lecture canceled 29.01. - K-Means and Spectral Clustering 31.01. - Hierarchical Clustering 05.02. - Dimensionality Reduction 07.02. - Statistical Learning Theory

### TIME AND LOCATION

Lecture: We, 14-16, Fr, 10-12, HS I, E1 3

Exercise Groups:

• Th, 12-14, SR 3 (U.11), E2 5
• Th, 14-16, SR 3 (U.11), E2 5
• Fr, 14-16, SR 014, E1 3

Exams: 19.2., 14.00-17.00 Re-exam: 1.4. , 14.00-17.00

• 50% of the points in the exercises (up to that point) are needed to take part in the exams (end-term/re-exam). In order to being admitted for the endterm and re-exam, you need to have presented properly once a solution in the exercise groups.
• An exam is passed if you get at least 50% of the points.
• The grading is based on the best result of the end-term and re-exam

### LECTURER

Prof. Dr. Matthias Hein

Office Hours: Mo, 16-18, Do, 16-18

Organization:
Syama Sundar Rangapuram
Office Hours: Mo, 15-16 and Tu: 13-14

### GENERAL INFORMATION

In a broader perspective machine learning tries to automatize the process of empirical sciences - namely extracting knowledge about natural phenomena from measured data with the goal to either understand better the underlying processes or to make good predictions. Machine learning methods are therefore widely used in different fields: bioinformatics, computer vision, information retrieval, computer linguistics, robotics,...

The lecture gives a broad introduction into machine learning methods. After the lecture the students should be able to solve and analyze learning problems.

List of topics (tentative)

• Reminder of probability theory
• Maximum Likelihood/Maximum A Posteriori Estimators
• Bayesian decision theory
• Linear classification and regression
• Kernel methods
• Model selection and evaluation of learning methods
• Feature selection
• Nonparametric methods
• Boosting, Decision trees
• Neural networks
• Structured Output
• Semi-supervised learning
• Unsupervised learning (Clustering, Independent Component Analysis)
• Dimensionality Reduction and Manifold Learning
• Statistical learning theory

Previous knowledge of machine learning is not required. The participants should be familiar with linear algebra, analysis and probability theory on the level of the local `Mathematics for Computer Scienticists I-III' lectures. In particular, attendees should be familiar with

• Discrete and continuous probability theory (marginals, conditional probability, random variables, expectation etc.)
The first three chapters of: L. Wasserman: All of Statistics, Springer, (2004) provide the necessary background
• Linear algebra (rank, linear systems, eigenvalues, eigenvectors (in particular for symmetric matrices), singular values, determinant)
A quick reminder of the basic ideas of linear algebra can be found in the tutorial of Mark Schmidt (I did not check it for correctness!). Apart from the LU factorization this summarizes all what is used in the lecture in a non-formal way.
• Multivariate analysis (integrals, gradient, Hessian, extrema of multivariate functions)

Type: Core lecture (Stammvorlesung), 9 credit points

### LITERATURE AND OTHER RESOURCES

The lecture will be partially based on the following books and partially on recent research papers:

• R.O. Duda, P.E. Hart, and D.G.Stork: Pattern Classification, Wiley, (2000).
• B. Schoelkopf and A. J. Smola: Learning with Kernels, MIT Press, (2002).
• J. Shawe-Taylor and N. Christianini: Kernel Methods for Pattern Analysis, Cambridge University Press, (2004).
• C. M. Bishop: Pattern recognition and Machine Learning, Springer, (2006).
• T. Hastie, R. Tibshirani, J. Friedman: The Elements of Statistical Learning, Springer, second edition, (2008).
• L. Devroye, L. Gyoerfi, G. Lugosi: A Probabilistic Theory of Pattern Recognition, Springer, (1996).
• L. Wasserman: All of Statistics, Springer, (2004).
• S. Boyd and L. Vandenberghe: Convex Optimization, Cambridge University Press, (2004).

Other resources:

• Matlab is accessible via our campus license. Details how to use it can be found here
Access from outside should be possible via ssh: ssh -X username@computername.studcs.uni-sb.de
• Material for Matlab:

### NEWS

Update: The lecture notes/slides of the statistical learning theory part have been updated. The lecture notes contain now the proof of the symmetrization lemma and the VC bound.

Office hours (Syama Sundar Rangapuram): Monday 15-16 and Tuesday 13-14

Google Group for the Lecture: We have set up a google group for the lecture. The idea is that discussions and comments/corrections are more quickly spread to all of you. You need to subscribe to the group to post or view messages. You can subscribe from any email account; if you are not using google account to subscribe, send a mail to subscribe and then give a blank reply to the "join request" mail you would receive (do not click on "join this group" button in that mail!). Members can post messages here: post

You are encouraged to submit in groups of up to three students where all students have to belong to the same group.

Appendix A.2. in the lecture notes added where the mixed case is discussed.

Group Assignment: can be found here .

The "test"' excercise sheet comes with a solution - try to solve it without. In the tutorials next week the solution and related problems will be discussed.