Linear Regression and Stochastic Gradient Descent

机器学习 实验

你可以点击这里查看中文版本。

Motivation of Experiment

1. Further understand of linear regression ，closed-form solution and Stochastic gradient descent.
2. Conduct some experiments under small scale dataset.
3. Realize the process of optimization and adjusting parameters.

Dataset

Linear Regression uses Housing in LIBSVM Data, including 506 samples and each sample has 13 features. You are expected to download scaled edition. After downloading, you are supposed to divide it into training set, validation set.

Environment for Experiment

python3, at least including following python package: sklearn，numpy，jupyter，matplotlib
It is recommended to install anaconda3 directly, which has built-in python package above.

Submit Deadline

2018-10-28 12:00 AM

Experiment Form

Complete Independently.

Experiment Step

*closed-form solution of Linear Regression *

1. Load the experiment data. You can use load_svmlight_file function in sklearn library.
2. Devide dataset. You should divide dataset into training set and validation set using train_test_split function. Test set is not required in this experiment.
3. Initialize linear model parameters. You can choose to set all parameter into zero, initialize it randomly or with normal distribution.
4. Select a Loss function and calculate the value of the Loss function of the training set,denoted as $loss$.
5. Get the formula of the closed-form solution, the process details the courseware ppt.
6. Get the value of parameter $W$ by the closed-form solution, and update the parameter $W$.
7. Get the $Loss$ ，$loss\_train$ under the training set and $loss\_val$ by validating under validation set.
8. Output the value of $Loss$,$loss\_train$ and $loss\_val$.

Linear Regression and Stochastic Gradient Descent

1. Load the experiment data. You can use load_svmlight_file function in sklearn library.
2. Devide dataset. You should divide dataset into training set and validation set using train_test_split function. Test set is not required in this experiment.
3. Initialize linear model parameters. You can choose to set all parameter into zero, initialize it randomly or with normal distribution.
4. Choose loss function and derivation: Find more detail in PPT.
5. Calculate $G$ toward loss function from each sample.
6. Denote the opposite direction of gradient $G$ as $D$.
7. Update model: $W_t=W_{t-1}+\eta D$. $\eta$ is learning rate, a hyper-parameter that we can adjust.
8. Get the loss $loss\_train$ under the training set and $loss\_val$ by validating under validation set.
9. Repeate step 5 to 8 for several times, andand output the value of $loss\_train$ as well as $loss\_val$.

Finishing experiment report according to result: The template of report can be found in example repository.

Evaluation

Requirement for Submission

Submission process

1.Access222.201.187.50:7001.
2.Click on the corresponding submission entry.
3.Fill in your name, student number, upload pdf format report and zip format code compression package.

Precautions

• Experiment reports and code can be uploaded multiple times, and multiple uploads will overwrite previously submitted files.
• After uploading, you can refresh the page and check if the upload is successful in the file list below.
• Teaching assistants save all uploaded results at the experimental deadline, and the files uploaded after the deadline are invalid.
• If you write an experiment report in Word, you need to export it to pdf format.
• The package format of the code file must be zip. Please do not submit the compressed file in rar format.
• Submit URL can only be accessed by campus network.
• The code is written in python language, the experimental report score standard English is better than Chinese, latex is better than word.

Any advice or idea is welcome to discuss with teaching assistant in QQ group.