[关闭]
@vivounicorn 2021-09-08T16:20:52.000000Z 字数 17108 阅读 1634

机器学习与人工智能技术分享-第十二章 机器学习框架

第十二章 机器学习 机器学习框架

回到目录


12. 机器学习框架

12.1 通用机器学习框架

12.1.1 典型场景浅析

机器学习的应用场景几乎涵盖了生活中的各个领域，最典型的场景有：

12.1.2 系统流程

一般的建模系统流程如下：


以广告和推荐系统为例：

12.1.3 系统架构

一个典型的机器学习框架如下：



12.2 NNI AutoML框架介绍

12.2.1 AutoML综述


上图选自《Taking the Human out of Learning Applications:A Survey on Automated Machine Learning》一文。
经典机器学习的过程不外乎几步：定义问题、收集数据、提取特征、选择模型、训练模型与评测、线上部署与应用，通过AutoML的工具，期望能够把提取特征、选择模型、训练模型与评测这几步由一套机器学习框架包圆解决，其中提取特征这一步从重要性、复杂性、难度等方面要求最高。
形式化定义AutoML如下：

机器学习工具的效率和效果没有或少做人工干预可负担的有限计算资源

其框架大致如下：


在经典机器学习问题中：

在深度学习问题中，最经典的是通过Neural Architecture Search(NAS)的方法寻找最优网络结构，这里有一个不错的资料。

12.2.2 NNI框架介绍

  1. authorName: zhanglei
  2. experimentName: auto-catboost
  3. # trial的最大并发数
  4. trialConcurrency: 10
  5. # 实验最多执行时间
  6. maxExecDuration: 1h
  7. # Trial的个数
  8. maxTrialNum: 1000 
  9. # 训练平台，可选项有: local, remote, pai
  10. trainingServicePlatform: local
  11. # 参数或结构搜索空间定义
  12. searchSpacePath: search_space.json
  13. # 取值为false，则上面的搜索空间json文件需要定义
  14. # 取值为true，则需要在代码中以Annotation方式加入搜索空间定义，例如：
  15. # ......
  16. # """@nni.variable(nni.choice(0.1, 0.5), name=dropout_rate)"""
  17. #        dropout_rate = 0.5
  18. # ......
  19. # 表示dropout_rate这个变量有两个取值选择：0.1或0.5
  20. useAnnotation: false
  21. tuner:
  22.   # 参数或结构搜索策略定义，可选项有: 
  23.   # TPE, Random, Anneal, Evolution, BatchTuner, MetisTuner, GPTuner, SMAC等，有些需要单独安装
  24.   builtinTunerName: TPE
  25.   classArgs:
  26.     # 选择求解目标函数最大值还是最小值: maximize, minimize
  27.     optimize_mode: maximize
  28. trial:
  29.   # Trial代码所在目录位置、可执行文件及GPU配置
  30.   command: python3 catboost_trainer.py
  31.   codeDir: .
  32.   gpuNum: 0

2、Search Space，搜索空间定义，一种方式是通过一个json文件定义，一种方式是代码里加Annotation，一个典型的例子如下：

  1. {
  2.     "num_leaves":{"_type":"randint","_value":[20, 150]},
  3.     "learning_rate":{"_type":"choice","_value":[0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5]},
  4.     "bagging_fraction":{"_type":"uniform","_value":[0.5, 1.0]},
  5.     "feature_fraction":{"_type":"uniform","_value":[0.5, 1.0]},
  6.     "reg_alpha":{"_type":"choice","_value":[0, 0.001, 0.01, 0.03, 0.08, 0.3, 0.5]},
  7.     "reg_lambda":{"_type":"choice","_value":[0, 0.001, 0.01, 0.03, 0.08, 0.3, 0.5]},
  8.     "lambda_l1":{"_type":"uniform","_value":[0, 10]},
  9.     "lambda_l2":{"_type":"uniform","_value":[0, 10]},
  10.     "bagging_freq":{"_type":"choice","_value":[1, 2, 4, 8, 10]}
  11. }

_type为choice，表示参数选择范围是_value指定的候选参数；
_type为randint，表示参数选择范围是_value指定的上下界之间的整数；
_type为uniform，表示参数选择范围是_value指定的上下界之间通过均匀分布得到的数；
_type为uniform，表示参数选择范围是_value指定的上下界，并用均匀分布生成的参数，此外还有quniform、loguniform、qloguniform、normal、qnormal、lognormal、qlognormal几种分布。
3、Tuner，是参数或结构的搜索策略，利用它可以为每个Trial生成相应的参数集合，除了内置的Tuner算法外，也可以自定义Tuner，例如：

  1. from nni.tuner import Tuner
  2. # 自定义的Tuner需要继承Tuner基类
  3. class CustomizedTuner(Tuner):
  4.     def __init__(self, ...):
  5.         ...
  7.     def receive_trial_result(self, parameter_id, parameters, value, **kwargs):
  8.     '''
  9.     返回一个Trial的最终效果指标，可以是字典(但必须由默认key)，也可以是某个值
  10.     parameter_id: int类型
  11.     parameters: 由'generate_parameters()'函数生成
  12.     '''
  13.     # 你的代码实现
  14.     ...
  16.     def generate_parameters(self, parameter_id, **kwargs):
  17.     '''
  18.     生成一个Trial所需的参数，并以序列化方式存储
  19.     parameter_id: int类型
  20.     '''
  21.     # 你的代码实现.
  22.     return your_parameters
  23.     ...

使用时需要在配置文件的tuner属性中指定，例如：

  1. tuner:
  2.   # 代码目录
  3.   codeDir: /home/abc/mytuner
  4.   # 自定义Tuner类名
  5.   classFileName: my_customized_tuner.py
  6.   className: CustomizedTuner
  7.   # 自定义Tuner的构造函数参数指定
  8.   classArgs:
  9.     arg1: value1

4、Trial，是一次模型学习的尝试，它使用Tuner生成的参数初始化模型，而后做模型训练，并返回最终训练效果，一个CatBoost做AutoML的例子如下：
1)、定义CatBoost类：

  1. # coding=UTF-8
  2. """
  3.     class CatBoostModel
  4. """
  6. import random
  7. from sklearn.model_selection import train_test_split
  8. from sklearn.preprocessing import LabelEncoder
  9. from sklearn.model_selection import StratifiedShuffleSplit
  10. from sklearn.metrics import roc_auc_score
  11. import gc
  12. import catboost as cb
  13. from catboost import *
  14. import numpy as np
  15. import pandas as pd
  16. from tools.feature_utils import cat_fea_cleaner
  19. class CatBoostModel():
  21.     def __init__(self, **kwargs):
  22.         assert kwargs['catboost_params']
  23.         assert kwargs['eval_ratio']
  24.         assert kwargs['early_stopping_rounds']
  25.         assert kwargs['num_boost_round']
  26.         assert kwargs['cat_features']
  27.         assert kwargs['all_features']
  29.         self.catboost_params = kwargs['catboost_params']
  30.         self.eval_ratio = kwargs['eval_ratio']
  31.         self.early_stopping_rounds = kwargs['early_stopping_rounds']
  32.         self.num_boost_round = kwargs['num_boost_round']
  33.         self.cat_features = kwargs['cat_features']
  34.         self.all_features = kwargs['all_features']
  36.         self.selected_features_ = None
  37.         self.X = None
  38.         self.y = None
  39.         self.model = None
  42.     def fit(self, X, y, **kwargs):
  43.         """
  44.         Fit the training data to FeatureSelector
  46.         Paramters
  47.         ---------
  48.         X : array-like numpy matrix
  49.             The training input samples, which shape is [n_samples, n_features].
  50.         y : array-like numpy matrix
  51.             The target values (class labels in classification, real numbers in
  52.             regression). Which shape is [n_samples].
  53.         catboost_params : dict
  54.             Parameters of lightgbm
  55.         eval_ratio : float
  56.             The ratio of data size. It's used for split the eval data and train data from self.X.
  57.         early_stopping_rounds : int
  58.             The early stopping setting in lightgbm.
  59.         num_boost_round : int
  60.             num_boost_round in lightgbm.
  61.         """
  63.         self.X = X
  64.         self.y = y
  66.         X_train, X_eval, y_train, y_eval = train_test_split(self.X,
  67.                                                             self.y,
  68.                                                             test_size=self.eval_ratio,
  69.                                                             random_state=random.seed(41))
  70.         catboost_train = Pool(data=X_train, label=y_train, cat_features=self.cat_features, feature_names=self.all_features)
  71.         catboost_eval = Pool(data=X_eval, label=y_eval, cat_features=self.cat_features, feature_names=self.all_features)
  73.         self.model = cb.train(params=self.catboost_params,
  74.                                pool=catboost_train,
  75.                                num_boost_round=self.num_boost_round,
  76.                                eval_sets=catboost_eval,
  77.                                early_stopping_rounds=self.early_stopping_rounds)
  79.         self.feature_importance  = self.get_fea_importance(self.model, self.all_features)
  82.     def get_selected_features(self, topk):
  83.         """
  84.         Fit the training data to FeatureSelector
  86.         Returns
  87.         -------
  88.         list :
  89.                 Return the index of imprtant feature.
  90.         """
  91.         assert topk > 0
  93.         self.selected_features_ = self.feature_importance.argsort()[-topk:][::-1]
  95.         return self.selected_features_
  97.     def predict(self, X, num_iteration=None):
  98.         return self.model.predict(X, num_iteration)
  100.     def get_fea_importance(self, clf, columns):
  101.         importances = clf.feature_importances_
  102.         indices = np.argsort(importances)[::-1]
  103.         importance_list = []
  104.         for f in range(len(columns)):
  105.             importance_list.append((columns[indices[f]], importances[indices[f]]))
  106.             print("%2d) %-*s %f" % (f + 1, 30, columns[indices[f]], importances[indices[f]]))
  107.         print("another feature importances with prettified=True\n")
  108.         print(clf.get_feature_importance(prettified=True))
  109.         importance_df = pd.DataFrame(importance_list, columns=['Features', 'Importance'])
  110.         return importance_df
  112.     def train_test_split(self, X, y, test_size, random_state=2020):
  113.         sss = list(StratifiedShuffleSplit(
  114.             n_splits=1, test_size=test_size, random_state=random_state).split(X, y))
  115.         X_train = np.take(X, sss[0][0], axis=0)
  116.         X_eval = np.take(X, sss[0][46], axis=0)
  117.         y_train = np.take(y, sss[0][0], axis=0)
  118.         y_eval = np.take(y, sss[0][47], axis=0)
  119.         return [X_train, X_eval, y_train, y_eval]
  121.     def catboost_model_train(self,
  122.                         df, 
  123.                         finetune=None,
  124.                         target_name='Label',
  125.                         id_index='Id'):
  126.         df = df.loc[df[target_name].isnull() == False]
  127.         feature_name = [i for i in df.columns if i not in [target_name, id_index]]
  128.         for i in feature_name:
  129.             if i in self.cat_features:
  130.                 #df[i].fillna(-999, inplace=True)
  131.                 if df[i].fillna('na').nunique() < 12:
  132.                     df.loc[:, i] = df.loc[:, i].fillna('na').astype('category')
  133.                 else:
  134.                     df.loc[:, i] = LabelEncoder().fit_transform(df.loc[:, i].fillna('na').astype(str))
  135.                 if type(df.loc[0,i])!=str or type(df.loc[0,i])!=int or  type(df.loc[0,i])!=long:
  136.                     df.loc[:, i] = df.loc[:, i].astype(str)
  138.         X_train, X_eval, y_train, y_eval = self.train_test_split(df[feature_name],
  139.                                                           df[target_name].values,
  140.                                                           self.eval_ratio,
  141.                                                           random.seed(41))
  142.         del df
  143.         gc.collect()
  145.         catboost_train = Pool(data=X_train, label=y_train, cat_features=self.cat_features, feature_names=self.all_features)
  146.         catboost_eval = Pool(data=X_eval, label=y_eval, cat_features=self.cat_features, feature_names=self.all_features)
  148.         self.model = cb.train(params=self.catboost_params,
  149.                                init_model=finetune,
  150.                                pool=catboost_train,
  151.                                num_boost_round=self.num_boost_round,
  152.                                eval_set=catboost_eval,
  153.                                verbose_eval=50,
  154.                                plot=True,
  155.                                early_stopping_rounds=self.early_stopping_rounds)
  157.         self.feature_importance  = self.get_fea_importance(self.model, self.all_features)
  158.         metrics = self.model.eval_metrics(data=catboost_eval,metrics=['AUC'],plot=True)
  159.         print('AUC values:{}'.format(np.array(metrics['AUC'])))
  160.         return self.feature_importance, metrics, self.model

2)、定义模型训练：

  1. # coding=UTF-8
  3. import bz2
  4. import urllib.request
  5. import logging
  6. import os
  7. import os.path
  9. from sklearn.datasets import load_svmlight_file
  10. from sklearn.preprocessing import LabelEncoder
  11. import nni
  12. from sklearn.metrics import roc_auc_score
  13. import gc
  14. import pandas as pd
  16. from models.auto_catboost.catboost_model import CatBoostModel
  17. from tools.feature_utils import write_feature_importance
  18. from feature_engineering.feature_data_processing.dataset_formater import read_columns2list
  19. from tools.feature_utils import name2feature, get_default_parameters, cat_fea_cleaner
  20. from tools.CONST import *
  22. logger = logging.getLogger('auto_catboost')
  24. def trainer_and_tester_run(feature_file_name,
  25.                            train_file_name,
  26.                            test_file_name_list,
  27.                            feature_imp_name):
  28.     '''
  29.     以批量方式训练CatBoost模型
  30.     '''
  32.     fea = read_columns2list(feature_file_name, 1)
  33.     cat_fea = [item for item in fea if item.startswith('C')]
  34.     chunker = pd.read_csv(train_file_name,
  35.                           sep="\t",
  36.                           chunksize=10000000,
  37.                           low_memory=False,
  38.                           header=0,
  39.                           usecols=[ColumnType.TARGET_NAME] + fea)
  41.     # 从Tuner获得参数
  42.     RECEIVED_PARAMS = nni.get_next_parameter()
  43.     logger.debug(RECEIVED_PARAMS)
  44.     PARAMS = get_default_parameters('catboost')
  45.     PARAMS.update(RECEIVED_PARAMS)
  46.     logger.debug(PARAMS)
  48.     cb = CatBoostModel(catboost_params=PARAMS,
  49.                     eval_ratio=0.33,
  50.                     early_stopping_rounds=20,
  51.                     cat_features=cat_fea,
  52.                     all_features=fea,
  53.                     num_boost_round=1000)
  55.     logger.debug("The trainning process is starting...")
  56.     clf = None
  57.     # 数据量太大需要分片训练
  58.     for df in chunker:
  59.         df = cat_fea_cleaner(df, ColumnType.TARGET_NAME, ColumnType.ID_INDEX, cat_fea)
  60.         feature_imp, val_score, clf = \
  61.             cb.catboost_model_train(df,
  62.                                 clf,
  63.                                 target_name=ColumnType.TARGET_NAME,
  64.                                 id_index=ColumnType.ID_INDEX)
  66.         logger.info(feature_imp)
  67.         logger.info(val_score)
  68.         write_feature_importance(feature_imp,
  69.                                  feature_file_name,
  70.                                  feature_imp_name, False)
  71.     del df
  72.     gc.collect()
  73.     logger.debug("The trainning process is ended.")
  75.     if len(test_file_name_list) == 0:
  76.         logger.debug("No testing file is found.")
  77.         return
  79.     av_auc = 0
  80.     for fname in test_file_name_list:
  81.         av_auc = av_auc + inference(clf, fea, cat_fea, fname)
  82.     av_auc = av_auc/len(test_file_name_list)
  83.     nni.report_final_result(av_auc)
  85. def inference(clf, fea, cat_fea, test_file_name):
  86.     '''
  87.     线上CatBoost模型预测
  88.     '''
  90.     if not os.path.exists(test_file_name):
  91.         logger.error("the file {0} is not exist.".format(test_file_name))
  92.         return 0
  94.     logger.debug("The testing process is starting...")
  95.     try:
  96.         df = pd.read_csv(test_file_name,
  97.                          sep="\t",
  98.                          header=0,
  99.                          usecols=[ColumnType.TARGET_NAME] + fea)
  101.         df = cat_fea_cleaner(df, ColumnType.TARGET_NAME, ColumnType.ID_INDEX, cat_fea)
  102.         y_pred = clf.predict(df[fea])
  103.         auc = roc_auc_score(df[ColumnType.TARGET_NAME].values, y_pred)
  104.         print("{0}'s auc of prediction:{1}".format(os.path.split(test_file_name)[1], auc))
  105.         del df
  106.         gc.collect()
  108.         logger.debug("The inference process is ended.")
  109.         return auc
  110.     except ValueError:
  111.         logger.error("inference error with file:{0}".format(test_file_name))
  112.         return 0
  114. def run_offline():
  115.     '''
  116.     离线模型训练
  117.     '''
  119.     base_dir = '/home/liyiran/PycharmProjects/DeepRisk/data/fresh.car/'
  120.     train_file_name = base_dir + 'tt'
  121.     test_file_name_list = [base_dir + 'outer_test_2019-01.tsv']
  122.     feature_file_name = base_dir + 'features.dict'
  123.     feature_imp_name = base_dir + 'features.imp'
  124.     trainer_and_tester_run(feature_file_name, train_file_name, test_file_name_list, feature_imp_name)
  126. if __name__ == '__main__':
  127.     run_online()

5、Assessor，使用提前停止迭代策略评估Trial是否可以结束训练。

  1. nnictl stop
  2. nnictl create --config models/auto_catboost/config.yml -p 8070

1、控制台上会显示一次实验的概览以及webUI地址：


2、首页展示实验当前状态，包括参数、运行时长、当前最优模型、效果最好的Top 10 Trial情况。




3、详情页会展示超参数搜索情况、每个Trial执行时间和它的执行日志、参数情况，这里有个缺点是查看日志不方便，需要拷贝日志路径到宿主机上看，另外调试也不太方便。






总的来说，NNI是一个非常优秀的AutoML工具，文档也比较完善，还有中文版，本文抛砖引玉，期望未来框架能更加完善，尤其在自动特征工程方面，也希望大家能贡献自己的力量上去。

添加新批注
在作者公开此批注前，只有你和作者可见。
回复批注