Methods for learning deep neural networks with fewer parameters are urgently needed, as the large storage and computational requirements of heavy neural networks largely prevent their widespread use on mobile devices. Training lightweight networks using the model of a teacher network-student network learning framework is a more flexible approach than algorithms that directly remove weights or convolutional kernels to obtain relatively large compression and acceleration ratios. However, in practice, it is difficult to determine which metric to utilize to select useful information from the teacher network. To overcome this challenge, we propose to use a generative adversarial network to learn a lightweight student neural network; specifically, the generator network is a student neural network with very few weight parameters, and the discriminator network is used as a helper to distinguish between features generated by the student neural network and the teacher neural network. By simultaneously optimizing the generator network and the discriminator network, the student neural network generated in this paper can generate features with the same distribution as the teacher neural network features for the input data.
essays 3： Reinforced Multi-label Image Classification by Exploring Curriculum
Humans and animals learn organized knowledge more efficiently than cluttered knowledge. Based on the mechanism of curriculum learning, we propose a method to strengthen multi-label classification to simulate the process of human from easy to difficult to predict labeling. This method allows an intensive learning intelligence to make label predictions sequentially based on the characteristics of the image and the predicted labels. In turn, it obtains the optimal strategy by looking for a way to maximize the cumulative reward, thus making the classification of many label images the most accurate. Our experiments on the PASCAL VOC2007 and PASCAL VOC2012 datasets demonstrate the necessity and effectiveness of this enhanced multi-label image classification approach in a real multi-label task.
essays 4： Learning with Single-Teacher Multi-Student
This paper investigates how a single complex generic model can be used to learn a series of lightweight specialized models, namely the Single-Teacher Multi-Student (STM) problem. Taking classical multiclassification and biclassification as an example, this paper revolves around how a pre-trained multiclassification model can be used to derive multiple biclassification models, where each biclassification model corresponds to a different class. In real-world scenarios, many problems can be viewed in this context; for example, making fast and accurate judgments based on a generic face recognition system for a specific suspect. However, direct inference using multi-classification models for dichotomous operations is inefficient, and training a dichotomous classifier from scratch often results in poor classification performance. In this paper, a gated support vector machine (gated SVM) model is proposed by considering the multi-classifier as the teacher and the target's binary classifier as the student. Each biclassifier in this model can give its own prediction in combination with the inferred results of the multiclassifier; moreover, each student can obtain the sample complexity measure given by the teacher's model, making the training process more adaptive. In practical experiments, the proposed model has achieved good results.
essays 5： Sequence-to-Sequence Learning via Shared Latent Representation