Supervisor: Oleg Bakhteev, PhD
| Link | |
|---|---|
| Paper | link |
| Thesis | link |
| Code | link |
| Slides | link |
The paper investigates the problem of structural pruning in neural networks. Structural pruning is the procedure of removing entire groups of parameters from a neural network, for example, filters in CNNs or skip connections. An appropriate pruning strategy can improve both generalization performance and inference efficiency. The main difficulty of structural pruning is that removing a layer requires removing or consistently modifying all dependent layers, especially across arbitrary or novel architectures.
The proposed method is based on the analysis of the deep learning computation graph and the estimation of the information flow propagated through it. By preserving the local dependency structure of the loss function, the method enables the estimation of the importance of operations in the computation graph in a few-shot setting, i.e., using several forward passes of a data subset through the analyzed model. To demonstrate the effectiveness of the proposed method, we conduct multiple experiments on randomly generated neural networks using the MNIST dataset, showing that our approach outperforms existing pruning methods. Furthermore, our method achieves 60.4% accuracy after pruning 43.9% of the parameters of ResNet-18 on the CIFAR-10 classification task without fine-tuning.