Structured Pruning in Deep Neural Networks with Trainable Probability Masks

F. Martinini; A. Enttsel; A. Marchioni; M. Mangia; R. Rovatti; G. Setti

doi:10.1109/MWSCAS57524.2023.10405945

Structured Pruning in Deep Neural Networks with Trainable Probability Masks

F. Martinini, A. Enttsel, A. Marchioni, M. Mangia, R. Rovatti, G. Setti

Electrical and Computer Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

The current trend of over-parameterized Deep Neural Networks makes the deployment on resource constrained systems challenging. To deal with this, optimization techniques, such as network pruning, can be adopted. We propose a novel pruning technique based on trainable probability masks that, when binarized, select the elements of the network to prune. Our method features i) an automatic selections of the elements to prune by jointly training the binary masks with the model, ii) the capability of controlling the pruning level through hyper-parameters of a novel regularization term. We assess the effectiveness of our method by employing it in the structured pruning of the fully connected layers of shallow and deep neural networks where it outperforms the magnitude-based pruning approaches.

Original language	English (US)
Title of host publication	2023 IEEE 66th International Midwest Symposium on Circuits and Systems, MWSCAS 2023
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1020-1024
Number of pages	5
ISBN (Electronic)	9798350302103
DOIs	https://doi.org/10.1109/MWSCAS57524.2023.10405945
State	Published - 2023
Event	2023 IEEE 66th International Midwest Symposium on Circuits and Systems, MWSCAS 2023 - Tempe, United States Duration: Aug 6 2023 → Aug 9 2023

Publication series

Name	Midwest Symposium on Circuits and Systems
ISSN (Print)	1548-3746

Conference

Conference	2023 IEEE 66th International Midwest Symposium on Circuits and Systems, MWSCAS 2023
Country/Territory	United States
City	Tempe
Period	08/6/23 → 08/9/23

Bibliographical note

Publisher Copyright:
© 2023 IEEE.

Keywords

Deep Neural Network
Network Pruning
Probability Mask
Trainable Binary Mask

ASJC Scopus subject areas

Electronic, Optical and Magnetic Materials
Electrical and Electronic Engineering

Access to Document

10.1109/MWSCAS57524.2023.10405945

Cite this

Martinini, F., Enttsel, A., Marchioni, A., Mangia, M., Rovatti, R., & Setti, G. (2023). Structured Pruning in Deep Neural Networks with Trainable Probability Masks. In 2023 IEEE 66th International Midwest Symposium on Circuits and Systems, MWSCAS 2023 (pp. 1020-1024). (Midwest Symposium on Circuits and Systems). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/MWSCAS57524.2023.10405945

@inproceedings{7751b05064ea4e969e6b1be181c33c00,

title = "Structured Pruning in Deep Neural Networks with Trainable Probability Masks",

abstract = "The current trend of over-parameterized Deep Neural Networks makes the deployment on resource constrained systems challenging. To deal with this, optimization techniques, such as network pruning, can be adopted. We propose a novel pruning technique based on trainable probability masks that, when binarized, select the elements of the network to prune. Our method features i) an automatic selections of the elements to prune by jointly training the binary masks with the model, ii) the capability of controlling the pruning level through hyper-parameters of a novel regularization term. We assess the effectiveness of our method by employing it in the structured pruning of the fully connected layers of shallow and deep neural networks where it outperforms the magnitude-based pruning approaches.",

keywords = "Deep Neural Network, Network Pruning, Probability Mask, Trainable Binary Mask",

author = "F. Martinini and A. Enttsel and A. Marchioni and M. Mangia and R. Rovatti and G. Setti",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 2023 IEEE 66th International Midwest Symposium on Circuits and Systems, MWSCAS 2023 ; Conference date: 06-08-2023 Through 09-08-2023",

year = "2023",

doi = "10.1109/MWSCAS57524.2023.10405945",

language = "English (US)",

series = "Midwest Symposium on Circuits and Systems",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1020--1024",

booktitle = "2023 IEEE 66th International Midwest Symposium on Circuits and Systems, MWSCAS 2023",

address = "United States",

}

Martinini, F, Enttsel, A, Marchioni, A, Mangia, M, Rovatti, R & Setti, G 2023, Structured Pruning in Deep Neural Networks with Trainable Probability Masks. in 2023 IEEE 66th International Midwest Symposium on Circuits and Systems, MWSCAS 2023. Midwest Symposium on Circuits and Systems, Institute of Electrical and Electronics Engineers Inc., pp. 1020-1024, 2023 IEEE 66th International Midwest Symposium on Circuits and Systems, MWSCAS 2023, Tempe, United States, 08/6/23. https://doi.org/10.1109/MWSCAS57524.2023.10405945

Structured Pruning in Deep Neural Networks with Trainable Probability Masks. / Martinini, F.; Enttsel, A.; Marchioni, A. et al.
2023 IEEE 66th International Midwest Symposium on Circuits and Systems, MWSCAS 2023. Institute of Electrical and Electronics Engineers Inc., 2023. p. 1020-1024 (Midwest Symposium on Circuits and Systems).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Structured Pruning in Deep Neural Networks with Trainable Probability Masks

AU - Martinini, F.

AU - Enttsel, A.

AU - Marchioni, A.

AU - Mangia, M.

AU - Rovatti, R.

AU - Setti, G.

PY - 2023

Y1 - 2023

N2 - The current trend of over-parameterized Deep Neural Networks makes the deployment on resource constrained systems challenging. To deal with this, optimization techniques, such as network pruning, can be adopted. We propose a novel pruning technique based on trainable probability masks that, when binarized, select the elements of the network to prune. Our method features i) an automatic selections of the elements to prune by jointly training the binary masks with the model, ii) the capability of controlling the pruning level through hyper-parameters of a novel regularization term. We assess the effectiveness of our method by employing it in the structured pruning of the fully connected layers of shallow and deep neural networks where it outperforms the magnitude-based pruning approaches.

AB - The current trend of over-parameterized Deep Neural Networks makes the deployment on resource constrained systems challenging. To deal with this, optimization techniques, such as network pruning, can be adopted. We propose a novel pruning technique based on trainable probability masks that, when binarized, select the elements of the network to prune. Our method features i) an automatic selections of the elements to prune by jointly training the binary masks with the model, ii) the capability of controlling the pruning level through hyper-parameters of a novel regularization term. We assess the effectiveness of our method by employing it in the structured pruning of the fully connected layers of shallow and deep neural networks where it outperforms the magnitude-based pruning approaches.

KW - Deep Neural Network

KW - Network Pruning

KW - Probability Mask

KW - Trainable Binary Mask

UR - http://www.scopus.com/inward/record.url?scp=85185385459&partnerID=8YFLogxK

U2 - 10.1109/MWSCAS57524.2023.10405945

DO - 10.1109/MWSCAS57524.2023.10405945

M3 - Conference contribution

AN - SCOPUS:85185385459

T3 - Midwest Symposium on Circuits and Systems

SP - 1020

EP - 1024

BT - 2023 IEEE 66th International Midwest Symposium on Circuits and Systems, MWSCAS 2023

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2023 IEEE 66th International Midwest Symposium on Circuits and Systems, MWSCAS 2023

Y2 - 6 August 2023 through 9 August 2023

ER -

Martinini F, Enttsel A, Marchioni A, Mangia M, Rovatti R, Setti G. Structured Pruning in Deep Neural Networks with Trainable Probability Masks. In 2023 IEEE 66th International Midwest Symposium on Circuits and Systems, MWSCAS 2023. Institute of Electrical and Electronics Engineers Inc. 2023. p. 1020-1024. (Midwest Symposium on Circuits and Systems). doi: 10.1109/MWSCAS57524.2023.10405945

Structured Pruning in Deep Neural Networks with Trainable Probability Masks

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this