Paper: PURSUhInT: In Search of Informative Hint Points Based on Layer Clustering for Knowledge Distillation

Informatics Institute faculty member Behçet Uğur Töreyin and research assistant Reyhan Kevser Keser coauthored paper titled 'PURSUhInT: In Search of Informative Hint Points Based on Layer Clustering for Knowledge Distillation' has been published in 'Expert Systems with Applications' on 1st March 2023.

DOI: https://doi.org/10.1016/j.eswa.2022.119040

Abstract: One of the most efficient methods for model compression is hint distillation, where the student model is injected with information (hints) from several different layers of the teacher model. Although the selection of hint points can drastically alter the compression performance, conventional distillation approaches overlook this fact and use the same hint points as in the early studies. Therefore, we propose a clustering based hint selection methodology, where the layers of teacher model are clustered with respect to several metrics and the cluster centers are used as the hint points. Our method is applicable for any student network, once it is applied on a chosen teacher network. The proposed approach is validated in CIFAR-100 and ImageNet datasets, using various teacher–student pairs and numerous hint distillation methods. Our results show that hint points selected by our algorithm results in superior compression performance compared to state-of-the-art knowledge distillation algorithms on the same student models and datasets.

Date 16 Mar 2024

ITU Informatics Institute provides graduate-level education and research in applied informatics, computer sciences, computational science and engineering, communication systems under the following programs.

Information and Communications Engineering Master of Science and Doctorate Program
Cybersecurity Engineering and Cryptography Master of Science and Doctorate Program
Geographical Information Technologies Master of Science and Doctorate Program
Information Technologies in Construction Management Professional Degree Master Program
Information Technologies Professional Degree Master Program
Computational Science and Engineering Master of Science and Doctorate Program
Computational Science and Engineering Master of Science and Doctorate Program
Satellite Communication and Remote Sensing Master of Science and Doctorate Program

Faculty members and students conduct research supported by national and international organızatıons in the fields of electromagnetic fields, communication systems/regulations, computational materials design, computational chemistry/biology, cryptography, signal/data processing/visualization, big data management, climate and ocean sciences,

List of Most Influential Scientists; Associate Professor. B. Uğur Töreyin (article by Dr. John PA Ioannidis, K. W. Boyack and J. Baas published in the journal PLOS Biology)
Beltus Nkwawir Wiysobunri, the best project award in the Science category, in the 2020 International Students Project Competition
Argenit company, of which Dr. Abdulkerim Çapar is among the founding partners, received the "National-International Supports" First Prize of ITU ARI Teknokent.
TÜBİTAK 2242 University Students Project Competition in Priority Areas: Istanbul region first place - Ahmet Burak Özyurt
Best Presentation Award at ICAT'18 Conference: Sena Efsun Cebeci, 2018
Tubitak Incentive Award; 2016 Assoc. Prof. Adem Tekin
“Technical Paper” and “Willis H. Carrier” Award by the American Heating, Cooling and Air Conditioning Association; 2016 Assist. Prof. Dr. H. Salih Erden
Science Heroes Association Young Scientist of the Year Award; 2016 Assoc. B. Uğur Töreyin Best Poster Award at PRACEdays 2016 conference; Samet Demir
ITU Most Successful Thesis Award; 2016 Hatice Gokcan

There is also a High Performance Computing Laboratory established with the support of the State Planning Organization within the Institute.

Programs

Candidate Students

Current Students

Web Page is Refreshing!

İTÜ Informatics Institute