Secure Federated Learning: model compression and Inverse Distillation

Breast cancer remains one of the leading causes of death in women, with late diagnosis significantly increasing health risks. This study will help to improve detection techniques for this disease, thanks to collaboration between hospitals via federated learning (FL), while guaranteeing data confidentiality using homomorphic encryption.We are
proposing an innovative and optimized architecture that enables models to be transmitted securely while reducing vulnerability to attacks. To achieve this, we exploit an innovative knowledge distillation approach, where a teaching neural network transfers its knowledge to a student.
Unlike traditional methods, our approach is based on a two-way exchange of knowledge, introducing an ‘inverse distillation’ mechanism: the student also contributes to refine the teacher model, enabling continuous mutual learning. This process optimizes the performance of the models while reducing their size, making them easier to encrypt and
transmit securely as part of Federated Learning process. The result is a compact neural network that performs as well as its teacher, with all layers protected by full homomorphic encryption. Our method improves the accuracy of models in terms of suspicion between malignant and benign tumours, while preserving the confidentiality of medical data by encrypting the model weights transmitted during federated learning. The
experimental results, obtained using mammography images from a Belgian hospital, show a significant improvement in detection accuracy while guaranteeing the security of hospital data. This work opens up promising prospects for training deep learning models in environments where data confidentiality is essential, enabling collaborative learning between institutions without compromising the security of their resources.