Handling partial occlusions in facial expression recognition with variational autoencoder

Abstract

Facial expression recognition (FER) is essential in various domains such as healthcare, road safety, and marketing, where real-time emotional feedback is crucial. Despite advancements in controlled settings such as well-lit, frontal, and unobstructed conditions, FER still faces significant challenges in natural, unconstrained environments. One of the most difficult issues is the presence of occlusions, which obscure key facial features. To overcome this, multiple strategies have been proposed, generally falling into two categories: those focused on analyzing visible facial regions and those aimed at reconstructing hidden facial features. In this study, we present a variational autoencoder (VAE)-based solution designed to reconstruct facial features obscured by occlusions. Experimental results show our VAE model optimized with the structural similarity index measure (SSIM) cost function achieves superior performance, with recognition rates of 91.2% for eye occlusions and 89.7% for mouth occlusions. The SSIM-optimized VAE effectively reconstructs occlude facial features while preserving structural details, demonstrating significant improvements over conventional approaches. This VAE-based solution proves particularly robust for real-world scenarios involving common facial obstructions like masks or sunglasses, making it valuable for applications in healthcare monitoring, driver safety systems, and human-computer interaction.