Ali Abusaleh

Multimodal signals, such as video and audio, are ordered chronologically. Processing these signals requires systems capable of performing incremental predictions and dynamic uncertainty assessments. This master thesis is about implementing and testing a framework that tracks events or states in real time, predicts them, interprets intermodal interactions, and provides confidence estimates. The proposed topic focuses on multimodal stream learning, wherein models update their internal representations continuously as new data arrives without access to the complete sequence. Unlike offline approaches, streaming systems process asynchronous, context-dependent signals where the same signals can have different meanings depending on accompanying visual or acoustic cues. For instance, the greeting "hello" (audio) accompanied by a smile (image) may imply friendliness, whereas the same greeting accompanied by a clenched fist (image) may imply a threat. This project has many potential applications, one of which is emotion recognition in video streams. Tasks include specifying a streaming-compatible application, acquiring relevant literature, implementing a basic system, and testing extensions such as online cross-modal fusion, temporal attention, and uncertainty-aware prediction layers.
Corresponding Lab Member: Ali Abusaleh and Alexander Mehler.

Accurately estimating sentiments in video data is a complex challenge that requires integrating visual, audio, and textual cues. This proposal involves developing a multimodal sentiment analysis pipeline that uses state-of-the-art models to extract embeddings from different modalities: Qwen2.5 for text, JEPA 2 for visuals, and WhisperX (pyannote) for audio. The embeddings then undergo projection into a shared latent space and fusion via cross-attention mechanisms to capture intermodal dependencies. A probabilistic output layer will then estimate sentiment distributions over fine-grained emotions. To accelerate the analysis process for real-time applications, the pipeline should use a stream-like vector projection method that updates latent space representations incrementally instead of reprocessing entire sequences. The pipeline will be built using DUUI to ensure modularity, scalability, and reproducibility. The goal of this thesis is to tackle limitations of unimodal sentiment analysis and traditional fusion methods to achieve efficient, accurate, and scalable multimodal sentiment analysis.
Corresponding Lab Member: Ali Abusaleh and Alexander Mehler.