Objectives of the chair
Recent AI models have achieved remarkable success in specific domains (e.g. vision, language, robotic agent control), and there is a push towards ever larger models combining multiple input and output modalities.
Inctheory, multimodal representations can help vision scientists by endowing sensory inputs with semantic information;
similarly, linguists can use them to ground NLP tokens in the sensorimotor environment and create a form of referential meaning;
Roboticists can also take advantage of these versatile representations for navigation and action planning.
But in practice, current models rely on brute-force training approaches using billions of labelled examples, while the datasets and computing resources available to academic and industrial researchers are typically much smaller.
Compared to artificial neural networks, real brains learn much more efficiently. We thus take inspiration from the cognitive science idea of a Global Workspace (GW) to build a novel class of AI systems.
The GW, a unique model of multimodal grounding (encompassing perception, action and semantic representations), can promote advances in perceptual models and support both top-down interactions (from language and semantics to perception and action) of interest to linguists and bottom-up interactions (from active perception and navigation to semantic abstractions) of interest to roboticists
The high-risk/high-gain hypothesis is that the modalities complement one another synergistically, such that the whole system is much more efficient than the sum of its parts, not just for multimodal tasks but also when evaluated in the initial domains (vision, NLP, robotics).
Building frugal perceptual and cognitive models that can support language grounding and embodiment and provide semantic representations to robotic agents is expected to have important beneficial consequences for ANITI’s industrial partners (e.g. Airbus, Linagora).
Research objectives
- Brain-inspired, frugal multimodal systems based on GW theory (PI: VanRullen)
- Perception with and without a GW (PI: Serre)
- Top-down track: from language semantics to situated conversational agents via GW grounding (PI:Asher)
- Bottom-up track: from mobile robots to situated conversational agents via GW grounding (PI: Stasse)
Industrial applications
- TUM, ICS Technical University of Munich, Institute of Cognitive Sciences
- University of Colorado @ Boulder, University of London on Minecraft
- University of Potsdam on COCOBOT project
- Imperial College London (UK) and Araya Inc (Japan) on ERC Glow
- Brown University, George Karniadakis via T.Serre
- Airbus Defense and Space: satellite image super-resolution (Computer Vision team: Mathieu LeGoff)
- Labsoft
Main scientific goals
- improved multimodal systems (more frugal, robust, grounded, explainable…) and novel neuroscienceinspired learning architectures
- novel explainability methods (for e.g. computer vision)
- better models of semantics at the clausal and discourse level for NLP
- transition to Industry 4.0, i.e., from hard-coded robots to more collaborative ones
- cobots that can reliably accomplish complex tasks on an industrial shop floor
- robots that are able to understand simple orders in an industrial context
- cobots endowed with advanced conversational and interactional skills for many safety-critical tasks in various areas (airport traffic management, single pilot operations, elderly care)