Multimodal learning

Learning from more than one kind of input, such as video, language, audio or robot state.

The inputs can provide complementary evidence, but their relationships must be preserved. Synchronization, units, coordinate frames and missing streams affect how examples can be interpreted. A correct label on one stream does not establish that the complete example is consistent across all inputs.

In practice

A language instruction, camera recording and robot action sequence need to describe the same demonstration.