Quality Alchemist (品質煉金術師): Exploring Knowledge Learning from Video Generative Modeling

2025年2月15日星期六

Exploring Knowledge Learning from Video Generative Modeling

The seminar named “Exploring Knowledge Learning from Video Generative Modeling” was co-organized by COMP, DSAI and IEEE Computational Intelligence Society on 14^th Feb 2025. Dr. Jiashi FENG (Head of Vision Research at ByteDance) was the guest speaker. His talk introduced recent efforts to leverage video generative modeling for learning from video data.

In the beginning, Dr. Jiashi FENG briefed existing AI models mostly learn knowledge from text and challenges of video data that needed an unsupervised learning approach.

Then he introduced video generative modeling that extracting knowledge might be achieved by compression.

And then he mentioned how to learn knowledge from video such as play the videos to AI model through solving tasks. One of example is to train Go game using video GoBench with Training set (10M 9x9 Go game video records) and Testing set (1000 matches).

After that Dr. FENG introduced Latent Dynamics Model (LDM) to improve video representation. LDM can extract multiple-step common patterns into the latent space that extracting knowledge from video.

The architecture of VideoWorld then discussed. VideoWorld explored learning knowledge from unlabeled videos and achieved promising performance for Go playing and simple robotic manipulation tasks.

Finally, he challenged the video generation model to investigate physical law learning such as classical mechanics.

The model can fill the gap through interpolation or extrapolation that were demonstrated.

Lastly, he concluded that video generative models demonstrate strong law extraction abilities for in-distribution data but struggle with out-of-distribution scenarios. Their performance is inconsistent in combinatoric settings, indicating a tendency to memorize rather than generalize.