AI-generated worlds look increasingly realistic, but they're still largely silent. WorldSonus is a video-to-audio system that adds stereo sound while the video is still streaming.
The swipe-through covers the paper's three challenges and approaches: chunked audio synthesis (real-time factor of 0.41), prompts scheduled by chunk index so sound events can change mid-stream, and stereo training data curated from stereo and ambisonic sources. The authors also say it matches or beats state-of-the-art bidirectional models on open-domain benchmarks.
Video: noizai.github.io/WorldSonus
Read the source: WorldSonus: Bringing Sound to Worlds