We Are Online Since 1998

The Multi-Modal Shift: Why Your Next Generative AI Course Must Include Video and 3D

hamzajaved
By hamzajaved
7 Min Read

Generative AI isn’t just about text any more. Today’s models can generate images, produce audio, create video, generate video sequences and even produce 3d assets from a natural language prompt. This is not a trend at all. This is a fundamental shift in the way AI understands and interacts with the world.

For the learner and the professional building up their skills in this area the implications are straightforward: a course that only covers large language models and text generation is already fundamentally flawed. The next generation of AI practitioners are going to need to be fluent in all the modalities and that includes video and 3d. That’s equally true of anyone doing agentic AI training where the agent has to be able to perceive and act across multiple modalities in order to complete a complex real world task.

Why Multi-Modal AI Matters Now

The move to multi-modal AI is being driven by both research progress and commercial demand. GPT-4o, Gemini and a number of other open source models are now capable of processing and generating across text, image, audio and video in a single, unified architecture. That is to say they can now understand a scene in a video, reason about spatial relationships in a 3D world and generate content across multiple modalities at the same time.

Industries are reacting pretty quickly. Film studios are using AI to generate concept art and pre-vis. Game developers are using 3D generative models to accelerate environment and character creation. Marketing teams are producing AI generated video content on a scale which was simply not possible before. Healthcare researchers are using 3D AI models to visualise molecular structures and medical imaging data.

If your AI education stops at text and static images you’re preparing for a version of the industry that doesn’t exist any more.

What Video Generation Requires You to Learn

Video generation is vastly more complex than image generation. A video generation model has to maintain temporal consistency so that the objects, the lighting and the motion all stay consistent from frame to frame. It has to have an understanding of physics, of scene composition and of camera movement that a static image model simply doesn’t need to have.

Key concepts a generative AI curriculum should cover for video include:

Diffusion based video models: Sora, Runway Gen-3 and Stable Video Diffusion all use diffusion models adapted for video. Understanding how they’re trained and conditioned is absolutely fundamental.

Video to video transformation: beyond the simple case of generating video from text, you need to know how to use a reference video as input and then use conditioning techniques to control the motion, style and content.

Latency and compute constraints: video generation is very resource intensive. Any practical curriculum needs to deal with optimisation techniques (model quantisation, resolution compromises, efficient inference pipelines etc).

These are also very much issues for agentic AI training, as autonomous agents are increasingly going to be required to process video as part of their perception stack (i.e. looking at recorded video, looking at live camera feeds, generating video outputs for downstream tasks etc).

Why 3D Generation Deserves a Dedicated Module

3d asset generation is one of the most dynamic areas of generative AI and has real economic significance. The global 3D content market is huge, covering gaming, architecture, industrial design, AR/VR and simulation. In all of those areas AI generated 3D assets have the potential to massively reduce time to production and cost.

Modern 3D generative AI uses several distinct technical approaches:

Neural Radiance Fields (Nerfs) work by learning to represent a 3D scene from a collection of 2d images. They do this by learning to model the way light interacts with the 3d geometry of the scene. Understanding Nerfs is a useful way of getting a handle on how AI can go from a set of 2D images to reconstructing a 3d scene.

Gaussian Splatting (a faster alternative to NeRF) represents the scene as a collection of 3D Gaussians and has become the go-to method for real time rendering.

Text to 3d pipelines: Shap-E, Point-E and the newer diffusion based 3D models can generate mesh or point cloud assets directly from text. That’s now the standard in a rapid prototyping environment.

A generative AI course that doesn’t cover 3D at all leaves the learner completely unprepared for entire industry verticals where that skillset is now a prerequisite.

Structuring a Complete Multi-Modal Curriculum

A well-designed generative AI program today would naturally progress from text and image generation, through to audio and video generation, with a specific module devoted to 3D asset generation and spatial AI. Alongside the technical skills you’d expect, you’d also need a good amount of practical experience in the area of evaluation (i.e. how do you measure the quality of the output, how consistent is it, how well does it conform to user intent etc).

For those of you who are specifically after agentic AI training, multi-modal fluency isn’t an add-on, it’s absolutely fundamental. An agent that can only work with text is operating with a very limited view of the world. An agent that can perceive and generate across video, audio and 3D is operating in the full richness of the real world.

Conclusion

The multi-modal shift is already well and truly underway. Video generation and 3D AI aren’t any more research playthings. They’re tools that are being used in production right across the board in creative, technical and scientific areas. A generative AI course that is relevant to this reality is one that will give the learner skills that are immediately applicable, are career relevant and are built for where the field is going rather than where it’s been.

​

 

Share This Article
Leave a comment
Need Help?
How can I help you?