SimDA: Simple Diffusion Adapter for Efficient Video Generation

Zhen Xing1, Qi Dai2, Han Hu2, Zuxuan Wu1, Yu-Gang Jiang1
1 Fudan University, 2MicroSoft Research Asia
Paper arXiv Code

Abstract

The recent wave of AI-generated content has witnessed the great development and success of Text-to-Image (T2I) technologies. By contrast, Text-to-Video (T2V) still falls short of expectations though attracting increasing interests. Existing works either train from scratch or adapt large T2I model to videos, both of which are computation and resource expensive.

In this work, we propose a Simple Diffusion Adapter (SimDA) that fine-tunes only 24M out of 1.1B parameters of a strong T2I model, adapting it to video generation in a parameter-efficient way. In particular, we turn the T2I model for T2V by designing light-weight spatial and temporal adapters for transfer learning. Besides, we change the original spatial attention to the proposed Latent-Shift Attention (LSA) for temporal consistency. With similar model architecture, we further train a video super-resolution model to generate high-definition (1024x1024) videos.

In addition to T2V generation in the wild, SimDA could also be utilized in one-shot video editing with only 2 minutes tuning. Doing so, our method could minimize the training effort with extremely few tunable parameters for model adaptation.

Text-to-Video Generation Results

A red Cardinal on a tree branch stands out when the snow is falling. A cat wearing sunglasses and working as a lifeguard at a pool. Time lapse at a fantasy landscape, 4k, high resolution. Xmas christmas tree holiday celebration winter snow animation gold background.
A dog wearing virtual reality goggles on the grass, 4k, high resolution. An astronaut flying in space, 4k, high resolution. Aerial view over snow covered mountains. A teddy bear wearing sunglasses playing the electric guitar high definition 4k.
fireworks being displayed for a crowd of people. a clownfish swimming through a coral reef. Close up of grapes on a rotating table. Beer pouring into glass, low angle video shot.
Standing on top a mountainside watching the sunset with the vivid pinks red orange showing from the fire colored sky. Sea waves with foam on white tropical sandy beach. Coffee pouring into a cup, 4K, high resolution. A beautiful sunrise on mars, Curiosity rover.

Text guided Video Editting Results





a jeep car is moving on the road. an AE86 car is moving on the road. a jeep car is moving on the road, cartoon style . a jeep car is moving in a forest, in Autumn.
a jeep car is moving on the road in a snow day a Porsche sports car is moving on the road. a jeep car is moving on the road, van Gogh’s style . a jeep car is moving on the desert at dark.
a man is dribbling a basketball wonder woman, wearing a cowboy hat, is dribbling a basketball Kobe Bryant is dribbling a basketball on the grass. James Bond is dribbling a basketball on the beach
A black swan is swiming in a pool A black swan is swiming in a snowy winter A swarovski blue crystal is swiming in a pool A yellow duck is swinming in a pool

Text-to-Video Generation Comparison


VDM[1]
CogVideo[2]
VideoFusion[3]
LVDM[4]
Ours
Mountain river. path in a tropical forest. snowfall in city. Forest in Autumn.
VDM[1]
CogVideo[2]
VideoFusion[3]
LVDM[4]
Ours
4K Illuminated Christmas Tree at Night During Snowstorm Dramatic ocean sunset Irrigation Canal in Western USA Water Sourced from the Colorado River 4K Aerial Video fire

BibTeX

If you use our work in your research, please cite: 


            @misc{xing2023simda,
            title={SimDA: Simple Diffusion Adapter for Efficient Video Generation}, 
            author={Zhen Xing and Qi Dai and Han Hu and Zuxuan Wu and Yu-Gang Jiang},
            year={2023},
            eprint={2308.09710},
            archivePrefix={arXiv},
            primaryClass={cs.CV}}