CoNo:

Tuning-free long video diffusion has been proposed to generate extended-duration videos with enriched content by reusing the knowledge from pre-trained short video diffusion model without retraining. However, most works overlook the fine-grained long-term video consistency modeling, resulting in limited scene consistency (i.e., unreasonable object or background transitions), especially with multiple text inputs. To mitigate this, we propose the Consistency Noise Injection, dubbed CoNo, which introduces the “look-back” mechanism to enhance the fine-grained scene transition between different video clips, and designs the long-term consistency regularization to eliminate the content shifts when extending video contents through noise prediction. In particular, the “look-back” mechanism breaks the noise scheduling process into three essential parts, where one internal noise prediction part is injected into two video-extending parts, intending to achieve a fine-grained transition between two video clips. The long-term consistency regularization focuses on explicitly minimizing the pixel-wise distance between the predicted noises of the extended video clip and the original one, thereby preventing abrupt scene transitions. Extensive experiments have shown the effectiveness of the above strategies by performing long-video generation under both single- and multi-text prompt conditions.

Illustration of the CoNo framework. We propose a “look-back” mechanism that inserts an internal noise prediction stage between two video extending stages to enhance scene consistency. To achieve this, we design the extending and internal initial noise shuffles and constrain the denoising trajectory using selected predicted noise (denoted as [s] in the figure). Additionally, we apply long-term consistency regularization between adjacent video clips to avoid abrupt content shifts. We obtain the final video by concatenating the frames marked with yellow boxes from different stages.

The ''look-back'' mechanism divides the video extension process into three crucial stages, where one internal noise prediction stage is inserted into two video extending stages, intending to ensure stable content transition through the inherent constrain of two-side contents at each reverse process (i.e., the predicted noises from left existing predicted frames and right extending frames). Notably maintaining the overall initial noise group of different video clips is crucial to guarantee the same contents/scenes, we also propose customized noise shuffle strategies for the above three stages, respectively. Concretely, we design the revised extending noise shuffle for the video extending stage, which recovers the noise order for guided frames after reversing the whole initial noises, thereby obviating the reverse-order repetitive content generation. For internal noise prediction, we directly inserted the initial noises at the end of the sequence into the middle position, resulting in an internal noise shuffle to ensure the same initial noise group.

We propose an explicit long-term consistency regularization, which minimizes the pixel-wise distance between predicted noises of the extended video clip and the original generated video clip.