Open-Sora Unveils Major Upgrade: Embracing Open Source with Single-Shot 16-Second Video Generation and 720p Resolution

Open-Sora has quietly updated in the open-source community, now supporting single-shot video generation up to 16 seconds long, with a maximum resolution of 720p, and capable of handling any aspect ratio for text-to-image, text-to-video, image-to-video, video-to-video, and infinitely long video generation requirements. Let's try out the effects.

 

Generate a landscape video for posting on YouTube.

 

Then generate a portrait video for TikTok.

Plus, with the ability to create 16-second long single-shot videos, now everyone can indulge in their screenwriting fantasies.

 

 

We have officially released the Open-Sora new Technical Report on GitHub [1]. This update primarily includes the following key features:

We, Colossal-AI team, have made key improvements to the STDiT architecture in previous Open-Sora, with the aim of enhancing the model's training stability and overall performance. For the current sequence prediction task, we have adopted best practices from large language models (LLMs), replacing the sinusoidal positional encoding in all attention layers with a more efficient Rotary Position Embedding(RoPE). Additionally, to enhance training stability, they have introduced QK-normalization technology, following the SD3 model architecture, to bolster the stability of half-precision training. To support the training requirements for multiple resolutions, various aspect ratios, and frame rates, the ST-DiT-2 architecture proposed by us is capable of automatically scaling positional embeddings and processing inputs of different sizes.

According to the Open-Sora new technical report, Open-Sora employs a multi-stage training approach, where each stage continues training based on the weights from the previous stage. Compared to single-stage training, this multi-stage training method more efficiently achieves the goal of high-quality video generation by introducing data in steps.

In the initial stage, most videos are trained at a resolution of 144p, with a mix of images and videos at 240p and 480p, lasting approximately one week with a total of 81k steps. The second stage elevates the resolution of most video data to 240p and 480p, with a training duration of one day, reaching 22k steps. The third stage further enhances the training to 480p and 720p, completing 4k steps in one day. The entire multi-stage training process is completed in about 9 days, improving the quality of video generation in multiple dimensions compared to Open-Sora 1.0.

Leveraging the characteristics of the Transformer, we can easily extend the DiT architecture to support both image-to-image and video-to-video tasks. We propose a masking strategy to support the conditioning of images and videos. By setting different masks, we can support a variety of generation tasks, including: image-to-video, looped video, video extension, autoregressive video generation, video stitching, video editing, and frame insertion.

Inspired by the UL2 method [2], we have introduced a random masking strategy during the model training phase. Specifically, during training, frames are randomly selected and unmasked, which includes but is not limited to unmask the first frame, the first k frames, the last k frames, and any random k frames. We found that based on previous experiments with Open-Sora, applying a masking strategy with a 50% probability allowed the model to better learn to handle image conditioning with just a small number of steps. In Open-Sora, we have adopted a pretraining approach from scratch using the masking strategy.

 

Additionally, we have provided a detailed guide for configuring the masking strategy for the inference phase, where a five-number tuple offers great flexibility and control in defining the masking strategy.

 

In the technical report of OpenAI Sora [3], they mentioned that training with the original video's resolution, aspect ratio, and length can increase sampling flexibility and improve framing and composition. Therefore, we proposed a bucket training strategy.

How is it implemented specifically? The so-called bucket is a triplet of (resolution, number of frames, aspect ratio). We have predefined a series of aspect ratios for videos of different resolutions to cover most common types of video aspect ratios. At the beginning of each training epoch, we shuffle the dataset and allocate samples to the corresponding buckets based on their characteristics. Specifically, we place each sample into a bucket that has a resolution and frame length that are less than or equal to the video.

 

To reduce the requirements for computational resources, we introduced two attributes: keep_prob andbatch_size for each （resolution，number of frames） pair, to decrease the computational cost and enable multi-stage training. In this way, we can control the number of samples in different buckets and balance the GPU load by searching for an optimal batch size for each bucket. We provide a detailed explanation of this in our technical report, and interested individuals can read more about it on GitHub:

We also provide a detailed guide for the data collection and processing phase. During the development of Open-Sora 1.0, we realized that the quantity and quality of data are crucial for nurturing a high-performance model. Therefore, we have been committed to expanding and optimizing the dataset. We have established an automated data processing pipeline that adheres to the principles of Singular Value Decomposition (SVD), covering scene segmentation, caption processing, diverse scoring and filtering, as well as the management system and conventions for the dataset. Similarly, we have shared the data processing scripts with the open-source community. Interested developers can now utilize these resources, in conjunction with the technical report and code, to efficiently process and optimize their datasets.

Comprehensive Performance Evaluation of Open-Sora

 

 Resolution：32*240p

 Resolution：64*360p

  Resolution：480*854p