Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generation

We visualize their completion rates for the stories across 7 classes, along with the average result for the entire set. Even the best model achieves an average completion rate of less than 50%, meaning it can successfully present fewer than half of the events in a simple short story on average.

We carefully design 423 prompts across 7 classes, and each prompt illustrates a short story containing 2-4 sequential events. For evaluation, we choose 3 top closed-source commercial models and 8 well-known open-source models, use them for text-to-video generation, and then combine the generated videos and the original prompts as input for VLM verifiers.

Different from previous detail-oriented evaluation that focus on fine-grained quality features, we let the VLM to judge how many events are successfully presented in the generated videos, and thus get the completion rate of the story in prompt.

We create the raw prompts by: (1) Retrieving and captioning the real-world videos. (2) Human Brainstorming. (3) Guided GPT-4o generation given examples from (1) and (2).

We will also filter the prompts in the following case: 1. Who uses the facial expression (like smiling) as a event. Because GPT-4o will blur the human face. 2. Who makes GPT-4o fails to give the completion rates (due to the reasons like security concern) for more than 2 models. 3. Whose corrsponding GPT-4o’s evaluation has relatively weak alignment with human feedback.

(Left) Word cloud of the StoryEval prompt suite. (Right) We visualize the proportion of 7 classes in the prompt suite using an UpSetplot. The bottom left of the figure shows the number of prompts in each class, while the right side displays the number of prompts belonging to each class or intersection group. For example, there are 14 examples that exactly belong to all three classes: ''Hard''', ''Creative'', and ''Object''.

Notably, ''Retrieval'' class includes the captions that are manually annotated for the videos retrieved from real-world, which means that in human aspect,the upper bound of this metric can be 100% if the retrieved videos are used as a baseline. But none of the T2V models here achieve completion rate higher than 45%.

Some Examples

Note: completion list denotes if each event is completed (1) or not (0). And videos are compressed for loading

Prompt: A man opens the refrigerator door, puts the elephant in, and then closes the door. (3 events)

Prompt: A balloon artist inflates a long balloon, twists it several times, and creates a dog shape. (3 events)

Prompt: A man takes off his hat, throws it into the air, and then it is taken by a passing eagle. (3 events)

Prompt: A woman pours some coffee beans into a steel cup, walks with the cup towards a coffee bean grinder, and then pours the beans from the steel cup into the grinder. (3 events)

Prompt: A man dribbles a basketball, and then throws it in a court (2 events)