※ The services mentioned in this article are developed and operated by the author personally. This is not for promotion, but an article sharing insights gained from product development using a video generation model.

Introduction

Video generation models are improving every year, but I still feel that producing videos of consistent quality each time is difficult. When generating 10 times with the same prompt, failures such as mismatched lip movements, broken hands, strange text appearing in the background, or unexpected people appearing in the video occasionally occur.

I am creating a small Web service called AI Rap Generators that produces 15-second vertical (9:16) rap videos using two adult portrait photos. The lyrics and beat are generated by AI, and the video generation uses Seedance 2.0.

In this article, I will summarize the following three points that were effective in improving stability.

1. Splitting the prompt into five blocks
2. Adding a one-line prohibition for each failure pattern
3. Shortening the lyrics to fit within 15 seconds

Note that the prompts in this article are simplified versions to illustrate the design concept.

About Seedance 2.0 (Prerequisites)

Within the scope of my research, Seedance 2.0 is a multi-modal audio-visual video generation model from ByteDance, with the following features:

• Generating video and audio simultaneously (Well-suited for content like rap where voice, beat, and movement are integrated)
• Accepting reference images (Allowing users to provide their portrait as a reference)
• Maximum generation length is around 15 seconds

However, the parameters and limitations may vary depending on the API provider. Please check the latest documentation of each provider when using it.

What couldn't be adjusted with narrative prompts

Initially, I was writing prompts as one paragraph per scene.

This way, when the result is bad, it's unclear which sentence is the cause. Sometimes, when trying to fix it, you might break another part.

Solution: Divide the prompt into 5 blocks

So, I divided all the scene prompts into the following 5 blocks.

Example of a rooftop scene (simplified).

By dividing into blocks, you can make adjustments like "changing only the Music to compare" or "adding one line to Constraints to see the effect"one change at a time. This was the biggest improvement.

Design Intent of Each Block

Scene: First fix the character, then fix the frame

• "Reference image 1 determines the character's identity, clothing, hairstyle, and physique," explicitly stating this to reduce the model's degrees of freedom.
• The composition is fixed as "fixed camera, upper body, vertical 9:16." I feel that upper body shots are easier to stabilize facial, hand, and mouth movements compared to full-body shots.
• The background is specified with concrete elements (night scene, string lights, palm trees, lounge furniture, reflective stone floor). The simpler and more clear the background, the less likely it is to have blurring between frames.
• Prohibit people from entering on their own, such as "no other people enter."

Music: Write using parameters, not just atmosphere

"Cool rap beat" is the same as not specifying anything. Specify it as follows.

• Fix the BPM (e.g., 110)
• List the instruments (kick, snare, hi-hat, sub-bass, piano, pad)
• Specify the vocals as "only one, clear" (adding chorus or harmonies makes facial movements difficult)

If the BPM is fixed, the length of the lyrics can be designed in advance.

Timeline: Instead of "dance," move one action every 3 seconds

Divide 15 seconds into 5 intervals of 3 seconds each, and write one main action for each interval.

Ending with "stop" gives an impression that the work is complete without seeming cut off in the middle.

Lyrics: Keep it short and let it be sung as is

• All in 6 lines, around 7 to 10 syllables per line (assuming English lyrics). Longer lines are more prone to lip movement misalignment.
• Write the title as 「perform exactly these」 to prevent the model from altering the lyrics on its own.
• Personalized elements like names and birthdays are only included in 1 to 2 lines.

Constraints: Add one line per failure pattern

This block showed the most growth. The common failures and their corresponding constraints are as follows.

What I noticed here is thatprompts aren't necessarily better the longer they are. Writing both 「what to do」 and 「what not to do」 briefly and specifically worked better.

Things to be cautious about in operation

• Change only one place at a time. Splitting into blocks is for this purpose.
• Record failures. Note which scenes had what failures, and link them to adding constraints.
• Check success rates per scene. Overall success rate alone doesn't show which scene's prompt is weak.
• Refund automatically when a failure occurs. Since generating content costs and failures can't be entirely avoided, I created a system from the start to prevent users from losing out.

Current limitations

• Videos are only 15 seconds long.
• Not supported for uploading your own music
• Only two adults are allowed as performers
• Due to AI generation, there may be variations in resemblance to the photos

Also, since we are handling real people's faces, we have set rules that only the performer themselves or an agreed adult can be used, no images of minors or public figures are accepted, AI-generated labels must be added to videos, and uploaded photos will be deleted after 7 days.

Pricing is on a credit system, with 10 credits per video. Credit packs start at $6.99 (no subscription required). Stages (scenes) can be viewed for free.

Summary

• Prompts are divided into the 5 blocks of Scene / Music / Timeline / Lyrics / Constraints, making adjustments easier.
• Each time a failure pattern is found, add one line to the Constraints.
• To fit into 15 seconds, lyrics should be short (6 lines, 7-10 syllables)
• It's more effective to specifically write 'what to do + what not to do' rather than a long prompt.

If you're developing using the same video generation model, please share what failure patterns you've encountered and the constraints you've added. The service is available at airapgenerators.com. Stages can be viewed for free. If you have any issues while using it, please let me know in the comments.