[I2V] How to achieve 19.5 FPS Inference speed?

[nanwanj]

Summary

We tested Helios-Distilled I2V inference speed on an H100 using scripts/inference/helios-distilled_i2v.sh, and the throughput we observe is lower than the claimed ~19.5 FPS.

Cuda version: 12.8

Environment

• GPU: NVIDIA H100
• Script: scripts/inference/helios-distilled_i2v.sh
• Model: Helios-Distilled (same as the script default)
• FlashAttention: log shows Flash Attn 3 is installed

Log snippet:

bash scripts/inference/helios-distilled_i2v.sh 
Fetching 4 files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 70197.56it/s]
Download complete: : 0.00B [00:00, ?B/s]              Flash Attn 3 is installed!

What we changed

We kept the script unchanged except:

1. Adjusted --num_frames for benchmarking (33, 99 and 264)
2. Updated pyramid_num_inference_steps_list from 2 2 2 to 1 1 1

Benchmark results

• 33 frames: 2.75 s → ~12 FPS
• 99 frames: 6 s → ~16.5 FPS
• 264 frames: 14.2 s → ~18.6 FPS

Expected behavior

Based on the reported numbers, we expected to get close to ~19.5 FPS on H100.
Based on demo video, 33 frames inference time should be around ~1.68 seconds on H100.

Actual behavior

Best throughput we see is ~18.6 FPS, which is still below the claimed ~19.5 FPS, even with --enable_compile and FlashAttention3 installed.

33 frames inference time from our testing is around ~2.75 seconds

Request / Question

Could you please guide us on how to improve I2V inference latency / throughput on H100 to match the reported performance?

Specifically, any recommendations on:

• required flags / settings to hit the reported FPS (compile mode, attention backend, bf16/fp16, SDPA/xformers, etc.)
• whether the claimed ~19.5 FPS assumes specific settings (resolution, num_frames, steps, stage2, batch size, warmup iterations, etc.)
• common bottlenecks or profiling tips for Helios I2V

Output of pip freeze is also attached.

installed.txt

Thank you!

The way we measure inference time is to update "infer_helios.py" and remove the comments at line 514-516 & line 548- 549, which is same to your demo video.

Helios/infer_helios.py

Lines 514 to 516 in c2f2ea5

	# import time
	# for _ in range(20):
	# start_time = time.time()

Helios/infer_helios.py

Lines 548 to 549 in c2f2ea5

	# elapsed_time = time.time() - start_time
	# print(f"Inference time: {elapsed_time:.2f} seconds ({elapsed_time/60:.2f} minutes)")

[nanwanj]

We’re reviewing the throughput demo video (helios_throughput.mp4) that shows ~19.5 FPS:

Video: https://github.com/PKU-YuanGroup/Helios-Page/blob/main/videos/helios_throughput.mp4

However, after comparing the demo code to the current repo’s infer_helios.py, we found they are not the same. Here is the diff:

DiffChecker (left = demo code, right = repo code): https://www.diffchecker.com/8u4GhDiZ/

Could you clarify:

• Is the demo using additional optimizations (e.g., different pipeline, settings, flags, compilation, attention impl, caching, etc.) that are not present in infer_helios.py?

• If possible, could you share the exact command line + environment details (GPU model, dtype, torch version, compile settings) + flag-settings used for that benchmark?

We’d like to reproduce the same measurement with the official code path.

[SHYuanBest]

@nanwanj Thanks for your great interest! I think Real-time systems are not only related to the GPU but also to other hardware components such as the CPU and memory. I just recorded a video using the latest code (no editing, one-shot), and the speed even reached 33/1.67 = 19.76 FPS. And I even didn't enable torch.set_float32_matmul_precision("high"), which should be faster.

https://www.youtube.com/watch?v=DnhzyzABf_E

[gn008]

@nanwanj Hi! Overall FPS can have many non-model factors, such as CPU speed (which affects FFMPEG decoding time), memory (which affects caching speed). Can you please tell us your hardware configuration such as CPU and memory settings? We are happy to check if it is the system differences that results in FPS difference.

[nanwanj]

@SHYuanBest Thanks for this record. It helps a lot!

@nanwanj Thanks for your great interest! I think Real-time systems are not only related to the GPU but also to other hardware components such as the CPU and memory. I just recorded a video using the latest code (no editing, one-shot), and the speed even reached 33/1.67 = 19.76 FPS. And I even didn't enable torch.set_float32_matmul_precision("high"), which should be faster.

https://www.youtube.com/watch?v=DnhzyzABf_E

[nanwanj]

@nanwanj Hi! Overall FPS can have many non-model factors, such as CPU speed (which affects FFMPEG decoding time), memory (which affects caching speed). Can you please tell us your hardware configuration such as CPU and memory settings? We are happy to check if it is the system differences that results in FPS difference.

Hi @gn008 — thanks for the quick reply!

Here are our system details on the machine we used for benchmarking:

Instance type: a3-highgpu-1g (26 vCPUs, 234 GB RAM, 1 × NVIDIA H100 80GB)

OS

Distributor ID: Ubuntu
Description:    Ubuntu 24.04.4 LTS
Release:        24.04
Codename:       noble

nvidia-smi

Thu Mar  5 05:04:26 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.211.01             Driver Version: 570.211.01     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:04:00.0 Off |                    0 |
| N/A   32C    P0             68W /  700W |       0MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

CPU (lscpu)

lscpu
Architecture:                x86_64
  CPU op-mode(s):            32-bit, 64-bit
  Address sizes:             52 bits physical, 57 bits virtual
  Byte Order:                Little Endian
CPU(s):                      26
  On-line CPU(s) list:       0-25
Vendor ID:                   GenuineIntel
  Model name:                Intel(R) Xeon(R) Platinum 8481C CPU @ 2.70GHz
    CPU family:              6
    Model:                   143
    Thread(s) per core:      2
    Core(s) per socket:      13
    Socket(s):               1
    Stepping:                8
    BogoMIPS:                5399.99
    Flags:                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_e
                             nhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 arat avx512vbmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid cldemote movdiri movdir64b fsrm md_clear 
                             serialize tsxldtrk amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities

Memory (/proc/meminfo, excerpt)

MemTotal:       241208512 kB
MemFree:        239571092 kB
MemAvailable:   238695840 kB
Buffers:           56028 kB
Cached:           862836 kB
SwapCached:            0 kB
Active:           365616 kB
Inactive:         630752 kB
Active(anon):      85708 kB
Inactive(anon):        0 kB
Active(file):     279908 kB
Inactive(file):   630752 kB
Unevictable:       25424 kB
Mlocked:           25424 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Zswap:                 0 kB
Zswapped:              0 kB
Dirty:                88 kB
Writeback:             0 kB
AnonPages:        103128 kB
Mapped:           109188 kB
Shmem:              1260 kB
KReclaimable:      45152 kB
Slab:             180312 kB
SReclaimable:      45152 kB
SUnreclaim:       135160 kB
KernelStack:        6656 kB
PageTables:         4040 kB
SecPageTables:         0 kB

If you need more information, please let me know.

Thanks!

[SHYuanBest]

@nanwanj My CPU configuration is as follows; it appears we are the same CPU, but I have more cores.

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         52 bits physical, 57 bits virtual
  Byte Order:            Little Endian
CPU(s):                  208
  On-line CPU(s) list:   0-207
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Xeon(R) Platinum 8481C CPU @ 2.70GHz
    CPU family:          6
    Model:               143
    Thread(s) per core:  2
    Core(s) per socket:  52
    Socket(s):           2
    Stepping:            8
    BogoMIPS:            5399.99

[nanwanj]

Thanks @SHYuanBest.

Just to confirm the environment used for your “Helios Throughput No-Cut” video:

1. Runtime: Was the script executed directly on the host, or inside Docker / Kubernetes (K8S)? We used to observe that even with identical settings, performance can differ between host and container environments, so we’d like to confirm your setup.
2. Dependencies: If possible, could you share the output of pip freeze (or pip list --format=freeze) so we can compare versions on our side?
3. OS: Were you running on Ubuntu? If so, which version?

Thanks!

[SHYuanBest]

1. K8S
2. requirements.txt

lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 12 (bookworm)
Release:        12
Codename:       bookworm

[nanwanj]

Thank you!

[xinjiu612]

@SHYuanBest i noticed the first step inference time is 08:49 min at https://www.youtube.com/watch?v=DnhzyzABf_E and the following steps is truely 19.5 FPS. I am not familiar with the video model. Can you explain the reason for it ?

[SHYuanBest]

@xinjiu612 Thanks for interest! We enabled torch.compile for Helios, which means the first time the script is launched, it takes a significant amount of time to compile. If the code was compiled before on the same machine, the next run will be much faster. You can refer to this article to cache the compile information, so that subsequent launches of the first step will also achieve 19.5 FPS.

[xinjiu612]

@SHYuanBest Thanks for your response! However, the first step inference time is still much longer than the others even when i test after the compiled code. My test was ran on A100.

[SHYuanBest]

Yeah, even after an initial compile, the second run isn't strictly compile-free. To bypass compilation completely, check out this link on how to cache your builds: https://huggingface.co/blog/zh/zerogpu-aoti

[lobo1586]

Hi @SHYuanBest thanks for the great work, i am following this thread and your recorded video, i noticed that in your youtube recording, it was running the distilled T2V script, and I was able to reproduce the speed (even slightly faster ) @ 20 fps using H100 and same environment.

however when it comes to I2V using scripts/inference/helios-distilled_i2v.sh , i can only get ~ 12 fps. here is my step to reproduce:

update scripts/inference/helios-distilled_t2v.sh with your code in youtube. ran the script, confirmed the speed is 19.5+ fps

then ran

rm -rf ~/.triton/cache/
rm -rf /tmp/torchinductor_*

and with scripts/inference/helios-distilled_i2v.sh looks like:

CUDA_VISIBLE_DEVICES=0 python infer_helios.py \
    --base_model_path "BestWishYsh/Helios-Distilled" \
    --transformer_path "BestWishYsh/Helios-Distilled" \
    --sample_type "i2v" \
    --image_path "example/wave.jpg" \
    --prompt "A towering emerald wave surges forward, its crest curling with raw power and energy. Sunlight glints off the translucent water, illuminating the intricate textures and deep green hues within the wave’s body. A thick spray erupts from the breaking crest, casting a misty veil that dances above the churning surface. As the perspective widens, the immense scale of the wave becomes apparent, revealing the restless expanse of the ocean stretching beyond. The scene captures the ocean’s untamed beauty and relentless force, with every droplet and ripple shimmering in the light. The dynamic motion and vivid colors evoke both awe and respect for nature’s might." \
    --num_frames 33 \
    --guidance_scale 1.0 \
    --is_enable_stage2 \
    --pyramid_num_inference_steps_list 1 1 1 \
    --enable_compile \
    --output_folder "./output_helios/helios-distilled"

i got speed of 2.65s per run, thats ~ 12 fps.

do you know if i2v natively has some part thats slower than t2v ? Thank you

[Mikexu007]

@SHYuanBest 您好，我用单卡H100，使用distilled模型做i2v，只能实现生成240frame需要90s左右，怎么差距那么大🤢