Skip to content

[bug] --skip_until_initial_step 使用時、--gradient_accumulation_steps > 1 で再開位置を過剰にスキップして学習が終了する #2407

Open
Open
[bug] --skip_until_initial_step 使用時、--gradient_accumulation_steps > 1 で再開位置を過剰にスキップして学習が終了する#2407

Description

@hybskgks28275
hybskgks28275
opened on Jul 20, 2026

事象

--resume で学習 state を再開し、--skip_until_initial_step を指定すると、--gradient_accumulation_steps が 2 以上の場合に、再開済み step 数より大幅に多くの epoch がスキップされます。
その結果、内部の step 数が負値まで減少し、学習が開始されずにすぐ終了します。

再現条件

--resume で state を再開
--skip_until_initial_step を指定
--gradient_accumulation_steps=4
再開 state の current_step=500
1 epoch あたりの dataloader batch 数が 100 の例

この条件では、500 optimizer step は 2000 dataloader batch に相当するため、本来は 20 epoch をスキップする必要があります。
しかし実際には 80 epoch がスキップされ、step 数が負値まで進みます。
再開 state は正常に読み込まれますが、--skip_until_initial_step 実行後に epoch のスキップ数が過剰となります。
最終的に学習ループへ入らず終了します。

ログ

running training / 学習開始
  num train images * repeats / 学習画像の数×繰り返し回数: 100
  num validation images * repeats / 学習画像の数×繰り返し回数: 0
  num reg images / 正則化画像の数: 0
  num batches per epoch / 1epochのバッチ数: 100
  num epochs / epoch数: 30
  batch size per device / バッチサイズ: 1
  gradient accumulation steps / 勾配を合計するステップ数 = 4
  total optimization steps / 学習ステップ数: 750
2026-07-21 07:38:46 INFO     skipping 500 steps / 500ステップをスキップします                      train_network.py:1461
                    INFO     text_encoder is not needed for training. deleting to save memory.     train_network.py:1494
2026-07-21 07:38:47 INFO     skipping epoch 1 because initial_step (multiplied) is 2000            train_network.py:1514
                    INFO     skipping epoch 2 because initial_step (multiplied) is 1900            train_network.py:1514
                    INFO     skipping epoch 3 because initial_step (multiplied) is 1800            train_network.py:1514
                    INFO     skipping epoch 4 because initial_step (multiplied) is 1700            train_network.py:1514
                    INFO     skipping epoch 5 because initial_step (multiplied) is 1600            train_network.py:1514
                    INFO     skipping epoch 6 because initial_step (multiplied) is 1500            train_network.py:1514
                    INFO     skipping epoch 7 because initial_step (multiplied) is 1400            train_network.py:1514
                    INFO     skipping epoch 8 because initial_step (multiplied) is 1300            train_network.py:1514
                    INFO     skipping epoch 9 because initial_step (multiplied) is 1200            train_network.py:1514
                    INFO     skipping epoch 10 because initial_step (multiplied) is 1100           train_network.py:1514
                    INFO     skipping epoch 11 because initial_step (multiplied) is 1000           train_network.py:1514
                    INFO     skipping epoch 12 because initial_step (multiplied) is 900            train_network.py:1514
                    INFO     skipping epoch 13 because initial_step (multiplied) is 800            train_network.py:1514
                    INFO     skipping epoch 14 because initial_step (multiplied) is 700            train_network.py:1514
                    INFO     skipping epoch 15 because initial_step (multiplied) is 600            train_network.py:1514
                    INFO     skipping epoch 16 because initial_step (multiplied) is 500            train_network.py:1514
                    INFO     skipping epoch 17 because initial_step (multiplied) is 400            train_network.py:1514
                    INFO     skipping epoch 18 because initial_step (multiplied) is 300            train_network.py:1514
                    INFO     skipping epoch 19 because initial_step (multiplied) is 200            train_network.py:1514
                    INFO     skipping epoch 20 because initial_step (multiplied) is 100            train_network.py:1514
                    INFO     skipping epoch 21 because initial_step (multiplied) is 0              train_network.py:1514
                    INFO     skipping epoch 22 because initial_step (multiplied) is -100           train_network.py:1514
                    INFO     skipping epoch 23 because initial_step (multiplied) is -200           train_network.py:1514
                    INFO     skipping epoch 24 because initial_step (multiplied) is -300           train_network.py:1514
                    INFO     skipping epoch 25 because initial_step (multiplied) is -400           train_network.py:1514
                    INFO     skipping epoch 26 because initial_step (multiplied) is -500           train_network.py:1514
                    INFO     skipping epoch 27 because initial_step (multiplied) is -600           train_network.py:1514
                    INFO     skipping epoch 28 because initial_step (multiplied) is -700           train_network.py:1514
                    INFO     skipping epoch 29 because initial_step (multiplied) is -800           train_network.py:1514
                    INFO     skipping epoch 30 because initial_step (multiplied) is -900           train_network.py:1514
                    INFO     skipping epoch 31 because initial_step (multiplied) is -1000          train_network.py:1514
                    INFO     skipping epoch 32 because initial_step (multiplied) is -1100          train_network.py:1514
                    INFO     skipping epoch 33 because initial_step (multiplied) is -1200          train_network.py:1514
                    INFO     skipping epoch 34 because initial_step (multiplied) is -1300          train_network.py:1514
                    INFO     skipping epoch 35 because initial_step (multiplied) is -1400          train_network.py:1514
                    INFO     skipping epoch 36 because initial_step (multiplied) is -1500          train_network.py:1514
                    INFO     skipping epoch 37 because initial_step (multiplied) is -1600          train_network.py:1514
                    INFO     skipping epoch 38 because initial_step (multiplied) is -1700          train_network.py:1514
                    INFO     skipping epoch 39 because initial_step (multiplied) is -1800          train_network.py:1514
                    INFO     skipping epoch 40 because initial_step (multiplied) is -1900          train_network.py:1514
                    INFO     skipping epoch 41 because initial_step (multiplied) is -2000          train_network.py:1514
                    INFO     skipping epoch 42 because initial_step (multiplied) is -2100          train_network.py:1514
                    INFO     skipping epoch 43 because initial_step (multiplied) is -2200          train_network.py:1514
                    INFO     skipping epoch 44 because initial_step (multiplied) is -2300          train_network.py:1514
                    INFO     skipping epoch 45 because initial_step (multiplied) is -2400          train_network.py:1514
                    INFO     skipping epoch 46 because initial_step (multiplied) is -2500          train_network.py:1514
                    INFO     skipping epoch 47 because initial_step (multiplied) is -2600          train_network.py:1514
                    INFO     skipping epoch 48 because initial_step (multiplied) is -2700          train_network.py:1514
                    INFO     skipping epoch 49 because initial_step (multiplied) is -2800          train_network.py:1514
                    INFO     skipping epoch 50 because initial_step (multiplied) is -2900          train_network.py:1514
                    INFO     skipping epoch 51 because initial_step (multiplied) is -3000          train_network.py:1514
                    INFO     skipping epoch 52 because initial_step (multiplied) is -3100          train_network.py:1514
                    INFO     skipping epoch 53 because initial_step (multiplied) is -3200          train_network.py:1514
                    INFO     skipping epoch 54 because initial_step (multiplied) is -3300          train_network.py:1514
                    INFO     skipping epoch 55 because initial_step (multiplied) is -3400          train_network.py:1514
                    INFO     skipping epoch 56 because initial_step (multiplied) is -3500          train_network.py:1514
                    INFO     skipping epoch 57 because initial_step (multiplied) is -3600          train_network.py:1514
                    INFO     skipping epoch 58 because initial_step (multiplied) is -3700          train_network.py:1514
                    INFO     skipping epoch 59 because initial_step (multiplied) is -3800          train_network.py:1514
                    INFO     skipping epoch 60 because initial_step (multiplied) is -3900          train_network.py:1514
                    INFO     skipping epoch 61 because initial_step (multiplied) is -4000          train_network.py:1514
                    INFO     skipping epoch 62 because initial_step (multiplied) is -4100          train_network.py:1514
                    INFO     skipping epoch 63 because initial_step (multiplied) is -4200          train_network.py:1514
                    INFO     skipping epoch 64 because initial_step (multiplied) is -4300          train_network.py:1514
                    INFO     skipping epoch 65 because initial_step (multiplied) is -4400          train_network.py:1514
                    INFO     skipping epoch 66 because initial_step (multiplied) is -4500          train_network.py:1514
                    INFO     skipping epoch 67 because initial_step (multiplied) is -4600          train_network.py:1514
                    INFO     skipping epoch 68 because initial_step (multiplied) is -4700          train_network.py:1514
                    INFO     skipping epoch 69 because initial_step (multiplied) is -4800          train_network.py:1514
                    INFO     skipping epoch 70 because initial_step (multiplied) is -4900          train_network.py:1514
                    INFO     skipping epoch 71 because initial_step (multiplied) is -5000          train_network.py:1514
                    INFO     skipping epoch 72 because initial_step (multiplied) is -5100          train_network.py:1514
                    INFO     skipping epoch 73 because initial_step (multiplied) is -5200          train_network.py:1514
                    INFO     skipping epoch 74 because initial_step (multiplied) is -5300          train_network.py:1514
                    INFO     skipping epoch 75 because initial_step (multiplied) is -5400          train_network.py:1514
                    INFO     skipping epoch 76 because initial_step (multiplied) is -5500          train_network.py:1514
                    INFO     skipping epoch 77 because initial_step (multiplied) is -5600          train_network.py:1514
                    INFO     skipping epoch 78 because initial_step (multiplied) is -5700          train_network.py:1514
                    INFO     skipping epoch 79 because initial_step (multiplied) is -5800          train_network.py:1514
                    INFO     skipping epoch 80 because initial_step (multiplied) is -5900          train_network.py:1514
                    INFO     unet dtype: torch.bfloat16, device: cuda:0                            train_network.py:1519

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions