My training command: torchrun --standalone --nnodes=1 --nproc_per_node=4 Open-Sora/scripts/train.py Open-Sora/configs/opensora-v1-2/train/stage1.py --data-path test123.csv
Here is my config file:
`
# Dataset settings
dataset = dict(
type="VariableVideoTextDataset",
transform_name="resize_crop",
)
# backup
# bucket_config = { # 20s/it
# "144p": {1: (1.0, 100), 51: (1.0, 30), 102: (1.0, 20), 204: (1.0, 8), 408: (1.0, 4)},
# # ---
# "256": {1: (0.5, 100), 51: (0.3, 24), 102: (0.3, 12), 204: (0.3, 4), 408: (0.3, 2)},
# "240p": {1: (0.5, 100), 51: (0.3, 24), 102: (0.3, 12), 204: (0.3, 4), 408: (0.3, 2)},
# # ---
# "360p": {1: (0.5, 60), 51: (0.3, 12), 102: (0.3, 6), 204: (0.3, 2), 408: (0.3, 1)},
# "512": {1: (0.5, 60), 51: (0.3, 12), 102: (0.3, 6), 204: (0.3, 2), 408: (0.3, 1)},
# # ---
# "480p": {1: (0.5, 40), 51: (0.3, 6), 102: (0.3, 3), 204: (0.3, 1), 408: (0.0, None)},
# # ---
# "720p": {1: (0.2, 20), 51: (0.3, 2), 102: (0.3, 1), 204: (0.0, None)},
# "1024": {1: (0.1, 20), 51: (0.3, 2), 102: (0.3, 1), 204: (0.0, None)},
# # ---
# "1080p": {1: (0.1, 10)},
# # ---
# "2048": {1: (0.1, 5)},
# }
# webvid
bucket_config = { # 12s/it
"144p": {1: (1.0, 475), 51: (1.0, 51), 102: ((1.0, 0.33), 27), 204: ((1.0, 0.1), 13), 408: ((1.0, 0.1), 6)},
# ---
"256": {1: (0.4, 297), 51: (0.5, 20), 102: ((0.5, 0.33), 10), 204: ((0.5, 0.1), 5), 408: ((0.5, 0.1), 2)},
"240p": {1: (0.3, 297), 51: (0.4, 20), 102: ((0.4, 0.33), 10), 204: ((0.4, 0.1), 5), 408: ((0.4, 0.1), 2)},
# ---
"360p": {1: (0.2, 141), 51: (0.15, 8), 102: ((0.15, 0.33), 4), 204: ((0.15, 0.1), 2), 408: ((0.15, 0.1), 1)},
"512": {1: (0.1, 141)},
# ---
"480p": {1: (0.1, 89)},
# ---
"720p": {1: (0.05, 36)},
"1024": {1: (0.05, 36)},
# ---
"1080p": {1: (0.1, 5)},
# ---
"2048": {1: (0.1, 5)},
}
grad_checkpoint = True
# Acceleration settings
num_workers = 8
num_bucket_build_workers = 16
dtype = "bf16"
plugin = "zero2"
# Model settings
model = dict(
type="STDiT3-XL/2",
from_pretrained=None,
qk_norm=True,
enable_flash_attn=True,
enable_layernorm_kernel=True,
freeze_y_embedder=True,
)
vae = dict(
type="OpenSoraVAE_V1_2",
from_pretrained="hpcai-tech/OpenSora-VAE-v1.2",
micro_frame_size=17,
micro_batch_size=4,
)
text_encoder = dict(
type="t5",
from_pretrained="DeepFloyd/t5-v1_1-xxl",
model_max_length=300,
shardformer=True,
)
scheduler = dict(
type="rflow",
use_timestep_transform=True,
sample_method="logit-normal",
)
# Mask settings
mask_ratios = {
"random": 0.05,
"intepolate": 0.005,
"quarter_random": 0.005,
"quarter_head": 0.005,
"quarter_tail": 0.005,
"quarter_head_tail": 0.005,
"image_random": 0.025,
"image_head": 0.05,
"image_tail": 0.025,
"image_head_tail": 0.025,
}
# Log settings
seed = 42
outputs = "outputs"
wandb = False
epochs = 1000
log_every = 10
ckpt_every = 200
# optimization settings
load = None
grad_clip = 1.0
lr = 1e-4
ema_decay = 0.99
adam_eps = 1e-15
warmup_steps = 1000
Here is the full terminal output including what commands I ran:
(pytorch) root@fa05c50c4f4a:/workspace/Open-Sora# CUDA_VISIBLE_DEVICES=0,1,2,3
(pytorch) root@fa05c50c4f4a:/workspace/Open-Sora# torchrun --standalone --nnodes=1 --nproc_per_node=4 Open-Sora/scripts/train.py Open-Sora/configs/opensora-v1-2/train/stage1.py --data-path test123.csv
[2024-08-11 18:36:51,286] torch.distributed.run: [WARNING] master_addr is only used for static rdzv_backend and when rdzv_endpoint is not specified.
[2024-08-11 18:36:51,287] torch.distributed.run: [WARNING]
[2024-08-11 18:36:51,287] torch.distributed.run: [WARNING] *****************************************
[2024-08-11 18:36:51,287] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
[2024-08-11 18:36:51,287] torch.distributed.run: [WARNING] *****************************************
[2024-08-11 18:37:01,496] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -7) local_rank: 0 (pid: 159) of binary: /opt/conda/envs/pytorch/bin/python
Traceback (most recent call last):
File "/opt/conda/envs/pytorch/bin/torchrun", line 33, in <module>
sys.exit(load_entry_point('torch==2.1.2', 'console_scripts', 'torchrun')())
File "/opt/conda/envs/pytorch/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/opt/conda/envs/pytorch/lib/python3.9/site-packages/torch/distributed/run.py", line 806, in main
run(args)
File "/opt/conda/envs/pytorch/lib/python3.9/site-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/opt/conda/envs/pytorch/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/opt/conda/envs/pytorch/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
===================================================
Open-Sora/scripts/train.py FAILED
---------------------------------------------------
Failures:
[1]:
time : 2024-08-11_18:37:01
host : fa05c50c4f4a
rank : 1 (local_rank: 1)
exitcode : -7 (pid: 160)
error_file: <N/A>
traceback : Signal 7 (SIGBUS) received by PID 160
[2]:
time : 2024-08-11_18:37:01
host : fa05c50c4f4a
rank : 2 (local_rank: 2)
exitcode : -7 (pid: 161)
error_file: <N/A>
traceback : Signal 7 (SIGBUS) received by PID 161
[3]:
time : 2024-08-11_18:37:01
host : fa05c50c4f4a
rank : 3 (local_rank: 3)
exitcode : -7 (pid: 162)
error_file: <N/A>
traceback : Signal 7 (SIGBUS) received by PID 162
---------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2024-08-11_18:37:01
host : fa05c50c4f4a
rank : 0 (local_rank: 0)
exitcode : -7 (pid: 159)
error_file: <N/A>
traceback : Signal 7 (SIGBUS) received by PID 159
My Docker container has access to all 4 GPUs which I made sure with nvidia-smi so I am not sure as to what the problem is because training works with 1,2 and 8 GPUs but not 4. Do I need to change something in my config file? Would greatly appreciate some help with this!