怎么并行训练？

[bash99]

我试图用
CUDA_VISIBLE_DEVICES=0,1 torchrun --nproc_per_node 2
这样并行训练，会直接报错

ValueError: You can't train a model that has been loaded in 8-bit precision on multiple devices in any distributed mode. In order to use 8-bit
models that have been loaded across multiple GPUs the solution is to use Naive Pipeline Parallelism. Therefore you should not specify that you
are under any distributed regime in your accelerate config.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 594501) of binary: /DaTa/mambaforge/bin/python3.10

完整log如下：

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /DaTa/dl/MedicalGPT_lora_train/chatGLM-6B-QLoRA/train_qlora.py:206 in         │
│ <module>                                                                                         │
│                                                                                                  │
│   203                                                                                            │
│   204 if __name__ == "__main__":                                                                 │
│   205 │   args = parse_args()                                                                    │
│ ❱ 206 │   train(args)                                                                            │
│   207                                                                                            │
│   208                                                                                            │
│                                                                                                  │
│ /DaTa/dl/MedicalGPT_lora_train/chatGLM-6B-QLoRA/train_qlora.py:200 in train   │
│                                                                                                  │
│   197 │   │   data_collator=data_collator                                                        │
│   198 │   )                                                                                      │
│   199 │                                                                                          │
│ ❱ 200 │   trainer.train(resume_from_checkpoint=resume_from_checkpoint)                           │
│   201 │   trainer.model.save_pretrained(hf_train_args.output_dir)                                │
│   202                                                                                            │
│   203                                                                                            │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/transformers/trainer.py:1645 in │
│ train                                                                                            │
│                                                                                                  │
│   1642 │   │   inner_training_loop = find_executable_batch_size(                                 │
│   1643 │   │   │   self._inner_training_loop, self._train_batch_size, args.auto_find_batch_size  │
│   1644 │   │   )                                                                                 │
│ ❱ 1645 │   │   return inner_training_loop(                                                       │
│   1646 │   │   │   args=args,                                                                    │
│   1647 │   │   │   resume_from_checkpoint=resume_from_checkpoint,                                │
│   1648 │   │   │   trial=trial,                                                                  │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/transformers/trainer.py:1756 in │
│ _inner_training_loop                                                                             │
│                                                                                                  │
│   1753 │   │   │   │   if self.use_apex:                                                         │
│   1754 │   │   │   │   │   model = self.accelerator.prepare(self.model)                          │
│   1755 │   │   │   │   else:                                                                     │
│ ❱ 1756 │   │   │   │   │   model, self.optimizer = self.accelerator.prepare(self.model, self.op  │
│   1757 │   │   │   else:                                                                         │
│   1758 │   │   │   │   # to handle cases wherein we pass "DummyScheduler" such as when it is sp  │
│   1759 │   │   │   │   model, self.optimizer, self.lr_scheduler = self.accelerator.prepare(      │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/accelerate/accelerator.py:1182  │
│ in prepare                                                                                       │
│                                                                                                  │
│   1179 │   │   elif self.distributed_type == DistributedType.MEGATRON_LM:                        │
│   1180 │   │   │   result = self._prepare_megatron_lm(*args)                                     │
│   1181 │   │   else:                                                                             │
│ ❱ 1182 │   │   │   result = tuple(                                                               │
│   1183 │   │   │   │   self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d i  │
│   1184 │   │   │   )                                                                             │
│   1185 │   │   │   result = tuple(self._prepare_one(obj, device_placement=d) for obj, d in zip(  │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/accelerate/accelerator.py:1183  │
│ in <genexpr>                                                                                     │
│                                                                                                  │
│   1180 │   │   │   result = self._prepare_megatron_lm(*args)                                     │
│   1181 │   │   else:                                                                             │
│   1182 │   │   │   result = tuple(                                                               │
│ ❱ 1183 │   │   │   │   self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d i  │
│   1184 │   │   │   )                                                                             │
│   1185 │   │   │   result = tuple(self._prepare_one(obj, device_placement=d) for obj, d in zip(  │
│   1186                                                                                           │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/accelerate/accelerator.py:1022  │
│ in _prepare_one                                                                                  │
│                                                                                                  │
│   1019 │   │   │   if isinstance(obj, torch.utils.data.DataLoader):                              │
│   1020 │   │   │   │   return self.prepare_data_loader(obj, device_placement=device_placement)   │
│   1021 │   │   │   elif isinstance(obj, torch.nn.Module):                                        │
│ ❱ 1022 │   │   │   │   return self.prepare_model(obj, device_placement=device_placement)         │
│   1023 │   │   │   elif isinstance(obj, torch.optim.Optimizer):                                  │
│   1024 │   │   │   │   optimizer = self.prepare_optimizer(obj, device_placement=device_placemen  │
│   1025 │   │   │   │   return optimizer                                                          │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/accelerate/accelerator.py:1247  │
│ in prepare_model                                                                                 │
│                                                                                                  │
│   1244 │   │   ):                                                                                │
│   1245 │   │   │   model_devices = set(model.hf_device_map.values())                             │
│   1246 │   │   │   if len(model_devices) > 1 and self.distributed_type != DistributedType.NO:    │
│ ❱ 1247 │   │   │   │   raise ValueError(                                                         │
│   1248 │   │   │   │   │   "You can't train a model that has been loaded in 8-bit precision on   │
│   1249 │   │   │   │   │   " In order to use 8-bit models that have been loaded across multiple  │
│   1250 │   │   │   │   │   " Therefore you should not specify that you are under any distribute  │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: You can't train a model that has been loaded in 8-bit precision on multiple devices in any distributed mode. In order to use 8-bit
models that have been loaded across multiple GPUs the solution is to use Naive Pipeline Parallelism. Therefore you should not specify that you
are under any distributed regime in your accelerate config.
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /DaTa/dl/MedicalGPT_lora_train/chatGLM-6B-QLoRA/train_qlora.py:206 in         │
│ <module>                                                                                         │
│                                                                                                  │
│   203                                                                                            │
│   204 if __name__ == "__main__":                                                                 │
│   205 │   args = parse_args()                                                                    │
│ ❱ 206 │   train(args)                                                                            │
│   207                                                                                            │
│   208                                                                                            │
│                                                                                                  │
│ /DaTa/dl/MedicalGPT_lora_train/chatGLM-6B-QLoRA/train_qlora.py:200 in train   │
│                                                                                                  │
│   197 │   │   data_collator=data_collator                                                        │
│   198 │   )                                                                                      │
│   199 │                                                                                          │
│ ❱ 200 │   trainer.train(resume_from_checkpoint=resume_from_checkpoint)                           │
│   201 │   trainer.model.save_pretrained(hf_train_args.output_dir)                                │
│   202                                                                                            │
│   203                                                                                            │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/transformers/trainer.py:1645 in │
│ train                                                                                            │
│                                                                                                  │
│   1642 │   │   inner_training_loop = find_executable_batch_size(                                 │
│   1643 │   │   │   self._inner_training_loop, self._train_batch_size, args.auto_find_batch_size  │
│   1644 │   │   )                                                                                 │
│ ❱ 1645 │   │   return inner_training_loop(                                                       │
│   1646 │   │   │   args=args,                                                                    │
│   1647 │   │   │   resume_from_checkpoint=resume_from_checkpoint,                                │
│   1648 │   │   │   trial=trial,                                                                  │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/transformers/trainer.py:1756 in │
│ _inner_training_loop                                                                             │
│                                                                                                  │
│   1753 │   │   │   │   if self.use_apex:                                                         │
│   1754 │   │   │   │   │   model = self.accelerator.prepare(self.model)                          │
│   1755 │   │   │   │   else:                                                                     │
│ ❱ 1756 │   │   │   │   │   model, self.optimizer = self.accelerator.prepare(self.model, self.op  │
│   1757 │   │   │   else:                                                                         │
│   1758 │   │   │   │   # to handle cases wherein we pass "DummyScheduler" such as when it is sp  │
│   1759 │   │   │   │   model, self.optimizer, self.lr_scheduler = self.accelerator.prepare(      │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/accelerate/accelerator.py:1182  │
│ in prepare                                                                                       │
│                                                                                                  │
│   1179 │   │   elif self.distributed_type == DistributedType.MEGATRON_LM:                        │
│   1180 │   │   │   result = self._prepare_megatron_lm(*args)                                     │
│   1181 │   │   else:                                                                             │
│ ❱ 1182 │   │   │   result = tuple(                                                               │
│   1183 │   │   │   │   self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d i  │
│   1184 │   │   │   )                                                                             │
│   1185 │   │   │   result = tuple(self._prepare_one(obj, device_placement=d) for obj, d in zip(  │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/accelerate/accelerator.py:1183  │
│ in <genexpr>                                                                                     │
│                                                                                                  │
│   1180 │   │   │   result = self._prepare_megatron_lm(*args)                                     │
│   1181 │   │   else:                                                                             │
│   1182 │   │   │   result = tuple(                                                               │
│ ❱ 1183 │   │   │   │   self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d i  │
│   1184 │   │   │   )                                                                             │
│   1185 │   │   │   result = tuple(self._prepare_one(obj, device_placement=d) for obj, d in zip(  │
│   1186                                                                                           │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/accelerate/accelerator.py:1022  │
│ in _prepare_one                                                                                  │
│                                                                                                  │
│   1019 │   │   │   if isinstance(obj, torch.utils.data.DataLoader):                              │
│   1020 │   │   │   │   return self.prepare_data_loader(obj, device_placement=device_placement)   │
│   1021 │   │   │   elif isinstance(obj, torch.nn.Module):                                        │
│ ❱ 1022 │   │   │   │   return self.prepare_model(obj, device_placement=device_placement)         │
│   1023 │   │   │   elif isinstance(obj, torch.optim.Optimizer):                                  │
│   1024 │   │   │   │   optimizer = self.prepare_optimizer(obj, device_placement=device_placemen  │
│   1025 │   │   │   │   return optimizer                                                          │
│                                                                                                  │
│ /DaTa/mambaforge/lib/python3.10/site-packages/accelerate/accelerator.py:1247  │
│ in prepare_model                                                                                 │
│                                                                                                  │
│   1244 │   │   ):                                                                                │
│   1245 │   │   │   model_devices = set(model.hf_device_map.values())                             │
│   1246 │   │   │   if len(model_devices) > 1 and self.distributed_type != DistributedType.NO:    │
│ ❱ 1247 │   │   │   │   raise ValueError(                                                         │
│   1248 │   │   │   │   │   "You can't train a model that has been loaded in 8-bit precision on   │
│   1249 │   │   │   │   │   " In order to use 8-bit models that have been loaded across multiple  │
│   1250 │   │   │   │   │   " Therefore you should not specify that you are under any distribute  │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: You can't train a model that has been loaded in 8-bit precision on multiple devices in any distributed mode. In order to use 8-bit
models that have been loaded across multiple GPUs the solution is to use Naive Pipeline Parallelism. Therefore you should not specify that you
are under any distributed regime in your accelerate config.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 594501) of binary: /DaTa/mambaforge/bin/python3.10
Traceback (most recent call last):
  File "/DaTa/mambaforge/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/DaTa/mambaforge/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346
  , in wrapper
    return f(*args, **kwargs)
  File "/DaTa/mambaforge/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
    run(args)
  File "/DaTa/mambaforge/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/DaTa/mambaforge/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/DaTa/mambaforge/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
train_qlora.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2023-07-06_12:13:14
  host      : zhongshanyanke
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 594502)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-07-06_12:13:14
  host      : zhongshanyanke
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 594501)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

[shuxueslpi]

多卡的我也在调，一样的报错，调好了尽快更新。
但我不是一直有多卡的环境，如果有人调好了，也可以公布一下……

[qingyue2014]

同样的问题，请问你调好了嘛

[bash99]

同样的问题，请问你调好了嘛

后来用MedicalGPT那边代码做到了并行训练

[CNUIGB]

同样的问题，请问你调好了嘛

后来用MedicalGPT那边代码做到了并行训练

是做到数据并行（DP）还是DDP呀，我想做DDP但是遇到同样的问题