You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
DEBUG 2024-12-03 16:02:35,806 | build dataset ... ...
/private/home/wuhao/Open-GroundingDino/datasets/rico/combined/ /private/home/wuhao/Open-GroundingDino/datasets/screen_annotation/json1_train.jsonl None
[rank1]: Traceback (most recent call last):
[rank1]: File "/private/home/wuhao/Open-GroundingDino/main.py", line 372, in
[rank1]: main(args)
[rank1]: File "/private/home/wuhao/Open-GroundingDino/main.py", line 187, in main
[rank1]: dataset_val = build_dataset(image_set='val', args=args, datasetinfo=dataset_meta["val"][0])
[rank1]: File "/private/home/wuhao/Open-GroundingDino/datasets/init.py", line 19, in build_dataset
[rank1]: return build_coco(image_set, args, datasetinfo)
[rank1]: File "/private/home/wuhao/Open-GroundingDino/datasets/coco.py", line 633, in build
[rank1]: dataset = CocoDetection(img_folder, ann_file,
[rank1]: File "/private/home/wuhao/Open-GroundingDino/datasets/coco.py", line 329, in init
[rank1]: super(CocoDetection, self).init(img_folder, ann_file)
[rank1]: File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/site-packages/torchvision/datasets/coco.py", line 37, in init
[rank1]: self.coco = COCO(annFile)
[rank1]: File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/site-packages/pycocotools/coco.py", line 82, in init
[rank1]: dataset = json.load(f)
[rank1]: File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/json/init.py", line 293, in load
[rank1]: return loads(fp.read(),
[rank1]: File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/json/init.py", line 346, in loads
[rank1]: return _default_decoder.decode(s)
[rank1]: File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/json/decoder.py", line 340, in decode
[rank1]: raise JSONDecodeError("Extra data", s, end)
[rank1]: json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 1489)
== total images: 15548
DEBUG 2024-12-03 16:02:37,054 | build dataset, done.
DEBUG 2024-12-03 16:02:37,054 | number of training dataset: 1, samples: 15548
/private/home/wuhao/Open-GroundingDino/datasets/rico/combined/ /private/home/wuhao/Open-GroundingDino/datasets/screen_annotation/json1_val.jsonl
loading annotations into memory...
[rank0]: Traceback (most recent call last):
[rank0]: File "/private/home/wuhao/Open-GroundingDino/main.py", line 372, in
[rank0]: main(args)
[rank0]: File "/private/home/wuhao/Open-GroundingDino/main.py", line 187, in main
[rank0]: dataset_val = build_dataset(image_set='val', args=args, datasetinfo=dataset_meta["val"][0])
[rank0]: File "/private/home/wuhao/Open-GroundingDino/datasets/init.py", line 19, in build_dataset
[rank0]: return build_coco(image_set, args, datasetinfo)
[rank0]: File "/private/home/wuhao/Open-GroundingDino/datasets/coco.py", line 633, in build
[rank0]: dataset = CocoDetection(img_folder, ann_file,
[rank0]: File "/private/home/wuhao/Open-GroundingDino/datasets/coco.py", line 329, in init
[rank0]: super(CocoDetection, self).init(img_folder, ann_file)
[rank0]: File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/site-packages/torchvision/datasets/coco.py", line 37, in init
[rank0]: self.coco = COCO(annFile)
[rank0]: File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/site-packages/pycocotools/coco.py", line 82, in init
[rank0]: dataset = json.load(f)
[rank0]: File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/json/init.py", line 293, in load
[rank0]: return loads(fp.read(),
[rank0]: File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/json/init.py", line 346, in loads
[rank0]: return _default_decoder.decode(s)
[rank0]: File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/json/decoder.py", line 340, in decode
[rank0]: raise JSONDecodeError("Extra data", s, end)
[rank0]: json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 1489)
[rank0]:[W1203 16:02:37.115402107 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
W1203 16:02:38.418425 131446 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 131549 closing signal SIGTERM
E1203 16:02:38.632854 131446 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 1 (pid: 131550) of binary: /private/home/wuhao/miniconda3/envs/groundingdino/bin/python
Traceback (most recent call last):
File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/site-packages/torch/distributed/launch.py", line 208, in
main()
File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/site-packages/typing_extensions.py", line 2853, in wrapper
return arg(*args, **kwargs)
File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/site-packages/torch/distributed/launch.py", line 204, in main
launch(args)
File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/site-packages/torch/distributed/launch.py", line 189, in launch
run(args)
File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/site-packages/torch/distributed/run.py", line 910, in run
elastic_launch(
File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 138, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
DEBUG 2024-12-03 16:02:35,806 | build dataset ... ...
/private/home/wuhao/Open-GroundingDino/datasets/rico/combined/ /private/home/wuhao/Open-GroundingDino/datasets/screen_annotation/json1_train.jsonl None
[rank1]: Traceback (most recent call last):
[rank1]: File "/private/home/wuhao/Open-GroundingDino/main.py", line 372, in
[rank1]: main(args)
[rank1]: File "/private/home/wuhao/Open-GroundingDino/main.py", line 187, in main
[rank1]: dataset_val = build_dataset(image_set='val', args=args, datasetinfo=dataset_meta["val"][0])
[rank1]: File "/private/home/wuhao/Open-GroundingDino/datasets/init.py", line 19, in build_dataset
[rank1]: return build_coco(image_set, args, datasetinfo)
[rank1]: File "/private/home/wuhao/Open-GroundingDino/datasets/coco.py", line 633, in build
[rank1]: dataset = CocoDetection(img_folder, ann_file,
[rank1]: File "/private/home/wuhao/Open-GroundingDino/datasets/coco.py", line 329, in init
[rank1]: super(CocoDetection, self).init(img_folder, ann_file)
[rank1]: File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/site-packages/torchvision/datasets/coco.py", line 37, in init
[rank1]: self.coco = COCO(annFile)
[rank1]: File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/site-packages/pycocotools/coco.py", line 82, in init
[rank1]: dataset = json.load(f)
[rank1]: File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/json/init.py", line 293, in load
[rank1]: return loads(fp.read(),
[rank1]: File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/json/init.py", line 346, in loads
[rank1]: return _default_decoder.decode(s)
[rank1]: File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/json/decoder.py", line 340, in decode
[rank1]: raise JSONDecodeError("Extra data", s, end)
[rank1]: json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 1489)
== total images: 15548
DEBUG 2024-12-03 16:02:37,054 | build dataset, done.
DEBUG 2024-12-03 16:02:37,054 | number of training dataset: 1, samples: 15548
/private/home/wuhao/Open-GroundingDino/datasets/rico/combined/ /private/home/wuhao/Open-GroundingDino/datasets/screen_annotation/json1_val.jsonl
loading annotations into memory...
[rank0]: Traceback (most recent call last):
[rank0]: File "/private/home/wuhao/Open-GroundingDino/main.py", line 372, in
[rank0]: main(args)
[rank0]: File "/private/home/wuhao/Open-GroundingDino/main.py", line 187, in main
[rank0]: dataset_val = build_dataset(image_set='val', args=args, datasetinfo=dataset_meta["val"][0])
[rank0]: File "/private/home/wuhao/Open-GroundingDino/datasets/init.py", line 19, in build_dataset
[rank0]: return build_coco(image_set, args, datasetinfo)
[rank0]: File "/private/home/wuhao/Open-GroundingDino/datasets/coco.py", line 633, in build
[rank0]: dataset = CocoDetection(img_folder, ann_file,
[rank0]: File "/private/home/wuhao/Open-GroundingDino/datasets/coco.py", line 329, in init
[rank0]: super(CocoDetection, self).init(img_folder, ann_file)
[rank0]: File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/site-packages/torchvision/datasets/coco.py", line 37, in init
[rank0]: self.coco = COCO(annFile)
[rank0]: File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/site-packages/pycocotools/coco.py", line 82, in init
[rank0]: dataset = json.load(f)
[rank0]: File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/json/init.py", line 293, in load
[rank0]: return loads(fp.read(),
[rank0]: File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/json/init.py", line 346, in loads
[rank0]: return _default_decoder.decode(s)
[rank0]: File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/json/decoder.py", line 340, in decode
[rank0]: raise JSONDecodeError("Extra data", s, end)
[rank0]: json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 1489)
[rank0]:[W1203 16:02:37.115402107 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
W1203 16:02:38.418425 131446 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 131549 closing signal SIGTERM
E1203 16:02:38.632854 131446 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 1 (pid: 131550) of binary: /private/home/wuhao/miniconda3/envs/groundingdino/bin/python
Traceback (most recent call last):
File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/site-packages/torch/distributed/launch.py", line 208, in
main()
File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/site-packages/typing_extensions.py", line 2853, in wrapper
return arg(*args, **kwargs)
File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/site-packages/torch/distributed/launch.py", line 204, in main
launch(args)
File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/site-packages/torch/distributed/launch.py", line 189, in launch
run(args)
File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/site-packages/torch/distributed/run.py", line 910, in run
elastic_launch(
File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 138, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/private/home/wuhao/miniconda3/envs/groundingdino/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
main.py FAILED
Failures:
<NO_OTHER_FAILURES>
Root Cause (first observed failure):
[0]:
time : 2024-12-03_16:02:38
host : dev-intern-intern1-ikz3-pod-0
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 131550)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
My JSON file has no issues, why is this happening
The text was updated successfully, but these errors were encountered: