for my thesis I’m working on Neural A* algorithm (https://github.com/omron-sinicx/neural-astar) and now my goal is use the algorithm on real maps provided by (https://github.com/micheleantonazzi/gibson-env-utilities) as well as the code to generate Voronoi graph. The main goal is to understand how the algorithm behaves with real enviroments in the context of robot applications.
This is my fork: https://github.com/SamuelMastrelli/neural-astar
I resized the images to 300×300 because of hardware (i have 16 gb ram and a 1050 ti 4 gb)
When i try to run the train_maps.py script, i receive this error:
scripts/train_maps.py:21: UserWarning:
The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
@hydra.main(config_path="config", config_name="train_maps")
torch.Size([5, 1, 300, 300]) torch.Size([5, 1, 300, 300]) torch.Size([5, 1, 300, 300]) torch.Size([5, 1, 300, 300])
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
| Name | Type | Params
-----------------------------------------------
0 | planner | NeuralAstar | 391 K
1 | vanilla_astar | VanillaAstar | 9
-----------------------------------------------
391 K Trainable params
18 Non-trainable params
391 K Total params
1.566 Total estimated model params size (MB)
Sanity Checking: 0it [00:00, ?it/s]C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagespytorch_lightningtrainerconnectorsdata_connector.py:224: PossibleUserWarning: The dataloader, val_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 4 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
rank_zero_warn(
Sanity Checking DataLoader 0: 0%| | 0/2 [00:00<?, ?it/s]C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagestorchfunctional.py:478: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at C:actions-runner_workpytorchpytorchbuilderwindowspytorchatensrcATennativeTensorShape.cpp:2895.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
Sanity Checking DataLoader 0: 50%|##### | 1/2 [00:08<00:08, 8.09s/it]C:actions-runner_workpytorchpytorchbuilderwindowspytorchatensrcATennativecudaIndexKernel.cu:91: block: [0,0,0], thread: [0,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:actions-runner_workpytorchpytorchbuilderwindowspytorchatensrcATennativecudaIndexKernel.cu:91: block: [0,0,0], thread: [1,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:actions-runner_workpytorchpytorchbuilderwindowspytorchatensrcATennativecudaIndexKernel.cu:91: block: [0,0,0], thread: [4,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
Error executing job with overrides: []
Traceback (most recent call last):
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagespytorch_lightningtrainercall.py", line 38, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagespytorch_lightningtrainertrainer.py", line 645, in _fit_impl
self._run(model, ckpt_path=self.ckpt_path)
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagespytorch_lightningtrainertrainer.py", line 1098, in _run
results = self._run_stage()
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagespytorch_lightningtrainertrainer.py", line 1177, in _run_stage
self._run_train()
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagespytorch_lightningtrainertrainer.py", line 1190, in _run_train
self._run_sanity_check()
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagespytorch_lightningtrainertrainer.py", line 1262, in _run_sanity_check
val_loop.run()
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagespytorch_lightningloopsloop.py", line 199, in run
self.advance(*args, **kwargs)
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagespytorch_lightningloopsdataloaderevaluation_loop.py", line 152, in advance
dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagespytorch_lightningloopsloop.py", line 199, in run
self.advance(*args, **kwargs)
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagespytorch_lightningloopsepochevaluation_epoch_loop.py", line 137, in advance
output = self._evaluation_step(**kwargs)
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagespytorch_lightningloopsepochevaluation_epoch_loop.py", line 234, in _evaluation_step
output = self.trainer._call_strategy_hook(hook_name, *kwargs.values())
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagespytorch_lightningtrainertrainer.py", line 1480, in _call_strategy_hook
output = fn(*args, **kwargs)
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagespytorch_lightningstrategiesstrategy.py", line 390, in validation_step
return self.model.validation_step(*args, **kwargs)
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagesneural_astarutilstraining.py", line 68, in validation_step
outputs = self.forward(map_designs, start_maps, goal_maps)
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagesneural_astarutilstraining.py", line 53, in forward
return self.planner(map_designs, start_maps, goal_maps)
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagestorchnnmodulesmodule.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagesneural_astarplannerastar.py", line 207, in forward
return self.perform_astar(
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagesneural_astarplannerastar.py", line 63, in perform_astar
astar_outputs = astar(
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagestorchnnmodulesmodule.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagesneural_astarplannerdifferentiable_astar.py", line 258, in forward
path_maps = backtrack(start_maps, goal_maps, parents, t)
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagesneural_astarplannerdifferentiable_astar.py", line 125, in backtrack
path_maps.view(num_samples, -1)[range(num_samples), loc] = 1
RuntimeError: CUDA error: device-side assert triggered
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "scripts/train_maps.py", line 57, in main
trainer.fit(module, train_loader, val_loader)
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagespytorch_lightningtrainertrainer.py", line 603, in fit
call._call_and_handle_interrupt(
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagespytorch_lightningtrainercall.py", line 63, in _call_and_handle_interrupt
trainer._teardown()
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagespytorch_lightningtrainertrainer.py", line 1161, in _teardown
self.strategy.teardown()
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagespytorch_lightningstrategiesstrategy.py", line 496, in teardown
self.lightning_module.cpu()
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packageslightning_liteutilitiesdevice_dtype_mixin.py", line 78, in cpu
return super().cpu()
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagestorchnnmodulesmodule.py", line 738, in cpu
return self._apply(lambda t: t.cpu())
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagestorchnnmodulesmodule.py", line 579, in _apply
module._apply(fn)
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagestorchnnmodulesmodule.py", line 579, in _apply
module._apply(fn)
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagestorchnnmodulesmodule.py", line 602, in _apply
param_applied = fn(param)
File "C:UsersUserDesktopuniTesineural-astar.venvlibsite-packagestorchnnmodulesmodule.py", line 738, in <lambda>
return self._apply(lambda t: t.cpu())
RuntimeError: CUDA error: device-side assert triggered
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
I’m using windows and git bash