unable to train on develop due to compile error (gcc version?)
I haven't been able to run the compiled version of the MVP on TREX (from develop branch, commit c09648b2)
When I run bash scripts/train/small_train.sh experiment=base location=trex, I get compilation errors.
While I haven't spent too much time investigating yet, it seems to me that there might be a missing dependency for GCC (default version on TREX is 8.5, I tried using module load gcc/14.2 and 10.2 to no avail). Am I missing something from my config?
I also tried deleting the torch_inductor_cache, reinstalling pixi env
Here is the error log:
load gcc/10.2.0 : OK
Error executing job with overrides: ['trainer.max_epochs=1', 'trainer.limit_train_batches=1', 'trainer.limit_val_batches=1', 'experiment=base', 'experiment_name=tmp']
Traceback (most recent call last):
File "/work/scratch/env/prissij/git/releo_mvp_2/train.py", line 30, in main
trainer, training_module, dm = instantiate_hydra(config, callbacks, loggers)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/work/scratch/env/prissij/git/releo_mvp_2/src/releo_mvp/training/hydra.py", line 166, in instantiate_hydra
n = get_num_input_channels(cached_sample, config.input_modalities)
File "/work/scratch/env/prissij/git/releo_mvp_2/src/releo_mvp/data/sample_utils.py", line 384, in get_num_input_channels
embedded_batch, _ = sample_position_embedding(
~~~~~~~~~~~~~~~~~~~~~~~~~^
batch, get_description(sub_sample), context_embedding
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_dynamo/eval_frame.py", line 574, in _fn
return fn(*args, **kwargs)
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1380, in __call__
return self._torchdynamo_orig_callable(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
frame, cache_entry, self.hooks, frame_state, skip=1
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1164, in __call__
result = self._inner_convert(
frame, cache_entry, hooks, frame_state, skip=skip + 1
)
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 547, in __call__
return _compile(
frame.f_code,
...<14 lines>...
skip=skip + 1,
)
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 986, in _compile
guarded_code = compile_inner(code, one_graph, hooks, transform)
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 715, in compile_inner
return _compile_inner(code, one_graph, hooks, transform)
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_utils_internal.py", line 95, in wrapper_function
return function(*args, **kwargs)
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 750, in _compile_inner
out_code = transform_code_object(code, transform)
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_dynamo/bytecode_transformation.py", line 1361, in transform_code_object
transformations(instructions, code_options)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 231, in _fn
return fn(*args, **kwargs)
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 662, in transform
tracer.run()
~~~~~~~~~~^^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_dynamo/symbolic_convert.py", line 2868, in run
super().run()
~~~~~~~~~~~^^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_dynamo/symbolic_convert.py", line 1052, in run
while self.step():
~~~~~~~~~^^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_dynamo/symbolic_convert.py", line 962, in step
self.dispatch_table[inst.opcode](self, inst)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_dynamo/symbolic_convert.py", line 3048, in RETURN_VALUE
self._return(inst)
~~~~~~~~~~~~^^^^^^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_dynamo/symbolic_convert.py", line 3033, in _return
self.output.compile_subgraph(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
self,
^^^^^
...<2 lines>...
),
^^
)
^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_dynamo/output_graph.py", line 1136, in compile_subgraph
self.compile_and_call_fx_graph(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
tx, pass2.graph_output_vars(), root, output_replacements
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_dynamo/output_graph.py", line 1382, in compile_and_call_fx_graph
compiled_fn = self.call_user_compiler(gm)
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_dynamo/output_graph.py", line 1432, in call_user_compiler
return self._call_user_compiler(gm)
~~~~~~~~~~~~~~~~~~~~~~~~^^^^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_dynamo/output_graph.py", line 1483, in _call_user_compiler
raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
e.__traceback__
) from None
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_dynamo/output_graph.py", line 1462, in _call_user_compiler
compiled_fn = compiler_fn(gm, self.example_inputs())
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_dynamo/repro/after_dynamo.py", line 130, in __call__
compiled_gm = compiler_fn(gm, example_inputs)
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/__init__.py", line 2340, in __call__
return compile_fx(model_, inputs_, config_patches=self.config)
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 1863, in compile_fx
return aot_autograd(
~~~~~~~~~~~~~
...<6 lines>...
cudagraphs=cudagraphs,
~~~~~~~~~~~~~~~~~~~~~~
)(model_, example_inputs_)
~^^^^^^^^^^^^^^^^^^^^^^^^^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_dynamo/backends/common.py", line 83, in __call__
cg = aot_module_simplified(gm, example_inputs, **self.kwargs)
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_functorch/aot_autograd.py", line 1155, in aot_module_simplified
compiled_fn = dispatch_and_compile()
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_functorch/aot_autograd.py", line 1131, in dispatch_and_compile
compiled_fn, _ = create_aot_dispatcher_function(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
functional_call,
^^^^^^^^^^^^^^^^
...<3 lines>...
shape_env,
^^^^^^^^^^
)
^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_functorch/aot_autograd.py", line 580, in create_aot_dispatcher_function
return _create_aot_dispatcher_function(
flat_fn, fake_flat_args, aot_config, fake_mode, shape_env
)
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_functorch/aot_autograd.py", line 830, in _create_aot_dispatcher_function
compiled_fn, fw_metadata = compiler_fn(
~~~~~~~~~~~^
flat_fn,
^^^^^^^^
...<2 lines>...
fw_metadata=fw_metadata,
^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 203, in aot_dispatch_base
compiled_fw = compiler(fw_module, updated_flat_args)
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_functorch/aot_autograd.py", line 489, in __call__
return self.compiler_fn(gm, example_inputs)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 1741, in fw_compiler_base
return inner_compile(
gm,
...<5 lines>...
boxed_forward_device_index=forward_device,
)
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 569, in compile_fx_inner
return wrap_compiler_debug(_compile_fx_inner, compiler_name="inductor")(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
gm,
^^^
example_inputs,
^^^^^^^^^^^^^^^
**kwargs,
^^^^^^^^^
)
^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_dynamo/repro/after_aot.py", line 102, in debug_wrapper
inner_compiled_fn = compiler_fn(gm, example_inputs)
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 685, in _compile_fx_inner
mb_compiled_graph = fx_codegen_and_compile(
gm, example_inputs, inputs_to_check, **graph_kwargs
)
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 1129, in fx_codegen_and_compile
return scheme.codegen_and_compile(gm, example_inputs, inputs_to_check, graph_kwargs)
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 1044, in codegen_and_compile
compiled_fn = graph.compile_to_module().call
~~~~~~~~~~~~~~~~~~~~~~~^^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_inductor/graph.py", line 2027, in compile_to_module
return self._compile_to_module()
~~~~~~~~~~~~~~~~~~~~~~~^^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_inductor/graph.py", line 2068, in _compile_to_module
mod = PyCodeCache.load_by_key_path(
key,
...<2 lines>...
attrs={**self.constants, **self.torchbind_constants},
)
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_inductor/codecache.py", line 2759, in load_by_key_path
mod = _reload_python_module(key, path)
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_inductor/runtime/compile_tasks.py", line 45, in _reload_python_module
exec(code, mod.__dict__, mod.__dict__)
~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mp/prissij/git/releo_mvp_2/scripts/train/../../torch_inductor_cache/u7/cu7fryskb3yswhlmmhb6w4tn2mtm4bkndlaukkh2m3raschic3w5.py", line 511, in <module>
async_compile.wait(globals())
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_inductor/async_compile.py", line 305, in wait
scope[key] = result.result()
~~~~~~~~~~~~~^^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_inductor/codecache.py", line 3253, in result
return self.result_fn()
~~~~~~~~~~~~~~^^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_inductor/codecache.py", line 2254, in future
result = get_result()
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_inductor/codecache.py", line 2044, in load_fn
future.result()
~~~~~~~~~~~~~^^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
~~~~~~~~~~~~~~~~~^^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/concurrent/futures/thread.py", line 59, in run
result = self.fn(*self.args, **self.kwargs)
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_inductor/codecache.py", line 2085, in _worker_compile_cpp
cpp_builder.build()
~~~~~~~~~~~~~~~~~^^
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_inductor/cpp_builder.py", line 1548, in build
status = run_compile_cmd(build_cmd, cwd=_build_tmp_dir)
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_inductor/cpp_builder.py", line 349, in run_compile_cmd
return _run_compile_cmd(cmd_line, cwd)
File "/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/_inductor/cpp_builder.py", line 343, in _run_compile_cmd
raise exc.CppCompileError(cmd, output) from e
torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
CppCompileError: C++ compile error
Command:
/work/softs/rh8/spack_install/linux-rhel8-x86_64/gcc-8.5.0/gcc-10.2.0-k3n2xho5b5377wjzyus5qglk5cfprjka/bin/g++ /home/mp/prissij/git/releo_mvp_2/scripts/train/../../torch_inductor_cache/j4/cj4o74hujmiuc37cdyu6pg6xe65dnsxn5kdjk6lvh6tah2vdzdfp.cpp -D TORCH_INDUCTOR_CPP_WRAPPER -D STANDALONE_TORCH_HEADER -D C10_USING_CUSTOM_GENERATED_MACROS -D CPU_CAPABILITY_AVX2 -shared -fPIC -O3 -DNDEBUG -fno-trapping-math -funsafe-math-optimizations -ffinite-math-only -fno-signed-zeros -fno-math-errno -fexcess-precision=fast -fno-finite-math-only -fno-unsafe-math-optimizations -ffp-contract=off -fno-tree-loop-vectorize -march=native -Wall -std=c++17 -Wno-unused-variable -Wno-unknown-pragmas -fopenmp -I/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/include/python3.13 -I/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/include -I/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/include/torch/csrc/api/include -I/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/include/TH -I/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/include/THC -mavx2 -mfma -mf16c -D_GLIBCXX_USE_CXX11_ABI=0 -ltorch -ltorch_cpu -ltorch_python -lgomp -L/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib -L/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/lib -o /home/mp/prissij/git/releo_mvp_2/scripts/train/../../torch_inductor_cache/j4/cj4o74hujmiuc37cdyu6pg6xe65dnsxn5kdjk6lvh6tah2vdzdfp.so
Output:
/home/mp/prissij/git/releo_mvp_2/scripts/train/../../torch_inductor_cache/j4/cj4o74hujmiuc37cdyu6pg6xe65dnsxn5kdjk6lvh6tah2vdzdfp.cpp: In lambda function:
/home/mp/prissij/git/releo_mvp_2/scripts/train/../../torch_inductor_cache/j4/cj4o74hujmiuc37cdyu6pg6xe65dnsxn5kdjk6lvh6tah2vdzdfp.cpp:159:60: error: no matching function for call to 'cos(at::vec::CPU_CAPABILITY::Vectorized<float>&)'
159 | auto tmp27 = std::cos(tmp26);
| ^
In file included from /usr/include/features.h:438,
from /work/softs/rh8/spack_install/linux-rhel8-x86_64/gcc-8.5.0/gcc-10.2.0-k3n2xho5b5377wjzyus5qglk5cfprjka/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/x86_64-pc-linux-gnu/bits/os_defines.h:39,
from /work/softs/rh8/spack_install/linux-rhel8-x86_64/gcc-8.5.0/gcc-10.2.0-k3n2xho5b5377wjzyus5qglk5cfprjka/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/x86_64-pc-linux-gnu/bits/c++config.h:518,
from /work/softs/rh8/spack_install/linux-rhel8-x86_64/gcc-8.5.0/gcc-10.2.0-k3n2xho5b5377wjzyus5qglk5cfprjka/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/utility:68,
from /work/softs/rh8/spack_install/linux-rhel8-x86_64/gcc-8.5.0/gcc-10.2.0-k3n2xho5b5377wjzyus5qglk5cfprjka/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/algorithm:60,
from /home/mp/prissij/git/releo_mvp_2/scripts/train/../../torch_inductor_cache/2r/c2rnilspx43ivnzu4uieul65kx65dfhfbptbh5og4wk6rqebuxoo.h:3,
from /home/mp/prissij/git/releo_mvp_2/scripts/train/../../torch_inductor_cache/j4/cj4o74hujmiuc37cdyu6pg6xe65dnsxn5kdjk6lvh6tah2vdzdfp.cpp:2:
/usr/include/bits/mathcalls.h:62:1: note: candidate: 'double cos(double)'
62 | __MATHCALL_VEC (cos,, (_Mdouble_ __x));
| ^~~~~~~~~~~~~~
In file included from /work/softs/rh8/spack_install/linux-rhel8-x86_64/gcc-8.5.0/gcc-10.2.0-k3n2xho5b5377wjzyus5qglk5cfprjka/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/cmath:45,
from /home/mp/prissij/git/releo_mvp_2/scripts/train/../../torch_inductor_cache/2r/c2rnilspx43ivnzu4uieul65kx65dfhfbptbh5og4wk6rqebuxoo.h:5,
from /home/mp/prissij/git/releo_mvp_2/scripts/train/../../torch_inductor_cache/j4/cj4o74hujmiuc37cdyu6pg6xe65dnsxn5kdjk6lvh6tah2vdzdfp.cpp:2:
/usr/include/bits/mathcalls.h:62:1: note: no known conversion for argument 1 from 'at::vec::CPU_CAPABILITY::Vectorized<float>' to 'double'
62 | __MATHCALL_VEC (cos,, (_Mdouble_ __x));
| ^
In file included from /home/mp/prissij/git/releo_mvp_2/scripts/train/../../torch_inductor_cache/2r/c2rnilspx43ivnzu4uieul65kx65dfhfbptbh5og4wk6rqebuxoo.h:5,
from /home/mp/prissij/git/releo_mvp_2/scripts/train/../../torch_inductor_cache/j4/cj4o74hujmiuc37cdyu6pg6xe65dnsxn5kdjk6lvh6tah2vdzdfp.cpp:2:
/work/softs/rh8/spack_install/linux-rhel8-x86_64/gcc-8.5.0/gcc-10.2.0-k3n2xho5b5377wjzyus5qglk5cfprjka/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/cmath:184:3: note: candidate: 'constexpr float std::cos(float)'
184 | cos(float __x)
| ^~~
/work/softs/rh8/spack_install/linux-rhel8-x86_64/gcc-8.5.0/gcc-10.2.0-k3n2xho5b5377wjzyus5qglk5cfprjka/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/cmath:184:13: note: no known conversion for argument 1 from 'at::vec::CPU_CAPABILITY::Vectorized<float>' to 'float'
184 | cos(float __x)
| ~~~~~~^~~
/work/softs/rh8/spack_install/linux-rhel8-x86_64/gcc-8.5.0/gcc-10.2.0-k3n2xho5b5377wjzyus5qglk5cfprjka/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/cmath:188:3: note: candidate: 'constexpr long double std::cos(long double)'
188 | cos(long double __x)
| ^~~
/work/softs/rh8/spack_install/linux-rhel8-x86_64/gcc-8.5.0/gcc-10.2.0-k3n2xho5b5377wjzyus5qglk5cfprjka/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/cmath:188:19: note: no known conversion for argument 1 from 'at::vec::CPU_CAPABILITY::Vectorized<float>' to 'long double'
188 | cos(long double __x)
| ~~~~~~~~~~~~^~~
/work/softs/rh8/spack_install/linux-rhel8-x86_64/gcc-8.5.0/gcc-10.2.0-k3n2xho5b5377wjzyus5qglk5cfprjka/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/cmath:196:5: note: candidate: 'template<class _Tp> constexpr typename __gnu_cxx::__enable_if<std::__is_integer<_Tp>::__value, double>::__type std::cos(_Tp)'
196 | cos(_Tp __x)
| ^~~
/work/softs/rh8/spack_install/linux-rhel8-x86_64/gcc-8.5.0/gcc-10.2.0-k3n2xho5b5377wjzyus5qglk5cfprjka/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/cmath:196:5: note: template argument deduction/substitution failed:
/work/softs/rh8/spack_install/linux-rhel8-x86_64/gcc-8.5.0/gcc-10.2.0-k3n2xho5b5377wjzyus5qglk5cfprjka/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/cmath: In substitution of 'template<class _Tp> constexpr typename __gnu_cxx::__enable_if<std::__is_integer<_Tp>::__value, double>::__type std::cos(_Tp) [with _Tp = at::vec::CPU_CAPABILITY::Vectorized<float>]':
/home/mp/prissij/git/releo_mvp_2/scripts/train/../../torch_inductor_cache/j4/cj4o74hujmiuc37cdyu6pg6xe65dnsxn5kdjk6lvh6tah2vdzdfp.cpp:159:60: required from here
/work/softs/rh8/spack_install/linux-rhel8-x86_64/gcc-8.5.0/gcc-10.2.0-k3n2xho5b5377wjzyus5qglk5cfprjka/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/cmath:196:5: error: no type named '__type' in 'struct __gnu_cxx::__enable_if<false, double>'
In file included from /work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/include/c10/util/complex.h:3,
from /work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/include/ATen/NumericUtils.h:14,
from /home/mp/prissij/git/releo_mvp_2/scripts/train/../../torch_inductor_cache/2r/c2rnilspx43ivnzu4uieul65kx65dfhfbptbh5og4wk6rqebuxoo.h:19,
from /home/mp/prissij/git/releo_mvp_2/scripts/train/../../torch_inductor_cache/j4/cj4o74hujmiuc37cdyu6pg6xe65dnsxn5kdjk6lvh6tah2vdzdfp.cpp:2:
/work/softs/rh8/spack_install/linux-rhel8-x86_64/gcc-8.5.0/gcc-10.2.0-k3n2xho5b5377wjzyus5qglk5cfprjka/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/complex:736:5: note: candidate: 'template<class _Tp> std::complex<_Tp> std::cos(const std::complex<_Tp>&)'
736 | cos(const complex<_Tp>& __z) { return __complex_cos(__z.__rep()); }
| ^~~
/work/softs/rh8/spack_install/linux-rhel8-x86_64/gcc-8.5.0/gcc-10.2.0-k3n2xho5b5377wjzyus5qglk5cfprjka/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/complex:736:5: note: template argument deduction/substitution failed:
/home/mp/prissij/git/releo_mvp_2/scripts/train/../../torch_inductor_cache/j4/cj4o74hujmiuc37cdyu6pg6xe65dnsxn5kdjk6lvh6tah2vdzdfp.cpp:159:60: note: 'at::vec::CPU_CAPABILITY::Vectorized<float>' is not derived from 'const std::complex<_Tp>'
159 | auto tmp27 = std::cos(tmp26);
| ^
In file included from /work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/include/c10/util/complex.h:665,
from /work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/include/ATen/NumericUtils.h:14,
from /home/mp/prissij/git/releo_mvp_2/scripts/train/../../torch_inductor_cache/2r/c2rnilspx43ivnzu4uieul65kx65dfhfbptbh5og4wk6rqebuxoo.h:19,
from /home/mp/prissij/git/releo_mvp_2/scripts/train/../../torch_inductor_cache/j4/cj4o74hujmiuc37cdyu6pg6xe65dnsxn5kdjk6lvh6tah2vdzdfp.cpp:2:
/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/include/c10/util/complex_math.h:168:40: note: candidate: 'template<class T> c10::complex<U> c10_complex_math::cos(const c10::complex<U>&)'
168 | C10_HOST_DEVICE inline c10::complex<T> cos(const c10::complex<T>& x) {
| ^~~
/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/include/c10/util/complex_math.h:168:40: note: template argument deduction/substitution failed:
/home/mp/prissij/git/releo_mvp_2/scripts/train/../../torch_inductor_cache/j4/cj4o74hujmiuc37cdyu6pg6xe65dnsxn5kdjk6lvh6tah2vdzdfp.cpp:159:60: note: 'at::vec::CPU_CAPABILITY::Vectorized<float>' is not derived from 'const c10::complex<U>'
159 | auto tmp27 = std::cos(tmp26);
| ^
In file included from /home/mp/prissij/git/releo_mvp_2/scripts/train/../../torch_inductor_cache/2r/c2rnilspx43ivnzu4uieul65kx65dfhfbptbh5og4wk6rqebuxoo.h:25,
from /home/mp/prissij/git/releo_mvp_2/scripts/train/../../torch_inductor_cache/j4/cj4o74hujmiuc37cdyu6pg6xe65dnsxn5kdjk6lvh6tah2vdzdfp.cpp:2:
/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/include/c10/util/BFloat16-math.h:110:10: note: candidate: 'template<class T, typename std::enable_if<is_reduced_floating_point_v<T>, int>::type <anonymous> > T std::cos(T)'
110 | inline T cos(T a) {
| ^~~
/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/include/c10/util/BFloat16-math.h:110:10: note: template argument deduction/substitution failed:
/work/scratch/env/prissij/git/releo_mvp_2/.pixi/envs/dev/lib/python3.13/site-packages/torch/include/c10/util/BFloat16-math.h:109:70: error: no type named 'type' in 'struct std::enable_if<false, int>'
109 | typename std::enable_if_t<is_reduced_floating_point_v<T>, int> = 0>
| ^
Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
You can suppress this exception and fall back to eager by setting:
import torch._dynamo
torch._dynamo.config.suppress_errors = True
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Edited by Julien Prissimitzis