Meet error when install about fairseq

[wwxxyy1996]

Dear Authors,

Thank you very much for your excellent work. I have a question about the environment setup. I’m using a requirements.txt file to configure the environment but am encountering an issue:

ERROR: Could not build wheels for fairseq, which is required to install pyproject.toml-based projects.

I suspect this might be related to the versions of gcc and g++(I have tried the version gcc/g++ 11, 12, 13). Despite spending a whole day adjusting these versions, the problem persists. However, when I downloaded the latest version of fairseq from its GitHub repository (https://github.com/facebookresearch/fairseq) and used Python 3.11 to install it, the installation was successful.

Could you please let me know which version of fairseq and which versions of gcc and g++ you used? Or do you have any suggestions on this matter? Thank you very much! I would greatly appreciate it.

Best wishes,
Xinyi

[wwxxyy1996]

Here is the error:

Building editable for fairseq (pyproject.toml) ... error
error: subprocess-exited-with-error

× Building editable for fairseq (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [111 lines of output]
running editable_wheel
creating /tmp/pip-wheel-6eepou0u/.tmp-f3ihlwqb/fairseq.egg-info
writing /tmp/pip-wheel-6eepou0u/.tmp-f3ihlwqb/fairseq.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-wheel-6eepou0u/.tmp-f3ihlwqb/fairseq.egg-info/dependency_links.txt
writing entry points to /tmp/pip-wheel-6eepou0u/.tmp-f3ihlwqb/fairseq.egg-info/entry_points.txt
writing requirements to /tmp/pip-wheel-6eepou0u/.tmp-f3ihlwqb/fairseq.egg-info/requires.txt
writing top-level names to /tmp/pip-wheel-6eepou0u/.tmp-f3ihlwqb/fairseq.egg-info/top_level.txt
writing manifest file '/tmp/pip-wheel-6eepou0u/.tmp-f3ihlwqb/fairseq.egg-info/SOURCES.txt'
reading manifest file '/tmp/pip-wheel-6eepou0u/.tmp-f3ihlwqb/fairseq.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file '/tmp/pip-wheel-6eepou0u/.tmp-f3ihlwqb/fairseq.egg-info/SOURCES.txt'
creating '/tmp/pip-wheel-6eepou0u/.tmp-f3ihlwqb/fairseq-1.0.0a0+b2ebdf7.dist-info'
creating /tmp/pip-wheel-6eepou0u/.tmp-f3ihlwqb/fairseq-1.0.0a0+b2ebdf7.dist-info/WHEEL
running build_py
running build_ext
building 'fairseq.libbleu' extension
creating /tmp/tmps57klyy9.build-temp/fairseq
creating /tmp/tmps57klyy9.build-temp/fairseq/clib
creating /tmp/tmps57klyy9.build-temp/fairseq/clib/libbleu
gcc -pthread -B /home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/psxxw13/anaconda3/envs/biomedgpt/include/python3.7m -c fairseq/clib/libbleu/libbleu.cpp -o /tmp/tmps57klyy9.build-temp/fairseq/clib/libbleu/libbleu.o -std=c++11 -O3
cc1plus: warning: command-line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
gcc -pthread -B /home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/psxxw13/anaconda3/envs/biomedgpt/include/python3.7m -c fairseq/clib/libbleu/module.cpp -o /tmp/tmps57klyy9.build-temp/fairseq/clib/libbleu/module.o -std=c++11 -O3
cc1plus: warning: command-line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
creating /tmp/tmpglt7n1hg.build-lib/fairseq
g++ -pthread -B /home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat -Wl,--sysroot=/ -pthread -shared -B /home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat -L/home/psxxw13/anaconda3/envs/biomedgpt/lib -Wl,-rpath=/home/psxxw13/anaconda3/envs/biomedgpt/lib -Wl,--no-as-needed -Wl,--sysroot=/ /tmp/tmps57klyy9.build-temp/fairseq/clib/libbleu/libbleu.o /tmp/tmps57klyy9.build-temp/fairseq/clib/libbleu/module.o -o /tmp/tmpglt7n1hg.build-lib/fairseq/libbleu.cpython-37m-x86_64-linux-gnu.so
/home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat/ld: warning: /home/psxxw13/anaconda3/envs/biomedgpt/lib/libstdc++.so: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat/ld: warning: /home/psxxw13/anaconda3/envs/biomedgpt/lib/libstdc++.so: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat/ld: warning: /lib/x86_64-linux-gnu/libm.so.6: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0008002
/home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat/ld: /lib/x86_64-linux-gnu/libm.so.6: unknown type [0x13] section .relr.dyn' /home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat/ld: skipping incompatible /lib/x86_64-linux-gnu/libm.so.6 when searching for /lib/x86_64-linux-gnu/libm.so.6 /home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat/ld: cannot find /lib/x86_64-linux-gnu/libm.so.6 inside / /home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat/ld: warning: /lib/x86_64-linux-gnu/libmvec.so.1: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0008002 /home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat/ld: /lib/x86_64-linux-gnu/libmvec.so.1: unknown type [0x13] section .relr.dyn'
/home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat/ld: skipping incompatible /lib/x86_64-linux-gnu/libmvec.so.1 when searching for /lib/x86_64-linux-gnu/libmvec.so.1
/home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat/ld: cannot find /lib/x86_64-linux-gnu/libmvec.so.1 inside /
collect2: error: ld returned 1 exit status
Traceback (most recent call last):
File "/tmp/pip-build-env-7vvj_fql/overlay/lib/python3.7/site-packages/setuptools/_distutils/unixccompiler.py", line 266, in link
self.spawn(linker + ld_args)
File "/tmp/pip-build-env-7vvj_fql/overlay/lib/python3.7/site-packages/setuptools/_distutils/ccompiler.py", line 1041, in spawn
spawn(cmd, dry_run=self.dry_run, **kwargs)
File "/tmp/pip-build-env-7vvj_fql/overlay/lib/python3.7/site-packages/setuptools/_distutils/spawn.py", line 71, in spawn
"command {!r} failed with exit code {}".format(cmd, exitcode)
distutils.errors.DistutilsExecError: command '/usr/bin/g++' failed with exit code 1

  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/tmp/pip-build-env-7vvj_fql/overlay/lib/python3.7/site-packages/setuptools/command/editable_wheel.py", line 155, in run
      self._create_wheel_file(bdist_wheel)
    File "/tmp/pip-build-env-7vvj_fql/overlay/lib/python3.7/site-packages/setuptools/command/editable_wheel.py", line 344, in _create_wheel_file
      files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
    File "/tmp/pip-build-env-7vvj_fql/overlay/lib/python3.7/site-packages/setuptools/command/editable_wheel.py", line 267, in _run_build_commands
      self._run_build_subcommands()
    File "/tmp/pip-build-env-7vvj_fql/overlay/lib/python3.7/site-packages/setuptools/command/editable_wheel.py", line 294, in _run_build_subcommands
      self.run_command(name)
    File "/tmp/pip-build-env-7vvj_fql/overlay/lib/python3.7/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/tmp/pip-build-env-7vvj_fql/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 1234, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-7vvj_fql/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-7vvj_fql/overlay/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 84, in run
      _build_ext.run(self)
    File "/tmp/pip-build-env-7vvj_fql/overlay/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
      self.build_extensions()
    File "/tmp/pip-build-env-7vvj_fql/overlay/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
      self._build_extensions_serial()
    File "/tmp/pip-build-env-7vvj_fql/overlay/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
      self.build_extension(ext)
    File "/tmp/pip-build-env-7vvj_fql/overlay/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 246, in build_extension
      _build_ext.build_extension(self, ext)
    File "/tmp/pip-build-env-7vvj_fql/overlay/lib/python3.7/site-packages/Cython/Distutils/build_ext.py", line 135, in build_extension
      super(build_ext, self).build_extension(ext)
    File "/tmp/pip-build-env-7vvj_fql/overlay/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 582, in build_extension
      target_lang=language,
    File "/tmp/pip-build-env-7vvj_fql/overlay/lib/python3.7/site-packages/setuptools/_distutils/ccompiler.py", line 765, in link_shared_object
      target_lang,
    File "/tmp/pip-build-env-7vvj_fql/overlay/lib/python3.7/site-packages/setuptools/_distutils/unixccompiler.py", line 268, in link
      raise LinkError(msg)
  distutils.errors.LinkError: command '/usr/bin/g++' failed with exit code 1
  /tmp/pip-build-env-7vvj_fql/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py:988: _DebuggingTips: Problem in editable installation.
  !!
  
          ********************************************************************************
          An error happened while installing `fairseq` in editable mode.
  
          The following steps are recommended to help debug this problem:
  
          - Try to install the project normally, without using the editable mode.
            Does the error still persist?
            (If it does, try fixing the problem before attempting the editable mode).
          - If you are using binary extensions, make sure you have all OS-level
            dependencies installed (e.g. compilers, toolchains, binary libraries, ...).
          - Try the latest version of setuptools (maybe the error was already fixed).
          - If you (or your project dependencies) are using any setuptools extension
            or customization, make sure they support the editable mode.
  
          After following the steps above, if the problem still persists and
          you think this is related to how setuptools handles editable installations,
          please submit a reproducible example
          (see https://stackoverflow.com/help/minimal-reproducible-example) to:
  
              https://github.com/pypa/setuptools/issues
  
          See https://setuptools.pypa.io/en/latest/userguide/development_mode.html for details.
          ********************************************************************************
  
  !!
    cmd_obj.run()
  error: command '/usr/bin/g++' failed with exit code 1
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building editable for fairseq
Building wheel for distance (setup.py) ... done
Created wheel for distance: filename=Distance-0.1.3-py3-none-any.whl size=16257 sha256=84814ae4870ac67442dc8e04285fcc17adc6a89e923445bffe1e54d63a0b7616
Stored in directory: /home/psxxw13/.cache/pip/wheels/80/1f/f0/543667b99d48d533fb25ce9e4a05189276c96df9e560880cf3
Successfully built distance
Failed to build fairseq
ERROR: Could not build wheels for fairseq, which is required to install pyproject.toml-based projects

[wwxxyy1996]

Moreover, although I can install the latest version of fairseq, using it can not run the evaluation code successfully.

[taokz]

The code is executable using the modified version of Fairseq provided in this repository, not the vanilla Fairseq. Therefore, simply installing Fairseq will not work. Please first uninstall Fairseq, ensure that pip==21.2.4 is installed, and then run pip install -e ./fairseq/ from the BiomedGPT home directory.

[taokz]

I have converted BiomedGPT-B to be compatible with the Transformers library. You can refer to this Colab notebook for more details: link. You can run the example directly.

[wwxxyy1996]

The code is executable using the modified version of Fairseq provided in this repository, not the vanilla Fairseq. Therefore, simply installing Fairseq will not work. Please first uninstall Fairseq, ensure that pip==21.2.4 is installed, and then run pip install -e ./fairseq/ from the BiomedGPT home directory.

Thank you very much for your assistance. The information you provided has been very helpful, but I am still encountering difficulties with the installation. Would it be possible for you to share the versions of Python, GCC, G++, and CUDA used in your environment? I noticed that you have adapted BiomedGPT-B to be compatible with the Transformers library, which is incredibly useful for me. However, having the specific versions would greatly aid me in fine-tuning the model later on. I truly appreciate your help. Thank you once again!

[wwxxyy1996]

I have converted BiomedGPT-B to be compatible with the Transformers library. You can refer to this Colab notebook for more details: link. You can run the example directly.

Thank you very much! It is very useful for me!

[wwxxyy1996]

The code is executable using the modified version of Fairseq provided in this repository, not the vanilla Fairseq. Therefore, simply installing Fairseq will not work. Please first uninstall Fairseq, ensure that pip==21.2.4 is installed, and then run pip install -e ./fairseq/ from the BiomedGPT home directory.

When I following this instruction, the error is

ERROR: Command errored out with exit status 1:
command: /home/psxxw13/anaconda3/envs/biomedgpt/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/Data/tmp/BiomedGPT/fairseq/setup.py'"'"'; file='"'"'/Data/tmp/BiomedGPT/fairseq/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps
cwd: /Data/tmp/BiomedGPT/fairseq/
Complete output (74 lines):
running develop
running egg_info
creating fairseq.egg-info
writing fairseq.egg-info/PKG-INFO
writing dependency_links to fairseq.egg-info/dependency_links.txt
writing entry points to fairseq.egg-info/entry_points.txt
writing requirements to fairseq.egg-info/requires.txt
writing top-level names to fairseq.egg-info/top_level.txt
writing manifest file 'fairseq.egg-info/SOURCES.txt'
reading manifest file 'fairseq.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'fairseq.egg-info/SOURCES.txt'
running build_ext
building 'fairseq.libbleu' extension
creating build
creating build/temp.linux-x86_64-cpython-37
creating build/temp.linux-x86_64-cpython-37/fairseq
creating build/temp.linux-x86_64-cpython-37/fairseq/clib
creating build/temp.linux-x86_64-cpython-37/fairseq/clib/libbleu
gcc -pthread -B /home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/psxxw13/anaconda3/envs/biomedgpt/include/python3.7m -c fairseq/clib/libbleu/libbleu.cpp -o build/temp.linux-x86_64-cpython-37/fairseq/clib/libbleu/libbleu.o -std=c++11 -O3
cc1plus: warning: command-line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
gcc -pthread -B /home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/psxxw13/anaconda3/envs/biomedgpt/include/python3.7m -c fairseq/clib/libbleu/module.cpp -o build/temp.linux-x86_64-cpython-37/fairseq/clib/libbleu/module.o -std=c++11 -O3
cc1plus: warning: command-line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
creating build/lib.linux-x86_64-cpython-37
creating build/lib.linux-x86_64-cpython-37/fairseq
g++ -pthread -B /home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat -Wl,--sysroot=/ -pthread -shared -B /home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat -L/home/psxxw13/anaconda3/envs/biomedgpt/lib -Wl,-rpath=/home/psxxw13/anaconda3/envs/biomedgpt/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-cpython-37/fairseq/clib/libbleu/libbleu.o build/temp.linux-x86_64-cpython-37/fairseq/clib/libbleu/module.o -o build/lib.linux-x86_64-cpython-37/fairseq/libbleu.cpython-37m-x86_64-linux-gnu.so
/home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat/ld: warning: /home/psxxw13/anaconda3/envs/biomedgpt/lib/libstdc++.so: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat/ld: warning: /home/psxxw13/anaconda3/envs/biomedgpt/lib/libstdc++.so: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat/ld: warning: /lib/x86_64-linux-gnu/libm.so.6: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0008002
/home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat/ld: /lib/x86_64-linux-gnu/libm.so.6: unknown type [0x13] section .relr.dyn' /home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat/ld: skipping incompatible /lib/x86_64-linux-gnu/libm.so.6 when searching for /lib/x86_64-linux-gnu/libm.so.6 /home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat/ld: cannot find /lib/x86_64-linux-gnu/libm.so.6 inside / /home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat/ld: warning: /lib/x86_64-linux-gnu/libmvec.so.1: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0008002 /home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat/ld: /lib/x86_64-linux-gnu/libmvec.so.1: unknown type [0x13] section .relr.dyn'
/home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat/ld: skipping incompatible /lib/x86_64-linux-gnu/libmvec.so.1 when searching for /lib/x86_64-linux-gnu/libmvec.so.1
/home/psxxw13/anaconda3/envs/biomedgpt/compiler_compat/ld: cannot find /lib/x86_64-linux-gnu/libmvec.so.1 inside /
collect2: error: ld returned 1 exit status
/tmp/pip-build-env-1hxsjhb3/overlay/lib/python3.7/site-packages/setuptools/init.py:84: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
!!

        ********************************************************************************
        Requirements should be satisfied by a PEP 517 installer.
        If you are using pip, you can try `pip install --use-pep517`.
        ********************************************************************************

!!
  dist.fetch_build_eggs(dist.setup_requires)
/tmp/pip-build-env-1hxsjhb3/overlay/lib/python3.7/site-packages/setuptools/command/develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` and ``easy_install``.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://github.com/pypa/setuptools/issues/917 for details.
        ********************************************************************************

!!
  easy_install.initialize_options(self)
/tmp/pip-build-env-1hxsjhb3/overlay/lib/python3.7/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
error: command '/usr/bin/g++' failed with exit code 1
----------------------------------------

ERROR: Command errored out with exit status 1: /home/psxxw13/anaconda3/envs/biomedgpt/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/Data/tmp/BiomedGPT/fairseq/setup.py'"'"'; file='"'"'/Data/tmp/BiomedGPT/fairseq/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.