tensorflow-2.3.1 in conda

[SamuelLarkin]

How to I install tensorflow-2.3.1 in a conda environment on trixie?

I'm getting a missing dependency on GLIBCXX_3.4.21 and from what I can find, trixie is using GLIBCXX_3.4.19

Traceback (most recent call last):
  File "../TTS/bin/train_tacotron.py", line 22, in <module>
    from TTS.tts.utils.synthesis import synthesis
  File "/gpfs/projects/DT/mtp/Project_SamuelL/tts/Mozilla-TTS.git/TTS/tts/utils/synthesis.py", line 4, in <module>
    import tensorflow as tf
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/Mozilla-tts/lib/python3.8/site-packages/tensorflow/__init__.py", line 41, in <module>
    from tensorflow.python.tools import module_util as _module_util
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/Mozilla-tts/lib/python3.8/site-packages/tensorflow/python/__init__.py", line 40, in <module>
    from tensorflow.python.eager import context
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/Mozilla-tts/lib/python3.8/site-packages/tensorflow/python/eager/context.py", line 32, in <module>
    from tensorflow.core.framework import function_pb2
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/Mozilla-tts/lib/python3.8/site-packages/tensorflow/core/framework/function_pb2.py", line 7, in <module>
    from google.protobuf import descriptor as _descriptor
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/Mozilla-tts/lib/python3.8/site-packages/google/protobuf/descriptor.py", line 48, in <module>
    from google.protobuf.pyext import _message
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/Mozilla-tts/lib/python3.8/site-packages/google/protobuf/pyext/_message.cpython-38-x86_64-linux-gnu.so)

I would like to be able to create a conda environment with the following requirement.txt

conda create --name Mozilla-tts  --yes cudatoolkit=10.1  cudnn python=3.8
conda activate Mozilla-tts
pip install -r requirement.txt

Where requirement.txt is

torch==1.6.0
tensorflow-gpu>=2.3.0
numpy
scipy>=0.19.0
numba==0.48
librosa==0.7.2
phonemizer>=2.2.0
unidecode==0.4.20
tensorboardX
matplotlib
Pillow
flask
tqdm
inflect
bokeh==1.4.0
pysbd
pyworld
soundfile
nose==1.3.7
cardboardlint==1.3.0
pylint==2.5.3
gdown
umap-learn
cython
pyyaml

[fieldsa]

I was not able to reproduce the error you encountered earlier. Using the steps listed above, it succeeded to load tensorflow from CVMFS wheelhouse using pip install from conda env.

Note: see bottom - I am using conda-forge channel and perhaps that upstream packages were update since your last attempt.

[fieldsa@hn2 ~]$ module load miniconda3-4.8.2-gcc-9.2.0-sbqd2xu
[fieldsa@hn2 ~]$ conda create --name Mozilla-tts  --yes cudatoolkit=10.1  cudnn python=3.8
Collecting package metadata (current_repodata.json): done
Solving environment: done  

## Package Plan ##

  environment location: /home/fieldsa/.conda/envs/Mozilla-tts

  added / updated specs:   
    - cudatoolkit=10.1
    - cudnn
    - python=3.8


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cudatoolkit-10.1.243       |       h036e899_9       428.2 MB  conda-forge
    libffi-3.4.2               |       h7f98852_5          57 KB  conda-forge
    libnsl-2.0.0               |       h7f98852_0          31 KB  conda-forge
    openssl-3.0.0              |       h7f98852_2         2.9 MB  conda-forge
    python-3.8.12              |hf930737_2_cpython        26.2 MB  conda-forge
    setuptools-59.1.0          |   py38h578d9bd_0         1.0 MB  conda-forge
    ------------------------------------------------------------
                                           Total:       458.4 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-1_gnu
  ca-certificates    conda-forge/linux-64::ca-certificates-2021.10.8-ha878542_0
  cudatoolkit        conda-forge/linux-64::cudatoolkit-10.1.243-h036e899_9
  cudnn              conda-forge/linux-64::cudnn-7.6.5.32-hc0a50b0_1
  ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.36.1-hea4e1c9_2
  libffi             conda-forge/linux-64::libffi-3.4.2-h7f98852_5
  libgcc-ng          conda-forge/linux-64::libgcc-ng-11.2.0-h1d223b6_11
  libgomp            conda-forge/linux-64::libgomp-11.2.0-h1d223b6_11
  libnsl             conda-forge/linux-64::libnsl-2.0.0-h7f98852_0
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-11.2.0-he4da1e4_11
  libzlib            conda-forge/linux-64::libzlib-1.2.11-h36c2ea0_1013
  ncurses            conda-forge/linux-64::ncurses-6.2-h58526e2_4
  openssl            conda-forge/linux-64::openssl-3.0.0-h7f98852_2
  pip                conda-forge/noarch::pip-21.3.1-pyhd8ed1ab_0
  python             conda-forge/linux-64::python-3.8.12-hf930737_2_cpython
  python_abi         conda-forge/linux-64::python_abi-3.8-2_cp38
  readline           conda-forge/linux-64::readline-8.1-h46c0cb4_0
  setuptools         conda-forge/linux-64::setuptools-59.1.0-py38h578d9bd_0
  sqlite             conda-forge/linux-64::sqlite-3.36.0-h9cd32fc_2
  tk                 conda-forge/linux-64::tk-8.6.11-h27826a3_1
  wheel              conda-forge/noarch::wheel-0.37.0-pyhd8ed1ab_1
  xz                 conda-forge/linux-64::xz-5.2.5-h516909a_1
  zlib               conda-forge/linux-64::zlib-1.2.11-h36c2ea0_1013



Downloading and Extracting Packages
  tuptools-59.1.0    | 1.0 MB    | ###################################################################################### | 100%
openssl-3.0.0        | 2.9 MB    | ###################################################################################### | 100%
libnsl-2.0.0         | 31 KB     | ###################################################################################### | 100%
libffi-3.4.2         | 57 KB     | ###################################################################################### | 100%
python-3.8.12        | 26.2 MB   | ###################################################################################### | 100%
cudatoolkit-10.1.243 | 428.2 MB  | ###################################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: \ b'By downloading and using the CUDA Toolkit conda packages, you accept the terms and conditions of the C
UDA End User License Agreement (EULA): https://docs.nvidia.com/cuda/eula/index.html\n'
b'By downloading and using the cuDNN conda packages, you accept the terms and conditions of the NVIDIA cuDNN EULA -\n  https://do
cs.nvidia.com/deeplearning/cudnn/sla/index.html\n'                                                                             do
ne
#
# To activate this environment, use
#
#     $ conda activate Mozilla-tts
# 
..
[fieldsa@hn2 ~]$ . activate Mozilla-tts

(Mozilla-tts) [fieldsa@hn2 ~]$ pip install -r requirements-mozilla-tts.txt
Looking in links: /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/nix/avx512, /cvmfs/soft.computecanada.ca/custom/python/wh
eelhouse/nix/avx2, /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/nix/generic, /cvmfs/soft.computecanada.ca/custom/python/
wheelhouse/generic
Processing /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic/torch-1.6.0+computecanada-cp38-cp38-linux_x86_64.whl
Processing /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic/tensorflow_gpu-2.3.0+computecanada-cp38-cp38-linux_x86_6
4.whl  
Processing /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/nix/generic/numpy-1.20.2+computecanada-cp38-cp38-linux_x86_64.wh
l
Processing /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/nix/generic/scipy-1.5.2+computecanada-cp38-cp38-linux_x86_64.whl
Processing /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic/numba-0.48.0+computecanada-cp38-cp38-linux_x86_64.whl
Collecting librosa==0.7.2
  Downloading librosa-0.7.2.tar.gz (1.6 MB)
     |████████████████████████████████| 1.6 MB 707 kB/s
  Preparing metadata (setup.py) ... done
Collecting phonemizer>=2.2.0
  Downloading phonemizer-3.0-py3-none-any.whl (87 kB)
     |████████████████████████████████| 87 kB 775 kB/s
Collecting unidecode==0.4.20
  Downloading Unidecode-0.04.20-py2.py3-none-any.whl (228 kB)
     |████████████████████████████████| 228 kB 678 kB/s
Processing /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic/tensorboardX-2.4+computecanada-py2.py3-none-any.whl
Processing /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic/matplotlib-3.4.2+computecanada-cp38-cp38-linux_x86_64.wh
l
Processing /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic/Pillow-7.2.0+computecanada-cp38-cp38-linux_x86_64.whl
Collecting flask
  Downloading Flask-2.0.2-py3-none-any.whl (95 kB)
     |████████████████████████████████| 95 kB 573 kB/s
Collecting tqdm
  Downloading tqdm-4.62.3-py2.py3-none-any.whl (76 kB)
     |████████████████████████████████| 76 kB 580 kB/s
Collecting inflect
  Downloading inflect-5.3.0-py3-none-any.whl (32 kB)
Collecting bokeh==1.4.0
  Downloading bokeh-1.4.0.tar.gz (32.4 MB)
     |████████████████████████████████| 32.4 MB 918 kB/s
  Preparing metadata (setup.py) ... done
Collecting pysbd
  Downloading pysbd-0.3.4-py3-none-any.whl (71 kB)
     |████████████████████████████████| 71 kB 844 kB/s
Collecting pyworld
..
Processing /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic/regex-2021.7.6+computecanada-cp38-cp38-linux_x86_64.whl
Collecting colorlog
  Downloading colorlog-6.6.0-py2.py3-none-any.whl (11 kB)
Processing /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic/tabulate-0.8.9+computecanada-py3-none-any.whl
Processing /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic/isodate-0.6.0+computecanada-py2.py3-none-any.whl
Collecting uritemplate>=3.0.0
  Downloading uritemplate-4.1.1-py2.py3-none-any.whl (10 kB)
Collecting rfc3986
  Downloading rfc3986-1.5.0-py2.py3-none-any.whl (31 kB)
Processing /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic/rsa-4.7.2+computecanada-py3-none-any.whl
Processing /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic/pyasn1_modules-0.2.8+computecanada-py2.py3-none-any.whl
Processing /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic/cachetools-4.2.4+computecanada-py3-none-any.whl
Processing /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic/requests_oauthlib-1.3.0+computecanada-py2.py3-none-any.whl
Processing /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic/pyasn1-0.4.8+computecanada-py2.py3-none-any.whl
Processing /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic/oauthlib-3.1.1+computecanada-py2.py3-none-any.whl
Building wheels for collected packages: librosa, bokeh, cardboardlint, pyworld, gdown
  Building wheel for librosa (setup.py) ... done
  Created wheel for librosa: filename=librosa-0.7.2-py3-none-any.whl size=1612900 sha256=c6a34b7d8c390e3339b5a2239941e3bc7e111925fbfe2e31a22dc9a4c99ed09b   
  Stored in directory: /home/fieldsa/.cache/pip/wheels/11/f0/b0/a8f9944f274bbc0f0159f2268f43dadcfa1cfe50a9007d8e1f
  Building wheel for bokeh (setup.py) ... done
  Created wheel for bokeh: filename=bokeh-1.4.0-py3-none-any.whl size=23689210 sha256=aaf7d951784603d806b31ddcfcd779d1a80be0f5639d51e3f68295b2e6727a35
  Stored in directory: /home/fieldsa/.cache/pip/wheels/4a/79/96/e953cfb5c24da5e5e03eb1ecb280ca88dce65661fb4d38c7b5
  Building wheel for cardboardlint (setup.py) ... done
  Created wheel for cardboardlint: filename=cardboardlint-1.3.0-py3-none-any.whl size=51322 sha256=5a9d1257cdd4d5bdd7fdcf00ec58bbad7d76ac1a7a0aff664484ce3f2d48137f
  Stored in directory: /home/fieldsa/.cache/pip/wheels/68/46/5f/6a82950ec6470ee1fe63d0b5fe31e1b484057276034e644511
  Building wheel for pyworld (pyproject.toml) ... done
  Created wheel for pyworld: filename=pyworld-0.3.0-cp38-cp38-linux_x86_64.whl size=722544 sha256=0eb21f5961af9762c21a76e8eae7a3afc83b0b43d8d284fda75b2013e1ca5e77
  Stored in directory: /home/fieldsa/.cache/pip/wheels/b7/9d/77/c12111ca99a5a889f7b3a44b55308f7bd230ea9dbaa2a99613
  Building wheel for gdown (pyproject.toml) ... done
  Created wheel for gdown: filename=gdown-4.2.0-py3-none-any.whl size=14262 sha256=b695313f69f32d94c45525d403afff035f8b9cad5506f6036f31be1372b0dfb8
  Stored in directory: /home/fieldsa/.cache/pip/wheels/2b/3c/51/52c46deda5cd1d59c6ce3d441ea5f3d155495dc294c4535a25
Successfully built librosa bokeh cardboardlint pyworld gdown
Installing collected packages: urllib3, six, pyasn1, idna, charset-normalizer, certifi, uritemplate, rsa, rfc3986, requests, pyth
on-dateutil, pyasn1-modules, oauthlib, isodate, cachetools, attrs, tabulate, requests-oauthlib, pycparser, numpy, llvmlite, googl
e-auth, csvw, colorlog, wrapt, Werkzeug, threadpoolctl, tensorboard-plugin-wit, tensorboard-data-server, soupsieve, scipy, regex,
 PySocks, pyparsing, protobuf, numba, MarkupSafe, markdown, lazy-object-proxy, joblib, grpcio, google-auth-oauthlib, clldutils, c
ffi, absl-py, tqdm, tornado, toml, termcolor, tensorflow-estimator, tensorboard, tbb, soundfile, segments, scikit-learn, resampy,
 pyyaml, Pillow, packaging, opt-einsum, mccabe, kiwisolver, keras-preprocessing, Jinja2, itsdangerous, isort, h5py, google-pasta,
 gast, future, filelock, dlinfo, decorator, cython, cycler, click, beautifulsoup4, audioread, astunparse, astroid, unidecode, uma
p-learn, torch, tensorflow-gpu, tensorboardX, pyworld, pysbd, pylint, phonemizer, nose, matplotlib, librosa, inflect, gdown, flas
k, cardboardlint, bokeh
Successfully installed Jinja2-3.0.3 MarkupSafe-2.0.1+computecanada Pillow-7.2.0+computecanada PySocks-1.7.1+computecanada Werkzeu
g-2.0.2+computecanada absl-py-1.0.0 astroid-2.5+computecanada astunparse-1.6.3+computecanada attrs-21.2.0+computecanada audioread
-2.1.6+computecanada beautifulsoup4-4.10.0+computecanada bokeh-1.4.0 cachetools-4.2.4+computecanada cardboardlint-1.3.0 certifi-2
021.10.8+computecanada cffi-1.14.5+computecanada charset-normalizer-2.0.7+computecanada click-8.0.3+computecanada clldutils-3.10.
1 colorlog-6.6.0 csvw-1.11.0 cycler-0.11.0+computecanada cython-0.29.24+computecanada decorator-5.1.0 dlinfo-1.2.1 filelock-3.3.2
+computecanada flask-2.0.2 future-0.18.2+computecanada gast-0.3.3+computecanada gdown-4.2.0 google-auth-2.3.3+computecanada googl
e-auth-oauthlib-0.4.6+computecanada google-pasta-0.2.0+computecanada grpcio-1.41.1+computecanada h5py-2.10.0+computecanada idna-3
.3+computecanada inflect-5.3.0 isodate-0.6.0+computecanada isort-4.3.21+computecanada itsdangerous-2.0.1+computecanada joblib-1.1
.0+computecanada keras-preprocessing-1.1.2+computecanada kiwisolver-1.3.1+computecanada lazy-object-proxy-1.6.0+computecanada lib
rosa-0.7.2 llvmlite-0.31.0+computecanada markdown-3.3.4+computecanada matplotlib-3.4.2+computecanada mccabe-0.6.1+computecanada n
ose-1.3.7+computecanada numba-0.48.0+computecanada numpy-1.18.4+computecanada oauthlib-3.1.1+computecanada opt-einsum-3.3.0+compu
tecanada packaging-21.2+computecanada phonemizer-3.0 protobuf-3.19.1+computecanada pyasn1-0.4.8+computecanada pyasn1-modules-0.2.
8+computecanada pycparser-2.21+computecanada pylint-2.5.3 pyparsing-2.4.7+computecanada pysbd-0.3.4 python-dateutil-2.8.2+compute
canada pyworld-0.3.0 pyyaml-6.0+computecanada regex-2021.7.6+computecanada requests-2.26.0+computecanada requests-oauthlib-1.3.0+
computecanada resampy-0.2.2+computecanada rfc3986-1.5.0 rsa-4.7.2+computecanada scikit-learn-0.23.0+computecanada scipy-1.4.1+com
putecanada segments-2.2.0 six-1.16.0+computecanada soundfile-0.10.3.post1+computecanada soupsieve-2.3.1 tabulate-0.8.9+computecan
ada tbb-2021.1.1+computecanada tensorboard-2.7.0 tensorboard-data-server-0.6.1+computecanada tensorboard-plugin-wit-1.8.0+compute
canada tensorboardX-2.4+computecanada tensorflow-estimator-2.3.0+computecanada tensorflow-gpu-2.3.0+computecanada termcolor-1.1.0+computecanada threadpoolctl-3.0.0+computecanada toml-0.10.2+computecanada torch-1.6.0+computecanada tornado-6.1+computecanada tqdm-4.62.3 umap-learn-0.4.2+computecanada unidecode-0.4.20 uritemplate-4.1.1 urllib3-1.26.7+computecanada wrapt-1.12.1+computecanada

(Mozilla-tts) [fieldsa@hn2 ~]$ which python
~/.conda/envs/Mozilla-tts/bin/python
(Mozilla-tts) [fieldsa@hn2 ~]$ python --version
Python 3.8.12
(Mozilla-tts) [fieldsa@hn2 ~]$ python
Python 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:57:06) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(@)>>> import tensorflow as tf
2021-11-15 16:29:41.880438: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
(@)>>> 

When I load it on cn135 for example:

(Mozilla-tts) [fieldsa@cn135 ~]$ cat ~/.condarc
channels:
#  - salilab
#  - bioconda
#  - defaults
#  - r
  - conda-forge
report_errors: false

(Mozilla-tts) [fieldsa@cn135 ~]$ python
Python 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:57:06)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(@)>>> import tensorflow as tf
2021-11-15 16:43:05.696985: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
(@)>>> tf.config.list_physical_devices()
2021-11-15 16:44:07.633893: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2021-11-15 16:44:07.667236: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:89:00.0 name: Tesla V100-SXM2-32GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 31.72GiB deviceMemoryBandwidth: 836.37GiB/s
2021-11-15 16:44:07.669153: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 1 with properties: 
pciBusID: 0000:8a:00.0 name: Tesla V100-SXM2-32GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 31.72GiB deviceMemoryBandwidth: 836.37GiB/s
2021-11-15 16:44:07.671055: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 2 with properties: 
pciBusID: 0000:b2:00.0 name: Tesla V100-SXM2-32GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 31.72GiB deviceMemoryBandwidth: 836.37GiB/s
2021-11-15 16:44:07.672922: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 3 with properties: 
pciBusID: 0000:b3:00.0 name: Tesla V100-SXM2-32GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 31.72GiB deviceMemoryBandwidth: 836.37GiB/s
2021-11-15 16:44:07.672976: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-11-15 16:44:07.711634: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2021-11-15 16:44:08.150815: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2021-11-15 16:44:09.129443: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2021-11-15 16:44:09.871181: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2021-11-15 16:44:10.156447: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2021-11-15 16:44:11.544995: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2021-11-15 16:44:11.559671: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0, 1, 2, 3
[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:XLA_CPU:0', device_type='XLA_CPU'), PhysicalDevice(name='/physical_device:XLA_GPU:0', device_type='XLA_GPU'), PhysicalDevice(name='/physical_device:XLA_GPU:1', device_type='XLA_GPU'), PhysicalDevice(name='/physical_device:XLA_GPU:2', device_type='XLA_GPU'), PhysicalDevice(name='/physical_device:XLA_GPU:3', device_type='XLA_GPU'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:2', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:3', device_type='GPU')]