Update Openxla-pin to Sep13, libtpu-pin to Sep13, jax to 0.4.33_nightly

[ManfeiBai]

update OpenXLA-pin update to Sep13's commit: 54052bcbe0cda5b0a7be7c8e22eb2bfef025786e

and also update libtpu version to Sep13

limit TPU CI tests XLA_EXPERIMENTAL=nonzero:masked_select:nms python3 test/ds/test_dynamic_shapes.py -v to only TPU v4 CI

[JackCaoG]

failure seems real

/github/home/.cache/bazel/_bazel_root/197a057057a49e5811107144e2d78508/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in <toplevel>
    ERROR: /__w/xla/xla/pytorch/xla/torch_xla/csrc/runtime/BUILD:500:14: no such package '@tsl//tsl/lib/core': BUILD file not found in directory 'tsl/lib/core' of external repository @tsl. Add a BUILD file to a directory to mark it as a package. and referenced by '//torch_xla/csrc/runtime:ifrt_computation_client_test'
    ERROR: Analysis of target '//torch_xla/csrc/runtime:ifrt_computation_client_test' failed; build aborted: Analysis failed

[JackCaoG]

hmm I don't know why TPU test still skipped. Let me trigger again.

[ManfeiBai]

failed with ERROR: Failed to query remote execution capabilities: UNAVAILABLE: Credentials failed to obtain metadata, looks like flaky, clicked rerun

[JackCaoG]

@ManfeiBai can you build this locally and check if pallas test can pass?

[lsy323]

You need to have a new commit after the tpu ci tag is applied to run tpu ci

[ManfeiBai]

@ManfeiBai can you build this locally and check if pallas test can pass?

tried locally on v4-8 and failed: https://gist.github.com/ManfeiBai/6dfc589f7d2f297c1954b675022dacee

met error like:

RuntimeError: Numpy is not available

RuntimeError: Bad StatusOr access: INVALID_ARGUMENT: Failed to parse the Mosaic module

RuntimeError: Bad StatusOr access: INTERNAL: Mosaic failed to compile TPU kernel: Bad lhs type in tpu.matmul

AssertionError: False is not true

[ManfeiBai]

try locally on v4-8 without update libtpu, PJRT_DEVICE=TPU python test/test_operations.py -v failed with:

======================================================================
ERROR: test_triangular_solve (__main__.TestOpBuilder)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/pytorch/xla/test/test_operations.py", line 2531, in test_triangular_solve
    self.runOpBuilderTest(
  File "/pytorch/xla/test/test_operations.py", line 2437, in runOpBuilderTest
    results = xu.as_list(aten_fn(*tensors, **kwargs))
  File "/pytorch/xla/test/test_operations.py", line 2524, in aten_fn
    return torch.triangular_solve(
RuntimeError: Calling torch.triangular_solve on a CPU tensor requires compiling PyTorch with BLAS. Please use PyTorch built with BLAS support.

----------------------------------------------------------------------

[ManfeiBai]

after skipped test/test_pallas.py and test/test_pallas_spmd.py, with libtpu 0821, other TPU CI tests passed;

next step is bring test/test_pallas.py and test/test_pallas_spmd.py back

[JackCaoG]

sg, you can ignore test/test_pallas_spmd.py and test/test_pallas.py for now, but we likely can not use 08/21 libtpu. Can you figure out why 08/22 libtpu will fail?

[ManfeiBai]

sg, you can ignore test/test_pallas_spmd.py and test/test_pallas.py for now, but we likely can not use 08/21 libtpu. Can you figure out why 08/22 libtpu will fail?

08/22 libtpu caused failure like:
tested test/tpu/run_tests.sh with libtpu0822, without test/test_pallas.py and test/test_pallas_spmd.py: https://gist.github.com/ManfeiBai/f215d9497d7d63b97c29df64ff69d107

failed due to:

free(): corrupted unsorted chunks
https://symbolize.stripped_domain/r/?trace=7f2ba0762ce1,7f2ba0a6013f,7f2ba07ab6d9&map= 
*** SIGABRT received by PID 627075 (TID 629631) on cpu 82 from PID 627075; stack trace: ***
PC: @     0x7f2ba0762ce1  (unknown)  raise
    @     0x7f2366fa91a1       1888  (unknown)
    @     0x7f2ba0a60140       2320  (unknown)
    @     0x7f2ba07ab6da  (unknown)  (unknown)
    @                0x1  (unknown)  (unknown)
https://symbolize.stripped_domain/r/?trace=7f2ba0762ce1,7f2366fa91a0,7f2ba0a6013f,7f2ba07ab6d9,0&map= 
E0905 20:27:41.467463  629631 coredump_hook.cc:316] RAW: Remote crash data gathering hook invoked.
E0905 20:27:41.467482  629631 client.cc:269] RAW: Coroner client retries enabled, will retry for up to 30 sec.
E0905 20:27:41.467487  629631 coredump_hook.cc:411] RAW: Sending fingerprint to remote end.
E0905 20:27:41.467508  629631 coredump_hook.cc:420] RAW: Cannot send fingerprint to Coroner: [NOT_FOUND] stat failed on crash reporting socket /var/google/services/logmanagerd/remote_coredump.socket (Is the listener running?): No such file or directory
E0905 20:27:41.467514  629631 coredump_hook.cc:472] RAW: Dumping core locally.
base/elfcore.c:2078 Failed to write mapping 2480 at 0x7f01c8000000 of size 58597376: No space left on device(28)
E0905 20:29:27.997053  629631 process_state.cc:805] RAW: Raising signal 6 with default behavior
test/tpu/run_tests.sh: line 41: 627075 Aborted                 (core dumped) python3 examples/data_parallel/train_resnet_spmd_data_parallel.py

and might due to the above error, /tmp is full, then caused another issue for tests after some above infos:

FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/pytorch/xla']

[ManfeiBai]

sg, you can ignore test/test_pallas_spmd.py and test/test_pallas.py for now, but we likely can not use 08/21 libtpu. Can you figure out why 08/22 libtpu will fail?

08/22 libtpu caused failure like: tested test/tpu/run_tests.sh with libtpu0822, without test/test_pallas.py and test/test_pallas_spmd.py: https://gist.github.com/ManfeiBai/f215d9497d7d63b97c29df64ff69d107

failed due to:

free(): corrupted unsorted chunks
https://symbolize.stripped_domain/r/?trace=7f2ba0762ce1,7f2ba0a6013f,7f2ba07ab6d9&map= 
*** SIGABRT received by PID 627075 (TID 629631) on cpu 82 from PID 627075; stack trace: ***
PC: @     0x7f2ba0762ce1  (unknown)  raise
    @     0x7f2366fa91a1       1888  (unknown)
    @     0x7f2ba0a60140       2320  (unknown)
    @     0x7f2ba07ab6da  (unknown)  (unknown)
    @                0x1  (unknown)  (unknown)
https://symbolize.stripped_domain/r/?trace=7f2ba0762ce1,7f2366fa91a0,7f2ba0a6013f,7f2ba07ab6d9,0&map= 
E0905 20:27:41.467463  629631 coredump_hook.cc:316] RAW: Remote crash data gathering hook invoked.
E0905 20:27:41.467482  629631 client.cc:269] RAW: Coroner client retries enabled, will retry for up to 30 sec.
E0905 20:27:41.467487  629631 coredump_hook.cc:411] RAW: Sending fingerprint to remote end.
E0905 20:27:41.467508  629631 coredump_hook.cc:420] RAW: Cannot send fingerprint to Coroner: [NOT_FOUND] stat failed on crash reporting socket /var/google/services/logmanagerd/remote_coredump.socket (Is the listener running?): No such file or directory
E0905 20:27:41.467514  629631 coredump_hook.cc:472] RAW: Dumping core locally.
base/elfcore.c:2078 Failed to write mapping 2480 at 0x7f01c8000000 of size 58597376: No space left on device(28)
E0905 20:29:27.997053  629631 process_state.cc:805] RAW: Raising signal 6 with default behavior
test/tpu/run_tests.sh: line 41: 627075 Aborted                 (core dumped) python3 examples/data_parallel/train_resnet_spmd_data_parallel.py

and might due to the above error, /tmp is full, then caused another issue for tests after some above infos:

FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/pytorch/xla']

this error appear with libtpu between Aug22 and Aug26, updated libtpu version to Aug27, this error disappear

[ManfeiBai]

Synced offline with @JackCaoG, will wait until libtpu is needed and ready to be updated with OpenXLA-commit

[bhavya01]

Can we also include the changes from https://github.com/pytorch/xla/pull/8008/files? We'll need that for the pallas tests to pass. I think we can directly include those in the pin update PR

[ManfeiBai]

Can we also include the changes from https://github.com/pytorch/xla/pull/8008/files? We'll need that for the pallas tests to pass. I think we can directly include those in the pin update PR

thanks, rebased, let's wait for CI test result