Pymc3 model fails to compile using reticulate, possibly related to m1 (apple silicon)

[elbamos]

The attached file works properly from within Jupyter, but produces a compile error when run from reticulate (1.20) using the same conda environment.

minimum reproducible example: pymc_test.py.zip

The compiler error I'm seeing is:

Python 3.8.6 (/opt/homebrew/Caskroom/miniforge/base/envs/pymc/bin/python)
Reticulate 1.20 REPL -- A Python interpreter in R.
CompileError: Compilation failed (return status=1):
/opt/homebrew/Caskroom/miniforge/base/envs/pymc/bin/clang++ -dynamiclib -g -O3 -fno-math-errno -Wno-unused-label -Wno-unused-variable -Wno-write-strings -Wno-c++11-narrowing -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -fPIC -undefined dynamic_lookup -I/opt/homebrew/Caskroom/miniforge/base/envs/pymc/lib/python3.8/site-packages/numpy/core/include -I/opt/homebrew/Caskroom/miniforge/base/envs/pymc/include/python3.8 -I/opt/homebrew/Caskroom/miniforge/base/envs/pymc/lib/python3.8/site-packages/aesara/link/c/c_code -L/opt/homebrew/Caskroom/miniforge/base/envs/pymc/lib -fvisibility=hidden -o /Users/__/.aesara/compiledir_macOS-11.4-arm64-arm-64bit-arm-3.8.6-64/tmpbn137xn6/m9a793963c6a845351f269f18de083e3b6952fddda6b2dc3a1eab6bc02aad38d1.so /Users/__/.aesara/compiledir_macOS-11.4-arm64-arm-64bit-arm-3.8.6-64/tmpbn137xn6/mod.cpp
In file included from /Users/__/.aesara/compiledir_macOS-11.4-arm64-arm-64bit-arm-3.8.6-64/tmpbn137xn6/mod.cpp:1:
In file included from /opt/homebrew/Caskroom/miniforge/base/envs/pymc/include/python3.8/Python.h:25:
/opt/homebrew/Caskroom/miniforge/base/envs/pymc/bin/../include/c++/v1/stdio.h:107:15: fatal error: 'stdio.h' file not found
#include_next <stdio.h>
              ^~~~~~~~~
1 error generated.

[kevinushey]

/opt/homebrew/Caskroom/miniforge/base/envs/pymc/bin/../include/c++/v1/stdio.h:107:15: fatal error: 'stdio.h' file not found
#include_next <stdio.h>
              ^~~~~~~~~

Do you have macOS command line tools installed? (What is the output of /usr/bin/clang --version in the terminal?)

[elbamos]

Do you have macOS command line tools installed? (What is the output of /usr/bin/clang --version in the terminal?)

Yup - I have fully functional R and python installations. From R I can compile R packages and Stan models. From Python I can compile python packages via pip and Pymc3 models.

% /usr/bin/clang --version
Apple clang version 12.0.0 (clang-1200.0.32.29)
Target: arm64-apple-darwin20.5.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

[kevinushey]

Can you also confirm the output of xcode-select -p? If that points to an Xcode installation, then changing it with:

sudo xcode-select -s /Library/Developer/CommandLineTools

may also help. (An alternate way to accomplish the same is to set the SDKROOT environment variable)

If that still doesn't help, then I'm not really sure what the issue could be beyond some kind of Anaconda incompatibility with arm64 macOS... (It would be worth trialing with a plain Python virtual environment, independent of Anaconda.)

[elbamos]

The xcode path was already set as:

% xcode-select -p
/Library/Developer/CommandLineTools

I don't think this can be an issue of anaconda not being compatible with arm64, since all of this runs perfectly from the same conda environment from within jupyter. It seems to me the issue has to be that reticulate, for whatever reason, perhaps because of something rstudio is doing in launching reticulate, doesn't match the environment aesara is expecting.

Calling os.environ._data.keys() from vanilla jupyter and and reticulate, before running any code, I see that jupyter has a host of environment variables related to compilation (e.g., LDFLAGS, CPPFLAGS, etc.) set while reticulate does not. A bit of searching around, and it looks like these may be coming from files located under env_path/etc/conda/activate.d

This may be naive, but is it possible that conda is executing the scripts under activate.d when the environment is activated, while reticulate is not?

[kevinushey]

Ah, you're probably right. When RStudio binds to an Anaconda Python installation, we don't run the associated activate script, so we don't pick up on the environment variables that would normally be set in those cases.

[elbamos]

Am I mistaken that that's going to be a pretty big issue? I think it breaks any conda package that depends on clang (pymc3, pystan?, tensorflow?, pytorch?, custom cython code?). Is it your thinking that this is confined to conda but the same packages should function properly in a virtualenv?

[elbamos]

FYI - something similar seems to happen with venv and virtualenv. The script runs fine from the console. From reticulate, however, I get a different compilation error:

> reticulate::source_python('~/Projects/experimentation/pymc_test.py')
Error in py_run_file_impl(file, local, convert) : 
  CompileError: Compilation failed (return status=1):
/usr/bin/clang++ -dynamiclib -g -O3 -fno-math-errno -Wno-unused-label -Wno-unused-variable -Wno-write-strings -Wno-c++11-narrowing -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -fPIC -undefined dynamic_lookup -I/opt/homebrew/lib/python3.9/site-packages/numpy/core/include -I/Users/__/.venv/test3/include/python3.9 -I/Users/__/.venv/test3/lib/python3.9/site-packages/aesara/link/c/c_code -L/Users/__/.venv/test3/lib -fvisibility=hidden -o /Users/__/.aesara/compiledir_macOS-11.4-arm64-arm-64bit-arm-3.9.5-64/tmpr1tajlp5/m9a793963c6a845351f269f18de083e3b6952fddda6b2dc3a1eab6bc02aad38d1.so /Users/__/.aesara/compiledir_macOS-11.4-arm64-arm-64bit-arm-3.9.5-64/tmpr1tajlp5/mod.cpp
/Users/__/.aesara/compiledir_macOS-11.4-arm64-arm-64bit-arm-3.9.5-64/tmpr1tajlp5/mod.cpp:1:10: fatal error: 'Python.h' file not found
#include <Python.h>
         ^~~~~~~~~~
1 error generated.

[jucor]

Uh oh. I'm about to try and wrap my PyMC python code for some biostatisticians R users, hopefully using reticulate, and I'm seeing this open issue and dreading the next few days... @elbamos @kevinushey , have you ever had any luck solving this in the last 2 years, by any chance, before I run into the minefield? ;-)

[t-kalinowski]

@jucor Many conda-compilation related issues go away, or change shape, in the moving landscape. If you run into problems please open a new issue and we'll do what we can to help. For minimizing risk of compilation/installation hurdles like this one, try use a venv if possible. You might also try something like this for pre-compiled binaries:

Sys.setenv(PIP_INDEX_URL = "https://packagemanager.posit.co/pypi/latest/simple")
reticulate::py_install(...)

[jucor]

Thank you, makes a lot of sense! I have used conda-forge for scientific packages for the last few years as they solved better than pip the dependencies on external scientific libraries (like lapack, blas, sundials, cvode, etc). Maybe that has changed? (I haven't tried pip as main tool in years)

[jucor]

Hi @t-kalinowski and @kevinushey

Is there any way we could pick up the environment variables that activate sets, please?

Indeed, I'm exploding on the same landlmine as @elbamos : pyMC failing to compile using reticulate on macOS. See the minimum reproducible example attached
TestReticulatedPyMC.rmd.gz file that causes the problem: you will see that, upon compiling the model, pyMC cannot find the basic complains of not finding the base headers like unistd.h. Clearly something wrong in one of the paths :/

And unfortunately, while reticulate recommends venv, pyMC strongly recommends conda (and I can't get it to work with venv), so I'm really stuck.

Thanks for any help!

[jucor]

I found a workaround by filling the SDKROOT variable, which clang knows to use:

if (Sys.info()['sysname'] == 'Darwin') {
  sdk_root=system2("xcrun", args="--show-sdk-path", stdout=TRUE, stderr=TRUE)
  Sys.setenv(SDKROOT=sdk_root)
}

So that helps the minimum example. Not sure how it will handle other issues linked to BLAS conda-env libraries, but I'm not there yet.

[t-kalinowski]

@jucor, thank you for posting back with your solution!

If you run into other errors, or want more help with this one, can you please open a new github issue, and include the MRE in the body of the issue instead of as an attachment?

[jucor]

With pleasure, if you can help me find how to put Rmarkdown blocks in the body of an issue 😅 (the code chunks in it okay havoc with the GitHub parser)

[t-kalinowski]

Flank the Rmd document text with lines containing 4 backticks, like so:

````
---
title: my title
---

prose

```r
code()
```

prose

````

[jucor]

Awesome, thanks a lot, I did not know that 4-backticks trick! Will definitely do next time.