emcee tutorial: multiprocessing issues on Mac

[JunHuiLiu2017]

In order to investigate the errors happening in which line, I only copy a part of codes to run. -->

#####Start
import phoebe
from phoebe import u # units
import numpy as np

logger = phoebe.logger('error')

## Create fake "observations"

We'll create the same fake "observations" as in the Inverse Paper Examples.

For the sake of efficiency for this example, we'll use the ellc backend. In practice, this is only safe for areas of the parameter space where phoebe and ellc are in sufficient agreement.

In[3]:

if name == 'main':
b = phoebe.default_binary()
b.set_value('ecc', 0.2)
b.set_value('per0', 25)
b.set_value('teff@primary', 7000)
b.set_value('teff@secondary', 6000)
b.set_value('sma@binary', 7)
b.set_value('incl@binary', 80)
b.set_value('q', 0.3)
b.set_value('t0_supconj', 0.1)
b.set_value('requiv@primary', 2.0)
b.set_value('vgamma', 80)

lctimes = phoebe.linspace(0, 10, 1005)
rvtimes = phoebe.linspace(0, 10, 105)
b.add_dataset('lc', compute_times=lctimes)
b.add_dataset('rv', compute_times=rvtimes)

b.add_compute('ellc', compute='fastcompute')
b.set_value_all('ld_mode', 'lookup')
b.run_compute(compute='fastcompute')

fluxes = b.get_value('fluxes@model') + np.random.normal(size=lctimes.shape) * 0.01
fsigmas = np.ones_like(lctimes) * 0.02

rvsA = b.get_value('rvs@primary@model') + np.random.normal(size=rvtimes.shape) * 10
rvsB = b.get_value('rvs@secondary@model') + np.random.normal(size=rvtimes.shape) * 10
rvsigmas = np.ones_like(rvtimes) * 20

In[4]:

b = phoebe.default_binary()

b.add_dataset('lc',
              compute_phases=phoebe.linspace(0,1,201),
              times=lctimes,
              fluxes=fluxes,
              sigmas=fsigmas,
              dataset='lc01')

b.add_dataset('rv',
              compute_phases=phoebe.linspace(0,1,201),
              times=rvtimes,
              rvs={'primary': rvsA, 'secondary': rvsB},
              sigmas=rvsigmas,
              dataset='rv01')

b.add_compute('ellc', compute='fastcompute')
#b.add_compute( compute='fastcompute')
b.set_value_all('ld_mode', 'lookup')


b.set_value('ecc', 0.2)
b.set_value('per0', 25)
b.set_value('teff@primary', 7000)
b.set_value('teff@secondary', 6000)
b.set_value('sma@binary', 7+0.01)
b.set_value('incl@binary', 80+0.1)
b.set_value('q', 0.3)
b.set_value('t0_supconj', 0.1-0.001)
b.set_value('requiv@primary', 2.0-0.05)
b.set_value('vgamma', 80)

b.run_compute(compute='fastcompute', model='orig_model')

_ = b.plot(x='phases', show=True)
_ = b.plot(x='phases', y='residuals', show=True)


b.add_solver('sampler.emcee', solver='emcee_solver')
print(b.filter(solver='emcee_solver'))


b.set_value('compute', solver='emcee_solver', value='fastcompute')

b.add_distribution({'sma@binary': phoebe.gaussian_around(0.1),
                   'incl@binary': phoebe.gaussian_around(5),
                   't0_supconj': phoebe.gaussian_around(0.001),
                   'requiv@primary': phoebe.gaussian_around(0.4),
                   'pblum@primary': phoebe.gaussian_around(0.2),
                   'sigmas_lnf@lc01': phoebe.uniform(-1e9, -1e4),
                   }, distribution='ball_around_guess')


b.run_compute(compute='fastcompute', sample_from='ball_around_guess',
              sample_num=20, model='init_from_model')

#######End

If I did not insert the main, it will report an error --->
'''
(base) liujunhuideMacBook:20210220phoebe liujunhui$ python test_emcee.py
100%|█████████████████████████████████████████████| 3/3 [00:00<00:00, 11.22it/s]
100%|█████████████████████████████████████████████| 3/3 [00:00<00:00, 37.53it/s]
ParameterSet: 11 parameters
comments@emcee_solver@solver:
compute@emcee_solver@solver: phoebe01
continue_from@emcee_solver@...: None
init_from@emcee_solver@solver: []
priors@emcee_solver@solver: []
nwalkers@emcee_solver@solver: 16
niters@emcee_solver@solver: 100
burnin_factor@emcee_solver@...: 2.0
thin_factor@emcee_solver@so...: 0.5
progress_every_niters@emcee...: 0
expose_failed@emcee_solver@...: True
(base) liujunhuideMacBook:20210220phoebe liujunhui$ python test_emcee.py
100%|█████████████████████████████████████████████| 3/3 [00:00<00:00, 8.50it/s]
100%|█████████████████████████████████████████████| 3/3 [00:00<00:00, 19.45it/s]
ParameterSet: 11 parameters
comments@emcee_solver@solver:
compute@emcee_solver@solver: phoebe01
continue_from@emcee_solver@...: None
init_from@emcee_solver@solver: []
priors@emcee_solver@solver: []
nwalkers@emcee_solver@solver: 16
niters@emcee_solver@solver: 100
burnin_factor@emcee_solver@...: 2.0
thin_factor@emcee_solver@so...: 0.5
progress_every_niters@emcee...: 0
expose_failed@emcee_solver@...: True
100%|█████████████████████████████████████████████| 3/3 [00:00<00:00, 5.16it/s]
100%|█████████████████████████████████████████████| 3/3 [00:00<00:00, 4.92it/s]
100%|█████████████████████████████████████████████| 3/3 [00:00<00:00, 7.16it/s]
100%|█████████████████████████████████████████████| 3/3 [00:00<00:00, 6.98it/s]
100%|█████████████████████████████████████████████| 3/3 [00:00<00:00, 16.25it/s]
100%|█████████████████████████████████████████████| 3/3 [00:00<00:00, 21.71it/s]
100%|█████████████████████████████████████████████| 3/3 [00:00<00:00, 17.25it/s]
100%|█████████████████████████████████████████████| 3/3 [00:00<00:00, 17.50it/s]
ParameterSet: 11 parameters
comments@emcee_solver@solver:
compute@emcee_solver@solver: phoebe01
continue_from@emcee_solver@...: None
init_from@emcee_solver@solver: []
priors@emcee_solver@solver: []
nwalkers@emcee_solver@solver: 16
niters@emcee_solver@solver: 100
burnin_factor@emcee_solver@...: 2.0
thin_factor@emcee_solver@so...: 0.5
progress_every_niters@emcee...: 0
expose_failed@emcee_solver@...: True
Traceback (most recent call last):
File "", line 1, in
File "/opt/anaconda3/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/opt/anaconda3/lib/python3.8/multiprocessing/spawn.py", line 125, in _main
prepare(preparation_data)
File "/opt/anaconda3/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/opt/anaconda3/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "/opt/anaconda3/lib/python3.8/runpy.py", line 265, in run_path
return _run_module_code(code, init_globals, run_name,
File "/opt/anaconda3/lib/python3.8/runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/opt/anaconda3/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/Users/liujunhui/Desktop/20210220phoebe/test_emcee.py", line 117, in
b.run_compute(compute='fastcompute', sample_from='ball_around_guess',
File "/opt/anaconda3/lib/python3.8/site-packages/phoebe/parameters/parameters.py", line 401, in _send_if_client
return fctn(self, *args, **kwargs)
File "/opt/anaconda3/lib/python3.8/site-packages/phoebe/frontend/bundle.py", line 10127, in run_compute
params = backends.SampleOverModel().run(self, computeparams.compute,
File "/opt/anaconda3/lib/python3.8/site-packages/phoebe/backend/backends.py", line 700, in run
pool = _pool.MultiPool()
File "/opt/anaconda3/lib/python3.8/site-packages/phoebe/pool/multipool.py", line 60, in init
super(MultiPool, self).init(processes, new_initializer,
File "/opt/anaconda3/lib/python3.8/multiprocessing/pool.py", line 212, in init
self._repopulate_pool()
File "/opt/anaconda3/lib/python3.8/multiprocessing/pool.py", line 303, in _repopulate_pool
return self._repopulate_pool_static(self._ctx, self.Process,
File "/opt/anaconda3/lib/python3.8/multiprocessing/pool.py", line 326, in _repopulate_pool_static
w.start()
File "/opt/anaconda3/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/opt/anaconda3/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
return Popen(process_obj)
File "/opt/anaconda3/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in init
super().init(process_obj)
File "/opt/anaconda3/lib/python3.8/multiprocessing/popen_fork.py", line 19, in init
self._launch(process_obj)
File "/opt/anaconda3/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 42, in _launch
prep_data = spawn.get_preparation_data(process_obj._name)
File "/opt/anaconda3/lib/python3.8/multiprocessing/spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "/opt/anaconda3/lib/python3.8/multiprocessing/spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

100%|███████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 8.78it/s]
100%|███████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 16.48it/s]
'''
After that, this error will happen after the normal running -->
'''
(base) liujunhuideMacBook:20210220phoebe liujunhui$ python test_emcee.py
100%|███████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 11.02it/s]
100%|███████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 22.35it/s]
ParameterSet: 11 parameters
comments@emcee_solver@solver:
compute@emcee_solver@solver: phoebe01
continue_from@emcee_solver@...: None
init_from@emcee_solver@solver: []
priors@emcee_solver@solver: []
nwalkers@emcee_solver@solver: 16
niters@emcee_solver@solver: 100
burnin_factor@emcee_solver@...: 2.0
thin_factor@emcee_solver@so...: 0.5
progress_every_niters@emcee...: 0
expose_failed@emcee_solver@...: True
33%|███████████████████████████▋ | 1/3 [00:00<00:00, 3.64it/s]
100%|███████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 9.94it/s]
100%|███████████████████████████████████████████████████████████████████████████████████| 3/3 [00:01<00:00, 2.46it/s]
100%|███████████████████████████████████████████████████████████████████████████████████| 3/3 [00:01<00:00, 2.71it/s]
100%|███████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 20.83it/s]
100%|███████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 18.61it/s]
100%|███████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 10.15it/s]
100%|███████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 17.73it/s]
ParameterSet: 11 parameters
comments@emcee_solver@solver:
compute@emcee_solver@solver: phoebe01
continue_from@emcee_solver@...: None
init_from@emcee_solver@solver: []
priors@emcee_solver@solver: []
nwalkers@emcee_solver@solver: 16
niters@emcee_solver@solver: 100
burnin_factor@emcee_solver@...: 2.0
thin_factor@emcee_solver@so...: 0.5
progress_every_niters@emcee...: 0
expose_failed@emcee_solver@...: True
Sun, 21 Feb 2021 21:06 BUNDLE ERROR Constraint 'asini@binary@orbit@constraint' raised the following error while setting the value of 'asini@binary@orbit@component'. Original error: solver='emcee_solver' not found
Sun, 21 Feb 2021 21:06 BUNDLE ERROR Constraint 'logg@primary@star@constraint' raised the following error while setting the value of 'logg@primary@star@component'. Original error: solver='emcee_solver' not found
Sun, 21 Feb 2021 21:06 BUNDLE ERROR Constraint 'logg@secondary@star@constraint' raised the following error while setting the value of 'logg@secondary@star@component'. Original error: solver='emcee_solver' not found
Sun, 21 Feb 2021 21:06 BUNDLE ERROR Constraint 'mass@primary@star@constraint' raised the following error while setting the value of 'mass@primary@star@component'. Original error: solver='emcee_solver' not found
Sun, 21 Feb 2021 21:06 BUNDLE ERROR Constraint 'mass@secondary@star@constraint' raised the following error while setting the value of 'mass@secondary@star@component'. Original error: solver='emcee_solver' not found
Sun, 21 Feb 2021 21:06 BUNDLE ERROR Constraint 'mass@primary@star@constraint' raised the following error while setting the value of 'mass@primary@star@component'. Original error: solver='emcee_solver' not found
Sun, 21 Feb 2021 21:06 BUNDLE ERROR Constraint 'sma@primary@star@constraint' raised the following error while setting the value of 'sma@primary@star@component'. Original error: solver='emcee_solver' not found
Sun, 21 Feb 2021 21:06 BUNDLE ERROR Constraint 'asini@primary@star@constraint' raised the following error while setting the value of 'asini@primary@star@component'. Original error: solver='emcee_solver' not found
....
'''
In fact, the version of python in my Mac is 3.8.3, and I install the package 'solve', 'emcee' and 'ellc'.

[kecnry]

ok, it looks to me like there is something different with your multiprocessing setup that isn't playing nicely with phoebe.

In order to double check, let's try to simplify the test case down to a few short snippets. Can you confirm that the following does run successfully for you?

import phoebe
b = phoebe.default_binary()
b.add_dataset('lc', compute_times=phoebe.linspace(0,1,11))
b.run_compute()

but that this fails with a similar error as before?

import phoebe
b = phoebe.default_binary()
b.add_dataset('lc', compute_times=phoebe.linspace(0,1,11))
b.add_distribution({'incl@binary': phoebe.uniform(80,90)}, distribution='dists01')
b.run_compute(sample_from='dists01', sample_num=4)

Do you have MPI installed and configured on your machine? If so, what happens if you run the snippet above in MPI instead? (If not or if you don't know how to use MPI, don't worry about it for now).

Can you also please let me know the version of the multiprocessing package you have installed (probably with something like conda list)?

I'll try to see if I can get someone to reproduce using anaconda on a mac and see if we can come up with a solution to get your configuration of multiprocessing working correctly with phoebe. If that doesn't work, I'll implement a way to disable the multiprocessing entirely which will then just run the sampling in serial mode on a single processor... but at least it should run!

Note that, if this is in fact the problem, this should only show up if you use sample_from in run_compute or if you call run_solver outside of MPI - as those are the cases that default to using multiprocessing.

[kecnry]

My apologies, that's what I get for just writing on the fly. You do need the add_dataset in the second-half as well as I've fixed below (but also to clarify: please run these in a single script so that we can test if the issue is caused by the reassignment of b).

import phoebe
b = phoebe.default_binary()
b.add_dataset('lc', compute_times=phoebe.linspace(0,1,11))
b.set_value('sma', component='binary', value=7)
b.run_compute()

b = phoebe.default_binary()
b.add_dataset('lc', compute_times=phoebe.linspace(0,1,11))
b.add_distribution({'incl@binary': phoebe.uniform(80,90)}, distribution='dists01')
b.add_solver('sampler.emcee')
b.run_compute(sample_from='dists01', sample_num=4)

[JunHuiLiu2017]

The first part was fine, and the second part was still running with no errors.

[kecnry]

I just released a bugfix (v 2.3.26) that now allows disabling multiprocessing. So, you'll want to place phoebe.multiprocessing_off() at the top of your script (or set the environment variable) and then this shouldn't be an issue anymore. Of course, this means that the sampling and solvers will run serially and therefore take longer - you'll eventually likely want to use MPI (preferably on a cluster with a large number of CPUs available) for any serious MCMC runs over a large number of parameters and/or iterations.

[JunHuiLiu2017]

My Mac system still does not run emcee.py successfully, because 2.3.26 was not installed successfully, and the error is due to the issues of g++ and clang upgrade. I tried using Linux system and successfully ran emcee.py. Thank you very much！

[kecnry]

Glad to hear you have it running on Linux! Other users are reporting similar issues with Mac updates, so I created a general bug report for that - please post any relevant details or follow along in that thread for a solution (hopefully).