Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Display python stdout when using knitr #1412

Merged
merged 7 commits into from
Jul 18, 2023

Conversation

dfalbel
Copy link
Member

@dfalbel dfalbel commented Jul 12, 2023

Fixes #1387 and fix #331

The approach used in this PR is to register a knitr hook that runs at the begining and end of each document chunk. In the begining of each chunk we remap Python's std streams to the same as R's (so knitr can capture them), at the end of the chunk we reset the streams to the default reticulate behavior (which is configurable with the RETICULATE_REMAP_OUTPUT_STREAMS env var).

Another approach would be to always set a output remap class in Python that would check at runtime if knitting is in progress by checking getOption('knitr.in.progress')), but this felt problematic because of possible concurrent calls, and the need to call R from the main thread.

This PR changes the default behavior of reticulate when rendering RMarkdown files, but users can easily go back to the previous behavior by setting include=FALSE in the chunk options. We also honor the RETICULATE_REMAP_OUTPUT_STREAMS if it's set:

  • if it's set to "0" we never set-up the hook and there's no output redirection.
  • If it's set to "1" we also don't set the hook as redirections are already in place.

Copy link
Member

@t-kalinowski t-kalinowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great @dfalbel. I left some comments in line.
Thank you!

R/package.R Show resolved Hide resolved
inst/python/rpytools/output.py Outdated Show resolved Hide resolved
tests/testthat/test-python-knitr-engine.R Outdated Show resolved Hide resolved
# if knitr is already in progress here, this means that python was initialized
# during a chunk execution. We have to force an instant remap as the hook won't
# have a chance to run for that chunk.
if (isTRUE(getOption('knitr.in.progress'))) set_output_streams(tty = FALSE)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might be misunderstand the order of events here. Does knitr call the knitr::knit_hooks with before = TRUE before it has resolved the chunk engines (meaning, before the reticulate namespace is loaded)?

My understanding is this code is either executing before knitr is loaded, or after, but never before reticulate has initialized python, and always before knitr has asked reticulate to process the chunk.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's possible that a code in an R engine tries to print to stdout from python (eg using reticulate), in such case the hook is called with before = TRUE and at this point reticulate might have not loaded python yet - or even reticulate might not yet been namespace attached yet.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, so it's possible this call to set_output_streams() will initialize python, yes?

Copy link
Member

@t-kalinowski t-kalinowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Look great, thank you!
Would you like to add a NEWS entry?

@t-kalinowski t-kalinowski merged commit a1512d7 into rstudio:main Jul 18, 2023
12 checks passed
@t-kalinowski
Copy link
Member

@dfalbel Can you please take a look at this failing test? I think it's related to this PR:
https://github.com/tidymodels/parsnip/actions/runs/5601457969/jobs/10253128187#step:8:581

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

knitr engine doesn't capture python stdout reticulate output in R chunks not captured in R Markdown
2 participants