Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🥷Add skip-execution tag support #1537

Merged
merged 6 commits into from
Sep 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/loud-points-listen.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"myst-execute": minor
---

Add skip-execution tag support
16 changes: 15 additions & 1 deletion docs/execute-notebooks.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ The following computational content will be executed:
In order to execute your MyST content, you must install a Jupyter Server and the kernel needed to execute your code (e.g., the [IPython kernel](https://ipython.readthedocs.io/en/stable/), the [Xeus Python kernel](https://github.com/jupyter-xeus/xeus-python), or the [IRKernel](https://irkernel.github.io/).)
:::

## Expected errors
## Expect a code-cell to fail

By default, MyST will stop executing a notebook if a cell raises an error.
If instead you'd like MyST to continue executing subsequent cells (e.g., in order to demonstrate an expected error message), add the `raises-exception` tag to the cell.
Expand All @@ -47,6 +47,20 @@ print("Hello" + 10001)
```
````

## Skip particular code-cells

Sometimes, you might have a notebook containing code that you _don't_ want to execute. For example, you might have code-cells that prompt the user for input, which should be skipped during a website build. MyST understands the same `skip-execution` cell-tag that other Jupyter Notebook tools (such as Jupyter Book) use to prevent a cell from being executed.

Using the {myst:directive}`code-cell` directive, the `skip-execution` tag can be added as follows:

````markdown
```{code-cell}
:tags: skip-execution

name = input("What is your name?")
```
````

## Cache execution outputs

When MyST executes your notebook, it will store the outputs in a cache in a folder called `execute/` in your MyST build folder.
Expand Down
64 changes: 50 additions & 14 deletions packages/myst-execute/src/execute.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import { select, selectAll } from 'unist-util-select';
import type { Logger } from 'myst-cli-utils';
import type { PageFrontmatter, KernelSpec } from 'myst-frontmatter';
import type { Kernel, KernelMessage, Session, SessionManager } from '@jupyterlab/services';
import type { Code, InlineExpression } from 'myst-spec-ext';
import type { Block, Code, InlineExpression, Output } from 'myst-spec-ext';
import type { IOutput } from '@jupyterlab/nbformat';
import type { GenericNode, GenericParent, IExpressionResult, IExpressionError } from 'myst-common';
import { NotebookCell, fileError } from 'myst-common';
Expand Down Expand Up @@ -106,7 +106,7 @@ async function evaluateExpression(kernel: Kernel.IKernelConnection, expr: string
*
* @param nodes array of executable nodes
*/
function buildCacheKey(kernelSpec: KernelSpec, nodes: (ICellBlock | InlineExpression)[]): string {
function buildCacheKey(kernelSpec: KernelSpec, nodes: (CodeBlock | InlineExpression)[]): string {
// Build an array of hashable items from an array of nodes
const hashableItems: {
kind: string;
Expand All @@ -118,7 +118,7 @@ function buildCacheKey(kernelSpec: KernelSpec, nodes: (ICellBlock | InlineExpres
hashableItems.push({
kind: node.type,
content: (select('code', node) as Code).value,
raisesException: !!node.data?.tags?.includes?.('raises-exception'),
raisesException: codeBlockRaisesException(node),
});
} else {
assert(isInlineExpression(node));
Expand All @@ -137,23 +137,55 @@ function buildCacheKey(kernelSpec: KernelSpec, nodes: (ICellBlock | InlineExpres
.digest('hex');
}

type ICellBlockOutput = GenericNode & {
/**
* Type narrowing Output to contain IOutput data
*
* TODO: lift this to the myst-spec definition
*/
type CodeBlockOutput = Output & {
data: IOutput[];
};

type ICellBlock = GenericNode & {
children: (Code | ICellBlockOutput)[];
/**
* Type narrowing Block to contain code-cells and code-cell outputs
*
* TODO: lift this to the myst-spec definition
*/

type CodeBlock = Block & {
kind: 'code';
data?: {
tags?: string[];
};
children: (Code | CodeBlockOutput)[];
};

/**
* Return true if the given node is a block over a code node and output node
*
* @param node node to test
*/
function isCellBlock(node: GenericNode): node is ICellBlock {
function isCellBlock(node: GenericNode): node is CodeBlock {
return node.type === 'block' && select('code', node) !== null && select('output', node) !== null;
}

/**
* Return true if the given code block is expected to raise an exception
*
* @param node block to test
*/
function codeBlockRaisesException(node: CodeBlock) {
return !!node.data?.tags?.includes?.('raises-exception');
}
/**
* Return true if the given code block should not be executed
*
* @param node block to test
*/
function codeBlockSkipsExecution(node: CodeBlock) {
return !!node.data?.tags?.includes?.('skip-execution');
}

/**
* Return true if the given node is an inlineExpression node
*
Expand All @@ -173,7 +205,7 @@ function isInlineExpression(node: GenericNode): node is InlineExpression {
*/
async function computeExecutableNodes(
kernel: Kernel.IKernelConnection,
nodes: (ICellBlock | InlineExpression)[],
nodes: (CodeBlock | InlineExpression)[],
opts: { vfile: VFile },
): Promise<{
results: (IOutput[] | IExpressionResult)[];
Expand All @@ -191,7 +223,7 @@ async function computeExecutableNodes(
results.push(outputs);

// Check for errors
const allowErrors = !!matchedNode.data?.tags?.includes?.('raises-exception');
const allowErrors = codeBlockRaisesException(matchedNode);
if (status === 'error' && !allowErrors) {
const errorMessage = outputs
.map((item) => item.traceback)
Expand Down Expand Up @@ -242,7 +274,7 @@ async function computeExecutableNodes(
* @param computedResult computed results for each node
*/
function applyComputedOutputsToNodes(
nodes: (ICellBlock | InlineExpression)[],
nodes: (CodeBlock | InlineExpression)[],
computedResult: (IOutput[] | IExpressionResult)[],
) {
for (const matchedNode of nodes) {
Expand Down Expand Up @@ -286,10 +318,14 @@ export async function kernelExecutionTransform(tree: GenericParent, vfile: VFile
const log = opts.log ?? console;

// Pull out code-like nodes
const executableNodes = selectAll(`block[kind=${NotebookCell.code}],inlineExpression`, tree) as (
| ICellBlock
| InlineExpression
)[];
const executableNodes = (
selectAll(`block[kind=${NotebookCell.code}],inlineExpression`, tree) as (
| CodeBlock
| InlineExpression
)[]
)
// Filter out nodes that skip execution
.filter((node) => !(isCellBlock(node) && codeBlockSkipsExecution(node)));

// Only do something if we have any nodes!
if (executableNodes.length === 0) {
Expand Down
49 changes: 49 additions & 0 deletions packages/myst-execute/tests/execute.yml
Original file line number Diff line number Diff line change
Expand Up @@ -268,3 +268,52 @@ cases:
# - "\e[0;31mValueError\e[0m: "
ename: ValueError
evalue: ''
- title: tree with bad executable code and `skip-execution` is not evaluated
before:
type: root
children:
- type: block
kind: notebook-code
data:
id: nb-cell-0
tags: skip-execution
identifier: nb-cell-0
label: nb-cell-0
html_id: nb-cell-0
children:
- type: code
lang: python
executable: true
value: raise ValueError
identifier: nb-cell-0-code
enumerator: 1
html_id: nb-cell-0-code
- type: output
id: T7FMDqDm8dM2bOT1tKeeM
identifier: nb-cell-0-output
html_id: nb-cell-0-output
data:
after:
type: root
children:
- type: block
kind: notebook-code
data:
id: nb-cell-0
tags: skip-execution
identifier: nb-cell-0
label: nb-cell-0
html_id: nb-cell-0
children:
- type: code
lang: python
executable: true
value: raise ValueError
identifier: nb-cell-0-code
enumerator: 1
html_id: nb-cell-0-code
- type: output
id: T7FMDqDm8dM2bOT1tKeeM
identifier: nb-cell-0-output
html_id: nb-cell-0-output
data:
Loading