Skip to content

Commit

Permalink
[PYG-19]💡Use with Cognite Functions (#285)
Browse files Browse the repository at this point in the history
* docs: update documentatioon

* docs: setup start page for cognite functions

* docs: how to deploy function

* docs: adjustments
  • Loading branch information
doctrino authored Aug 18, 2024
1 parent 5e247c9 commit 6ccf292
Show file tree
Hide file tree
Showing 6 changed files with 108 additions and 18 deletions.
6 changes: 3 additions & 3 deletions cognite/pygen/_build.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,8 @@ def build_wheel(
the package will be [external_id:snake] of the first data model given, while
the client name will be [external_id:pascal_case]
client_name: The name of the client class. For example, `APMClient`. See above for more details.
default_instance_space: The default instance space to use for the generated SDK. Defaults to the
instance space of the first data model given.
default_instance_space: The default instance space to use for the generated SDK. If not provided,
the space must be specified when creating, deleting, and retrieving nodes and edges.
output_dir: The location to output the generated SDK wheel. Defaults to "dist".
format_code: Whether to format the generated code using black. Defaults to True.
config: The configuration used to control how to generate the SDK.
Expand Down Expand Up @@ -107,7 +107,7 @@ def generate_pyproject_toml(build_dir: Path, package_name: str) -> None:
pyproject_toml.write_text(
f"""[project]
name = "{package_name}"
version = "0.1.0"
version = "1.0.0"
dependencies = [
"cognite-sdk>={cognite_sdk_version}",
"pydantic>={PYDANTIC_VERSION}",
Expand Down
6 changes: 3 additions & 3 deletions cognite/pygen/_generator.py
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ def generate_sdk(
the client name will be [external_id:pascal_case]
client_name: The name of the client class. For example, `APMClient`. See above for more details.
default_instance_space: The default instance space to use for the generated SDK. If not provided,
all nodes and edges as well as filtering must specify instance space in the generated SDK.
the space must be specified when creating, deleting, and retrieving nodes and edges.
output_dir: The location to output the generated SDK. Defaults to the current working directory.
logger: A logger function to log progress. Defaults to print.
pydantic_version: The version of pydantic to use. Defaults to "infer" which will use
Expand Down Expand Up @@ -163,8 +163,8 @@ def generate_sdk_notebook(
the package will be [external_id:snake] of the first data model given, while
the client name will be [external_id:pascal_case]
client_name: The name of the client class. For example, `APMClient`. See above for more details.
default_instance_space: The default instance space to use for the generated SDK. Defaults to the
instance space of the first data model given.
default_instance_space: The default instance space to use for the generated SDK. If not provided,
the space must be specified when creating, deleting, and retrieving nodes and edges.
config: The configuration used to control how to generate the SDK.
clean_pygen_temp_dir: Whether to clean the temporary directory used to store the generated SDK.
Defaults to True.
Expand Down
97 changes: 97 additions & 0 deletions docs/cognite_functions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
## Introduction

After you have built your logic with the newly generated data model SDK, you might want to deploy it to
Cognite Functions.

Cognite Functions enables Python code to be hosted and executed in the cloud, on demand or by using a schedule.

A few useful links for Cognite Functions:

- [Cognite Functions documentation](https://docs.cognite.com/cdf/functions/)
- Deploy and manage your Cognite Functions with [Cognite Toolkit](https://docs.cognite.com/cdf/deploy/cdf_toolkit/)
- Expected folder structure for Cognite Toolkit [here](https://docs.cognite.com/cdf/deploy/cdf_toolkit/references/configs#functions)
- [What is a wheel?](https://realpython.com/python-wheels/)

## Deploying your generated SDK to Cognite Functions

Cognite Functions supports private packages in your function deployment, and `pygen` can generate a `wheel` package of
your data model SDK. To generate a `wheel` package, use the [build_wheel](api/api.html#cognite.pygen.build_wheel) function.

For example, given the data model (`power-models`, `Windmill`, `1`) we generated in the
[Generation](usage/generation.html) guide,
we can build a `wheel` package with the following code:

```python
from cognite.client import CogniteClient
from cognite.pygen import build_wheel

client = CogniteClient()

build_wheel(("power-models", "Windmill", 1), client)
```

This will output a `wheel` package, `windmill-1.0.0-py3-none-any.whl`, in the `dist` folder of the current working directory.
**Note** The version of the package will always be `1.0.0` as the version of a data model can be an arbitrary string,
while the version of a package must be a valid semantic version.

To deploy generated SDK to Cognite Functions, you can use the `cognite-toolkit` CLI with the following structure,
```
📦functions/
┣ 📂my_function - The function folder
┃ ┣ 📦windmill-1.0.0-py3-none-any.whl - The generated wheel package
┃ ┣ 📜__init__.py - Empty file (required to make the function into a package)
┃ ┣ 📜handler.py - Module with script inside a handle function
┃ ┗ 📜requirements.txt - Explicitly states the dependencies needed to run the handler.py script.
┣ 📜my_function.Function.yaml - The configuration file for the function
┣ 📜my_schedule.Schedule.yaml - The configuration file for the function schedule(s)
```

=== "handler.py"

```python
from cognite.client import CogniteClient
from windmill WindmillClient


def handle(client: CogniteClient, data: dict):
windmill_client = WindmillClient(client)
windmills = windmill_client.windmill.list(limit=10)
print(windmills)
return "Success"

```

=== "requirements.txt"

```txt
# The Cognite SDK should be specified and match the version used to generate the SDK.
cognite-sdk==7.54.4
# This is the syntax for adding a private wheel in a Cognite Function.
# Note the 'function' prefix is not a placeholder, it is a required prefix for private packages.
function/windmill-1.0.0-py3-none-any.whl
```

=== "my_function.Function.yaml"

```yaml
name: my_function
externalId: fn_my_function
owner: Anonymous
description: 'Demo of a function with a pygen generated SDK'
runtime: 'py311'
```

=== "my_schedule.Schedule.yaml"

```yaml
- name: "daily-8am-utc"
functionExternalId: fn_my_function
description: "Run every day at 8am UTC"
cronExpression: "0 8 * * *"
data:
some: "data"
```

After you have the folder structure set up, you can deploy the function using the `cognite-toolkit` CLI,
go to [Deploy](https://docs.cognite.com/cdf/deploy/cdf_toolkit/guides/configure_deploy_modules)
for more information on how to use the CLI.
10 changes: 1 addition & 9 deletions docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,23 +9,15 @@

```
python -m venv venv
```
```
venv\Scripts\activate.bat
```
```
pip install cognite-pygen[all]
```

=== "Mac/Linux"

``` bash
```bash
python -m venv venv
```
``` bash
source venv/bin/activate
```
``` bash
pip install "cognite-pygen[all]"
```

Expand Down
5 changes: 3 additions & 2 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,13 +31,14 @@ nav:
- Creating and Deleting: usage/creating_deleting.ipynb
- Working with Time Series: usage/timeseries.ipynb
- Working with Edges: usage/edge_properties.ipynb
- Cognite Functions: cognite_functions.md
- Extra:
- Generating Mock Data: extra/mock_data.ipynb
- Pygen Config:
- Home: config/index.md
- Naming: config/naming.md
- Filtering: config/filtering_methods.md
- Example Docs:
- Example SDK Docs:
- Home: example_docs/index.md
- Omni Client: example_docs/omni_client.md
- Omni APIs: example_docs/omni_apis.md
Expand All @@ -46,7 +47,7 @@ nav:
- Windmill APIs: example_docs/wind_apis.md
- Windmill Data Classes: example_docs/wind_data_classes.md

- API:
- Reference - Code API:
- Pygen: api/api.md
- Demo: api/demo.md
- Exceptions: api/exceptions.md
Expand Down
2 changes: 1 addition & 1 deletion tests/test_unit_slow/test_generator/test_build.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ def test_build_wheel() -> None:
data_model = OMNI_SDK.load_data_model()
local_dir = Path(__file__).parent / "dist"
build_wheel(data_model, top_level_package="omni_sdk", output_dir=local_dir)
expected_file = local_dir / "omni_sdk-0.1.0-py3-none-any.whl"
expected_file = local_dir / "omni_sdk-1.0.0-py3-none-any.whl"
assert expected_file.exists()

# Clean up
Expand Down

0 comments on commit 6ccf292

Please sign in to comment.