Docs revamp (#180)

* Add methods to API docs and use typehints

Closes #160 #161

* Use the Pydata theme for docs

Closes #164

* Add methods to API docs and use typehints

Closes #160 #161

* Use the Pydata theme for docs

Closes #164

* Hide the top menu navigation from the index page

* Docs revamp

Start of a big revamp of the docs.
- Uses Pydata theme
- Enables MyST markdown and converts a lot of the docs to it (rst heading levels and git conflict formatting conflict, which was a fun discovery)
- Re-arranges things along the Diátaxis framework lines, or at least how they are being used in the PyData space
- Adds the vision or 'Why Xpublish' page
- Breaks the tutorial into sections and scaffolds sections for use of plugins

Closes #163 #165 #162 #166

* Update xarray-contrib to xpublish-community

* Pre-commit format markdown and docs code snippets

* Dataset router plugin tutorial and other doc tweaks

* Use literalinclude rather than duplicating doc code

* Remove some under-construction alerts

.pre-commit-config.yaml

@@ -9,6 +9,20 @@ repos:
       - id: check-yaml
       - id: double-quote-string-fixer
 
+  - repo: https://github.com/executablebooks/mdformat
+    rev: 0.7.16
+    hooks:
+      - id: mdformat
+        additional_dependencies:
+          - mdformat-myst
+
+  - repo: https://github.com/adamchainz/blacken-docs
+    rev: "1.13.0"
+    hooks:
+      - id: blacken-docs
+        additional_dependencies:
+          - black==23.1.0
+
   - repo: https://github.com/psf/black
     rev: 23.3.0
     hooks:
@@ -33,3 +47,5 @@ repos:
     hooks:
       - id: prettier
         language_version: system
+        exclude_types:
+          - markdown # managed by mdformat

README.rst

@@ -37,10 +37,10 @@ Here is an example of directly accessing the data from within Python:
    from fsspec.implementations.http import HTTPFileSystem
 
    fs = HTTPFileSystem()
-   http_map = fs.get_mapper('http://0.0.0.0:9000')
+   http_map = fs.get_mapper("http://0.0.0.0:9000")
 
    # open as a zarr group
-   zg = zarr.open_consolidated(http_map, mode='r')
+   zg = zarr.open_consolidated(http_map, mode="r")
 
    # or open as another Xarray Dataset
    ds = xr.open_zarr(http_map, consolidated=True)

docs/requirements.txt

@@ -3,3 +3,5 @@ sphinx-autosummary-accessors
 pydata-sphinx-theme
 sphinx-autodoc-typehints
 autodoc_pydantic
+myst-nb
+sphinx-design

docs/source/api.rst → docs/source/api.md

@@ -1,17 +1,17 @@
+```{eval-rst}
 .. currentmodule:: xpublish
+```
 
-#############
-API reference
-#############
+# API reference
 
-Top-level Rest class
-====================
+## Top-level Rest class
 
-The :class:`~xpublish.Rest` class can be used for publishing a
-:class:`xarray.Dataset` object or a collection of Dataset objects.
+The {class}`~xpublish.Rest` class can be used for publishing a
+{class}`xarray.Dataset` object or a collection of Dataset objects.
 
 The main interfaces to Xpublish that many users may use.
 
+```{eval-rst}
 .. autosummary::
    :toctree: generated/
 
@@ -22,11 +22,13 @@ The main interfaces to Xpublish that many users may use.
    Rest.serve
    Rest.register_plugin
    Rest.dependencies
+```
 
 There are also a handful of methods that are more likely to be used
 when subclassing `xpublish.Rest` to modify functionality, or are used
 by plugin dependencies.
 
+```{eval-rst}
 .. autosummary::
    :toctree: generated/
 
@@ -37,78 +39,92 @@ by plugin dependencies.
    Rest.init_cache_kwargs
    Rest.init_app_kwargs
    Rest.plugin_routers
+```
 
-There is also a specialized version of :class:`xpublish.Rest` for use
+There is also a specialized version of {class}`xpublish.Rest` for use
 when only a single dataset is being served, instead of a collection
 of datasets.
 
+```{eval-rst}
 .. autosummary::
    :toctree: generated/
 
    SingleDatasetRest
    SingleDatasetRest.setup_datasets
+```
 
-For serving a single dataset the :class:`~xpublish.SingleDatasetRest` is used instead.
+For serving a single dataset the {class}`~xpublish.SingleDatasetRest` is used instead.
 
+```{eval-rst}
 .. autosummary::
    :toctree: generated/
 
    SingleDatasetRest
+```
 
-Dataset.rest (xarray accessor)
-==============================
+## Dataset.rest (xarray accessor)
 
-This accessor extends :py:class:`xarray.Dataset` with the same interface than
-:class:`~xpublish.SingleDatasetRest`. It is a convenient method for publishing one single
+This accessor extends {py:class}`xarray.Dataset` with the same interface than
+{class}`~xpublish.SingleDatasetRest`. It is a convenient method for publishing one single
 dataset. Proper use of this accessor should be like:
 
-.. code-block:: python
-
-   >>> import xarray as xr         # first import xarray
-   >>> import xpublish             # import xpublish (the dataset 'rest' accessor is registered)
-   >>> ds = xr.Dataset()           # create or load an xarray Dataset
-   >>> ds.rest(...)                # call the 'rest' accessor on the dataset
-   >>> ds.rest.<meth_or_prop>      # access to the methods and properties listed below
+```
+>>> import xarray as xr         # first import xarray
+>>> import xpublish             # import xpublish (the dataset 'rest' accessor is registered)
+>>> ds = xr.Dataset()           # create or load an xarray Dataset
+>>> ds.rest(...)                # call the 'rest' accessor on the dataset
+>>> ds.rest.<meth_or_prop>      # access to the methods and properties listed below
+```
 
+```{eval-rst}
 .. currentmodule:: xarray
+```
 
 **Calling the accessor**
 
+```{eval-rst}
 .. autosummary::
    :toctree: generated/
    :template: autosummary/accessor_callable.rst
 
    Dataset.rest
+```
 
 **Properties**
 
+```{eval-rst}
 .. autosummary::
    :toctree: generated/
    :template: autosummary/accessor_attribute.rst
 
    Dataset.rest.app
    Dataset.rest.cache
+```
 
 **Methods**
 
+```{eval-rst}
 .. autosummary::
    :toctree: generated/
    :template: autosummary/accessor_method.rst
 
    Dataset.rest.serve
+```
 
-FastAPI dependencies
-====================
+## FastAPI dependencies
 
-The functions below are defined in module ``xpublish.dependencies`` and can
-be used as `FastAPI dependencies <https://fastapi.tiangolo.com/tutorial/dependencies>`_
+The functions below are defined in module `xpublish.dependencies` and can
+be used as [FastAPI dependencies](https://fastapi.tiangolo.com/tutorial/dependencies)
 when creating custom API endpoints directly.
 
-When creating routers with plugins, instead use ``xpublish.Dependency`` that will be
-passed in to the ``Plugin.app_router`` or ``Plugin.dataset_router`` method.
+When creating routers with plugins, instead use `xpublish.Dependency` that will be
+passed in to the `Plugin.app_router` or `Plugin.dataset_router` method.
 
+```{eval-rst}
 .. currentmodule:: xpublish.dependencies
+```
 
+```{eval-rst}
 .. autosummary::
    :toctree: generated/
 
@@ -119,13 +135,17 @@ passed in to the ``Plugin.app_router`` or ``Plugin.dataset_router`` method.
    get_zmetadata
    get_plugins
    get_plugin_manager
+```
 
-Plugins
+## Plugins
 
-Plugins are inherit from the :class:`~xpublish.Plugin` class, and implement various hooks.
+Plugins are inherit from the {class}`~xpublish.Plugin` class, and implement various hooks.
 
+```{eval-rst}
 .. currentmodule:: xpublish
+```
 
+```{eval-rst}
 .. autosummary::
    :toctree: generated/
 
@@ -137,3 +157,4 @@ Plugins are inherit from the :class:`~xpublish.Plugin` class, and implement vari
    plugins.manage.find_default_plugins
    plugins.manage.load_default_plugins
    plugins.manage.configure_plugins
+```

docs/source/conf.py

@@ -46,11 +46,16 @@
     'sphinxcontrib.autodoc_pydantic',
     'sphinx_autosummary_accessors',
     'sphinx_autodoc_typehints',
+    'sphinx_design',
+    'myst_parser',
 ]
 
+myst_enable_extensions = []
+myst_heading_anchors = 6
+
 extlinks = {
-    'issue': ('https://github.com/xarray-contrib/xpublish/issues/%s', 'GH#'),
-    'pr': ('https://github.com/xarray-contrib/xpublish/pull/%s', 'GH#'),
+    'issue': ('https://github.com/xpublish-community/xpublish/issues/%s', 'GH#'),
+    'pr': ('https://github.com/xpublish-community/xpublish/pull/%s', 'GH#'),
 }
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates', sphinx_autosummary_accessors.templates_path]
@@ -140,7 +145,7 @@
     'icon_links': [
         {
             'name': 'GitHub',
-            'url': 'https://github.com/xarray-contrib/xpublish',  # required
+            'url': 'https://github.com/xpublish-community/xpublish',  # required
             'icon': 'fa-brands fa-square-github',
             'type': 'fontawesome',
         }
@@ -149,7 +154,7 @@
 }
 
 html_context = {
-    'github_user': 'xarray-contrib',
+    'github_user': 'xpublish-community',
     'github_repo': 'xpublish',
     'github_version': 'main',
     'doc_path': 'docs/source/',

docs/source/contributing.rst

@@ -16,7 +16,7 @@ Feature requests and feedback
 Do you like Xpublish? Share some love on Twitter or in your blog posts!
 
 We'd also like to hear about your propositions and suggestions.  Feel free to
-`submit them as issues <https://github.com/xarray-contrib/xpublish>`_ and:
+`submit them as issues <https://github.com/xpublish-community/xpublish>`_ and:
 
 * Explain in detail how they should work.
 * Keep the scope as narrow as possible.  This will make it easier to implement.
@@ -26,7 +26,7 @@ We'd also like to hear about your propositions and suggestions.  Feel free to
 Report bugs
 -----------
 
-Report bugs for Xpublish in the `issue tracker <https://github.com/xarray-contrib/xpublish>`_.
+Report bugs for Xpublish in the `issue tracker <https://github.com/xpublish-community/xpublish>`_.
 
 If you are reporting a bug, please include:
 
@@ -45,7 +45,7 @@ fix the bug itself.
 Fix bugs
 --------
 
-Look through the `GitHub issues for bugs <https://github.com/xarray-contrib/xpublish/labels/type:%20bug>`_.
+Look through the `GitHub issues for bugs <https://github.com/xpublish-community/xpublish/labels/type:%20bug>`_.
 
 Talk to developers to find out how you can fix specific bugs.
 
@@ -81,7 +81,7 @@ Preparing Pull Requests
 -----------------------
 
 #. Fork the
-   `xpublish GitHub repository <https://github.com/xarray-contrib/xpublish>`__.  It's
+   `xpublish GitHub repository <https://github.com/xpublish-community/xpublish>`__.  It's
    fine to use ``xpublish`` as your fork repository name because it will live
    under your user.
 
@@ -134,5 +134,5 @@ Preparing Pull Requests
     head-fork: YOUR_GITHUB_USERNAME/xpublish
     compare: your-branch-name
 
-    base-fork: xarray-contrib/xpublish
+    base-fork: xpublish-community/xpublish
     base: main

docs/source/ecosystem/index.md

@@ -0,0 +1,31 @@
+# Ecosystem
+
+Xpublish's ecosystem is made up of plugins and servers, and the folks who build and run them.
+
+## Connect
+
+We have two main venues for discussing Xpublish and it's ecosystem, Github Discussions and Slack.
+
+### Github Discussions
+
+For longer form discussions, we can be found in [Github Discussions](https://github.com/xpublish-community/community/discussions?discussions_q=).
+
+### Slack
+
+Xpublish has a channel (`#xpublish`) on [ESIP](https://www.esipfed.org/)'s (Earth Science Information Partners) Slack. (Insert Rich's justification here about ESIP being the biggest unbrella that he can find) [Join here](https://join.slack.com/t/esip-all/shared_invite/zt-1omjufm9z-iH8Gf7gmmsm2SiS5Xh6BlQ)
+
+## Server distributions
+
+- [XREDS](https://github.com/asascience-open/xreds) from RPS
+- [Xpublish Host](https://github.com/axiom-data-science/xpublish-host) from Axiom Data Science
+
+## Plugins
+
+- [OGC EDR](https://github.com/xpublish-community/xpublish-edr/)
+- [OpenDAP](https://github.com/xpublish-community/xpublish-opendap/)
+- [WMS](https://github.com/xpublish-community/xpublish-wms)
+- [Intake](https://github.com/axiom-data-science/xpublish-intake)
+
+## Experiments
+
+We have a separate Github organization ([xpublish-experiments](https://github.com/xpublish-experiments)) for things that you've tried and want to share with the community, but don't necessarily want to maintain.

docs/source/getting-started/index.md

@@ -0,0 +1,13 @@
+# Getting started
+
+The getting started guide aims to get you using Xpublish productively as quickly as possible.
+It is designed as an entry point for new users, and it provided an introduction to Xpublish’s main concepts.
+
+```{toctree}
+---
+hidden:
+---
+why-xpublish
+installation
+tutorial/index
+```

docs/source/getting-started/tutorial/dataset-provider-plugin.md

@@ -0,0 +1,14 @@
+# Building a dataset provider plugin
+
+So far, we've been eagerly loading datasets for Xpublish to serve, but this tends not to scale well between memory needs and slow startup. Xpublish plugins can also be __Dataset Providers__ and handle loading of datasets on request.
+
+This also allows organizations to quickly be able to adapt Xpublish to work in their own environment, rather than needing Xpublish to explicitly support it.
+
+```{literalinclude} dataset-provider-plugin.py
+```
+
+With this plugin, Xpublish can serve the same datasets as we explictly defined and loaded in [serving multiple datasets](./serving-multiple-datasets.md), as well as any others supported by [`xr.tutorial`](https://github.com/pydata/xarray/blob/main/xarray/tutorial.py)
+
+```{note}
+For more details on building dataset provider plugins, please see the [plugin user guide](../../user-guide/plugins.md#dataset-provider-plugins)
+```

docs/source/getting-started/tutorial/dataset-provider-plugin.py

@@ -0,0 +1,24 @@
+import xarray as xr
+from requests import HTTPError
+
+from xpublish import Plugin, Rest, hookimpl
+
+
+class TutorialDataset(Plugin):
+    name = 'xarray-tutorial-dataset'
+
+    @hookimpl
+    def get_datasets(self):
+        return list(xr.tutorial.file_formats)
+
+    @hookimpl
+    def get_dataset(self, dataset_id: str):
+        try:
+            return xr.tutorial.open_dataset(dataset_id)
+        except HTTPError:
+            return None
+
+
+rest = Rest({})
+rest.register_plugin(TutorialDataset())
+rest.serve()