Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Draft: #756 - implement python workflow submissions #762

Open
wants to merge 24 commits into
base: 1.9.latest
Choose a base branch
from
Open

Draft: #756 - implement python workflow submissions #762

wants to merge 24 commits into from
+270 −0

Commits on Aug 8, 2024

  1. databricks#756 - stub out implementation for python workflow submissions

    Kyle Valade committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    f03b3e5 View commit details
    Browse the repository at this point in the history

Commits on Aug 12, 2024

  1. Make this runnable

    Kyle Valade committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    1f8a17e View commit details
    Browse the repository at this point in the history

  2. Allow for job updates

    Kyle Valade committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    16cb9df View commit details
    Browse the repository at this point in the history

  3. doccomments to code

    Kyle Valade committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    f92af82 View commit details
    Browse the repository at this point in the history

  4. Allow for existing cluster id; pull notebook paths + dirs into proper… 

    …ties
    Kyle Valade committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    36be7c3 View commit details
    Browse the repository at this point in the history

  5. Build path from dir, not vice versa

    Kyle Valade committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    8affd9b View commit details
    Browse the repository at this point in the history

  6. Remove comments

    Kyle Valade committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    2e5e09d View commit details
    Browse the repository at this point in the history

  7. linting 

    Signed-off-by: Kyle Valade <kylevalade@rivian.com>
    Kyle Valade committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    0ba3c1d View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2024

  1. Allow user to specify an existing workflow/job id

    Kyle Valade committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    b4dfbe7 View commit details
    Browse the repository at this point in the history

  2. Don't override job name - provide a default

    Kyle Valade committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    4a46b79 View commit details
    Browse the repository at this point in the history

  3. Allow additional tasks to get added to the workflow as 'post_hook_tasks'

    Kyle Valade committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    afc1b5a View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2024

  1. Allow for additional model task settings; allow permissions to be set… 

    … on job

Signed-off-by: Kyle Valade <kylevalade@rivian.com>
    Kyle Valade committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    df173aa View commit details
    Browse the repository at this point in the history

  2. Update permissions regardless of whether the job is new or not 

    Signed-off-by: Kyle Valade <kylevalade@rivian.com>
    Kyle Valade committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    0078058 View commit details
    Browse the repository at this point in the history

  3. Don't skip setting job grants if they aren't defined, as they should … 

    …be reset

Signed-off-by: Kyle Valade <kylevalade@rivian.com>
    Kyle Valade committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    3664df2 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. Use different run endpoint for result status

    Kyle Valade committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    b3f6dbf View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2024

  1. If a workflow is already running, return the active run id instead of… 

    … attempting to trigger a new run
    Kyle Valade committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    3f100c4 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2024

  1. Allow for serverless tasks

    Kyle Valade committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    13033ed View commit details
    Browse the repository at this point in the history

  2. BROKEN: starting to merge with 1.9.latest. Need to fix api_client calls

    Kyle Valade committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    0985af6 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. Add new API classes to support workflow jobs

    Kyle Valade committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    dfaa6cc View commit details
    Browse the repository at this point in the history

  2. Add lots of types

    Kyle Valade committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    81996f4 View commit details
    Browse the repository at this point in the history

  3. Fix types for python 3.8

    Kyle Valade committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    214372b View commit details
    Browse the repository at this point in the history

  4. Missed a Tuple

    Kyle Valade committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    c19b196 View commit details
    Browse the repository at this point in the history

  5. Use FolderApi; change dbt namespacing placement

    Kyle Valade committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    e9bc5fc View commit details
    Browse the repository at this point in the history

  6. Queue each workflow run behind an active run

    Kyle Valade committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    743ca00 View commit details
    Browse the repository at this point in the history