[Algorithm] GAIL #2273

BY571 · 2024-07-08T08:58:01Z

Description

Add the algorithm Generative Adversarial Imitation Learning GAIL

Motivation and Context

Why is this change required? What problem does it solve?
If it fixes an open issue, please link to the issue here.
You can use the syntax close #15213 if this solves the issue #15213

I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

What types of changes does your code introduce? Remove all that do not apply:

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds core functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation (update in the documentation)
Example (update in the folder of examples)

Checklist

Go over all the following points, and put an x in all the boxes that apply.
If you are unsure about any of these, don't hesitate to ask. We are here to help!

I have read the CONTRIBUTION guide (required)
My change requires a change to the documentation.
I have updated the tests accordingly (required for a bug fix or a new feature).
I have updated the documentation accordingly.

pytorch-bot · 2024-07-08T08:58:04Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2273

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 5 New Failures, 1 Cancelled Job, 7 Unrelated Failures

As of commit c488bcd with merge base 788710f ():

NEW FAILURES - The following jobs have failed:

Continuous Benchmark (PR) / CPU Pytest benchmark (gh)
Workflow failed! Resource not accessible by integration
Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
Workflow failed! Resource not accessible by integration
Habitat Tests on Linux / tests (3.9, 12.1) / linux-job (gh)
RuntimeError: Command docker exec -t 642784d066744e775b31bf49eb9dc5085eb84a2a4a5d6895ca6e87b3ba7ba062 /exec failed with exit code 139
Wheels / build-wheel-windows (3.11, 3.11) (gh)
Process completed with exit code 1.
Wheels / build-wheel-windows (3.8, 3.8) (gh)
Process completed with exit code 1.

CANCELLED JOB - The following job was cancelled. Please retry:

Wheels / build-wheel-windows (3.12, 3.12) (gh)

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

Wheels / build-wheel-windows (3.10, 3.10.3) (gh) (matched win rule in flaky-rules.json)
##[error]The operation was canceled.
Wheels / build-wheel-windows (3.9, 3.9) (gh) (matched win rule in flaky-rules.json)
##[error]The operation was canceled.

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

Build Windows Wheels / pytorch/rl (pytorch/rl, python packaging/wheel/relocate.py, test/smoke_test.py, torchrl) / upload / wheel-py3_9-cpu (gh) (trunk failure)
Unable to find any artifacts for the associated workflow
Build Windows Wheels / pytorch/rl (pytorch/rl, python packaging/wheel/relocate.py, test/smoke_test.py, torchrl) / upload / wheel-py3_9-cuda11_8 (gh) (trunk failure)
Unable to find any artifacts for the associated workflow
Build Windows Wheels / pytorch/rl (pytorch/rl, python packaging/wheel/relocate.py, test/smoke_test.py, torchrl) / upload / wheel-py3_9-cuda12_1 (gh) (trunk failure)
Unable to find any artifacts for the associated workflow
Build Windows Wheels / pytorch/rl (pytorch/rl, python packaging/wheel/relocate.py, test/smoke_test.py, torchrl) / upload / wheel-py3_9-cuda12_4 (gh) (trunk failure)
Unable to find any artifacts for the associated workflow
Unit-tests on Windows / unittests-cpu / windows-job (gh) (trunk failure)
Process completed with exit code 1.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

vmoens

Let's just add the device in the config and we're good to go!
Thanks for this, fab work

sota-implementations/gail/gail.py

vmoens · 2024-07-30T21:59:10Z

@BY571 The examples fail with:

100% 48/48 [00:00<00:00, 53.31it/s]
  python .github/unittest/helpers/coverage_run_parallel.py sota-implementations/gail/gail.py \
  collector.total_frames=48 \
  loss.mini_batch_size=10 \
  collector.frames_per_batch=16 \
  logger.mode=offline \
  logger.backend=
/pytorch/rl/sota-implementations/gail/gail.py:31: UserWarning: 
The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
  @hydra.main(config_path="", config_name="config")
omegaconf.errors.ConfigAttributeError: Key 'collector' is not in struct

BY571 · 2024-07-31T08:06:56Z

@BY571 The examples fail with:

100% 48/48 [00:00<00:00, 53.31it/s]
  python .github/unittest/helpers/coverage_run_parallel.py sota-implementations/gail/gail.py \
  collector.total_frames=48 \
  loss.mini_batch_size=10 \
  collector.frames_per_batch=16 \
  logger.mode=offline \
  logger.backend=
/pytorch/rl/sota-implementations/gail/gail.py:31: UserWarning: 
The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
  @hydra.main(config_path="", config_name="config")
omegaconf.errors.ConfigAttributeError: Key 'collector' is not in struct

should be fixed also the device selection @vmoens.

vmoens · 2024-08-03T00:25:14Z

@BY571 I think we need to tell that the backend is gymansium, not gym:

  python .github/unittest/helpers/coverage_run_parallel.py sota-implementations/gail/gail.py \
  ppo.collector.total_frames=48 \
  ppo.loss.mini_batch_size=10 \
  ppo.collector.frames_per_batch=16 \
  logger.mode=offline \
  logger.backend=
/pytorch/rl/sota-implementations/gail/gail.py:31: UserWarning: 
The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
  @hydra.main(config_path="", config_name="config")
/pytorch/rl/env/lib/python3.9/site-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
  ret = run_job(
Error executing job with overrides: ['ppo.collector.total_frames=48', 'ppo.loss.mini_batch_size=10', 'ppo.collector.frames_per_batch=16', 'logger.mode=offline', 'logger.backend=']
Traceback (most recent call last):
  File "/pytorch/rl/sota-implementations/gail/gail.py", line 281, in <module>
    main()
  File "/pytorch/rl/env/lib/python3.9/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/pytorch/rl/env/lib/python3.9/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/pytorch/rl/env/lib/python3.9/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/pytorch/rl/env/lib/python3.9/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/pytorch/rl/env/lib/python3.9/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/pytorch/rl/env/lib/python3.9/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/pytorch/rl/env/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/pytorch/rl/env/lib/python3.9/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/pytorch/rl/env/lib/python3.9/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/pytorch/rl/sota-implementations/gail/gail.py", line 72, in main
    actor, critic = make_ppo_models(cfg.env.env_name)
  File "/pytorch/rl/sota-implementations/gail/ppo_utils.py", line 121, in make_ppo_models
    proof_environment = make_env(env_name, device="cpu")
  File "/pytorch/rl/sota-implementations/gail/ppo_utils.py", line 31, in make_env
    env = GymEnv(env_name, device=device, from_pixels=from_pixels, pixels_only=False)
  File "/pytorch/rl/torchrl/envs/libs/gym.py", line 571, in __call__
    instance: GymWrapper = super().__call__(*args, **kwargs)
  File "/pytorch/rl/torchrl/envs/common.py", line 176, in __call__
    instance: EnvBase = super().__call__(*args, **kwargs)
  File "/pytorch/rl/torchrl/envs/libs/gym.py", line 1371, in __init__
    super().__init__(**kwargs)
  File "/pytorch/rl/torchrl/envs/libs/gym.py", line 784, in __init__
    super().__init__(**kwargs)
  File "/pytorch/rl/torchrl/envs/common.py", line 3047, in __init__
    self._env = self._build_env(**kwargs)  # writes the self._env attribute
  File "/pytorch/rl/torchrl/envs/libs/gym.py", line 1423, in _build_env
    env = self.lib.make(env_name, **kwargs)
  File "/pytorch/rl/env/lib/python3.9/site-packages/gym/envs/registration.py", line 676, in make
    return registry.make(id, **kwargs)
  File "/pytorch/rl/env/lib/python3.9/site-packages/gym/envs/registration.py", line 518, in make
    spec = self.spec(path)
  File "/pytorch/rl/env/lib/python3.9/site-packages/gym/envs/registration.py", line 540, in spec
    return self.env_specs[id]
  File "/pytorch/rl/env/lib/python3.9/site-packages/gym/envs/registration.py", line 376, in __getitem__
    self._assert_version_exists(namespace, name, version)
  File "/pytorch/rl/env/lib/python3.9/site-packages/gym/envs/registration.py", line 369, in _assert_version_exists
    raise error.VersionNotFound(message)
gym.error.VersionNotFound: Environment version `v4` for `HalfCheetah` could not be found. `HalfCheetah` provides the versioned environments: [ `v2`, `v3` ].

vmoens · 2024-08-05T20:25:57Z

Still an error in the CI

  File "/pytorch/rl/sota-implementations/gail/gail.py", line 281, in <module>
    main()
  File "/pytorch/rl/env/lib/python3.9/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/pytorch/rl/env/lib/python3.9/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/pytorch/rl/env/lib/python3.9/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/pytorch/rl/env/lib/python3.9/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/pytorch/rl/env/lib/python3.9/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/pytorch/rl/env/lib/python3.9/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/pytorch/rl/env/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/pytorch/rl/env/lib/python3.9/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/pytorch/rl/env/lib/python3.9/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/pytorch/rl/sota-implementations/gail/gail.py", line 168, in main
    expert_data.set(
  File "/pytorch/rl/env/lib/python3.9/site-packages/tensordict/base.py", line 4294, in set
    return self._set_tuple(
  File "/pytorch/rl/env/lib/python3.9/site-packages/tensordict/_td.py", line 2216, in _set_tuple
    return self._set_str(
  File "/pytorch/rl/env/lib/python3.9/site-packages/tensordict/_td.py", line 2164, in _set_str
    value = self._validate_value(
  File "/pytorch/rl/env/lib/python3.9/site-packages/tensordict/base.py", line 8553, in _validate_value
    raise RuntimeError(
RuntimeError: batch dimension mismatch, got self.batch_size=torch.Size([256]) and value.shape=torch.Size([16, 6]).

I will try to patch it

BY571 added 4 commits July 4, 2024 13:19

fix norm

6d7c5c4

update docs

2d31f33

update comments

79bda13

add sota-example-test

1391b50

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 8, 2024

vmoens added the new algo New algorithm request or PR label Jul 8, 2024

BY571 added 7 commits July 8, 2024 14:53

update collection data slice

f444b72

update docstring

244b7ab

update config and objective with gp param

db635d7

init cost tests gail

434622c

update cost test

baca70f

Merge branch 'main' into gail

956567f

add gail cost tests

8e7713f

BY571 marked this pull request as ready for review July 11, 2024 09:40

Merge remote-tracking branch 'origin/main' into gail

714c35c

vmoens reviewed Jul 30, 2024

View reviewed changes

sota-implementations/gail/gail.py Outdated Show resolved Hide resolved

BY571 added 5 commits July 31, 2024 09:56

Merge branch 'main' into gail

a05bef3

Update config

b31da8a

update gail device

63885b0

update example tests

739332c

Merge branch 'gail' of github.com:BY571/rl into gail

7a9919c

Merge remote-tracking branch 'origin/main' into gail

3fd3c32

BY571 and others added 4 commits August 5, 2024 15:09

gymnasium backend

9455fef

Merge branch 'gail' of https://github.com/BY571/rl into gail

fba43d2

Merge remote-tracking branch 'origin/main' into gail

5e41d89

Merge remote-tracking branch 'origin/main' into gail

6c3f7d2

vmoens added 13 commits August 6, 2024 13:32

Merge remote-tracking branch 'origin/main' into gail

415443b

fixes

4926d80

init

cbd5dfa

Merge branch 'pin-mujoco' into gail

70e1f49

amend

b8ca705

Merge branch 'pin-mujoco' into gail

6a00bda

amend

f0c225f

amend

511fa95

amend

4bc316b

Merge branch 'pin-mujoco' into gail

2f7e64c

amend

3d43e42

Merge branch 'pin-mujoco' into gail

63398d1

amend

c488bcd

vmoens approved these changes Aug 6, 2024

View reviewed changes

vmoens merged commit 4348c84 into pytorch:main Aug 6, 2024
59 of 68 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Algorithm] GAIL #2273

[Algorithm] GAIL #2273

BY571 commented Jul 8, 2024 •

edited

Loading

pytorch-bot bot commented Jul 8, 2024 •

edited

Loading

vmoens left a comment

vmoens commented Jul 30, 2024

BY571 commented Jul 31, 2024

vmoens commented Aug 3, 2024 •

edited

Loading

vmoens commented Aug 5, 2024

[Algorithm] GAIL #2273

[Algorithm] GAIL #2273

Conversation

BY571 commented Jul 8, 2024 • edited Loading

Description

Motivation and Context

Types of changes

Checklist

pytorch-bot bot commented Jul 8, 2024 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2273

❌ 5 New Failures, 1 Cancelled Job, 7 Unrelated Failures

vmoens left a comment

Choose a reason for hiding this comment

vmoens commented Jul 30, 2024

BY571 commented Jul 31, 2024

vmoens commented Aug 3, 2024 • edited Loading

vmoens commented Aug 5, 2024

BY571 commented Jul 8, 2024 •

edited

Loading

pytorch-bot bot commented Jul 8, 2024 •

edited

Loading

vmoens commented Aug 3, 2024 •

edited

Loading