Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BugFix] Fix LSTM in GAE with vmap #2376

Merged
merged 1 commit into from
Aug 7, 2024
Merged

[BugFix] Fix LSTM in GAE with vmap #2376

merged 1 commit into from
Aug 7, 2024

Conversation

vmoens
Copy link
Contributor

@vmoens vmoens commented Aug 7, 2024 •
edited
Loading

No description provided.

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Aug 7, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2376

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 9 Unrelated Failures

As of commit 1152c07 with merge base 607db8b (image):

NEW FAILURES - The following jobs have failed:

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 7, 2024
@vmoens vmoens linked an issue Aug 7, 2024 that may be closed by this pull request
[BUG] Calculation of GAE fails with recurrent critic #2372
Closed
3 tasks
@vmoens vmoens added the bug Something isn't working label Aug 7, 2024
@vmoens vmoens mentioned this pull request Aug 7, 2024
Closed
3 tasks
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 7, 2024

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 91. Improved: $\large\color{#35bf28}5$. Worsened: $\large\color{#d91a1a}3$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 59.7776ms 58.9419ms 16.9659 Ops/s 17.0795 Ops/s $\color{#d91a1a}-0.67\%$
test_sync 40.2981ms 33.1563ms 30.1602 Ops/s 31.7429 Ops/s $\color{#d91a1a}-4.99\%$
test_async 67.5349ms 30.7164ms 32.5559 Ops/s 32.6730 Ops/s $\color{#d91a1a}-0.36\%$
test_simple 0.4916s 0.4225s 2.3666 Ops/s 2.4265 Ops/s $\color{#d91a1a}-2.47\%$
test_transformed 0.6276s 0.5721s 1.7480 Ops/s 1.7489 Ops/s $\color{#d91a1a}-0.05\%$
test_serial 1.3345s 1.2729s 0.7856 Ops/s 0.7971 Ops/s $\color{#d91a1a}-1.45\%$
test_parallel 1.1821s 1.1096s 0.9012 Ops/s 0.9072 Ops/s $\color{#d91a1a}-0.66\%$
test_step_mdp_speed[True-True-True-True-True] 0.2168ms 25.2035μs 39.6770 KOps/s 40.2042 KOps/s $\color{#d91a1a}-1.31\%$
test_step_mdp_speed[True-True-True-True-False] 64.9890μs 14.6731μs 68.1520 KOps/s 68.3680 KOps/s $\color{#d91a1a}-0.32\%$
test_step_mdp_speed[True-True-True-False-True] 71.6450μs 14.5462μs 68.7464 KOps/s 69.8840 KOps/s $\color{#d91a1a}-1.63\%$
test_step_mdp_speed[True-True-True-False-False] 40.3230μs 8.4903μs 117.7816 KOps/s 120.1074 KOps/s $\color{#d91a1a}-1.94\%$
test_step_mdp_speed[True-True-False-True-True] 62.6270μs 26.7022μs 37.4501 KOps/s 37.5526 KOps/s $\color{#d91a1a}-0.27\%$
test_step_mdp_speed[True-True-False-True-False] 61.5350μs 16.0969μs 62.1238 KOps/s 61.8134 KOps/s $\color{#35bf28}+0.50\%$
test_step_mdp_speed[True-True-False-False-True] 41.6880μs 16.0614μs 62.2610 KOps/s 63.1974 KOps/s $\color{#d91a1a}-1.48\%$
test_step_mdp_speed[True-True-False-False-False] 39.2530μs 9.9325μs 100.6797 KOps/s 100.9125 KOps/s $\color{#d91a1a}-0.23\%$
test_step_mdp_speed[True-False-True-True-True] 74.8290μs 28.4813μs 35.1108 KOps/s 35.1460 KOps/s $\color{#d91a1a}-0.10\%$
test_step_mdp_speed[True-False-True-True-False] 65.0610μs 17.7019μs 56.4912 KOps/s 56.5427 KOps/s $\color{#d91a1a}-0.09\%$
test_step_mdp_speed[True-False-True-False-True] 68.5380μs 16.0382μs 62.3513 KOps/s 63.2619 KOps/s $\color{#d91a1a}-1.44\%$
test_step_mdp_speed[True-False-True-False-False] 63.6690μs 10.0149μs 99.8513 KOps/s 100.7380 KOps/s $\color{#d91a1a}-0.88\%$
test_step_mdp_speed[True-False-False-True-True] 60.9440μs 29.8252μs 33.5287 KOps/s 33.6962 KOps/s $\color{#d91a1a}-0.50\%$
test_step_mdp_speed[True-False-False-True-False] 51.2160μs 19.0304μs 52.5474 KOps/s 51.9306 KOps/s $\color{#35bf28}+1.19\%$
test_step_mdp_speed[True-False-False-False-True] 69.3590μs 17.5770μs 56.8924 KOps/s 57.6007 KOps/s $\color{#d91a1a}-1.23\%$
test_step_mdp_speed[True-False-False-False-False] 51.7170μs 11.3103μs 88.4147 KOps/s 89.0216 KOps/s $\color{#d91a1a}-0.68\%$
test_step_mdp_speed[False-True-True-True-True] 81.2210μs 28.6131μs 34.9490 KOps/s 35.0751 KOps/s $\color{#d91a1a}-0.36\%$
test_step_mdp_speed[False-True-True-True-False] 44.2530μs 17.7284μs 56.4066 KOps/s 57.0389 KOps/s $\color{#d91a1a}-1.11\%$
test_step_mdp_speed[False-True-True-False-True] 72.7550μs 18.7293μs 53.3924 KOps/s 53.4066 KOps/s $\color{#d91a1a}-0.03\%$
test_step_mdp_speed[False-True-True-False-False] 37.6100μs 11.1917μs 89.3521 KOps/s 90.3066 KOps/s $\color{#d91a1a}-1.06\%$
test_step_mdp_speed[False-True-False-True-True] 81.9730μs 29.7954μs 33.5622 KOps/s 33.3745 KOps/s $\color{#35bf28}+0.56\%$
test_step_mdp_speed[False-True-False-True-False] 48.0190μs 19.1434μs 52.2373 KOps/s 52.6338 KOps/s $\color{#d91a1a}-0.75\%$
test_step_mdp_speed[False-True-False-False-True] 76.0920μs 19.9778μs 50.0555 KOps/s 48.2095 KOps/s $\color{#35bf28}+3.83\%$
test_step_mdp_speed[False-True-False-False-False] 33.8330μs 12.5643μs 79.5906 KOps/s 79.8766 KOps/s $\color{#d91a1a}-0.36\%$
test_step_mdp_speed[False-False-True-True-True] 3.0222ms 31.5493μs 31.6964 KOps/s 31.8215 KOps/s $\color{#d91a1a}-0.39\%$
test_step_mdp_speed[False-False-True-True-False] 70.7620μs 20.6077μs 48.5255 KOps/s 48.6313 KOps/s $\color{#d91a1a}-0.22\%$
test_step_mdp_speed[False-False-True-False-True] 46.2360μs 19.9144μs 50.2148 KOps/s 50.1826 KOps/s $\color{#35bf28}+0.06\%$
test_step_mdp_speed[False-False-True-False-False] 61.5840μs 12.6733μs 78.9062 KOps/s 79.5795 KOps/s $\color{#d91a1a}-0.85\%$
test_step_mdp_speed[False-False-False-True-True] 90.7290μs 32.9141μs 30.3821 KOps/s 30.6346 KOps/s $\color{#d91a1a}-0.82\%$
test_step_mdp_speed[False-False-False-True-False] 71.8340μs 22.1117μs 45.2249 KOps/s 45.6658 KOps/s $\color{#d91a1a}-0.97\%$
test_step_mdp_speed[False-False-False-False-True] 70.1300μs 21.4468μs 46.6271 KOps/s 46.9438 KOps/s $\color{#d91a1a}-0.67\%$
test_step_mdp_speed[False-False-False-False-False] 44.1320μs 13.9412μs 71.7298 KOps/s 72.4837 KOps/s $\color{#d91a1a}-1.04\%$
test_values[generalized_advantage_estimate-True-True] 12.0561ms 9.7296ms 102.7791 Ops/s 106.4963 Ops/s $\color{#d91a1a}-3.49\%$
test_values[vec_generalized_advantage_estimate-True-True] 37.0235ms 33.4866ms 29.8627 Ops/s 28.0601 Ops/s $\textbf{\color{#35bf28}+6.42\%}$
test_values[td0_return_estimate-False-False] 0.2315ms 0.1812ms 5.5184 KOps/s 6.0802 KOps/s $\textbf{\color{#d91a1a}-9.24\%}$
test_values[td1_return_estimate-False-False] 24.4680ms 23.9899ms 41.6843 Ops/s 42.3742 Ops/s $\color{#d91a1a}-1.63\%$
test_values[vec_td1_return_estimate-False-False] 36.1296ms 33.3527ms 29.9826 Ops/s 28.1558 Ops/s $\textbf{\color{#35bf28}+6.49\%}$
test_values[td_lambda_return_estimate-True-False] 34.8407ms 34.4632ms 29.0165 Ops/s 29.1975 Ops/s $\color{#d91a1a}-0.62\%$
test_values[vec_td_lambda_return_estimate-True-False] 35.9334ms 33.2504ms 30.0749 Ops/s 27.9307 Ops/s $\textbf{\color{#35bf28}+7.68\%}$
test_gae_speed[generalized_advantage_estimate-False-1-512] 13.0696ms 8.4272ms 118.6641 Ops/s 122.3701 Ops/s $\color{#d91a1a}-3.03\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 2.3002ms 1.9177ms 521.4464 Ops/s 520.5329 Ops/s $\color{#35bf28}+0.18\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.4388ms 0.3599ms 2.7789 KOps/s 2.8320 KOps/s $\color{#d91a1a}-1.87\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 46.9442ms 40.4075ms 24.7479 Ops/s 24.6715 Ops/s $\color{#35bf28}+0.31\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 3.3500ms 3.0284ms 330.2058 Ops/s 328.4400 Ops/s $\color{#35bf28}+0.54\%$
test_dqn_speed 1.8974ms 1.3216ms 756.6725 Ops/s 773.5822 Ops/s $\color{#d91a1a}-2.19\%$
test_ddpg_speed 3.6511ms 2.7454ms 364.2424 Ops/s 370.8747 Ops/s $\color{#d91a1a}-1.79\%$
test_sac_speed 9.8062ms 7.9964ms 125.0566 Ops/s 126.3634 Ops/s $\color{#d91a1a}-1.03\%$
test_redq_speed 13.7276ms 12.6743ms 78.8997 Ops/s 80.1285 Ops/s $\color{#d91a1a}-1.53\%$
test_redq_deprec_speed 15.5322ms 12.7570ms 78.3885 Ops/s 79.2823 Ops/s $\color{#d91a1a}-1.13\%$
test_td3_speed 8.2229ms 7.9174ms 126.3037 Ops/s 127.2762 Ops/s $\color{#d91a1a}-0.76\%$
test_cql_speed 37.6549ms 35.2668ms 28.3552 Ops/s 28.4656 Ops/s $\color{#d91a1a}-0.39\%$
test_a2c_speed 8.4203ms 7.2873ms 137.2247 Ops/s 138.3734 Ops/s $\color{#d91a1a}-0.83\%$
test_ppo_speed 9.2137ms 7.5546ms 132.3693 Ops/s 133.2845 Ops/s $\color{#d91a1a}-0.69\%$
test_reinforce_speed 7.5647ms 6.4621ms 154.7475 Ops/s 155.9367 Ops/s $\color{#d91a1a}-0.76\%$
test_iql_speed 33.1108ms 32.0009ms 31.2491 Ops/s 31.3578 Ops/s $\color{#d91a1a}-0.35\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 7.4151ms 4.9569ms 201.7373 Ops/s 209.6057 Ops/s $\color{#d91a1a}-3.75\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 1.2997ms 0.4754ms 2.1035 KOps/s 2.0997 KOps/s $\color{#35bf28}+0.18\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.6242ms 0.4530ms 2.2074 KOps/s 2.2071 KOps/s $\color{#35bf28}+0.02\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 5.1770ms 4.7937ms 208.6056 Ops/s 214.0638 Ops/s $\color{#d91a1a}-2.55\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.8496ms 0.4733ms 2.1129 KOps/s 1.8355 KOps/s $\textbf{\color{#35bf28}+15.12\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.6338ms 0.4456ms 2.2441 KOps/s 2.2613 KOps/s $\color{#d91a1a}-0.76\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 1.8161ms 1.6844ms 593.6835 Ops/s 594.8478 Ops/s $\color{#d91a1a}-0.20\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 2.3105ms 1.6046ms 623.1936 Ops/s 627.5205 Ops/s $\color{#d91a1a}-0.69\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 5.4453ms 4.9641ms 201.4451 Ops/s 201.3997 Ops/s $\color{#35bf28}+0.02\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.8929ms 0.6117ms 1.6347 KOps/s 1.5957 KOps/s $\color{#35bf28}+2.45\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8152ms 0.5862ms 1.7060 KOps/s 1.7071 KOps/s $\color{#d91a1a}-0.06\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 5.4275ms 4.8690ms 205.3809 Ops/s 208.0389 Ops/s $\color{#d91a1a}-1.28\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.8715ms 0.4857ms 2.0587 KOps/s 2.0921 KOps/s $\color{#d91a1a}-1.59\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.6741ms 0.4581ms 2.1829 KOps/s 2.1950 KOps/s $\color{#d91a1a}-0.55\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 7.7853ms 4.8844ms 204.7315 Ops/s 206.4976 Ops/s $\color{#d91a1a}-0.86\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 1.6404ms 0.4718ms 2.1196 KOps/s 2.1368 KOps/s $\color{#d91a1a}-0.81\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.6519ms 0.4519ms 2.2126 KOps/s 2.2490 KOps/s $\color{#d91a1a}-1.62\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 5.2497ms 4.9225ms 203.1502 Ops/s 204.7528 Ops/s $\color{#d91a1a}-0.78\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.9243ms 0.6216ms 1.6087 KOps/s 1.6298 KOps/s $\color{#d91a1a}-1.30\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.7699ms 0.5872ms 1.7031 KOps/s 1.7168 KOps/s $\color{#d91a1a}-0.80\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.1266s 8.3108ms 120.3247 Ops/s 163.8239 Ops/s $\textbf{\color{#d91a1a}-26.55\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 17.0150ms 13.0158ms 76.8295 Ops/s 79.2107 Ops/s $\color{#d91a1a}-3.01\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 1.6111ms 1.1116ms 899.5997 Ops/s 865.5770 Ops/s $\color{#35bf28}+3.93\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1090s 5.8090ms 172.1458 Ops/s 172.9842 Ops/s $\color{#d91a1a}-0.48\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 16.9503ms 12.8921ms 77.5667 Ops/s 79.2810 Ops/s $\color{#d91a1a}-2.16\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 1.8172ms 1.1298ms 885.1291 Ops/s 849.5600 Ops/s $\color{#35bf28}+4.19\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1067s 5.8940ms 169.6644 Ops/s 116.6256 Ops/s $\textbf{\color{#35bf28}+45.48\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 16.8166ms 12.9755ms 77.0681 Ops/s 77.5829 Ops/s $\color{#d91a1a}-0.66\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 6.2240ms 1.3753ms 727.1073 Ops/s 778.2051 Ops/s $\textbf{\color{#d91a1a}-6.57\%}$

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 7, 2024

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 94. Improved: $\large\color{#35bf28}2$. Worsened: $\large\color{#d91a1a}1$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 0.1072s 0.1069s 9.3519 Ops/s 9.3049 Ops/s $\color{#35bf28}+0.51\%$
test_sync 95.3003ms 91.8506ms 10.8872 Ops/s 10.9111 Ops/s $\color{#d91a1a}-0.22\%$
test_async 0.2542s 90.7148ms 11.0236 Ops/s 11.3073 Ops/s $\color{#d91a1a}-2.51\%$
test_single_pixels 0.1203s 0.1178s 8.4898 Ops/s 8.4286 Ops/s $\color{#35bf28}+0.73\%$
test_sync_pixels 77.4547ms 76.5539ms 13.0627 Ops/s 13.1039 Ops/s $\color{#d91a1a}-0.31\%$
test_async_pixels 0.2029s 71.6403ms 13.9586 Ops/s 13.9073 Ops/s $\color{#35bf28}+0.37\%$
test_simple 0.8620s 0.7876s 1.2697 Ops/s 1.3141 Ops/s $\color{#d91a1a}-3.38\%$
test_transformed 1.0840s 1.0165s 0.9838 Ops/s 0.9798 Ops/s $\color{#35bf28}+0.41\%$
test_serial 2.2563s 2.1858s 0.4575 Ops/s 0.4545 Ops/s $\color{#35bf28}+0.67\%$
test_parallel 1.9078s 1.8748s 0.5334 Ops/s 0.5262 Ops/s $\color{#35bf28}+1.36\%$
test_step_mdp_speed[True-True-True-True-True] 76.9010μs 38.0419μs 26.2868 KOps/s 25.8450 KOps/s $\color{#35bf28}+1.71\%$
test_step_mdp_speed[True-True-True-True-False] 46.8210μs 21.0086μs 47.5995 KOps/s 46.9720 KOps/s $\color{#35bf28}+1.34\%$
test_step_mdp_speed[True-True-True-False-True] 49.5610μs 21.5990μs 46.2984 KOps/s 47.8487 KOps/s $\color{#d91a1a}-3.24\%$
test_step_mdp_speed[True-True-True-False-False] 32.9910μs 11.9975μs 83.3509 KOps/s 83.0455 KOps/s $\color{#35bf28}+0.37\%$
test_step_mdp_speed[True-True-False-True-True] 74.9510μs 40.4960μs 24.6938 KOps/s 24.1810 KOps/s $\color{#35bf28}+2.12\%$
test_step_mdp_speed[True-True-False-True-False] 46.6800μs 23.4821μs 42.5856 KOps/s 42.1750 KOps/s $\color{#35bf28}+0.97\%$
test_step_mdp_speed[True-True-False-False-True] 53.6200μs 23.8820μs 41.8726 KOps/s 42.2101 KOps/s $\color{#d91a1a}-0.80\%$
test_step_mdp_speed[True-True-False-False-False] 36.9900μs 14.2992μs 69.9341 KOps/s 69.4804 KOps/s $\color{#35bf28}+0.65\%$
test_step_mdp_speed[True-False-True-True-True] 73.4510μs 42.3317μs 23.6229 KOps/s 23.1673 KOps/s $\color{#35bf28}+1.97\%$
test_step_mdp_speed[True-False-True-True-False] 51.5400μs 25.7409μs 38.8487 KOps/s 37.9142 KOps/s $\color{#35bf28}+2.46\%$
test_step_mdp_speed[True-False-True-False-True] 52.1800μs 23.5799μs 42.4090 KOps/s 42.4498 KOps/s $\color{#d91a1a}-0.10\%$
test_step_mdp_speed[True-False-True-False-False] 37.7110μs 14.3164μs 69.8498 KOps/s 69.6040 KOps/s $\color{#35bf28}+0.35\%$
test_step_mdp_speed[True-False-False-True-True] 84.6110μs 44.0962μs 22.6777 KOps/s 21.9025 KOps/s $\color{#35bf28}+3.54\%$
test_step_mdp_speed[True-False-False-True-False] 53.2010μs 27.9788μs 35.7413 KOps/s 35.1583 KOps/s $\color{#35bf28}+1.66\%$
test_step_mdp_speed[True-False-False-False-True] 52.8800μs 25.6726μs 38.9520 KOps/s 39.0577 KOps/s $\color{#d91a1a}-0.27\%$
test_step_mdp_speed[True-False-False-False-False] 45.5210μs 16.5245μs 60.5161 KOps/s 60.4043 KOps/s $\color{#35bf28}+0.19\%$
test_step_mdp_speed[False-True-True-True-True] 74.9610μs 41.9849μs 23.8181 KOps/s 22.8279 KOps/s $\color{#35bf28}+4.34\%$
test_step_mdp_speed[False-True-True-True-False] 51.5500μs 25.6790μs 38.9423 KOps/s 38.1852 KOps/s $\color{#35bf28}+1.98\%$
test_step_mdp_speed[False-True-True-False-True] 54.0600μs 27.9435μs 35.7865 KOps/s 35.0532 KOps/s $\color{#35bf28}+2.09\%$
test_step_mdp_speed[False-True-True-False-False] 38.6100μs 16.0631μs 62.2545 KOps/s 61.0516 KOps/s $\color{#35bf28}+1.97\%$
test_step_mdp_speed[False-True-False-True-True] 79.0810μs 44.4717μs 22.4862 KOps/s 21.6995 KOps/s $\color{#35bf28}+3.63\%$
test_step_mdp_speed[False-True-False-True-False] 54.4610μs 27.8124μs 35.9551 KOps/s 35.2601 KOps/s $\color{#35bf28}+1.97\%$
test_step_mdp_speed[False-True-False-False-True] 59.6710μs 30.3318μs 32.9687 KOps/s 32.7879 KOps/s $\color{#35bf28}+0.55\%$
test_step_mdp_speed[False-True-False-False-False] 44.1010μs 18.3365μs 54.5361 KOps/s 53.8791 KOps/s $\color{#35bf28}+1.22\%$
test_step_mdp_speed[False-False-True-True-True] 3.8313ms 47.0087μs 21.2726 KOps/s 20.6871 KOps/s $\color{#35bf28}+2.83\%$
test_step_mdp_speed[False-False-True-True-False] 53.7800μs 30.7796μs 32.4891 KOps/s 32.4301 KOps/s $\color{#35bf28}+0.18\%$
test_step_mdp_speed[False-False-True-False-True] 49.3200μs 30.8635μs 32.4007 KOps/s 32.8482 KOps/s $\color{#d91a1a}-1.36\%$
test_step_mdp_speed[False-False-True-False-False] 40.7300μs 18.5550μs 53.8938 KOps/s 54.1754 KOps/s $\color{#d91a1a}-0.52\%$
test_step_mdp_speed[False-False-False-True-True] 75.9610μs 48.8404μs 20.4749 KOps/s 19.9290 KOps/s $\color{#35bf28}+2.74\%$
test_step_mdp_speed[False-False-False-True-False] 59.7900μs 32.7245μs 30.5582 KOps/s 30.2542 KOps/s $\color{#35bf28}+1.00\%$
test_step_mdp_speed[False-False-False-False-True] 60.8310μs 32.6114μs 30.6642 KOps/s 30.7987 KOps/s $\color{#d91a1a}-0.44\%$
test_step_mdp_speed[False-False-False-False-False] 37.0100μs 20.5817μs 48.5870 KOps/s 48.7663 KOps/s $\color{#d91a1a}-0.37\%$
test_values[generalized_advantage_estimate-True-True] 25.6192ms 24.7106ms 40.4685 Ops/s 40.3373 Ops/s $\color{#35bf28}+0.33\%$
test_values[vec_generalized_advantage_estimate-True-True] 92.1392ms 2.7455ms 364.2320 Ops/s 352.3441 Ops/s $\color{#35bf28}+3.37\%$
test_values[td0_return_estimate-False-False] 93.6910μs 68.1886μs 14.6652 KOps/s 15.0614 KOps/s $\color{#d91a1a}-2.63\%$
test_values[td1_return_estimate-False-False] 56.2234ms 55.6773ms 17.9607 Ops/s 17.9946 Ops/s $\color{#d91a1a}-0.19\%$
test_values[vec_td1_return_estimate-False-False] 1.3699ms 1.1020ms 907.4227 Ops/s 918.8239 Ops/s $\color{#d91a1a}-1.24\%$
test_values[td_lambda_return_estimate-True-False] 88.7898ms 88.1333ms 11.3465 Ops/s 11.3746 Ops/s $\color{#d91a1a}-0.25\%$
test_values[vec_td_lambda_return_estimate-True-False] 1.2676ms 1.0874ms 919.6306 Ops/s 923.2481 Ops/s $\color{#d91a1a}-0.39\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 26.1348ms 25.0145ms 39.9768 Ops/s 40.1772 Ops/s $\color{#d91a1a}-0.50\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 0.9450ms 0.7273ms 1.3750 KOps/s 1.3793 KOps/s $\color{#d91a1a}-0.31\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.7705ms 0.6736ms 1.4845 KOps/s 1.5007 KOps/s $\color{#d91a1a}-1.08\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 1.5330ms 1.4744ms 678.2482 Ops/s 680.5098 Ops/s $\color{#d91a1a}-0.33\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 0.7360ms 0.6896ms 1.4502 KOps/s 1.4665 KOps/s $\color{#d91a1a}-1.11\%$
test_dqn_speed 1.5543ms 1.3944ms 717.1630 Ops/s 703.9839 Ops/s $\color{#35bf28}+1.87\%$
test_ddpg_speed 3.1325ms 2.8378ms 352.3813 Ops/s 351.4669 Ops/s $\color{#35bf28}+0.26\%$
test_sac_speed 8.5566ms 8.1012ms 123.4380 Ops/s 123.2263 Ops/s $\color{#35bf28}+0.17\%$
test_redq_speed 11.7939ms 10.3367ms 96.7428 Ops/s 96.6922 Ops/s $\color{#35bf28}+0.05\%$
test_redq_deprec_speed 11.3454ms 10.7592ms 92.9434 Ops/s 92.1929 Ops/s $\color{#35bf28}+0.81\%$
test_td3_speed 8.1109ms 7.9982ms 125.0286 Ops/s 124.9721 Ops/s $\color{#35bf28}+0.05\%$
test_cql_speed 27.1445ms 25.5726ms 39.1044 Ops/s 39.6325 Ops/s $\color{#d91a1a}-1.33\%$
test_a2c_speed 6.2029ms 5.6210ms 177.9043 Ops/s 178.3117 Ops/s $\color{#d91a1a}-0.23\%$
test_ppo_speed 6.2030ms 5.9667ms 167.5957 Ops/s 167.3250 Ops/s $\color{#35bf28}+0.16\%$
test_reinforce_speed 4.7519ms 4.5378ms 220.3712 Ops/s 220.3639 Ops/s $+0.00\%$
test_iql_speed 20.2676ms 19.4628ms 51.3800 Ops/s 51.3499 Ops/s $\color{#35bf28}+0.06\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 6.8966ms 6.6973ms 149.3147 Ops/s 150.4714 Ops/s $\color{#d91a1a}-0.77\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.1040s 0.5898ms 1.6954 KOps/s 1.9463 KOps/s $\textbf{\color{#d91a1a}-12.89\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.6900ms 0.4903ms 2.0394 KOps/s 2.0373 KOps/s $\color{#35bf28}+0.10\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 6.9750ms 6.6345ms 150.7265 Ops/s 152.5105 Ops/s $\color{#d91a1a}-1.17\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.8145ms 0.5083ms 1.9675 KOps/s 1.9700 KOps/s $\color{#d91a1a}-0.13\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.7046ms 0.4867ms 2.0547 KOps/s 2.0683 KOps/s $\color{#d91a1a}-0.66\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 2.1365ms 1.9644ms 509.0503 Ops/s 513.3908 Ops/s $\color{#d91a1a}-0.85\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 2.0831ms 1.8731ms 533.8771 Ops/s 538.9141 Ops/s $\color{#d91a1a}-0.93\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 6.8760ms 6.7880ms 147.3186 Ops/s 147.5986 Ops/s $\color{#d91a1a}-0.19\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.7860ms 0.6603ms 1.5144 KOps/s 1.5090 KOps/s $\color{#35bf28}+0.36\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.7588ms 0.6349ms 1.5752 KOps/s 1.5603 KOps/s $\color{#35bf28}+0.95\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 6.8174ms 6.6934ms 149.4009 Ops/s 150.3548 Ops/s $\color{#d91a1a}-0.63\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.7579ms 0.5145ms 1.9435 KOps/s 1.9383 KOps/s $\color{#35bf28}+0.27\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.6667ms 0.4909ms 2.0373 KOps/s 1.9972 KOps/s $\color{#35bf28}+2.01\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 6.9761ms 6.5924ms 151.6887 Ops/s 151.8060 Ops/s $\color{#d91a1a}-0.08\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 1.0039ms 0.5072ms 1.9715 KOps/s 1.9512 KOps/s $\color{#35bf28}+1.04\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.6184ms 0.4914ms 2.0348 KOps/s 2.0364 KOps/s $\color{#d91a1a}-0.08\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 7.0363ms 6.8597ms 145.7784 Ops/s 146.7772 Ops/s $\color{#d91a1a}-0.68\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.5712ms 0.6629ms 1.5085 KOps/s 1.5094 KOps/s $\color{#d91a1a}-0.06\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8388ms 0.6393ms 1.5641 KOps/s 1.5593 KOps/s $\color{#35bf28}+0.31\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.1373s 7.8220ms 127.8452 Ops/s 128.1852 Ops/s $\color{#d91a1a}-0.27\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 0.1316s 18.7067ms 53.4568 Ops/s 53.6631 Ops/s $\color{#d91a1a}-0.38\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 2.3177ms 1.2552ms 796.6729 Ops/s 796.3135 Ops/s $\color{#35bf28}+0.05\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1206s 7.5384ms 132.6541 Ops/s 134.3956 Ops/s $\color{#d91a1a}-1.30\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 19.0212ms 16.4306ms 60.8621 Ops/s 61.7822 Ops/s $\color{#d91a1a}-1.49\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 2.2706ms 1.2693ms 787.8113 Ops/s 734.0839 Ops/s $\textbf{\color{#35bf28}+7.32\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1182s 7.6342ms 130.9887 Ops/s 131.6152 Ops/s $\color{#d91a1a}-0.48\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 19.0430ms 16.3224ms 61.2656 Ops/s 61.4505 Ops/s $\color{#d91a1a}-0.30\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 2.4584ms 1.4107ms 708.8693 Ops/s 659.4292 Ops/s $\textbf{\color{#35bf28}+7.50\%}$

@vmoens vmoens merged commit 342450e into main Aug 7, 2024
61 of 68 checks passed
@vmoens vmoens deleted the patch-rnn-gae branch August 7, 2024 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
bug Something isn't working CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[BUG] Calculation of GAE fails with recurrent critic
2 participants
@vmoens @facebook-github-bot