Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] Pass replay buffers to MultiaSyncDataCollector #2387

Merged
merged 13 commits into from
Aug 13, 2024
Merged

[Feature] Pass replay buffers to MultiaSyncDataCollector #2387

merged 13 commits into from
Aug 13, 2024

Conversation

vmoens
Copy link
Contributor

@vmoens vmoens commented Aug 10, 2024 •
edited
Loading

@vmoens
Update
a61bf28 
[ghstack-poisoned]
This was referenced Aug 10, 2024
Merged
Closed
@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Aug 10, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2387

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 21 Unrelated Failures

As of commit 763042d with merge base 2b975da (image):

NEW FAILURE - The following job has failed:

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

vmoens added a commit that referenced this pull request Aug 10, 2024
@vmoens
[Feature] Pass replay buffers to MultiaSyncDataCollector
0bd4961 
ghstack-source-id: fb08f18ea55044e0e72053b3ed5aa0054c229753
Pull Request resolved: #2387
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 10, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 10, 2024 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 91. Improved: $\large\color{#35bf28}2$. Worsened: $\large\color{#d91a1a}5$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 60.0323ms 58.8111ms 17.0036 Ops/s 17.4039 Ops/s $\color{#d91a1a}-2.30\%$
test_sync 41.7116ms 32.7919ms 30.4953 Ops/s 31.1810 Ops/s $\color{#d91a1a}-2.20\%$
test_async 73.2955ms 31.4172ms 31.8297 Ops/s 32.0965 Ops/s $\color{#d91a1a}-0.83\%$
test_simple 0.4877s 0.4178s 2.3934 Ops/s 2.3982 Ops/s $\color{#d91a1a}-0.20\%$
test_transformed 0.6373s 0.5753s 1.7381 Ops/s 1.7515 Ops/s $\color{#d91a1a}-0.76\%$
test_serial 1.3340s 1.2680s 0.7887 Ops/s 0.7989 Ops/s $\color{#d91a1a}-1.29\%$
test_parallel 1.1979s 1.1132s 0.8983 Ops/s 0.9151 Ops/s $\color{#d91a1a}-1.84\%$
test_step_mdp_speed[True-True-True-True-True] 0.2262ms 24.6227μs 40.6129 KOps/s 40.3058 KOps/s $\color{#35bf28}+0.76\%$
test_step_mdp_speed[True-True-True-True-False] 42.1790μs 14.0716μs 71.0653 KOps/s 69.7307 KOps/s $\color{#35bf28}+1.91\%$
test_step_mdp_speed[True-True-True-False-True] 42.1290μs 13.9635μs 71.6152 KOps/s 70.4487 KOps/s $\color{#35bf28}+1.66\%$
test_step_mdp_speed[True-True-True-False-False] 50.7440μs 8.1400μs 122.8498 KOps/s 122.8358 KOps/s $\color{#35bf28}+0.01\%$
test_step_mdp_speed[True-True-False-True-True] 79.6050μs 26.1435μs 38.2504 KOps/s 37.5601 KOps/s $\color{#35bf28}+1.84\%$
test_step_mdp_speed[True-True-False-True-False] 85.7300μs 15.4951μs 64.5367 KOps/s 63.2409 KOps/s $\color{#35bf28}+2.05\%$
test_step_mdp_speed[True-True-False-False-True] 46.3160μs 15.6683μs 63.8233 KOps/s 63.8012 KOps/s $\color{#35bf28}+0.03\%$
test_step_mdp_speed[True-True-False-False-False] 55.8440μs 9.6574μs 103.5476 KOps/s 103.1086 KOps/s $\color{#35bf28}+0.43\%$
test_step_mdp_speed[True-False-True-True-True] 79.8180μs 27.7208μs 36.0740 KOps/s 35.4040 KOps/s $\color{#35bf28}+1.89\%$
test_step_mdp_speed[True-False-True-True-False] 53.6500μs 17.2622μs 57.9302 KOps/s 56.8828 KOps/s $\color{#35bf28}+1.84\%$
test_step_mdp_speed[True-False-True-False-True] 49.5320μs 15.6169μs 64.0334 KOps/s 63.0714 KOps/s $\color{#35bf28}+1.53\%$
test_step_mdp_speed[True-False-True-False-False] 57.9680μs 9.5847μs 104.3330 KOps/s 102.5068 KOps/s $\color{#35bf28}+1.78\%$
test_step_mdp_speed[True-False-False-True-True] 77.5550μs 29.0064μs 34.4752 KOps/s 34.0913 KOps/s $\color{#35bf28}+1.13\%$
test_step_mdp_speed[True-False-False-True-False] 51.8570μs 18.6723μs 53.5554 KOps/s 52.4413 KOps/s $\color{#35bf28}+2.12\%$
test_step_mdp_speed[True-False-False-False-True] 56.0350μs 16.8779μs 59.2490 KOps/s 58.2263 KOps/s $\color{#35bf28}+1.76\%$
test_step_mdp_speed[True-False-False-False-False] 45.2740μs 11.0526μs 90.4766 KOps/s 89.2658 KOps/s $\color{#35bf28}+1.36\%$
test_step_mdp_speed[False-True-True-True-True] 72.4650μs 28.2267μs 35.4275 KOps/s 35.1795 KOps/s $\color{#35bf28}+0.70\%$
test_step_mdp_speed[False-True-True-True-False] 49.0710μs 17.4655μs 57.2557 KOps/s 56.5352 KOps/s $\color{#35bf28}+1.27\%$
test_step_mdp_speed[False-True-True-False-True] 70.8920μs 18.2960μs 54.6569 KOps/s 55.1610 KOps/s $\color{#d91a1a}-0.91\%$
test_step_mdp_speed[False-True-True-False-False] 50.4140μs 10.8440μs 92.2169 KOps/s 91.2660 KOps/s $\color{#35bf28}+1.04\%$
test_step_mdp_speed[False-True-False-True-True] 66.3330μs 29.6753μs 33.6981 KOps/s 33.8069 KOps/s $\color{#d91a1a}-0.32\%$
test_step_mdp_speed[False-True-False-True-False] 55.6530μs 18.6002μs 53.7629 KOps/s 52.9383 KOps/s $\color{#35bf28}+1.56\%$
test_step_mdp_speed[False-True-False-False-True] 47.9290μs 20.0942μs 49.7656 KOps/s 50.8280 KOps/s $\color{#d91a1a}-2.09\%$
test_step_mdp_speed[False-True-False-False-False] 56.8160μs 12.1962μs 81.9927 KOps/s 80.5385 KOps/s $\color{#35bf28}+1.81\%$
test_step_mdp_speed[False-False-True-True-True] 3.3141ms 31.4215μs 31.8254 KOps/s 31.8385 KOps/s $\color{#d91a1a}-0.04\%$
test_step_mdp_speed[False-False-True-True-False] 67.2050μs 20.0645μs 49.8392 KOps/s 48.8850 KOps/s $\color{#35bf28}+1.95\%$
test_step_mdp_speed[False-False-True-False-True] 48.9010μs 19.5917μs 51.0421 KOps/s 50.4343 KOps/s $\color{#35bf28}+1.21\%$
test_step_mdp_speed[False-False-True-False-False] 59.7010μs 12.2103μs 81.8979 KOps/s 80.2569 KOps/s $\color{#35bf28}+2.04\%$
test_step_mdp_speed[False-False-False-True-True] 0.1107ms 32.0146μs 31.2358 KOps/s 30.7774 KOps/s $\color{#35bf28}+1.49\%$
test_step_mdp_speed[False-False-False-True-False] 52.3580μs 21.4431μs 46.6350 KOps/s 45.6118 KOps/s $\color{#35bf28}+2.24\%$
test_step_mdp_speed[False-False-False-False-True] 74.6890μs 20.5289μs 48.7118 KOps/s 48.3078 KOps/s $\color{#35bf28}+0.84\%$
test_step_mdp_speed[False-False-False-False-False] 58.8500μs 13.4953μs 74.1000 KOps/s 72.7151 KOps/s $\color{#35bf28}+1.90\%$
test_values[generalized_advantage_estimate-True-True] 10.1454ms 9.6727ms 103.3840 Ops/s 105.4677 Ops/s $\color{#d91a1a}-1.98\%$
test_values[vec_generalized_advantage_estimate-True-True] 40.6083ms 35.5459ms 28.1326 Ops/s 28.3136 Ops/s $\color{#d91a1a}-0.64\%$
test_values[td0_return_estimate-False-False] 0.2234ms 0.1677ms 5.9631 KOps/s 5.7502 KOps/s $\color{#35bf28}+3.70\%$
test_values[td1_return_estimate-False-False] 26.9411ms 23.6824ms 42.2255 Ops/s 41.9842 Ops/s $\color{#35bf28}+0.57\%$
test_values[vec_td1_return_estimate-False-False] 37.6261ms 35.5682ms 28.1150 Ops/s 28.2269 Ops/s $\color{#d91a1a}-0.40\%$
test_values[td_lambda_return_estimate-True-False] 37.2504ms 34.3978ms 29.0716 Ops/s 29.1999 Ops/s $\color{#d91a1a}-0.44\%$
test_values[vec_td_lambda_return_estimate-True-False] 43.0533ms 35.9017ms 27.8538 Ops/s 28.0459 Ops/s $\color{#d91a1a}-0.68\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 8.7709ms 8.3892ms 119.2013 Ops/s 122.1436 Ops/s $\color{#d91a1a}-2.41\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 2.3161ms 1.9757ms 506.1613 Ops/s 504.1972 Ops/s $\color{#35bf28}+0.39\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.4924ms 0.3603ms 2.7758 KOps/s 2.7990 KOps/s $\color{#d91a1a}-0.83\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 49.9135ms 45.9746ms 21.7511 Ops/s 21.8764 Ops/s $\color{#d91a1a}-0.57\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 4.0062ms 3.0420ms 328.7271 Ops/s 328.7638 Ops/s $\color{#d91a1a}-0.01\%$
test_dqn_speed 6.2001ms 1.3019ms 768.1319 Ops/s 775.7317 Ops/s $\color{#d91a1a}-0.98\%$
test_ddpg_speed 3.5352ms 2.6749ms 373.8428 Ops/s 375.2644 Ops/s $\color{#d91a1a}-0.38\%$
test_sac_speed 8.4084ms 7.8334ms 127.6585 Ops/s 127.4705 Ops/s $\color{#35bf28}+0.15\%$
test_redq_speed 14.1441ms 12.7032ms 78.7202 Ops/s 80.2697 Ops/s $\color{#d91a1a}-1.93\%$
test_redq_deprec_speed 13.7943ms 12.7972ms 78.1419 Ops/s 79.2221 Ops/s $\color{#d91a1a}-1.36\%$
test_td3_speed 8.9090ms 7.8207ms 127.8656 Ops/s 128.5812 Ops/s $\color{#d91a1a}-0.56\%$
test_cql_speed 36.5128ms 35.2983ms 28.3300 Ops/s 25.9476 Ops/s $\textbf{\color{#35bf28}+9.18\%}$
test_a2c_speed 8.6305ms 7.3036ms 136.9197 Ops/s 138.0935 Ops/s $\color{#d91a1a}-0.85\%$
test_ppo_speed 8.7751ms 7.5944ms 131.6768 Ops/s 132.8008 Ops/s $\color{#d91a1a}-0.85\%$
test_reinforce_speed 7.6099ms 6.4577ms 154.8551 Ops/s 156.0711 Ops/s $\color{#d91a1a}-0.78\%$
test_iql_speed 33.7224ms 31.8966ms 31.3513 Ops/s 31.5147 Ops/s $\color{#d91a1a}-0.52\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 5.1503ms 4.8335ms 206.8914 Ops/s 210.8174 Ops/s $\color{#d91a1a}-1.86\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.7900ms 0.4776ms 2.0938 KOps/s 2.1053 KOps/s $\color{#d91a1a}-0.55\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.6541ms 0.4559ms 2.1936 KOps/s 2.2340 KOps/s $\color{#d91a1a}-1.81\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 8.0620ms 4.8624ms 205.6593 Ops/s 212.9784 Ops/s $\color{#d91a1a}-3.44\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 1.4029ms 0.4762ms 2.1001 KOps/s 2.1263 KOps/s $\color{#d91a1a}-1.23\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.6312ms 0.4506ms 2.2192 KOps/s 2.2502 KOps/s $\color{#d91a1a}-1.37\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 2.3255ms 1.6833ms 594.0879 Ops/s 596.0065 Ops/s $\color{#d91a1a}-0.32\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 1.9001ms 1.5914ms 628.3874 Ops/s 631.9046 Ops/s $\color{#d91a1a}-0.56\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 7.4174ms 4.9801ms 200.7975 Ops/s 207.3280 Ops/s $\color{#d91a1a}-3.15\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.0169ms 0.6155ms 1.6246 KOps/s 1.6506 KOps/s $\color{#d91a1a}-1.58\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8032ms 0.5908ms 1.6927 KOps/s 1.7233 KOps/s $\color{#d91a1a}-1.78\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 5.2511ms 4.9461ms 202.1777 Ops/s 211.3520 Ops/s $\color{#d91a1a}-4.34\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 1.3477ms 0.4848ms 2.0627 KOps/s 2.1034 KOps/s $\color{#d91a1a}-1.93\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.6174ms 0.4585ms 2.1809 KOps/s 2.2149 KOps/s $\color{#d91a1a}-1.54\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 5.1622ms 4.8416ms 206.5451 Ops/s 214.6743 Ops/s $\color{#d91a1a}-3.79\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.1326s 0.6618ms 1.5111 KOps/s 2.1355 KOps/s $\textbf{\color{#d91a1a}-29.24\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.6263ms 0.4524ms 2.2105 KOps/s 2.2379 KOps/s $\color{#d91a1a}-1.23\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 5.3663ms 5.0727ms 197.1331 Ops/s 208.1221 Ops/s $\textbf{\color{#d91a1a}-5.28\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.4067ms 0.6461ms 1.5479 KOps/s 1.6435 KOps/s $\textbf{\color{#d91a1a}-5.82\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 7.6149ms 0.5966ms 1.6761 KOps/s 1.6969 KOps/s $\color{#d91a1a}-1.23\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.1185s 6.0054ms 166.5161 Ops/s 168.2147 Ops/s $\color{#d91a1a}-1.01\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 16.7261ms 12.8176ms 78.0178 Ops/s 77.2864 Ops/s $\color{#35bf28}+0.95\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 1.6086ms 1.1040ms 905.7691 Ops/s 894.3119 Ops/s $\color{#35bf28}+1.28\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1130s 8.0320ms 124.5019 Ops/s 173.2519 Ops/s $\textbf{\color{#d91a1a}-28.14\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 17.1948ms 12.7338ms 78.5313 Ops/s 77.6094 Ops/s $\color{#35bf28}+1.19\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 6.2466ms 1.2359ms 809.1260 Ops/s 885.6418 Ops/s $\textbf{\color{#d91a1a}-8.64\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1111s 6.0055ms 166.5133 Ops/s 122.8026 Ops/s $\textbf{\color{#35bf28}+35.59\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 17.3413ms 12.8597ms 77.7625 Ops/s 76.3141 Ops/s $\color{#35bf28}+1.90\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 6.5008ms 1.3817ms 723.7231 Ops/s 724.8052 Ops/s $\color{#d91a1a}-0.15\%$

@vmoens vmoens mentioned this pull request Aug 10, 2024
Merged
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 10, 2024 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 94. Improved: $\large\color{#35bf28}2$. Worsened: $\large\color{#d91a1a}5$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 0.1049s 0.1047s 9.5498 Ops/s 9.5679 Ops/s $\color{#d91a1a}-0.19\%$
test_sync 91.6926ms 88.9345ms 11.2442 Ops/s 11.2537 Ops/s $\color{#d91a1a}-0.08\%$
test_async 0.2557s 88.1670ms 11.3421 Ops/s 11.4588 Ops/s $\color{#d91a1a}-1.02\%$
test_single_pixels 0.1151s 0.1146s 8.7268 Ops/s 8.6823 Ops/s $\color{#35bf28}+0.51\%$
test_sync_pixels 76.0341ms 75.6952ms 13.2109 Ops/s 13.1498 Ops/s $\color{#35bf28}+0.46\%$
test_async_pixels 0.1393s 70.0458ms 14.2764 Ops/s 14.2809 Ops/s $\color{#d91a1a}-0.03\%$
test_simple 0.7486s 0.7463s 1.3399 Ops/s 1.3137 Ops/s $\color{#35bf28}+2.00\%$
test_transformed 1.0809s 1.0031s 0.9969 Ops/s 1.0256 Ops/s $\color{#d91a1a}-2.80\%$
test_serial 2.2016s 2.1267s 0.4702 Ops/s 0.4774 Ops/s $\color{#d91a1a}-1.51\%$
test_parallel 1.9312s 1.8686s 0.5352 Ops/s 0.5347 Ops/s $\color{#35bf28}+0.09\%$
test_step_mdp_speed[True-True-True-True-True] 0.2514ms 37.2088μs 26.8753 KOps/s 26.7257 KOps/s $\color{#35bf28}+0.56\%$
test_step_mdp_speed[True-True-True-True-False] 0.1185ms 20.7338μs 48.2305 KOps/s 48.3124 KOps/s $\color{#d91a1a}-0.17\%$
test_step_mdp_speed[True-True-True-False-True] 99.7110μs 21.0069μs 47.6034 KOps/s 49.0832 KOps/s $\color{#d91a1a}-3.01\%$
test_step_mdp_speed[True-True-True-False-False] 32.8510μs 11.6684μs 85.7012 KOps/s 85.1838 KOps/s $\color{#35bf28}+0.61\%$
test_step_mdp_speed[True-True-False-True-True] 0.1290ms 39.6365μs 25.2292 KOps/s 25.6451 KOps/s $\color{#d91a1a}-1.62\%$
test_step_mdp_speed[True-True-False-True-False] 49.8310μs 22.7704μs 43.9166 KOps/s 43.6411 KOps/s $\color{#35bf28}+0.63\%$
test_step_mdp_speed[True-True-False-False-True] 44.7920μs 22.7806μs 43.8969 KOps/s 44.0926 KOps/s $\color{#d91a1a}-0.44\%$
test_step_mdp_speed[True-True-False-False-False] 35.5710μs 13.8932μs 71.9779 KOps/s 71.8967 KOps/s $\color{#35bf28}+0.11\%$
test_step_mdp_speed[True-False-True-True-True] 66.3510μs 41.8128μs 23.9161 KOps/s 23.9511 KOps/s $\color{#d91a1a}-0.15\%$
test_step_mdp_speed[True-False-True-True-False] 44.1910μs 25.1781μs 39.7170 KOps/s 39.7458 KOps/s $\color{#d91a1a}-0.07\%$
test_step_mdp_speed[True-False-True-False-True] 41.5510μs 23.1969μs 43.1092 KOps/s 44.4368 KOps/s $\color{#d91a1a}-2.99\%$
test_step_mdp_speed[True-False-True-False-False] 43.9510μs 14.0826μs 71.0094 KOps/s 71.4462 KOps/s $\color{#d91a1a}-0.61\%$
test_step_mdp_speed[True-False-False-True-True] 59.3110μs 43.9868μs 22.7341 KOps/s 22.9181 KOps/s $\color{#d91a1a}-0.80\%$
test_step_mdp_speed[True-False-False-True-False] 42.3920μs 27.3184μs 36.6053 KOps/s 36.7204 KOps/s $\color{#d91a1a}-0.31\%$
test_step_mdp_speed[True-False-False-False-True] 44.2200μs 25.4109μs 39.3532 KOps/s 39.6959 KOps/s $\color{#d91a1a}-0.86\%$
test_step_mdp_speed[True-False-False-False-False] 42.2000μs 16.2015μs 61.7228 KOps/s 62.1680 KOps/s $\color{#d91a1a}-0.72\%$
test_step_mdp_speed[False-True-True-True-True] 67.5620μs 41.5723μs 24.0545 KOps/s 24.1190 KOps/s $\color{#d91a1a}-0.27\%$
test_step_mdp_speed[False-True-True-True-False] 42.2210μs 25.4462μs 39.2986 KOps/s 39.9008 KOps/s $\color{#d91a1a}-1.51\%$
test_step_mdp_speed[False-True-True-False-True] 0.1060ms 27.2884μs 36.6456 KOps/s 36.6214 KOps/s $\color{#35bf28}+0.07\%$
test_step_mdp_speed[False-True-True-False-False] 35.6700μs 15.9086μs 62.8592 KOps/s 63.5333 KOps/s $\color{#d91a1a}-1.06\%$
test_step_mdp_speed[False-True-False-True-True] 65.3220μs 43.7835μs 22.8397 KOps/s 23.0301 KOps/s $\color{#d91a1a}-0.83\%$
test_step_mdp_speed[False-True-False-True-False] 51.8810μs 27.2950μs 36.6368 KOps/s 36.8377 KOps/s $\color{#d91a1a}-0.55\%$
test_step_mdp_speed[False-True-False-False-True] 0.1386ms 29.3123μs 34.1154 KOps/s 34.0561 KOps/s $\color{#35bf28}+0.17\%$
test_step_mdp_speed[False-True-False-False-False] 39.5310μs 18.0142μs 55.5117 KOps/s 56.0578 KOps/s $\color{#d91a1a}-0.97\%$
test_step_mdp_speed[False-False-True-True-True] 4.2467ms 46.5115μs 21.5000 KOps/s 21.7917 KOps/s $\color{#d91a1a}-1.34\%$
test_step_mdp_speed[False-False-True-True-False] 59.5010μs 29.8868μs 33.4596 KOps/s 33.2249 KOps/s $\color{#35bf28}+0.71\%$
test_step_mdp_speed[False-False-True-False-True] 49.2810μs 29.1404μs 34.3167 KOps/s 34.0901 KOps/s $\color{#35bf28}+0.66\%$
test_step_mdp_speed[False-False-True-False-False] 46.2100μs 18.2398μs 54.8252 KOps/s 55.7569 KOps/s $\color{#d91a1a}-1.67\%$
test_step_mdp_speed[False-False-False-True-True] 0.1372ms 47.5484μs 21.0312 KOps/s 21.3346 KOps/s $\color{#d91a1a}-1.42\%$
test_step_mdp_speed[False-False-False-True-False] 57.3710μs 31.7136μs 31.5322 KOps/s 31.6447 KOps/s $\color{#d91a1a}-0.36\%$
test_step_mdp_speed[False-False-False-False-True] 58.0710μs 31.0469μs 32.2093 KOps/s 31.9866 KOps/s $\color{#35bf28}+0.70\%$
test_step_mdp_speed[False-False-False-False-False] 0.1040ms 20.1699μs 49.5788 KOps/s 50.2561 KOps/s $\color{#d91a1a}-1.35\%$
test_values[generalized_advantage_estimate-True-True] 24.8107ms 24.2420ms 41.2506 Ops/s 41.4864 Ops/s $\color{#d91a1a}-0.57\%$
test_values[vec_generalized_advantage_estimate-True-True] 0.1021s 2.9374ms 340.4421 Ops/s 360.3101 Ops/s $\textbf{\color{#d91a1a}-5.51\%}$
test_values[td0_return_estimate-False-False] 0.2077ms 67.7637μs 14.7572 KOps/s 15.6242 KOps/s $\textbf{\color{#d91a1a}-5.55\%}$
test_values[td1_return_estimate-False-False] 54.5828ms 54.3229ms 18.4084 Ops/s 18.6725 Ops/s $\color{#d91a1a}-1.41\%$
test_values[vec_td1_return_estimate-False-False] 1.3462ms 1.0762ms 929.1851 Ops/s 928.1877 Ops/s $\color{#35bf28}+0.11\%$
test_values[td_lambda_return_estimate-True-False] 93.4198ms 86.3954ms 11.5747 Ops/s 11.8093 Ops/s $\color{#d91a1a}-1.99\%$
test_values[vec_td_lambda_return_estimate-True-False] 1.4085ms 1.0745ms 930.6673 Ops/s 931.7633 Ops/s $\color{#d91a1a}-0.12\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 24.5038ms 24.2689ms 41.2051 Ops/s 41.8112 Ops/s $\color{#d91a1a}-1.45\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 0.9409ms 0.7102ms 1.4081 KOps/s 1.4002 KOps/s $\color{#35bf28}+0.56\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.8162ms 0.6627ms 1.5091 KOps/s 1.5137 KOps/s $\color{#d91a1a}-0.31\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 1.6082ms 1.4644ms 682.8887 Ops/s 685.8178 Ops/s $\color{#d91a1a}-0.43\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 0.8528ms 0.6758ms 1.4798 KOps/s 1.4830 KOps/s $\color{#d91a1a}-0.22\%$
test_dqn_speed 7.6442ms 1.3384ms 747.1541 Ops/s 744.4890 Ops/s $\color{#35bf28}+0.36\%$
test_ddpg_speed 3.1222ms 2.7378ms 365.2569 Ops/s 366.7684 Ops/s $\color{#d91a1a}-0.41\%$
test_sac_speed 0.1056s 8.7087ms 114.8278 Ops/s 128.3650 Ops/s $\textbf{\color{#d91a1a}-10.55\%}$
test_redq_speed 11.8543ms 10.2021ms 98.0190 Ops/s 97.8623 Ops/s $\color{#35bf28}+0.16\%$
test_redq_deprec_speed 11.5756ms 11.0341ms 90.6283 Ops/s 89.2298 Ops/s $\color{#35bf28}+1.57\%$
test_td3_speed 7.9860ms 7.8159ms 127.9446 Ops/s 128.3931 Ops/s $\color{#d91a1a}-0.35\%$
test_cql_speed 26.1444ms 25.2585ms 39.5906 Ops/s 39.6905 Ops/s $\color{#d91a1a}-0.25\%$
test_a2c_speed 6.4665ms 5.6169ms 178.0335 Ops/s 176.9994 Ops/s $\color{#35bf28}+0.58\%$
test_ppo_speed 7.7518ms 5.9718ms 167.4534 Ops/s 168.6815 Ops/s $\color{#d91a1a}-0.73\%$
test_reinforce_speed 4.8613ms 4.4640ms 224.0127 Ops/s 217.4713 Ops/s $\color{#35bf28}+3.01\%$
test_iql_speed 20.9616ms 19.6251ms 50.9551 Ops/s 50.4675 Ops/s $\color{#35bf28}+0.97\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 6.6904ms 6.4976ms 153.9023 Ops/s 154.0783 Ops/s $\color{#d91a1a}-0.11\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 1.0587ms 0.5226ms 1.9135 KOps/s 1.9117 KOps/s $\color{#35bf28}+0.10\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.6900ms 0.5035ms 1.9863 KOps/s 1.9842 KOps/s $\color{#35bf28}+0.10\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 6.7921ms 6.4411ms 155.2527 Ops/s 156.9911 Ops/s $\color{#d91a1a}-1.11\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 2.0009ms 0.5169ms 1.9345 KOps/s 1.9445 KOps/s $\color{#d91a1a}-0.51\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.7117ms 0.5019ms 1.9923 KOps/s 2.0044 KOps/s $\color{#d91a1a}-0.61\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 2.2545ms 1.9945ms 501.3754 Ops/s 513.4528 Ops/s $\color{#d91a1a}-2.35\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 5.6665ms 1.9066ms 524.4920 Ops/s 539.7341 Ops/s $\color{#d91a1a}-2.82\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 6.7826ms 6.6079ms 151.3343 Ops/s 151.6511 Ops/s $\color{#d91a1a}-0.21\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.1299s 0.7869ms 1.2708 KOps/s 1.4732 KOps/s $\textbf{\color{#d91a1a}-13.74\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8392ms 0.6487ms 1.5416 KOps/s 1.5388 KOps/s $\color{#35bf28}+0.18\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 6.8018ms 6.5436ms 152.8202 Ops/s 154.4415 Ops/s $\color{#d91a1a}-1.05\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 1.6820ms 0.5204ms 1.9217 KOps/s 1.9138 KOps/s $\color{#35bf28}+0.41\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.6809ms 0.4994ms 2.0022 KOps/s 1.9842 KOps/s $\color{#35bf28}+0.91\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 6.7598ms 6.4325ms 155.4598 Ops/s 156.2260 Ops/s $\color{#d91a1a}-0.49\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.7105ms 0.5157ms 1.9390 KOps/s 1.9273 KOps/s $\color{#35bf28}+0.61\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 4.2243ms 0.5046ms 1.9819 KOps/s 1.9750 KOps/s $\color{#35bf28}+0.35\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 6.8733ms 6.6760ms 149.7909 Ops/s 150.7811 Ops/s $\color{#d91a1a}-0.66\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.8674ms 0.6720ms 1.4880 KOps/s 1.4914 KOps/s $\color{#d91a1a}-0.23\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8441ms 0.6590ms 1.5175 KOps/s 1.5332 KOps/s $\color{#d91a1a}-1.02\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.1536s 8.1057ms 123.3692 Ops/s 127.5839 Ops/s $\color{#d91a1a}-3.30\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 18.4356ms 15.9786ms 62.5838 Ops/s 64.0314 Ops/s $\color{#d91a1a}-2.26\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 2.1875ms 1.2746ms 784.5447 Ops/s 758.7694 Ops/s $\color{#35bf28}+3.40\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1291s 7.6033ms 131.5224 Ops/s 97.5811 Ops/s $\textbf{\color{#35bf28}+34.78\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 18.0132ms 15.7937ms 63.3164 Ops/s 63.2860 Ops/s $\color{#35bf28}+0.05\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 6.9990ms 1.3628ms 733.8079 Ops/s 767.6007 Ops/s $\color{#d91a1a}-4.40\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1311s 10.3369ms 96.7405 Ops/s 129.4344 Ops/s $\textbf{\color{#d91a1a}-25.26\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 18.8426ms 16.1363ms 61.9721 Ops/s 62.9701 Ops/s $\color{#d91a1a}-1.58\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 2.2277ms 1.3026ms 767.6758 Ops/s 672.8504 Ops/s $\textbf{\color{#35bf28}+14.09\%}$

@vmoens
Update
f6e1335 
[ghstack-poisoned]
@vmoens
Update
67d407d 
[ghstack-poisoned]
@vmoens
Update
8bf3ad2 
[ghstack-poisoned]
@vmoens vmoens mentioned this pull request Aug 10, 2024
Open
@vmoens
Update
93a32de 
[ghstack-poisoned]
@vmoens
Update
47c463f 
[ghstack-poisoned]
@vmoens
Update
8e9d28d 
[ghstack-poisoned]
@vmoens
Update
ef3d608 
[ghstack-poisoned]
@vmoens
Update
9df29d1 
[ghstack-poisoned]
@vmoens
Update
cdba311 
[ghstack-poisoned]
@vmoens
Update
cd51882 
[ghstack-poisoned]
@vmoens
Update
91a276e 
[ghstack-poisoned]
@vmoens
Update
763042d 
[ghstack-poisoned]
@vmoens vmoens merged commit 763042d into gh/vmoens/15/base Aug 13, 2024
14 of 21 checks passed
vmoens added a commit that referenced this pull request Aug 13, 2024
@vmoens
[Feature] Pass replay buffers to MultiaSyncDataCollector
a0c12cd 
ghstack-source-id: 7275208e2f02560229ca83c999cd9b0ae68aaf4f
Pull Request resolved: #2387
@vmoens vmoens added the enhancement New feature or request label Aug 13, 2024
@vmoens vmoens deleted the gh/vmoens/15/head branch August 13, 2024 19:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@vmoens @facebook-github-bot