Skip to content

Code for the paper "WCSAC: Worst-Case Soft Actor Critic for Safety-Constrained Reinforcement Learning"

License

Notifications You must be signed in to change notification settings

AlgTUDelft/WCSAC

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

62 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

WCSAC

Portions of the code in saclag.py and wcsac.py are adapted from Safety Starter Agents.

Warning: If you want to use the WCSAC algorithm in Safety Gym, make sure to install Safety Gym according to the instructions on the Safety Gym repo.

The various utilities here are copied over from Spinning Up in Deep RL. Two custom environments (StaticEnv-v0 and DynamicEnv-v0) are created within wcsac.py and saclag.py.

Installation & Operation

Two ways to install this package:

Direct installation

To install this package:

git clone https://github.com/AlgTUDelft/WCSAC.git

cd WCSAC/

pip install -e .

Through [Safety Starter Agents]

So you can install Safety Starter Agents, and add wcsac.py and saclag.py of our package to /path/to/safety-starter-agents/safe_rl/sac. Then you can follow the instructions on the Safety Starter Agents to use our algorithms as using their given baselines.

Reproduce Experiments

Example: If you want to test WCSAC with risk level 0.5 in StaticEnv-v0, run:

cd /path/to/wcsac

python wcsac.py --env 'StaticEnv-v0' -s SEED --cl 0.5 --cost_lim d --logger_kwargs_str '{"output_dir": "./xxx"}'

where SEED is the random seed, d is the real-world safety threshold, and '{"output_dir": "./xxx"}' indicates where to store the data.

Cite the Paper

If you use WCSAC code in your paper, please cite:

@inproceedings{WCSAC,
  title     = {{WCSAC: Worst-Case Soft Actor Critic for Safety-Constrained Reinforcement Learning}},
  author    = {
      Yang, Qisong and 
      Sim{\~{a}}o, Thiago D. and
      Tindemans, Simon H. and
      Spaan, Matthijs T. J.
  },
  booktitle = {Thirty-Fifth AAAI Conference on Artificial Intelligence},
  pages     = {10639--10646},
  year      = {2021}
}
@article{WCSAC-IQN,
  author       = {Qisong Yang and
                  Thiago D. Sim{\~{a}}o and
                  Simon H. Tindemans and
                  Matthijs T. J. Spaan},
  title        = {Safety-constrained reinforcement learning with a distributional safety
                  critic},
  journal      = {Mach. Learn.},
  volume       = {112},
  number       = {3},
  pages        = {859--887},
  year         = {2023}
}

About

Code for the paper "WCSAC: Worst-Case Soft Actor Critic for Safety-Constrained Reinforcement Learning"

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages