Mujoco Python to Unity

[mg638]

The feature, motivation and pitch

hello. I would like to demonstrate the process of reproducing mujoco training data in Python and reproducing that data into a Unity Mujoco model using Unity's Mujoco plugin. Is there anything that can be helpful when creating a feature like this? Even if you try to find the information, it's not easy.

Alternatives

No response

Additional context

No response

[Balint-H]

Hello,

Can you explain a bit more what exactly are you trying to do? Transfer a control policy learned in python to Unity? Record motion trajectories as data in python, and use them/replay them in Unity?

[mg638]

Thank you for your reply.
To be exact, the Mujoco model is learned with Pytorch and a PT file is created. After that, I plan to use the learned PT file to play motion in Unity's Mujoco model.

[Balint-H]

If you can convert your model to ONNX format, you can load it in Unity for inference (with Unity Sentis or Barracuda, or check out the ML-Agents package, which uses PyTorch as well, so they probably have converter script examples).

I don't cover the steps for transferring policies from PyTorch , but here's my tutorial on neural network policies with MuJoCo in Unity: https://github.com/Balint-H/mj-unity-tutorial

I have had success in transferring policies learned externally in the past, so it is possible. Be aware that the physics settings of your Unity scene may be different than the one you trained externally (e.g. timesteps, solver, iterations, etc). Make sure you configure it as closely as possible so the system dynamics remain consistent, otherwise the policy may underperform after the shift.

[mg638]

Thank you for your reply.
Do you think it is possible to apply MjData generated using a pt file in C++ to a Unity Mujoco model without using ONNX? I have successfully sent the values ​​of qpos, xpos, etc. to Unity, but the coordinates are moving but the model keeps falling over.

[Balint-H]

Hello, its still not clear to me, do you want to run inference and simulation outside Unity and stream only qpos/xpos to visualize in Unity?

[mg638]

Yes, that's right. The value is being used in Unity's mjdata.