Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed TypeError when retraining #14

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

szymonk92
Copy link

The error 'TypeError: Population must be a sequence or set. For dicts, use list(d)' sometimes occurred, so I made this small fix for next users who might play with it.

The error 'TypeError: Population must be a sequence or set.  For dicts, use list(d)' sometimes occurred
@yenchenlin
Copy link
Owner

Can you describe your environment?
I never see TypeError: Population must be a sequence or set

@szymonk92
Copy link
Author

Sure,

  • Ubuntu 14.04
  • Tensorflow 0.8
  • Python 3.4

@szymonk92
Copy link
Author

szymonk92 commented Apr 17, 2016

It looks like this:

...
TIMESTEP 10000 / STATE observe / EPSILON 0.1 / ACTION 0 / REWARD 0.1 / Q_MAX -1.201076e-03
TIMESTEP 10001 / STATE explore / EPSILON 0.1 / ACTION 0 / REWARD 0.1 / Q_MAX -1.623187e-03
Traceback (most recent call last):
  File "deep_q_network.py", line 215, in <module>
    main()
  File "deep_q_network.py", line 212, in main
    playGame()
  File "deep_q_network.py", line 209, in playGame
    trainNetwork(s, readout, h_fc1, sess)
  File "deep_q_network.py", line 153, in trainNetwork
    minibatch = random.sample(D, BATCH)
  File "/home/cave/anaconda3/envs/tensorflow/lib/python3.4/random.py", line 311, in sample
    raise TypeError("Population must be a sequence or set.  For dicts, use list(d).")
TypeError: Population must be a sequence or set.  For dicts, use list(d).

@yenchenlin
Copy link
Owner

Big thanks for your report.
What I'm thinking is that D here is a deque instead of a dicts, so I don't know whether it's the right way to fix.
Also, we need to know what cause it happen.

Can you provide the parameter you are using? e.g. OBSERVE, EXPLORE

@szymonk92
Copy link
Author

I just followed 'How to reproduce?'. So have exactly this same parameters as you wrote. But I have some problems, most of a time the bird is going up and can't go through pipes. That might be related.

@yenchenlin
Copy link
Owner

yenchenlin commented Apr 17, 2016

But I have some problems, most of a time the bird is going up and can't go through pipes.

That's normal. You should leave it run in the background and check it tomorrow 😈
(make sure you are using a GPU)

That might be related.

No, I don't think so. There may be something worng when the system switch from OBSERVE state to EXPLORE state since you crashed at TIMESTEP 10001.

@szymonk92
Copy link
Author

Tomorrow? Okey:balloon:
All the time when it switch from OBSERVE to EXPLORE I have this same error..

@yenchenlin
Copy link
Owner

yenchenlin commented Apr 17, 2016

Nice! So I think we find the problem.
I will have a closer look into this tomorrow since I am currently relatively busy.

BTW, why the bird keeps going up and can't go through the pipes is because it still got high probability to choose a random action. And since the FPS of the game is 30, i.e. agent can make an action every 0.03s, it has a high probability to choose jump, which results in what you observed.

However, after some epochs of training, it will keep sampling those bad memories to fix its behavior. And when the ϵ finally anneals down to some relatively low value, agent will start to follow the policy it learned. (no more random action!)

At that time, you can see the bird flying like a ninja.

Hope this helps! 🍻

@yenchenlin
Copy link
Owner

Hey @szymonk92 , how's it going?

@yenchenlin
Copy link
Owner

Any updates? @szymonk92

@szymonk92
Copy link
Author

It's fine, but I had to add list(D) to make it work.
Did you test your version on Python 3.4?

@patrick-llgc
Copy link

@yenchenlin For python versions below 3.5, random.sample() cannot take deque as argument. As @szymonk92 mentioned, use list(D) as a fix for python version below 3.5.

I would suggest put list(D) in the code as it works for most versions of python.

@BPDanek
Copy link

BPDanek commented Apr 11, 2019

I had the same issue in python 3.4; when explore mode transitioned to observe there was a bug. By making deque, D a list the issue was fixed.
I will retract my pull request

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants