Current Events > Deep Reinforcement Learning in Python!

(+) Yeah! (+)

I just implemented a deep reinforcement learning agent for Ms. Pacman using Deep Q-Network and OpenAI Gym.

What an amazing time we live in.

Run the code yourself: https://github.com/moduIo/Deep-Learning/blob/master/Reinforcement%20Learning/dqn.ipynb
---

# 3x3 Conv
model.add(Conv2D(8, (3, 3), padding='same', input_shape=self.state_size))
model.add(Activation('relu'))

# 2x2 Pooling
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())

# FC Layers
model.add(Dense(16, activation='relu'))
model.add(Dense(self.action_size, activation='linear'))

model.compile(loss='mse', optimizer=Adam(lr=self.learning_rate))


only 7 lines of code needed to construct a neural NW? This is making my tensorflow code looking too verbose
---

FightingGames posted...

# 3x3 Conv
model.add(Conv2D(8, (3, 3), padding='same', input_shape=self.state_size))
model.add(Activation('relu'))

# 2x2 Pooling
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())

# FC Layers
model.add(Dense(16, activation='relu'))
model.add(Dense(self.action_size, activation='linear'))

model.compile(loss='mse', optimizer=Adam(lr=self.learning_rate))


only 7 lines of code needed to construct a neural NW? This is making my tensorflow code looking too verbose

Yeah Keras is so simple, but really TF is barely more verbose. I want to get better at TF to implement custom architectures. DQN is essentially just a CNN and regression haha
---

I'm using this architecture now:

# 3x3 Conv
model.add(Conv2D(16, (3, 3), padding='same', input_shape=self.state_size))
model.add(Activation('relu'))

# 2x2 Pooling
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())

# FC Layers
model.add(Dense(32, activation='relu'))
model.add(Dense(16, activation='relu'))
model.add(Dense(self.action_size, activation='linear'))

With batch size of 8 to reduce overhead since I'm just using my Macbook Air. I wish I had some real hardware so I could use a deeper CNN. Currently my best Pacman got 1060 points which is non-random.
---

The agent currently walks left as it's policy with a 10% random movement chance. 4 epochs in and still pretty stupid so far lol
---

unrelated, but i'm a sbu alumni

what a coincidene

Haha nice there are actually at least 4 of us here. SBU is a great school I'm leaving in a week though to move to Denver... I hate the island so much.
---

https://automatetheboringstuff.com/
---

100 games later and my agent is still stupid af.

I think I would need to run for 10,000's of iterations to learn something.
---

Completely refactored my code along with helpful comments: https://github.com/moduIo/Deep-Q-network

A lot of things were wrong with the original code which was from a tutorial. I added the target network along with some other changes to hopefully make the agent learn
---

Wish I knew programming
---

PonyLivesMatter posted...

Wish I knew programming

It's never too late to learn haha.. just like any skill programming is best honed by practice.
---

I modified the reward function so that pacman is constantly losing 1 point an action. This change and 33 games later and my pacman finally (sometimes) knows how to run the map to get points!

1310 point game atm, although the best was 2260 so far.
---

I've been coding as a hobby for the past 15+ years and have dabbled in everything from simple GUI apps to 2D tiling graphics engines to homegrown operating systems (x86 only). Where should I look to learn more about AI and data analysis?

IllegalAlien posted...

PonyLivesMatter posted...

Wish I knew programming

It's never too late to learn haha.. just like any skill programming is best honed by practice.

do you need school to learn programming?

I did write a "sticker calculator" app for my job in Python, for oil change stickers at a car dealership. the sticker machines are hard-coded by serial port input to a set interval (5k miles), but Ford recommends 7.5k miles in their modern maintenance schedules, so I wrote a GUI app to print out the right mileage difference to punch into the printer machine's keyboard. apparently all the rest of the staff use it, but I wrote it because I was bored one day and didn't feel like doing the math in my head >_____>

ImAMarvel posted...

Fuck, I need to get back to learning Python. I had to work every day this week and was busy over the weekend so I haven't had the time.

I have no idea what the fuck anyone is talking about ITT though.

same

treewojima posted...

I've been coding as a hobby for the past 15+ years and have dabbled in everything from simple GUI apps to 2D tiling graphics engines to homegrown operating systems (x86 only). Where should I look to learn more about AI and data analysis?

I would watch Andrew Ng's course material on Coursera, and read his lecture notes from Stanford if you want more rigorous treatment.

Best thing to do: grab some data and do analysis with it.

scikit-learn provides the Python API for (classical) machine learning, and is extremely easy to use.

Keras, etc provide deep learning APIs. I like Keras API because it's simple and terse like scikit-learn.

The hardest part is gaining intuition in the problem. This requires a decent understanding of the math and trial/error. I didn't know my agent was doing so poorly until I kept trying to make it better haha
---

skimmedmilktea posted...

IllegalAlien posted...

PonyLivesMatter posted...

Wish I knew programming

It's never too late to learn haha.. just like any skill programming is best honed by practice.

do you need school to learn programming?

I don't think so. The main thing is practice. Although, I'm pretty deep in academia, programming is a way of thought. You train yourself to think in logical procedures, which you can naturally express in your code.

It's just like playing an instrument or sport, some people have natural inclinations, but otherwise most of the field improves with practice.
---

the realization that I might be a natural programmer frightens me, since I often curse at my projects when they don't work and move on. or when I get bored of them