Reinforcement learning with tensorflow

Please note that I'm using Linux mint.

Reinforcement learning with tensorflow.
Clone the project from github (it's a forked project which is originally dennybritz's):

$ git clone https://github.com/lechatthecat/reinforcement-learning-with-tensorflow

(Note: project will be downloaded in the currect folder.)

Then run the program:

$ cd reinforcement-learning-with-tensorflow/DQN
$ python3.5 dqn.py

Now reinforcement learning starts:

After 700 episodes.

The learned weights are saved and can be loaded later.

Detail

When it is started, it populates the replay memory at first (50000 steps by default). This isn't learning at this point.

After them, it starts learning through 500 episodes training. The higher the mean_q is, the better the machine is learning.

Press ctrl + c to stop the training.

Fizzbuzz zero

AlphaGoZero and Fizzbuzz Zero

You might know that AlphaGo played Go with Lee Sedol (one of the best players of Go) and AlphaGo won all but the fourth game. This proved that an artificial intelligence, which learned by reinfocement learning, can even surpass the best human player. AlphaGo used many new moves and skills, that were still unknown to human players at that time, so many people were impressed by how AlphaGo was strong and creative.

Although this surprised the world, the subsequent model AlphaGoZero has surprised the world even more by playing 100 games of Go with AlphaGo and won all the games. More surprisingly, although AlphaGo learned from human's go games, AlphaGoZero didn't learn from them. AlphaGoZero learned Go all by himself from scratch. Now artificial intelligence doesn't even need humans to learn Go games and is much stronger than humans.

In this post, I will introduce a program that is similar to AlphaGoZero. This is called FizzBuzzZero. This make two AI players fight with each other and make them learn how to play FizzBuzz (like AlphaGoZero learned how to play Go). It's more simple project, so it can be applied to other fields. Regardless of the strength, Fizzbuzz zero has simple structure.

How to use Fizzbuzz zero

At first, download the project.

$ git clone https://github.com/ymgaq/FizzBuzzZero

Then let it play fizzbuzz.

$ cd FizzBuzzZero
$ python fizzbuzzzero.py --learn 

This AI didn't even know how to count at first. What he knew was only the rule. But after 100 games, he answered perfectly (accuracy 100%).

AlphaGoZero-like project

suragnair/alpha-zero-general claims to be a clean implementation based on AlphaZero for any game in any framework. This can be interesting too.

Download it this way:

$ git clone https://github.com/suragnair/alpha-zero-general.git

Start training a model for Othello this way:

$ cd alpha-zero-general
$ python main.py

Training of Othero.

Chess & Shogi

AlphaZero is a subsequent model of AlphaGoZero and can learn Shogi and Chess in addition to Go. According to the DeepMind's research paper, it has become unbelievably strong at Shogi & Chess after days of training.

It might be good to do similar research with these AlphaZero-like projects and python-shogi & python-chess.

Symfony 4 Framework: install and setup with virtualbox/vagrant

At first, prepare a development environment for PHP:
Basic preparation of PHP for development

Then log-in to the virtual machine:

Then move to the vagrant directory:

$ cd /vagrant

Install composer

Install the composer:

$ curl -sS https://getcomposer.org/installer | php

$ sudo mv composer.phar /usr/local/bin/composer

Check if you could install it successfully:

$ composer

If this is displayed after running "composer" command, you have installed it successfully.

Then run the following:

$ composer self-update

Install Symfony 4.0

Run the following to install symfony:

$ cd /vagrant
$ composer create-project symfony/website-skeleton my-symfony-project

You will have your symfony project:

Checking for Security Vulnerabilities

Symfony has a utility called "Security Checker" (or sec-checker) to check if your project's dependencies contain any known security vulnerability. This is NOT mandatory, but if you want it, you can install it by:

$ cd my-symfony-project/
$ composer require sec-checker --dev

Also, you can install a tool to check if all requirements are met:

$ cd my-symfony-project/
$ composer require symfony/requirements-checker

Configuration of Symfony

We will check if symfony is working. Go to the URL like:
http://192.168.33.10/my-symfony-project/public/

If you see the following, your symfony is working correctly.

--- optional below ---
If you see errors because it can't write in the log file, run the following command to change the folder's owner.
For Apache users:

$ sudo chown -R apache:apache /vagrant/my-symfony-project/{path to the log folder}

For Nginx (with php-fpm) users:

$ sudo chown -R php-fpm:php-fpm /vagrant/my-symfony-project/{path to the log folder}

If this doesn't work, change/add the following line in Vagrantfile.
For Apache users:

config.vm.synced_folder "./", "/vagrant", type: "virtualbox", owner: 'vagrant', group: 'apache', mount_options: ['dmode=777', 'fmode=777']

For Nginx (with php-fpm) users:

config.vm.synced_folder ".", "/vagrant", type: "virtualbox", owner: 'vagrant', group: 'php-fpm', mount_options: ['dmode=777', 'fmode=777']

And "vagrant reload".

If even that doesn't work, it might be because of SE linux. Turn off SE linux this way:

$ sudo getenforce
Enforcing
$ sudo setenforce 0
$ sudo getenforce
Permissive

--- optional above ---

Database

You can configure for your database from here:

My mysql's user name is root, password is also root, so the database URL is:
mysql://root:root@127.0.0.1:3306/testdb

You don't need to have a database named "testdb" because we will create the db on console. Run the following command:

$ cd /vagrant/my-symfony-project/
$ php bin/console doctrine:database:create

Database was created:

An error occurred while loading the web debug toolbar.

You might see this error at the bottom:

This is a bug according to this page:
https://github.com/symfony/symfony-docs/issues/9178#issuecomment-363006113

References

Symfony doc: https://symfony.com/doc/current/doctrine.html

Basic preparation of PHP for development

Prepare development environment for PHP 

1. Virtual box and Vagrant

2. Install Apache

3. Install MySQL
(Or 3. Install MariaDB)

4. Install PHP

How to use junit on inteliJ IDEA

Prerequisites

I am using Windows8.1 for my OS.
Suppose we are using Windows 8.1.

JUnit

At first, we will add junit library. Open File -> Project Structure.

Then open "Library". Click on green + . Search for junit and find something like "junit:junit:4.12". Click on it and add it as library.

We need to add a directory for tests. Create the directory:

Click OK.

Now you have a "Test" directory.

Right click on it and mark the directory as "Test Sources Root"

We will add a test now. We will make a test for this method:

public class TestJunit {
    public int test(){
        return 10;
    }
}

This is supposed to return 10.

You can use the "Create Test" intention action by pressing Alt+Enter.

Press Alt+Enter on the class name.

You will see that this will appear.

Click "Create Test". Then you will see this:

If you opened this dialog for the fist time, to add the required library jars to the module dependencies, you need to click "Fix".

Original image is from: https://www.jetbrains.com/help/idea/configuring-testing-libraries.html

Now tick methods for which you want to test and then click OK.

Your groovy script for test is added here:

Write as follows in the TestTestJunit.groovy:

import TestJunit;
import org.junit.Assert;
import org.junit.Test;

public class TestTestJunit {
    @Test
    public void unit_test1(){
        TestJunit jtest = new TestJunit();
        Assert.assertEquals(10, jtest.test());
        Assert.assertEquals("This is not 10!!!",10, jtest.test());
       
        System.out.println("Unit test1 finished without error.");
    }
}

Now run the test (right click inside unit_test1 method):

Test finished.

Just for a try, we can change the returned value and test the method again:

Test result:

You can see it is correctly detecting that the returned value is different from the expected value.

How to use Nablarch

Nablarch is a minor Japanese-made java framework. The error message etc can be Japanese, so I am not sure you will like it. Official document is here (Japanese). I'm sure you will find little questions & examples on internet (like stackoverflow). Official examples are here (Japanese).

According to these documents, Nablarch is supposed to be developed by IntelliJ IDEA.

I am using Windows8.1 for my OS.
Suppose we are using Windows 8.1.

The following is how to install and build Nablarch.

JDK

At first, we will install JDK.
Install from here: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

Then set the path for Java home. Open the control panel and choose the "System and Security" --> "System".
Click the Advanced tab and the button Environment Variables.

Click New... at the System Variables or User variables.
Use JDK_HOME as the variable name and "C:\Program Files\Java\(your jdk folder)" as the path.

If you have installed wrong version of Java and want to change the version, download the right version of Java and change the path of JAVA_HOME to the new JDK's path. If this doesn't work and if you are using windows8, check if there is "C:\ProgramData\Oracle\Java\javapath" in the "Path" in System Variables. If there is, delete "C:\ProgramData\Oracle\Java\javapath" from the path. Otherwise the Java version doesn't change on the command prompt.

Check if JDK is installed successfully by running a command "java -version" on command prompt.

java -version

java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

Maven

Then install Maven.
Visit the official site of Maven: http://maven.apache.org/download.cgi
And install Maven (like "apache-maven-3.5.2-bin.zip") from there.

After the installation, add both M2_HOME and MAVEN_HOME variables in the Windows environment, and point it to your Maven folder.

Maven is a project management tool, which is aimed for Java projects.
Available commands of Maven:

install
clean
compile
package
validate
test-compile
test
integration-test
verify
deploy

Check if Mave is installed successfully by running the following command.

mvn -version

Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T16:58:1
3+09:00)
Maven home: C:\Program Files\Apache\maven\bin\..
Java version: 1.8.0_151, vendor: Oracle Corporation
Java home: C:\Program Files\Java\jdk1_8\jre
Default locale: en_GB, platform encoding: MS932
OS name: "windows 8.1", version: "6.3", arch: "amd64", family: "windows"
'cmd' is not recognized as an internal or external command,
operable program or batch file.

You can find setting.xml here:
C:\Program Files\Apache\maven\conf

Copy the setting.xml to ${user.home}/.m2 directory.  This will be user-specific settings.

We don't use it yet. When it is necessary, configure and activate it. But how setting.xml file can be used:

<settings>
  <!-- *snip* -->
  <profiles>
    <profile>
      <id>my-repository</id>
      <repositories>
        <repository>
          <id>my-repository-release</id>
          <url><!-- URL of Project Local Release Repository--></url>
          <releases>
            <enabled>true</enabled>
          </releases>
          <snapshots>
            <enabled>false</enabled>
          </snapshots>
        </repository>
        <repository>
          <id>my-repository-snapshot</id>
          <url><!-- URL of Project Local Snapshot Repository --></url>
          <releases>
            <enabled>false</enabled>
          </releases>
          <snapshots>
            <enabled>true</enabled>
          </snapshots>
        </repository>
      </repositories>
      <pluginRepositories>
        <pluginRepository>
          <id>my-repository-release</id>
          <url><!-- URL of Project Local Release Repository --></url>
          <releases>
            <enabled>true</enabled>
          </releases>
          <snapshots>
            <enabled>false</enabled>
          </snapshots>
        </pluginRepository>
        <pluginRepository>
          <id>my-repository-snapshot</id>
          <url><!-- URL of Project Local Snapshot Repository --></url>
          <releases>
            <enabled>false</enabled>
          </releases>
          <snapshots>
            <enabled>true</enabled>
          </snapshots>
        </pluginRepository>
      </pluginRepositories>
    </profile>
  </profiles>

  <!-- Activate this repository -->
  <activeProfiles>
    <activeProfile>my-repository</activeProfile>
  </activeProfiles>
  <!-- *snip* -->
</settings>

Nablarch example web

Create a folder and clone the project:

$ mkdir ${user.home}\Documents\nablarch-example
$ git clone https://github.com/nablarch/nablarch-example-web.git

Please replace ${user.home} with your user home directory.

And we get the example:

Now create and configure database and create entity classes:

$ cd nablarch-example-web
$ mvn generate-resources

Now build the application:

$ cd nablarch-example-web
$ mvn generate-resources

Start the application:

$ mvn waitt:run

Wait for a while and this page is opened automatically:

Use these to sign in:
Sign-in ID: 10000001
Password: pass123-

After sigining in:

How to use a trained model of TF Detect in Android

1. How to use Tensorflow Object Detection API
2. How to train for Tensorflow Object Detection API
3. How to use Tensorboard
4. How to use a trained model of TF Detect in Android

We trained a model in previous posts. We will use it for android.

At first, install tensorflow and test if demo apps work.
How to use tensorflow demo apps on android

If it works in your PC, now we change the model used for TF detect.

Open this directory:

$HOME/Documents/tensorflow/tensorflow/examples/android/assets

Maybe it should be something like ~/Documents/tensorflow/tensorflow/examples/android/assets or /home/yourname/Documents/tensorflow/tensorflow/examples/android/assets

Then move your trained .pb file here. Also we will create a label file.

The inside of the thora_label.txt is:

???
thora

Now open "DetectorActivity".

Change the path and point them to the files we created in assets folder like this:

Now start the demo app in your phone! It should detect based on your trained model.
This is a capture of my demo app. You can see that it is detecting Thora's face.

How to go back to previous version in git

$ git clone [remote_address_here] my_repo
$ cd my_repo
$ git reset --hard [ENTER HERE THE COMMIT HASH YOU WANT]

For example

$ git clone https://github.com/tensorflow/models.git
$ cd ./models
$ git reset --hard 490813bdb3499290633919a9867eb0bb6d346d87 

References:
https://stackoverflow.com/questions/3555107/git-clone-particular-version-of-remote-repository/18037563
https://github.com/tensorflow/models/issues/4002