Hi! It’s been a year.

Hi my blog!

It is really long time no see 🙂

I saw my last post was before went ISEF. So it is almost 1 year.

Really many things were happend… I got 4th award on ISEF, did some internships, had a interview on tv-show.

I will post about them later, if I reminded. (hehe)

These days I really study hard for university application, because this semester is my last semester that actually evaluated by university.

So I could not post here frequntly.

About programming I still always adicted on programming too 🙂

I bought MS surface Go and Laptop! They are very awesome to code!

Oh I should make a post about my coding environment too. It should be helpful.

I am really impressed this codex is still alive, even the last post of others is last year.

I think might Mr.Miller moved homepage for educating to somewhere better, or changed some course whatever, I hope he is still living well 🙂 (I will email after got

I hope everyone in Cairo is doing well!

I still sometimes remind my life in Cairo, and it was really impressive memory for me.

Yeah, it is midnight here Korea now, and let me finish this post! See ya

I will go to ISEF in this Saturday.

Long time no see Codex, I am doing well at Korea for a while!

I was so busy, because of school, and competition stuffs.

And finally, in this Saturday, I will go to Pittsburgh to compete on ISEF.

This is the final version of my project poster for booth display.

I feel a little bit nervous about presentation because I didn’t speak in English for about 2 years 🙁

However, I hope I will be okay as like as how I lived in Egypt because I was really bad at English at that time.

I will do my best in Pittsburgh, good luck me!

Calibrate Tracking Result With Linear Regression

I applied linear regression to gaze tracking. The tracking result is getting really nice now. It has only about 1~2 cm tracking error. So we can use this for real application I think.

Actually, I tried to implement support vector regression, but it was way too hard for me… I will try it later.

I think I did my best to calibrating tracking result. Now to improve result more, I have to put more variable data to NN!

I Should Apply CapsNet For My Network.

Prof. Hinton who made CNN, release brand new neural network for compute image that called Capsule network.

It manage image’s data as Capsule that contain some high-level image feature like stroke.

This is seems nice solution for my network, because gaze track task is very spacial depend.

Actually I planned to implement this next year, but I saw other guys starting to use CapsNet already in Facebook.

I need to update alwayyys.

Gaze tracking is now acceptable quality!

You can see tracking test since 30 sec.

I applied updated neural network and I manually applied simple linear regression. Just like ptNew = (ptOld + biasVec) * scale.

I will implement some machine learning tech for calibration now. Just like support vector regression. :3

I think I will use NN for it too. NN is the most simple to use it if I have many data and nice machine. Simple multi-layered perceptron is easy.

What’s wrong with CNN regression?

In my koi17 project, I am using convolutional neural network regression model.

My model is working with standard VGG16 architecture.

But the performance of NN is not enough for my goal, and it also seems under fitted too.

So I decided stack more layer and change the architecture to ResNet. I stacked 4 of the residual block.

In a usual classification problem, ResNet must be working much better than VGG… But in my case, it wasn’t.

ResNet worked worse then VGG, and then I tested shallow DenseNet too, but it showed up higher bias and test-acc is overfitted too…

I think deeper CNN is not a good solution in regression problem because ConvNet ignores geometric placements.


I have no solution to solve this problem yet…


-Update 2017-11-16

I found what is the problem with my model. I made too narrow space for CNN. I passed 4×4 image for last of 5~6 layers and it made overfitting because filter size was 3×3, so 9 of 16 nodes were fully connected. So it increased more fully connected ratio in my network so it got overfitted.

I removed some pooling layer on top of the model so the last layer got the 8×8 image to compute. And it solved the problem.

Progress Report: Optimize Neural Network And Some Landmark Detectors.

I’m going to try to summarize my works on Neural Action for this few months.

Neural Network (Gaze Estimation)

Model Changes

I changed the model structure to use ROI image of the face too.

It made my NN result is strong to face rotations. I will post model structure images later too.

While change model structure, I did really A LOT of parameter optimizations too.

Model Optimizations

SELU Implemented

I added SELU (scaled exponential linear units). It amazing brand-new NN normalizing technology. I love it a lot. I mean ALOT.

When I use BN (batch normalizations) for NN training, my training error is ~8.5 degree, and then apply magical SELU, training error going to be ~6 degrees. 50% of benefit… 🙂

Weight Decay

I added weight decay to training loss. Weight decay is weight normalize technology. It makes a sum of L2Loss of weights gonna smaller (preventing weight explode).

Learning-rate Decay

I added learning rate decay. Learning rate decay is decay learning rate while training. L.r decay helps to settle NN to lower loss. I applied exponential learning rate decay every 12 epochs.

I checked out this helped my NN error decreases ~2 degrees.

Parameter Optimizations

Hmm, I did parameter optimizations too, but I am not sure what I actually did in this section. Parameters in NN is really chaos. We can’t manage them all by brute force with just 1 machine.

Ex) Loss functions, learning rate, conv channels, filter size, input size, output functions, hidden node counts, batch size, weight decay rate, learning rate decay rate, decay steps, optimizer algorithm, layer depths, feature sharing. And there are much more parameters if add more NN optimization technics.

SELU vs RELU(with BN) Compare

Mean error is an error between estimated gaze vector. You can change it to degrees like this.

edegree=tan-1 ( eerror )

  • RELU Error/Loss Graph. You can see larger overfitting gaps then SELU.

  • SELU Error/Loss Graph

Both graphs are at 103 epoch. I couldn’t get more than 103 epoch because while training my RELU NN, bugs occurred.

But SELU one is keep trained until 250 epoch.

This is the final graph of SELU training one.

Facial Landmark Detector Changes

OpenFace (by Tadas)

I wrote about C# wrapper of it on my previous post. It is lastest facial landmark detector.

Compare to previous backend “flandmark”, it’s really high accuracy. Actually, compare to OpenFace, flandmark’s performance and accuracy are almost useless 🙁

Compare between Flandmark

  • This is translate/test look vector graph.

  • And this is a graph of OpenFace

You can see Flandmark have a lot of detection error. So it should affect to train dataset generator too.

So by changing landmark library to OpenFace, I could improve landmark/dataset accuracy.