The eye of the Python, an eye tracking system. From zero to... what eye learned.

I am Samuel (Samu for friends). With a curious mind I studied computer science, then focused in machine learning and IoT as a professional career. I haven't given up on my plan to take over the world; that's why my coworkers know a crazy idea is coming out, when I can't hide any longer a mischievous smile. I like to meet people and understand other points of view, and in return I like to show what I can do and teach what I have mastered. But, what drives me crazy is rollerskating with disco music. </div>

Abstract

Tags: data-science use-case python ai deep learning

Is it possible to predict the point in the screen where a person is looking at? Easy to say but hard to do. An eye tracking system is the perfect project to learn the difficulties of applied machine learning. From gathering training data to building the final software with an acceptable performance.

Description

In this talk I will show how to build an eye tracking system from scratch.

Once the approach (machine learning) and the tools (Python ecosystem) are set, the important tasks are:

• Making users addicted to a game built with Pygame in order to generate data.
• Unleashing the power of deep learning over GPU with Tensorflow to train an Artificial Neural Network.
• Exploiting the training model in real time so as to control a computer with the eyes with PyAutoGUI.

The path is full of pitfalls and every clear single step to the goal turns out to be a mountain of small but very important subtasks. Every iteration is a continuous struggle just to gain a bit of accuracy.

Despite these efforts, at the very end we will see the difference between the theoretical and the real world. Our engineering skills will determine the usability of the new interface that allows us to move the mouse with our eyes; and we will learn that things don't need to be perfect, because humans are ...