Update, I let my colleague try it. For him it works every single time.
Just to be sure any of the corrections (lenses, glasses, etc) would be out of the question I tried those and for each the calibration failed.
Edit: I think I notice my eyes getting distracted all the time. Especially when the white dot is covered by the yellow and uncovered, like they are trying to find out where the yellow is going to. Now I know I am far from an expert on these things, but nevertheless;
Would it be a good option to be able to choose the moving dot from a set of dots? (if noise-data is the actual cause of the calibration failing for me)