Gaming and desktop mouse control with the Rex

[gigertron]

desktop control: http://youtu.be/2IjTZcbXYQY
… this is what I’m using at work and at home right now.

Spore: http://youtu.be/elmzclePbu8
Peggle: 😀 http://youtu.be/ErgHfJABRGk

The mouse driving code is all based on the Rex devkit MinimalGazeDataStream sample. Simple stuff.

… other videos in the series don’t have the gamepad hybrid support and are much less satisfying.

[Robert [Tobii]]

Nice video series about how eye tracking can be used to play games using mouse emulation. We actually showed a part of your Spore video on our sprint demo last week. Good that you show both in which games it works well, but also the cases when it does not work at all.

While mouse emulation techniques works quite well for some games, our long-term goal is of course that the game developers add native support for eye tracking input, or allow API:s or plugins for this. I found a blog post by another eye tracking company that illustrates the differences between mouse emulation an native eye tracking support:
http://gaming.mirametrix.com/review/active-application-unity-s2-eye-tracker/

It seems that you @gigertron has realized that native eye tracking support is the way to go. Now we just need to get the message out to the game developers out there.

[gigertron]

Hey cool! Thanks! It’s nice to know that someone’s looking at the results of my experiments (and at least listening to my opinions).

I just reviewed a game that works damned near-perfectly: Hearthstone. This one’s playable with just the tracker and a mouse button.

[Cheng Guo]

Hi gigertron:

Thanks for the videos. It is nice to see how eye trackers with existing games. I have one question for you. How did you make the cursor appear so stable? The raw data I get from the eye tracker (with lightly filtered option turned on) jitters.

Thanks!

[gigertron]

I’m using a basic fast-response damped control system. It’s simpler than it sounds. Basically I take the difference between the cursor’s current location and the eye tracker input location, (we’ll call that ‘error’) and multiply that error by some damping factor like 0.1. Then we add the error to the current cursor position.

The eye tracking data is smooth (ish…) over time, so damping it in this way makes for a stable point.

I add an extra layer of stability where I don’t move my cursor at all unless the error is above a certain amount, like 10 pixels.

[Dennis Lo]

Hi Gigertron,

Thank you for sharing your build on github, just wondering if you have managed to update your cursor control program for the newer EyeX engine? If so, will you be able to upload that on your github as I am experiencing some difficulty programming something similar but is running into a bit difficulty with filtering and etc – so just trying to see where I may have gone wrong in the code.

Thanks in advance 🙂

[Robert [Tobii]]

Great that you are sharing and collaborating. I’ll add some helping instructions regarding the API changes.

Here is the code, currently written for EyeX Engine 0.8.3:
https://github.com/gigertron/EyeTracker

To update the MinimalGazeDataStream.cpp to latest EyeX Engine API, here are the steps:
1) Change type of hContext from TX_HANDLE to TX_CONTEXTHANDLE
2) On txInitializeSystem, add a fourth NULL parameter
3) On txReleaseContext, remove the last two parameters
4) Rename txCommitSnapshot to txCommitSnapshotAsync

5) Replace OnSnapshotCommitted with this snippet:

void TX_CALLCONVENTION OnSnapshotCommitted(TX_CONSTHANDLE hAsyncData, TX_USERPARAM param)
{
	// check the result code using an assertion.
	// this will catch validation errors and runtime errors in debug builds. in release builds it won't do anything.

	TX_RESULT result = TX_RESULT_UNKNOWN;
	txGetAsyncDataResultCode(hAsyncData, &result);
	assert(result == TX_RESULT_OK || result == TX_RESULT_CANCELLED);
}

6) Change the first part of HandleEvent (before the if statements) to:

void TX_CALLCONVENTION HandleEvent(TX_CONSTHANDLE hObject, TX_USERPARAM userParam)
{
	TX_HANDLE hEvent = TX_EMPTY_HANDLE;
	TX_HANDLE hBehavior = TX_EMPTY_HANDLE;

    txGetAsyncDataContent(hObject, &hEvent);

7) Change the last line of HandleEvent to

txReleaseObject(&hEvent);

[torque]

Hello and thank you for the code & nice video.

In addition to Robert’s fix above, I needed the change below to build.

In HandleEvent(), between 6) and 7) change hObject to hEvent.
From

	if (txGetEventBehavior(hObject, &hBehavior, TX_INTERACTIONBEHAVIORTYPE_GAZEPOINTDATA) == TX_RESULT_OK) {
		OnGazeDataEvent(hBehavior);
		txReleaseObject(&hBehavior);
	}

	if (txGetEventBehavior(hObject, &hBehavior, TX_INTERACTIONBEHAVIORTYPE_FIXATIONDATA) == TX_RESULT_OK) {
		OnFixationDataEvent(hBehavior);
		txReleaseObject(&hBehavior);
	}

to

	if (txGetEventBehavior(hEvent, &hBehavior, TX_INTERACTIONBEHAVIORTYPE_GAZEPOINTDATA) == TX_RESULT_OK) {
		OnGazeDataEvent(hBehavior);
		txReleaseObject(&hBehavior);
	}
	
	if (txGetEventBehavior(hEvent, &hBehavior, TX_INTERACTIONBEHAVIORTYPE_FIXATIONDATA) == TX_RESULT_OK) {
		OnFixationDataEvent(hBehavior);
		txReleaseObject(&hBehavior);
	}

[Remo]

I’ve been using EyeXMouse which seems to be based on gigertron’s work and have had a degree of success I’d like to share.

Hearthstone using EyeXMouse:
http://youtu.be/FYN-7EPbcvs?list=UUumqIFsH1ZTQw3wpHfQbaOg
Works great! Slow pace and large hotspots make for an enjoyable experience.

Osu! using EyeXmouse:
http://youtu.be/VH4yduiveEY?list=UUumqIFsH1ZTQw3wpHfQbaOg

This is a bit more difficult because of the faster pace and smaller targets. This demonstrates the limits in averaging delay and calibration.

At the end of the video I continue to use EyeXMouse control to close the process. You can see how difficult it is to hit such small targets. I am confident this can be improved with tuning and calibration.

[Remo]

After a little more tuning and with a perspective camera…

[Videojocs Adaptats]

Hello.
I have some problems.
When I run your software y get a requester that says thats missing the MSVCP120D.dll
I’ve installed the MS C runtime redistributable but with no luck.
Can you helpme?
Thanks

[Sage]

I have the same problem as Videojocs. Any assistance would be greatly appreciated!

[Robert [Tobii]]

The problem is that the EyeXMouse.exe file available on GitHub seems to be build in debug mode, in that case you need Visual Studio 2013 installed to be able to run it. Otherwise, check out the source code and rebuild in Release mode.

[Author]

Posts