Pointer Teleport

[Mark]

The pointer motion and teleport control needs more settings and to work in ALL Windows applications. I appreciate that proper integration through an SDK is preferable but there needs to be some fall back option to an emulated virtual mouse in conjunction with a physical mouse if this is going to catch on.

What are the chances of the EyeX Experience expanding on some basic pointer control functionality for all Windows applications through mouse emulation?

[Mark]

I see now you have the Tobii Windows Control application, that is what you need for EyeX and for it to work within a DirectX application.

[JeffKang]

I don’t have the EyeX yet, but I’m very curious as to how well the Tobii Windows Control will function.

Unless the EyeX is extremely more accurate than I imagine, I still think that the community will have to come up with some software features to select the smallest interface elements, like individual characters.

This is a crappy mockup of something I’ve been envisioning:

Generate/project large-button, touch/eye-tracking UI from existing non-touch/non-eye-tracking UI

Tag nearby elements w/ color & ID – project to large element (screenshot and mock-up)

[JeffKang]

Here is a GIF of the LabelControl AutoHotkey script with its element detection: i.imgur.com/INB0Jt1.gif
donationcoder/com/Software/Skrommel/LabelControl/LabelControl.ahk

Vimium is an extension for Chrome
https://github.com/philc/vimium

[Anders]

Hi Mark,
the EyeX for Windows software provides the pointer motion and teleport features, that are intended to improve the mouse/touchpad user experience. Now it’s clear that you found some cases where these features don’t match your needs. Could you please point out what those cases are? (DirectX?)

[Anders]

Jeff, you’re quite right that small interface elements are difficult to select using eye tracking. We’re currently prototyping a solution based on a two-step workflow. It will be interesting to see how it works out for example on crowded web pages.

[Mark]

Anders thanks for the reply, let me explain in more detail what the use case is.

To start I think we need a suitable reference and for that I am going to choose a PC game that is free to download and use :
Stalker : Lost Alpha (http://www.moddb.com/mods/lost-alpha)

I picked this example because it uses DirectX9 or 10 rendering, its single player and non-competitive it has some unique gameplay drivers that requires a player to look around a lot in any given scene. The developers in question have access to the source code so there is potential for Tobii EyeX SDK implementation too. Its also a familiar title to many people…

So if I were a PC gamer I would expect two things from a Tobii EyeX:
1. Default gaze interaction in all games where I can teleport the cursor to my gaze position as an enhancement to my keyboard/mouse
2. More detailed interaction in Gaze-enabled titles much like the DeusEx demo and not unlike the TrackIR solution

Now if we go back to the Lost Alpha use case, what I would expect by default is to view a scene with my eyes and at the click of a button (or a physical wink of the eye) have the viewport in game centre on that position ‘as if my mouse had guided it there’. That could be some item in a room or a character in landscape vista.

What I would desire but not expect would be using my eyes to drive dialogs and options with AI characters in game as an enhancement. I would also desire a toggle on/off for continuous gaze tracking.

I suspect this means the EyeX or a virtual controller being recognized as a DirectX input device in conjunction with the mouse (as not all games will support many input devices).

[Robert [Tobii]]

Hi Mark,

Thank you for the clarification. We could not run Stalker: Lost Alpha due to a debug.assert() crash at startup, but I see your point.

The idea of the Pointer Teleport functions in EyeX for Windows is to work everywhere in Windows. If there are use cases where it does not work or of you have have feedback or feature requests, please locate the EyeX notification area icon and select “Send feedback”.

Note that the concepts in EyeX for Windows is based on the desktop use case and that the solution implemented therein is not enough to be compatible with all full-screen games. There is probably room for another more gaming-focused product.

The idea of creating a “virtual” input device combining eye tracking + mouse is very interesting, although I think it is hard to create something that works universally well in all games. The TrackIR solution could also be an option as you say. Tobii is constantly researching and innovating in this area and more products or prototypes related to gaming will be presented later on. Until then we appreciate any help from 3rd party developers wanting to contribute in making computer games even more immersive and effortless to interact with.

[Mark]

Another example (as you are struggling with LA, use a 64bit os btw) would be MechWarrior Online : https://mwomercs.com/

This is a free2play title based on CryEngine 3 (Dx9 or Dx11 rendering), principally a multiplayer game of large robotic tanks fighting it out but there is also ‘Testing Grounds’ mode which is a single person experience design for you to try out the game mechanics and different chassis on offer. So you can install this with no cost and play a demo mode by yourself without distraction from other players.

MWO has an interesting effect with the Tobii EyeX because in the initial menus configuring the game, the settings, your customisations and loadouts etc the gaze interaction seems to function. However when you launch into a battle or the single player testing grounds it all stops working. Again it would appear that when the game shifts into its DirectX mode the EyeX cannot use its Windows Control.

The point here is I think to give players enough control options to setup and configure EyeX to work with basic gaze features (like teleport) in all games, and then for 3rd parties to work on the special features and interactions. You don’t want to follow TrackIR’s example of having to use an SDK to get even basic functionality because it fails to get enough support in enough titles.

What happened to the DeusEX:HR demo with Gaze interaction?

[Robert [Tobii]]

Ok, thanks for the example. I have forwarded your suggestion to the Gaming dev team. The DeusEX demo was developed to work at exhibitions, but would need some quality improvements before it can be published externally.

[Mark]

So a year on now, are we any closer to having mouse emulation or teleport working in a DirectX application?

[Author]

Posts