Full precise mouse cursor control without a mouse.

[Aleksey]

Perhaps the offer was already on the forum, but my knowledge of English is not enough to check it. I propose to implement full control of the mouse cursor with the help of eye tracking, head tracking and macros.
In fact, I have already made a similar system using free software. To do this, I needed
Soft
1. eViacam (head tracking)
2. key manager (setup macros)
3. Tobii Experience
Hardware
1. webcam, 30 fps.
2. your device.
It works as follows.
1. Warp the cursor by the button assigned in your application (right alt)

2 In the key manager program, we set 2 scenarios for the Caps Lock key.
2.1 If the position was not accurate. Your device does not have enough accuracy for 100 percent hits for example, in the text I need to move the cursor to a certain position… according to my feelings, the accuracy of positioning the mouse cursor when using only eye tracking is at best 1/4 and only in the middle of the screen, and not on its edges, in games such accuracy is usually enough, but I do not write about games, but about text editing (programming, journalism, writing, etc.) and then such accuracy becomes insufficient:
-I hold down the Caps Lock key for 1 second and the key manager program through the macro enables head tracking in the eViacam application for 1.5 seconds.
– With the help of the head tracking, during this time, I bring the cursor to the desired location.
– And after 1,5 second, the same macro emulates a left mouse click and disables tracking eViacam.
2.2 If the position was accurate:
If the cursor is in the right place, then just use the second script for the capslock key. Short press capslock = left mouse click, two short presses =2 mouse clicks, and so on.

This implementation is quite a working one, but it has its own significant drawbacks.
1. I need an additional webcam.
2. I need to use additional software, the terms of use of which may change significantly or it will no longer be supported on the new OS.
3. You have to configure several different applications and few people will do this, and then if something stops working, then sit and figure it out.
4. The most important thing. To use such a scenario, I have to use at least 2 keyboard keys. And the button assigned to the mouse warp becomes unavailable in the key manager. That is, you can assign a command to it, but it will not be executed. There is also a disadvantage that 1.5 seconds may not be enough to navigate the cursor and you will get a mouse click in an unnecessary place for you, or this time will be too much and you will have to sit still until the macro works.

So I suggest the following implementation in your application using the example of the same capslock button.
1. single click = single click of the left mouse button.
2. if you hold down the button, then the cursor makes a warp to the place you are looking at and while you hold the button down, you can place the cursor in the desired position with your head, then release the button and the system disables head tracking and makes a left click.
Thus, if you work with a large volume of text, you do not need to take your hands off the keyboard to move the print carriage in the text editor to the desired position without errors.

In fact, in the current implementation of your software, “warp cursor by button” without precise cursor positioning and left mouse click is a useless thing. In any case, you will have to take your hand off the keyboard and take the mouse or reach for the touchpad. In my already finished and described above implementation, I clearly showed that you clearly did not realize even half of the potential of your device, even taking the shortcomings into my implementation.
Also, my implementation allows you to solve dead zones at the edges of the display and especially in the corners, and head tracking is not tied to 1 monitor, and with the help of my implementation, I can also freely enable cursor control using the head and on the second monitor.
Thanks for attention.

[Hongbin Zou]

Try Precision Gaze Mouse.
https://precisiongazemouse.org/

[Aleksey]

Perhaps my English is so bad that you do not understand what I wrote. This program does not allow me to do this logic.
if (CapsLock – pressed 500 ms){
int count = 0
while (CapsLock – pressed){
if (count == 0){
press&release F11
count=1
}
}
press&release leftButtonMouse
press&release F11
}
else if (CapsLock – press&released){press&release leftButtonMouse}

i use logic used in syntax Java
And your program threw non-interceptable exceptions 3 times in a row, with a complete hang of all control, solved by restarting the OS. So no, that alone is enough not to use it

[Hongbin Zou]

I don’t think there are any problems with your English.
The “Precision Gaze Mouse” program is not mine. I mentioned it just because it claims to do “precise mouse cursor control” in a similar way with what you described. If you are not satisfied, or it is error prone, then ignore it and stick with your own way.
I believe tobii knows everything about the potential of their devices. The reason they don’t add those features is not on the tech side.

[Aleksey]

And in General… a few hours of dancing and learning the script language called AutoHotkey… they let me write this script.

CapsLock::
Send {F11 down}{F11 up}
KeyWait CapsLock
Send {LButton down}{LButton up}
Send {F11 down}{F11 up}
return

CapsLock:: setup capslock hotkey and push it key
Send {F11}{…} send command to eViacam to enable tracking via webcam
KeyWait CapsLock – wait release CapsLock. while you hold CapsLock headtracking work and script wait
Send {LButton}{…} – send emulated click left button mouse, after release CapsLock
Send {F11}{…} – send command to eViacam to disable tracking
return – If there is no caller to which to return, Return will do an Exit instead.

Create 1 script file, copy, paste, save, close, run, and enjoy. The script works until the PC restarts, there seems to be an opportunity to do autorun, but I’m too lazy to understand, and I use sleep mode, so this problem does not worry me. If you fast make double click CapsLock and don’t move your head (deadzone setup in eViacam), then it is like double click mouse.

The only thing that gets in the way, the warp button and the enable headtracking button are still different buttons, since Toby Experience intercepts the control of the warp button and does not allow it to be used together. And i not tested it in game, when control may be intercept

[Grant [Tobii]]

Hi @fallgeratoor, thanks you so much for sharing your project and describing it for us, if you have any links that might be useful for other users to try it out that would be great.

I was not sure if you were looking for assistance, however we can say that mouse emulation and computer control overall is handled by our sister company Tobii Dynavox which specialised in this area, which is why at Tobii Tech we have not prioritised these features but instead have placed our focus on gaming.

Should you have any specific questions for us, please do let us know. Best Wishes.

[Aleksey]

Rather than questions, they are error messages and suggestions.
1. The warp key stops working if you change the input language (I checked it on the Russian layout, the right alt button).
2. I also noticed the dead zones at the edges of the screen, and at this point I thought, why not make the screen physical 1920*1080, and the actual/virtual 2000*1200 (random numbers only indicate a larger size), when you bring your eyes to the edge of the display, the image on it shifts and a hidden area opens (in fact, the same thing that you do in elite dangerous, when the player looks at the edge of the screen and his character sees the side control panels). And in this hidden area, you can place the same browser bookmarks, music management, program list, start panel, and so on. Moreover, everything that was in the dead zone will be shifted to the center of the screen and will be in the right place, you only need to correctly configure the timings and trigger zones.

Tobii Dynavox costs approximately $ 3,000, and eye tracking costs $ 250… maybe in normal countries, 3 thousand dollars is not very much, but alas, Russia is not one of the normal countries and the saddest thing is that it does not strive to become one. You will never see such a number of disabled people on the streets, because they simply can not leave the house, and not because we do not have them. Therefore, the question is how a disabled person can earn 3 thousand dollars in the country, when the environment is not calculated for him, despite the fact that the average salary in the country is about 250 dollars after taxes and without taking into account mandatory payments (electricity, Internet, heating, etc.).
I think the reason is somewhere in this area

[Aleksey]

On the first fault. I figured out what was going on, some “overmind” (sarcasm) thought that in the Russian layout, when you click on the right alt (I had a warp on this key, because I don’t use it anywhere else), the right alt is not pressed, but 2 keys are pressed: LCtrl+RAlt. In English layout only RAlt pressed… my face twisted when i understand it. Sorry, I need to calm down, because I don’t have any censored words. Palm-face

[Grant [Tobii]]

Hi Alexsey, yes I understand your frustration with scenario on pricing naturally however there are a few user built solutions for mouse control, etc that should hopefully close this gap for you. Unfortunately, from an official Tobii perspective, we must allow dynavox to specialise in this area to avoid competing directly within this sector.
Hopefully, you can find a solution to your needs at any rate.

[Aleksey]

Okay, since this has nothing to do with you, let’s focus on the games.

Add the ability to throw the cursor warp in 3d shooters. For example, this is done in Dying Light, where when shooting through the sight, it automatically points to the place where the player is looking. In fact, you do not need such difficulties as are made in the same Division, where the interface is highlighted, hiding places, and so on. For example, in the same Battlefield, eye-pointing will be a sufficient degree of integration for this type of game when you click the right mouse button. The main thing is that this is not perceived as a dishonest game, because playing for the same sniper at close range, quickscopes will be much easier to perform, on the other hand, ping in 4 ms and 44 ms is also a real and legal cheat. You have this feature disabled in game, moreover, it is disabled even when the mouse cursor disappears in full-screen mode when watching a video, which is annoying. And it is bug IMHO.

You have 2 functions in the settings now – warp by mouse movement and warp by pressing a button in the OS. I suggest adding 3 variant – warp mouse settings in the game when you click on the right mouse button, as in the dying light.
By doing so, you will remove the restriction of integration into games that your device supports, and limited integration is a strong stopping factor when buying.

[Grant [Tobii]]

Thanks for your feedback Aleksey, yes we are aware that mouse warp is disabled for full screen application which is designed to avoid interference with a games own eye tracking configuration, however, I will pass along your comments to the development team for consideration.

[Shawn Fumo]

Aleksey, my first thought of using eye/head tracking for mouse movement is very similar to your approach, so I’m glad to know that it works in practice. Basically as you said, using a warp based on eye movement and then head tracking for more precise control. Right now I’m using the mouse keys support built into the firmware of my keyboard (a Moonlander) combined with my own AutoHotkey script that lets you warp the cursor anywhere within a 9×9 grid on the screen using a combined key chord (basically I hold a key with my thumb which enables a layer and sets the keys I want AHK to pick up to things like ctrl-shift-alt-win-1). That works ok for the most part, but it does seem like having eye/head tracking would be even better.

Maybe it’d be even better if head tracking was always enabled, with it just adjusting the pointer position by small amounts. So you can have your head in a neutral position and press a key to activates the warp, and move the head to fine-tune. Then use regular mousekeys keys to left or right click or scroll wheel or whatever.

I still need to look at what the other free alternatives by users do, but I feel like it shouldn’t be that hard to do what I just described using their API. It might even be possible to call their DLLs using AHK, since it can call DLLs directly, though that probably isn’t ideal compared to something like C#.

To Tobii, I do think it is a little sucky that you wouldn’t implement something like this due to worrying about competing with a sister product. I have to think the other products must have special capabilities for them to be $3k. If your product and a simple functionality to move a mouse cursor based on a combo of eye and head tracking is actually competitive, that doesn’t speak well of the expensive product. It doesn’t need to be about disabilities and medical applications. Personally, I just like the ability to move my pointer without taking my hands off of the keyboard. I’m sure most of the people using the product would be for gaming no matter what. That doesn’t mean it wouldn’t be a nice addition to use it for normal productivity tasks without needing unofficial software solutions.

[Aleksey]

I use a bundle of eViacam – AutoHotkey + Tobii Eye Tracker

That device is for 3k dollars, it is designed for completely paralyzed people and maybe 3 thousand is justified money for it, because the software there is a little more advanced, at least judging by the videos, although the accuracy of eye positioning leaves much to be desired. The question here is that not everyone needs such a highly specialized device… And in my opinion, such a policy of protectionism violates one of the important principles of sales -if you can sell goods with a small margin, but with high turnover. It’s like a store selling 1 apple for $100 just once a month, hoping that next month they will come to you again for 1 apple for $100, instead of selling 100 apples for $1.5 and significantly expand their market, because the more people try apples, the more people will come for apples. You can expand the basic product not only for the gaming field of application, get a stable turnover from which you can subsidize your other company that produces devices for 3k and even sell them at a loss for say 1 thousand, but solve an important problem for people with disabilities… I would do something like this if I had to worry about another subsidiary

[Author]

Posts