Problem with receiving interaction events

[Veronika]

Hej,
I am working on a prototype solution which involves Tobii. The solution uses Tobii .NET SDK. I have faced a problem with receiving interaction events.
The documentation says: “The engine then sends events to the application that owns the interactor, with information about the user’s interaction.”

My question are:
1. How exactly the engine finds out the application that owns the interactor?
1.1. Is it based on WindowID of the interactor or it is retrieved somehow through the interactor reference?
2. Is it possible to redirect the events to another application, that is not owning the interactor?

Thank you in advance!

[Jenny [Tobii]]

Hi Veronika,

Yes, that description in the documentation is a simplified description of how it works.

To answer your questions I have to make a distinction between region-bound interactors and global interactors, because the engine treats these different kinds of interactors differently. Region-bound interactors are interactors that have an EyeX behavior that is limited to a certain region on the screen – for example an Activatable button. Global interactors are interactors with EyeX behaviors that are not tied to a specific region on the screen, but will rather receive a stream of events – for example a data stream of gaze point events of where on the screen the user is looking.

Based on your questions, I am assuming that what you are trying to achieve is regarding region-bound interactors, so I will answer them for the region-bound case. I am further assuming that you want to EyeX enable an application for which you don’t have access to the source code, so that the WinForms or WPF EyeX frameworks included in the .NET SDK are not feasible.

Answers:
1. The engine keeps track of which client has sent it a certain region-bound interactor in a snapshot by prefixing the interactor id with a unique client id. So, when the engine has discovered that the user has interacted with a certain interactor, it can direct the event (with the client prefix removed from the interactor id) to the client that sent the interactor to the engine. So, the “owning” client is really the client that sent the interactor in a snapshot.
1.1 While the answer above is correct for a single interactor and client, it does not give the whole picture of how it works. The window id is also important, and so are the bounds of the snapshot that includes the interactor. When the engine receives a snapshot with interactors with window ids, it will consider this snapshot the complete and latest description of everything that is within the snapshot bounds for the respective window. Let’s say we have two snapshots sent after each other to the engine, and that they have the same snapshot bounds, but the first snapshot contains a single interactor with id = ‘A’, and window id = 1, and the second snapshot contains a single interactor with id = ‘B’, and window id = 1. From the engine’s perspective, as soon as the second snapshot is received, the interactor ‘B’ is the only existing interactor in window 1 within the snapshot bounds (no matter of the bounds of the individual interactors). If interactors ‘A’ and ‘B’ in the previous example would instead have different window id:s, then after the second snapshot was received, they would both be considered to exist, in different windows. If the snapshot bounds of the two snapshots were non-overlapping, they would update the engine’s information about the different regions independenlty, without interfering with each other. In conclusion, it is therefore important to include all the currently existing interactors within the snapshot bounds of the respective windows in each and every snapshot sent to the engine.
2. Now, can a client application get events for an interactor that “belongs” to another window – say a button or region within another application’s window? Yes, it can, simply by sending a snapshot to the engine with the correct bounds and window id set for the interactor. But, please consider the following when EyeX enabling another application’s window: if that other application is itself a client application to the EyeX Engine, you would end up in a situation where two client applications are sending snapshots about the same window and regions of the screen simultaneously to the engine. Since the interactor ids will always be distinct due to the client prefix, the interactor information in the snapshots would constantly replace each other and the engine’s view of existing interactors, and it would be a race condition which client application would get a certain interaction event.

When it comes to the WinForms and WPF EyeX frameworks in the .NET SDK, they take care of the creation and sending of snapshots with region-bound interactors to the EyeX Engine under the hood. If you are using any of these frameworks and are not receiving interaction events, then there is a different kind of problem.

For further reading and understanding of the differences between region-bound and global interactors, please see the sections “Interactors” and “The Query-Snapshot Cycle” in the Developer’s Guide.

[Veronika]

Hello Jenny!
Thank you for the detailed answer. It was very helpful.

However, I did not suceed to track UI objects of another application.
Here is my situation: i have my own windows forms application where i initialize Tobii and everything related to it. It also has some UI, but I am not interested to track it. I am interested in UI of another application, which is a third-party windows application which source code i cannot access (but I can programmatically detect some UI elements of this application, their boundaries and so on).

What i did is that i took a windows form example from you examples set for .NET and slightly modified it for my needs. In two words, in my application I initialize Tobii and register an interactor that belongs to another application.

When I am debugging my code, i see that Tobii sends queries regarding interactors only if i am looking at the window of my application, even though i have no interactors registered in the boundaries of this window (only an interactor from the third-party application window). When I am looking outside my application window, Tobii does not even send the queries to my application(function HandleQuery which handles a query from the EyeX Engine is never called).

It seems that Tobii is in some way bounded to the initial application boundaries. Or am i doing something completely wrong ?

[Jenny [Tobii]]

Hi Veronika,

This is going to be another long reply… 🙂

Since you are basing your EyeX implementation on one of the WinForms samples, I guessing there are two main reasons why it doesn’t work as desired.

One reason is that your application probably is hooked up with a FormsEyeXHost that expects all interactors to be GUI components of the application itself. Another reason is something you have to add that I missed in my reply above: in order to receive queries from another application, a query handler has to be registered explicitly for that application’s process id.

Let’s start with the first reason. The idea with the Forms code in the EyeXFramework is to provide code that can take an object that inherits from Window.Forms.Control and create an EyeX interactor that correspond to the location, width, height and z ordering of that control, and also add the behaviors to the interactor that have been mapped to that control using a BehaviorMap.

In your case, as you explained, you already have code that can find and describe a GUI component or region of another application that you want to be an EyeX interactor with certain behaviors. So, what you actually need is not a FormsEyeXHost with its associated code, but rather your own implementation of an EyeX host that derives from the EyeXHost class found in the EyeXFramework namespace.

There you need to override the HandleQuery(…) and HandleEvent(…) methods. In the HandleQuery method you should create a snapshot and create and add interactors that are within the bounds of the query:

// using the method implemented in EyeXHost:
var snapshot = CreateSnapshot(query);

Then, for each region in the other application that you want to be an interactor:

var interactor = snapshot.CreateInteractor(interactorId, parentInteractorId, windowId); // interactor id has to be unique
var bounds = interactor.CreateBounds(InteractionBoundsType.Rectangular);
bounds.SetRectangularData(x, y, width, height);
interactor.Z = z; // only needed if interactor regions overlap

Then you add the behavior to the interactor. This is how you would add a “Foo” behavior to an interactor:

var behaviorParameters = new FooParams { SomeFooSpecificParam = someValue };
interactor.SetFooBehavior(ref behaviorParameters);

Finally, you should commit the snapshot:

// using the method implemented in EyeXHost:
CommitSnapshot(snapshot);

In the HandleEvent method you have to find out which interactor it concerns:
var interactorId = event_.InteractorId;

Then you loop over the behaviors and try and extract the event type and event parameters and invoke an appropriate response in the application. How to extract depends on the event type. Please, see ActivatableBehavior.HandleEvent(…) and GazeAwareBehavior.HandleEvent(…) how to do that.

Remember that all callbacks to HandleQuery and HandleEvent are called on a worker thread, so take care to dispatch to a UI thread if needed.

Now, for the other part of the problem: that you don’t receive queries for the other application. The EyeX Engine creates queries around the current gaze point of the user. It finds out which windows fall within the query bounds and looks up the process id of the process each window belongs to. If there is a registered query handler for that process, it will call the specified callback function for that handler.

In your case, you only get queries for your own application since you have hooked it up with the EyeXHost base class that registers a query handler for the current process, in other words your application’s process.

To make your application receive queries also for the other application, you have to register a query handler for the other application’s process: expose the private InteractionContext _context in EyeXHost to your derived class (by for example making it protected), and override the EyeXHost.Init() method in your derived class, where you add the registration:

public override void Init()
{
    base.Init();
    // given a string processId representing the process id of the other application:
    _context.RegisterQueryHandler(processId, HandleQuery);
}

When you get the queries from the engine, make sure you answer correctly for the window ids in the query. Remember that you could gaze-enable both your own application and the other application at the same time, by using the FormsEyeXHost and the Forms EyeXFramework code for your own application, and the EyeXHost derivative described above to enable the other application.

[Author]

Posts