Unreal Engine Issues

[Eric Chu]

I have been working with the Tobii Plug in for the Unreal Engine for months now, and the largest problem I have with it is that it is very inconsistent when reacting with EyeX Actors, meaning that there are times that the gaze point would be on an EyeX Actor and it wouldn’t react unless I move around and get close to the Actor.

The inconsistency is making development really hard, due to the fact that game play feels really inconsistent and there is nothing that we can really do to remedy it.

If anyone has any experience with this or has run into a similar problem and help or empathy would be appreciated 😀

[Temaran]

Hello there!

I can’t be sure what is causing the exact problems you are experiencing, but it is most likely related to how the interactors work right now. Since they don’t by default work well with occlusion, partial or full, I would recommend to roll your own solution for interactors until we have a working solution (which is still in an underterminant amount of time in the future I’m afraid).

My best naïve solution so far has been to first test all actors in the scene against a convex shape (frustum works best of course) I align in the direction of the gaze point and then to iterate over the actors I found and test a raycast against each one to test for occulsion. This of course still doesn’t work well with partial occlusion, but at least it is fast enough to make it workable until I can get the budget to do the good solution.

If you are working in blueprint only, I think the best way would be to simply spherecast instead of doing the convex shape test, but this will have really bad results to objects far away.

Both of these naïve methods are actually in the SDK, although not in their latest versions (in EyeXMathHelpers.cpp I believe).
If you tell me more about the game scenario I might be able to give you more directed tips for that, but as a general rule I think this is as good as it can get until the good solution.
If you have any questions on how to get this into the game, please tell me and I could probably put together and send you some sample code that makes it a bit clearer.

Best regards,
Temaran

[Eric Chu]

Hello

We are trying to use the EyeXActors in a Horror Setting that has an enemy that paths and follows the player around and has an EyeX Actor attached to it. It definitely works, but the results are inconsistent where I would be looking at the visual representation of the Gaze Point and the EyeXActor wouldn’t return the Boolean of the function “Has Gaze”.

I also set up a testing situation where the enemy would stand still and you would look at him from a distance activating the aggro. It wouldn’t activate consistently, only when I got really close to him or found a random sweet spot.

I would offer you a copy of the project for a better view on what the problem is, but the project recently went through some large changes and would be confusing to navigate through.

And I am currently using Blueprint Only, I had some issues with C++ in the Unreal Engine in general so I stopped using it part way through development but I am not opposed to making a few classes.

Thanks,
Eric

[Temaran]

Hello!

In that case it is most likely the standard implementation like I suspected. The standard imlementation uses a series of spherecasts to approximate a frustum, but once again, it will not work well with occlusion, and the approximation in conjunction with the accuracy offset will “miss” certain spots.

Please behold my amazing drawing skills as I try to illustrate the problem:

Here is the code for the frustum also:

Frustum.h:

#pragma once

struct FFrustum : public FConvexVolume
{
public:
	/**
	* Calculates a frustum from a part of the screen that has the same side plane slopes as the main view, and allows the user to specify a custom far plane distance
	*
	* @param Corners - The corners of a rectangle on the screen that defines the new frustum's near plane
	* @param DistanceToFarPlane - If this is 0 the same far plane as the view is used. Otherwise, it is the distance to the far plane of the new frustum.
	* @param View - The view to calculate the frustum from. This will decide frustum side plane slopes, near plane position etc.
	* @param OutFrustum - The new frustum.
	*/
	static FFrustum CalculateFrustumFromSceneView(FVector2D Corners[], float DistanceToFarPlane, const FSceneView* const View);

	int32 LeftPlaneIndex;
	int32 RightPlaneIndex;
	int32 TopPlaneIndex;
	int32 BottomPlaneIndex;
	int32 BackPlaneIndex;

public:
// 	bool IntersectsFrustum(FHitResult& HitOut, AActor* InActor);
// 	bool IntersectsVertices(FHitResult& HitOut, AActor* InActor, UPrimitiveComponent* InComponent);
	void DrawDebugVisualization(UWorld* World);

private:
//	void InitHitResult(FHitResult& HitOut, AActor* Actor, UPrimitiveComponent* Component, const FVector& Location, const FVector& Normal, int32 FaceIndex, const FName& BoneName = TEXT(""));
	static bool GetIntersection(const FPlane& Plane1, const FPlane& Plane2, const FPlane& Plane3, FVector& OutIntersectionPoint);
};

frustum.cpp

#include "Frustum.h"

FFrustum FFrustum::CalculateFrustumFromSceneView(FVector2D Corners[], float DistanceToFarPlane, const FSceneView* const View)
{
	FFrustum OutFrustum;

	// Deproject the four corners of the selection box
	FVector WorldPoint[4];
	FVector WorldDir;
	for (int i = 0; i < 4; ++i)
	{
		View->DeprojectFVector2D(Corners[i], WorldPoint[i], WorldDir);
	}

	// Use the camera position and the selection box to create and add the bounding planes
	FVector CamPoint = View->ViewLocation;
	for (int i = 0; i < 4; i++)
	{
		OutFrustum.Planes.Add(FPlane(WorldPoint[i], WorldPoint[(i + 1) % 4], CamPoint));
	}

	//TODO: For some reason the far plane of the frustum does not seem to have any effect on the selection... Maybe we need a matching near plane for this to work? It seems the near plane is not needed, so it is not used.
	FPlane FarPlane;
	if (View->ViewProjectionMatrix.GetFrustumFarPlane(FarPlane))
	{
		if (0 != DistanceToFarPlane)
		{
			FVector ViewDirection = View->GetViewDirection();
			FVector FarPlaneBasePosition = View->ViewLocation + ViewDirection * DistanceToFarPlane;
			FVector FarPlaneNormal = FarPlane;

			FarPlane = FPlane(FarPlaneBasePosition, FarPlaneNormal);
		}

		OutFrustum.Planes.Add(FarPlane.Flip());
	}

	OutFrustum.Init();

	OutFrustum.LeftPlaneIndex = 3;
	OutFrustum.RightPlaneIndex = 1;
	OutFrustum.TopPlaneIndex = 0;
	OutFrustum.BottomPlaneIndex = 2;
	OutFrustum.BackPlaneIndex = 4;

	return OutFrustum;
}

void FFrustum::DrawDebugVisualization(UWorld* World)
{
	//Test all plane permutations
	TArray<FVector> FrustumVertices;
	for (int32 FirstIndex = 0; FirstIndex < Planes.Num(); FirstIndex++)
	{
		for (int32 SecondIndex = FirstIndex + 1; SecondIndex < Planes.Num(); SecondIndex++)
		{
			for (int32 ThirdIndex = SecondIndex + 1; ThirdIndex < Planes.Num(); ThirdIndex++)
			{
				FVector CurrentIntersectionPoint;
				if (GetIntersection(Planes[FirstIndex], Planes[SecondIndex], Planes[ThirdIndex], CurrentIntersectionPoint) && 
					!FrustumVertices.ContainsByPredicate([=](FVector VectorToTest) { return VectorToTest.Equals(CurrentIntersectionPoint, 0.0001f); }))
				{
					FrustumVertices.Add(CurrentIntersectionPoint);
				}
			}
		}
	}

	//Generate lines between all permutations of vertices
	for (int32 FirstIndex = 0; FirstIndex < FrustumVertices.Num(); FirstIndex++)
	{
		for (int32 SecondIndex = FirstIndex + 1; SecondIndex < FrustumVertices.Num(); SecondIndex++)
		{
			DrawDebugLine(World, FrustumVertices[FirstIndex], FrustumVertices[SecondIndex], FColor::Red);
		}
	}
}

bool FFrustum::GetIntersection(const FPlane& Plane1, const FPlane& Plane2, const FPlane& Plane3, FVector& OutIntersectionPoint)
{
	//Find the determinant of the normal matrix
	float det = Plane1.X * (Plane2.Y * Plane3.Z - Plane2.Z * Plane3.Y)
		- Plane1.Y * (Plane2.X * Plane3.Z - Plane2.Z * Plane3.X)
		+ Plane1.Z * (Plane2.X * Plane3.Y - Plane2.Y * Plane3.X);

	// If the determinant is 0, that means parallel Planes, no intersection
	if (FMath::IsNearlyZero(det, 0.0001f))
		return false;

	OutIntersectionPoint = (FVector::CrossProduct(Plane2, Plane3) * Plane1.W +
		FVector::CrossProduct(Plane3, Plane1) * Plane2.W +
		FVector::CrossProduct(Plane1, Plane2) * Plane3.W) / det;

	return true;
}

Usage (update the gaze frustum):

	FVector2D BoundSize = EyetrackingBoundSize * 5.0f;
	float MaximumSelectionDistance = 1000;

	UWorld* World = GetWorld();
	const FEyeXGazePoint GazePointData = IEyeXPlugin::Get().GetGazePoint(EEyeXGazePointDataMode::Unfiltered);

	ULocalPlayer* LocalPlayer = Cast<ULocalPlayer>(Player);

	if (!GazePointData.bHasValue || NULL == LocalPlayer || NULL == World)
	{
		return;
	}

	FVector2D GazePoint = GazePointData.Value;

	FVector2D Corners[4];
	const FVector2D HalfSize = BoundSize / 2;
	Corners[0] = FVector2D(GazePoint.X - HalfSize.X, GazePoint.Y - HalfSize.Y); // Upper Left Corner	
	Corners[1] = FVector2D(GazePoint.X + HalfSize.X, GazePoint.Y - HalfSize.Y); // Upper Right Corner
	Corners[2] = FVector2D(GazePoint.X + HalfSize.X, GazePoint.Y + HalfSize.Y); // Lower Right Corner
	Corners[3] = FVector2D(GazePoint.X - HalfSize.X, GazePoint.Y + HalfSize.Y); // Lower Left Corner

	FSceneViewFamily ViewFamily(FSceneViewFamily::ConstructionValues(
		LocalPlayer->ViewportClient->Viewport,
		World->Scene,
		LocalPlayer->ViewportClient->EngineShowFlags)
		.SetRealtimeUpdate(true));

	FVector ViewLocation;
	FRotator ViewRotation;
	FSceneView* View = LocalPlayer->CalcSceneView(&ViewFamily, ViewLocation, ViewRotation, LocalPlayer->ViewportClient->Viewport);

	Frustum = FFrustum::CalculateFrustumFromSceneView(Corners, MaximumSelectionDistance, View);

Then you just test your point / primitive against the frustum to determine which object has focus and which do not. After an initial test I recommend also testing with a raycast afterwards to determine occlusion. It will still have similar problems to the base implementations when it comes to partial occlusion, but at least this method will have less “strange misses”.

I hope this helps! And please tell me if you have any problems.

Best regards,
Temaran

[Author]

Posts