On an HTC Vive, in the unity_pupil_plugin_vr, in the "3D Calibration Demo" scene, after a successful calibration the MarketWith3DCalibration.cs starts throwing NullPtrException on EyeData.get_Raw3D ().

Does not affect the 2D Calibration.

Tested with

• Unity 2017.3.0 and 2017.2.1
• the 1866fae commit and the v0.3.0.1 release commit
• the 1.1.2 and 1.2.7 Pupil Service (the 1.2.7 does not actually finish the calibration at all so that's not affected).

Editor log attached:
Editor.log
.

I've been trying to build a hololens project using the the il2cpp backend instead of .net. The il2cpp backend gives access to additional namespaces that are not included in the .net backend. The il2cpp backend converts the c# project to c++ and will become the preferred backend in the next couple years.

Everything builds fine but when I deploy the app I see a crippling memory leak (~20mb increase per second) that crashes the app after about 30 seconds. After a lot of testing I've been able to trace the leak to happening when gaze events are subscribed to. Specifically the leak occurs in the UDPCommunication.cs script in the Socket_MessageReceived function. For reasons unknown and suggested by this discussion thread the line:

Stream streamIn = args.GetDataStream().AsStreamForRead();

seems to cause the problem. I believe the memory allocated in this line is not properly garbage collected leading to the leak.

Rewriting the offending section to not use Stream (see code below) appears to correct the memory leak for IL2CPP-based builds.

private void Socket_MessageReceived(Windows.Networking.Sockets.DatagramSocket sender, Windows.Networking.Sockets.DatagramSocketMessageReceivedEventArgs args)
{
    using (var reader = args.GetDataReader())
    {
        var buf = new byte[reader.UnconsumedBufferLength];
        reader.ReadBytes(buf);
        if (ExecuteOnMainThread.Count == 0)
        {
            ExecuteOnMainThread.Enqueue(() => {InterpreteUDPData(buf);});
        }
    }
}

The solution would be to check to make sure both eye windows are open in service and, if not, issue the appropriate error message.

Hello,
I opened the PupilDataDemo scene in Unity and I tried to display the "diameter" values. What I get is a float between around 20 and 140.
While it is written that the pupil is operating in mm.
Is it a matter of measurement unit or the values do not represent the actual pupil diameter?

In the pupil_capture application, I see the correct values (from around 0 to 8mm).
Thank you!

In reference of this issue #53. I'm using HMD-eyes 0.63 and PupilService 1.10.20 and It's still unable to detect blinks. I've used debug to print all received messages and I've got only these data: gaze.2d.0 and gaze.2d.1. Even BlinkDemo Default Scene can't detect any blinks.

In the Gaze Demo, I do get the "connected" message, but the eye images are not visible, and when I try and calibrate, I get the following message:

FormatterNotRegisteredException: System.Collections.Generic.Dictionary2[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.Object, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]] is not registered in this resolver. resolver:StandardResolver MessagePack.FormatterResolverExtensions.GetFormatterWithVerify[T] (MessagePack.IFormatterResolver resolver) (at Assets/Plugins/Pupil/3rd-party/MessagePack/IFormatterResolver.cs:35) MessagePack.MessagePackSerializer.Serialize[T] (T obj, MessagePack.IFormatterResolver resolver) (at Assets/Plugins/Pupil/3rd-party/MessagePack/MessagePackSerializer.cs:64) MessagePack.MessagePackSerializer.Serialize[T] (T obj) (at Assets/Plugins/Pupil/3rd-party/MessagePack/MessagePackSerializer.cs:55) PupilLabs.RequestController+Request.SendRequestMessage (System.Collections.Generic.Dictionary2[TKey,TValue] data) (at Assets/Plugins/Pupil/Scripts/Request.cs:76)
PupilLabs.RequestController.StopPlugin (System.String name) (at Assets/Plugins/Pupil/Scripts/RequestController.cs:118)
PupilLabs.FrameListener.Disable () (at Assets/Plugins/Pupil/Scripts/FrameListener.cs:53)
PupilLabs.RequestController.Disconnect () (at Assets/Plugins/Pupil/Scripts/RequestController.cs:90)
PupilLabs.RequestController.OnDisable () (at Assets/Plugins/Pupil/Scripts/RequestController.cs:45)

See here

	public static float FloatFromDictionary(Dictionary<string,object> source, string key)
	{
		object value_o;
		source.TryGetValue ("confidence", out value_o);
		return (float)(double)value_o;
	}

It's almost certainly true that "confidence" needs to be replaced by key. I'm surprised this hasn't come up before because it seems to affect timestamps in recordings here.

I'm just learning the delegate system in Unity, but I think there might be a typo in the following code found here:

	public static event OnCalibrationEndDeleg OnCalibrationEnded;
	public static event OnCalibrationEndDeleg OnCalibrationFailed;
	public static event OnConnectedDelegate OnConnected;
	public static event OnConnectedDelegate OnDisconnected;

Again, I'm a novice here, so there's a chance that there's a good reason for this, but my intuition is that OnCalibrationFailed should be an event of OnCalibrationFailedDeleg, not OnCalibrationEndDeleg. Same thing with OnDisconnected and OnDisconnectedDelegate.

I am unable to import Pupil.Import.Package.VR.v0.61.unitypackage into an existing Unity 2018.3.6f1 project that has been pre-installed with SteamVR. As soon as I imported it, Unity complained that ILGenerator, LocalGuilder, ModuleBuilder, AssemblyBuilder, TypeBuilder, FieldBuilder are not found. When I changed the build target to UWP, these errors are gone. However, a new error stating that "FFMpegOut" is not found appears instead.

Start/stop calibration with c button on keyboard in addition to button in GUI.

Hi,

I would like to use the hmd_calibration python scripts to connect to the pupil_service, but it seems that this script is not up to date (it doesn't run with python 3 without modifications and other errors arise as wel).
Could you please check if the script runs with the latest pupil service and update it if it is not?

Thanks!

The UnityPlayer.dll required for a built demo to run is missing from the release package.

I tried running it with a UnityPlayer.dll generated from building another project, but it doesn't work either (I'd guess that the editor/engine versions have to be matched between the executable and the used DLL).

User should be able to use pupil_capture or pupil_service with the Unity3d plugin.

User should set the path to the executable e.g. path/to/pupil_capture.exe or path/to/pupil_service.exe

Remove UI fields for setting calibration points from Settings>Calibration. Calibration points can either be set by API (network communication) or via a game object in Unity.

Set default points for 2d and 3d mode.

Default points for 2d mode are:

[(0,0), (0,0.5), (0,1.0), (0.5,1.0), (1.0,1.0), (1.0,0.5), (1.0,0), (0.5,0), (0.5,0.5)]

Default points for 3d mode (in meters) are:

[(0.5,0.5,1.0), (0,0,1.0), (0,1.0,1.0),  (1.0,1.0,1.0), (1.0,0,1.0),  (0.5,0.5,1.0)]

In the file "unity_pupil_plugin_hololens/Assets/pupil_plugin/Scripts/Pupil/PupilMarker.cs" you have the line:

material = new Material (Resources.Load<Material> ("Materials/MarkerMaterial"));

But there is no such file. If you adjust the path to just "MarkerMaterial" everything works fine.
Maybe you should fix this.

User should be able to start/stop calibration by clicking c on the keyboard. Currently user can only start 2d calibration with c - but should also be able to stop the calibration with c.

When deploying the example project to the Hololens I cannot connect to pupil capture with following seutp:

• Unity 2017.4.1f1
• pupil capture 1.10.20
• hmd-eyes-0.5.1 unity pupil plugin for the hololens

As you can see on the picture I cannot start the calibration. I followed the instructions explained here: https://github.com/pupil-labs/hmd-eyes/tree/v0.5.1/unity_pupil_plugin_hololens

Am I using the wrong IP address for the PupilSettings? I tried it once with the remote one of the hololens relay and once with the ip address of my pc.

The examples for Hololens/Unity can be built using Unity.
However, when trying to deploy them to the Hololens (using visual studio) it fails...with error code 1

Hello,
I have the issure that the markers cant be seen through my pupil labs VR Vive headset. The objects are out of range.
Is there a quick way to fix that?

This will make it possible to reconnect after a Disable/Enable cycle.

Hi,
I think the calibration is only sending 2D data and no depth (z component)?

Links are no longer valid after docs update. Please fix links in the README.

Hi.

I properly installed all necessary packages in "https://docs.pupil-labs.com/" and want to connect pupil-labs devices with HTC-Vive device.

We want to collect coordinated data where people are looking at in a 360 video using the pupil device and it seems that to do so, we can test it using this library.

Right now, we want to make connection between Unity and this pupil device in the HTC-Vive device and followed all steps in "Changing standard settings via PupilGazeTracker".

After playing, the current scene shows us the "Connecting to pupil." message and we can see the message in the HTC-Vive device as well. However, the Unity stops often and it doesn't show the word "Connected" as described in 3 in this section, "Changing standard settings via PupilGazeTracker".

Do you have any ideas to do what we want? Thanks.

The example is for reference only but having in in python3 is still required to make it work.

Our game/experiment generates an error when the subject looks away from the fixation point and it would be great if that didn't happen during blinks or momentary tracking glitches. It seems the easiest way to do this would be to throw away data that didn't pass a confidence threshold.

In fact this is already done here (I think) but it is hard-coded to 0.6. Please make this user-configurable.

Please remove this. Its not useful.

Your methods for gaze visualization are biasing estimates of gaze accuracy in your favor.

And, here's a brief explanation...

Collisions are detected by casting a sphere of a fixed size in the direction of the gaze ray.

The gaze blob is then set to the position of hit.point (the collision location).
...which is offset from the gaze ray!

The more honest way to do this would be to place the gaze blob at the distance of the hit.point, but keep it centered on the gaze ray.

The code I'm referencing is here:

if (Physics.SphereCast(origin, sphereCastRadius, direction, out RaycastHit hit, Mathf.Infinity))
            {
                Debug.DrawRay(origin, direction * hit.distance, Color.yellow);

                projectionMarker.position = hit.point;
}

I get compiler errors after switching the platform to UWP. Since I'm going to deplay to the Holo-Lens, I imported the MixedRealityToolkit and switched the plattform to UWP.

right after switching I see the following error in the console:

so I tried to fix that but simply doing

public class Heatmap : MonoBehaviour 
{
#if !UNITY_WSA
    //  ...
#endif
}

but now I see there is even more

Am I doing something wrong or is pupil-labs currently not compatible with UWP?

Update

It worked now using the Pupil.Import.Package.HoloLens.unitypackage from v0.5.1 is it supposed to be like that?

@kornellvarga - the Calibrate button in the GUI of your branch is not currently triggering calibration.should_start. c button on the keyboard still works to trigger calibration.

I add unity_pupil_plugin_vr and play Calibration scene

but PupilGazeTracker can't get PupilData._2D.GetEyeGaze

That value is always (0,0)

How do I get eye gaze and eye position?

When I press 'c' to begin calibration there is an exception error.

TerminatingException: CheckContextTerminated - yes, is terminated.
NetMQ.Core.SocketBase.CheckContextTerminated ()
NetMQ.Core.SocketBase.TrySend (NetMQ.Msg& msg, TimeSpan timeout, Boolean more)
NetMQ.NetMQSocket.TrySend (NetMQ.Msg& msg, TimeSpan timeout, Boolean more)
NetMQ.OutgoingSocketExtensions.Send (IOutgoingSocket socket, NetMQ.Msg& msg, Boolean more)
NetMQ.OutgoingSocketExtensions.SendFrame (IOutgoingSocket socket, System.Byte[] data, Int32 length, Boolean more)
NetMQ.OutgoingSocketExtensions.SendMoreFrame (IOutgoingSocket socket, System.Byte[] data, Int32 length)
NetMQ.OutgoingSocketExtensions.SendMultipartMessage (IOutgoingSocket socket, NetMQ.NetMQMessage message)
PupilGazeTracker._sendRequestMessage (System.Collections.Generic.Dictionary`2 data) (at Assets/Scripts/PupilGazeTracker.cs:1788)
PupilGazeTracker.StartCalibration () (at Assets/Scripts/PupilGazeTracker.cs:2096)
PupilGazeTracker.Update () (at Assets/Scripts/PupilGazeTracker.cs:880)

This is the offending line:

--System--
Unity 5.6.2f1
Windows 10 x64
pupil_v0912_windows_x64
Latest Drivers

I suggest calling camera.main once and then referencing it thorugh a field in code.

Typo. Default name for Pupil Service File Name default is et to pupil_service.ex for Windows OS. Please fix to correct suffix .exe.

In 2d calibration mode either remove grid entirely or replace the grid with a simple plane so that user can see the bounds of the calibration space. The grid is distracting during calibration.

I'm trying to get 'diameter' value in 'base_data' from 'PupilTools.gazeDictionary' after 2D calibration process.
But it seems that 'base_data' can't be cast in spite of using function BaseData() or DictionaryFromDictionary() in PupilTools.
How can I get values in base_data from PupilTools.gazeDictionary?

--Environment--
OS: Windows 10 x64
Unity: 2017.3.1
Pupil Capture: v1.8-26

NullReferenceException: Object reference not set to an instance of an object:
PupilManager.OnCalibtaionStarted () (at Assets/pupil_plugin/Scripts/PupilManager.cs:93)
PupilTools.StartCalibration () (at Assets/pupil_plugin/Scripts/Networking/PupilTools.cs:364)
PupilGazeTracker.Update () (at Assets/pupil_plugin/Scripts/Pupil/PupilGazeTracker.cs:105)

@kornellvarga - Set IP address to 127.0.0.1 by default - this is localhost.

From what I understand, the 2D and 3D market scene demos differ both in pupil-to-gaze mapping algorithm as well as the way in which the gaze data is represented. More specifically...

In the 2D market scene demo:

1. the 2D pupil-to-gaze mapping algorithm is used. However, this algorithm seems to use only the centroid of the detected pupil, and calibraiton quality is quickly degraded by small shifts of the helmet on the face. I do not recommend that anyone use this method for research.

2. The eye-in-head vectors are represented by a series of points mapped to a 2D plane at a fixed distance.

In the 3D market scene demo:

1. the 3D pupil-to-gaze mapping is used. From my experience with the mobile tracker, this algorithm can work very well. Unfortunately, it does not fare well in this demo. I do not know why. Poor eye images?

2. The eye-in-head vectors are represented by single point placed along the cyclopean gaze vector at the depth of gaze estimated from the convergence of the left and right gaze vectors.

There are two issues here.

One is that folks are never quite sure what is implied the 2D vs 3D marketplace demo, because the issues of mapping algorithm and data representation are conflated. Is it called 2D simply because the gaze direction is represented as discs on a 2D plane? On both Discord and in personal communication, I have been asked the question "What algorithm is used in the 2D marketplace demo?" (or some variant of the question) several times.

The second issue is that, for the 3D gaze mapping algorithm to succeed, it must meet a HIGHER criterion than the 2D mapping: that is, it must also be sufficiently accurate for the recovery of depth of gaze. I have never seen this done with a conventional eye tracker, and I believe the estimated depth of gaze is extremely inaccurate beyond 3m even with a very high quality eye tracker.

I strongly suggest that you use the same method of data representation for both the 2D and 3D trackers: the discs on the screen. You may give the option to try and recover depth of gaze, but please do not make it the default representation.

I also **strongly * recommend an explicit explanation of what differs between the 2D and 3D marketplace demos.

Thanks!

• gD

Hello

I have a minimal Unity project with imported only SteamVR plugin and hmd-eyes plugin.
If there is chosen API compatibility level '.NET Standard 2.0', there appears massive wave of missing types in MessagePack (those from .Net 4).
Once I switch into .NET 4.x, all errors dissappears, but then every calibration is being failed with result shown in picture below

Problem with failing was existing there for some time using .NET 4, but MessagePack missing types has appeared when Steam released update of SteamVR app.

Unity v. 2018.3.5f1
SteamVR unity plugin v. 2.2.0
hmd-eyes plugin v. latest
pupil service v. 1.10-20-g6936560
OpenVR v. 1.0.3

I'm trying to get blink data through blink demo, but there is no logout message(blink detected, etc..) in the unity console. How can I get blink data? I can get the gaze data after calibration.

Thank you for your last help.
Thanks to that, I am now getting the Gaze value without any problems.
There are three kinds of help I need right now.
 

1. Is there a way to measure pupil size?
   I want to find the pupil size as well as the eye position.

2. I am getting the value of Gaze. How much of this value is actually moving and can I get the angle moving?

3. https://www.youtube.com/watch?v=2cTJ1IWmOGY
   I want to get the image of Pupil Service from Unity like the video of this link.
   It can be received in real time, but it is okay to have only one image of both eyes when you want.
   Unity_pupil_plugin_vr / Assets / pupil_plugin / Scripts / Pupil / FramePublishing.cs It seems that you are trying to get an image from this class, but it does not seem to have been developed yet.
   Can you tell when it will be developed?

I would like you to let me know if you can easily read the document.

My environment is using win10, unity, pupil version 1.3.9.

@kornellvarga - Pupil Service App Path requires trailing slash in order for Pupil Service application bundle to be found.

Hi,
I have a 120Hz HTC vive binocular add-on and I would like to know the camera specifications and physical dimensions of the image sensor. I am not sure where I can find these. Any help on this would be appreciated.

Don't show the htc vive scene view during calibration - it is distracting.

Capture rate in GUI should use eye rate or pupil instead of gaze data.

I'm having trouble accessing the "gaze" data on my own unity application using the Pupil Labs Unity plugin.

For calibration I am using the pupil manager prefab with an added "WorldCamera" that starts as activated, is deactivated on calibration start up, then reactivated again after calibration is successful.

After the calibration procedure, I deactivate the calibration camera and activate the "WorldCamera." After doing such, I set the following lines of code:

PupilSettings.Instance.currentCamera = Camera.main; // I confirmed this is the camera I want it to be
PupilTools.IsGazing = true;
PupilTools.SubscribeTo("gaze");

After reading the documentation I expected the code in the function InitializeSubscriptionSocket of connection.cs to be called every time the hardware detects a gaze:

subscriptionSocketForTopic[topic].ReceiveReady += (s, a) => 
{
int i = 0;
NetMQMessage m = new NetMQMessage();
while(a.Socket.TryReceiveMultipartMessage(ref m)) {
...
switch(msgType) {
...
case "gaze":
case "pupil.0":
case "pupil.1":
	var dictionary = MessagePackSerializer.Deserialize<Dictionary<string,object>> (mStream);
...
        if (confidence > 0.6f && msgType == "gaze")
                                PupilTools.gazeDictionary = dictionary;
...

Such that when I call PupilData._2D.GazePosition or PupilData._3D.GazePosition I would receive meaningful data.

However, after the calibration function ends the subscriptionsocket is never called. I suspect this isn't just for the gaze subscription either.

My suspicion is that something got deactivated which makes the subsciptions void. However, I have yet to prove this.

Any suggestions?

For a project I used your code and deployed the Unity project to my Android device. After receiving data for up to 2 minutes flawlessly, it just stops without printing any information or raising any exception. The SubscriberSocket in the connection just doesn't have new data to deliver.

However, in very rare cases I managed to get an exception:
03-01 10:40:00.086 13450 13594 E Unity : ArgumentOutOfRangeException: Number must be either non-negative and less than or equal to Int32.MaxValue or -1. 03-01 10:40:00.086 13450 13594 E Unity : Parameter name: millisecondsTimeout 03-01 10:40:00.086 13450 13594 E Unity : at System.Threading.WaitHandle.WaitOne (System.Int32 millisecondsTimeout, System.Boolean exitContext) [0x00013] in <d7ac571ca2d04b2f981d0d886fa067cf>:0 03-01 10:40:00.086 13450 13594 E Unity : at System.Threading.WaitHandle.WaitOne (System.Int32 millisecondsTimeout) [0x00000] in <d7ac571ca2d04b2f981d0d886fa067cf>:0 03-01 10:40:00.086 13450 13594 E Unity : at System.Collections.Concurrent.BlockingCollection1[T].TryTake (T& item, System.Int32 timeout) [0x00087] in <20599ba1ee5a49a39b6681a06025b37e>:0 03-01 10:40:00.086 13450 13594 E Unity : at AsyncIO.DotNet.CompletionPort.GetMultipleQueuedCompletionStatus (System.Int32 timeout, AsyncIO.CompletionStatus[] completionStatuses, System.Int32& removed) [0x0002d] in <20599ba1ee5a49a39b6681a06025b37e>:0 03-01 10:40:00.086 13450 13594 E Unity : at NetMQ.Core.Utils.Proactor.Loop () [0x00014] in <3f7bfb8e0fb64c5398262c3b6b8f6c44>:0 03-01 10:40:00.086 13450 13594 E Unity : at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00014] in <d7ac571ca2d04b2f981d0d886fa067cf>:0 03-01 10:40:00.086 13450 13594 E Unity : at System.

It seems like your AsyncIO DLL is rather outdated. For me the 0.1.69 .NET 4.0 version from (https://www.nuget.org/packages/AsyncIO/) works fine. Maybe someone bothers to update the dependency.

Regards,
Marcel