pydsgz / deepvog Goto Github PK
View Code? Open in Web Editor NEWPupil segmentation and gaze estimation using fully convolutional neural networks
License: GNU General Public License v3.0
Pupil segmentation and gaze estimation using fully convolutional neural networks
License: GNU General Public License v3.0
I tried running the "test_if_model_work.py" file. The test_image.png file included with the python and h5 files didn't work, so I just took an image off google images. However, the output image is black. The numbers in the outputted array are on the order of 10^-5, 10^-6 and 10^-7. When I tried scaling these to the 0-255 range, the output was just grayscale noise. I've included the image for completeness. I only slightly modified test_if_model_works, and did not modify anything else. Here is the slightly modified test_if_model_work code I used:
(Edit: I downloaded all of the codes in the DeepVOG folder, but it is not clear which ones are dependencies and which is the main code that runs everything.)
def test_if_model_work():
model = load_DeepVOG()
img = np.zeros((1, 240, 320, 3))
reader=ski.imread("test_image.png")/255
reader.resize((240,320))
img[:,:,:,:] = reader.reshape(1, 240, 320, 1)
prediction = model.predict(img)
ski.imsave("test_prediction.png", prediction[0,:,:,1])
#print(prediction)
viewer = ImageViewer(np.uint8(prediction[0,:,:,1]*10000000))
viewer.show()
raise TypeError('Keras symbolic inputs/outputs do not '
TypeError: Keras symbolic inputs/outputs do not implement __len__
. You may be trying to pass Keras symbolic inputs/outputs to a TF API that does not register dispatching, preventing Keras from automatically converting the API call to a lambda layer in the Functional Model. This error will also get raised if you try asserting a symbolic input/output directly
The variable resizing
is passed into _preprocess_image
but in the body of the code, the variable shape_correct
is passed which has opposite definition as resizing. Suggest changing resizing in the function definition to be shape_correct
and change line 277 to be if not shape_correct:
.
I assume that the aspect ratio here should be the same:
Lines 23 to 25 in 7bad240
i.e ori_video_shape[0] / ori_video_shape[1] ~= sensor_size [0] / sensor_size[1]
(within floating point tolerance)?
If that is the case, then the inputs can be checked with an assertion, etc
Hi,
I'm trying to run the demo as instructed and I get the following:
tensorflow.python.framework.errors_impl.OperatorNotAllowedInGraphError: using a
tf.Tensoras a Python
bool is not allowed in Graph execution. Use Eager execution or decorate this function with @tf.function.
.
I'm using windows 7 (with an i7 processor and w/o GPU)... not sure if it has any connection.
Thanks in advance!
When trying out the visualization branch, i noticed that the eye center was off from what it supposed to be. The pupil ellipse and Gaze vector were accurate but the blue line from the gaze to eye center did not form a continuous vector as it should have.
I am assuming the error is in the unprojection but I have not tracked through the math.
How exactly do we upload the annotation to the DeepVOG model once images are annotated? Are we to make a separate script for it or can you provide one?
Hey,
Good work. We are looking to similar lines but with really low dimensional data. We are currently getting data from Webcam.
Can we please have dataset links as well? We and the community can greatly benefit from the same.
Hello, i need a labeling tool, please tell me how to use. Then I want to use your Deepvog to fine-tune my data. What can I do?
Hello, what should I do if I want to fine-tune it? Please be more detailed, thank you.
hello, thank you for sharing your project. I installed it as read.me, and started fit step, but I got the error "No unprojected gaze lines or ellipse centres were added (not yet initalized). Use add_to_fitting() function to add them first ". Besides, i want konw that whether can i utilize the project to estimate gaze of video from DMS(driver monitor system) NIR camera ?
I just downloaded the code and tested the fit on the demo video included in the repo and I get the following eye model
{ "eye_centre": [ [ -189.91085622309689 ], [ 129.4567167957034 ], [ 3333.3333333333335 ] ], "aver_eye_radius": 1286.1201062769812 }
Shouldn't the units be in mm? Do these numbers make sense? Also I was expecting the algorithm to estimate the depth (z) of the eye center but it always reports the given initial z value. Is that correct?
Hi there,
I need the dataset for that project to use it in my own project
I am running fitting on your demo script using the command:
python -m deepvog --fit ./demo.mp4 ./demo_eyeball_model.json -m -b 32
I've seen that in your paper (Section 3.1.5 - Inference Speed) you run your program at 130Hz for batch sizes of 32, however when I run your program on your demo files (even without visualisation) I am averaging around 15Hz.
I am using a machine with the following specs:
CPU - Intel Xeon 12-core 2.5Ghz w/ Windows 10
GPU - Nvidia GeForce RTX 2080 Ti
RAM - 64GB
Python - 3.6.1
Tensorflow-gpu - 1.15.0
CUDA - 10.0
cuDNN - 7.6.5
Is there anything obvious that I am missing here that could lead to the weak performance?
import deepvog is not working
.model.DeepVOG_model cannot be found
running on Google Colab
Good day!
Thanks for your research and implementation. Can I get the pupil diameter?
deepvog/inferer.py Line 46
self.mm2px_scaling = np.linalg.norm(self.ori_video_shape) / np.linalg.norm(self.sensor_size)#mm转换为像素 self.model = model self.confidence_fitting_threshold = 0.96 self.eyefitter = SingleEyeFitter(focal_length=self.flen * self.mm2px_scaling, pupil_radius=2 * self.mm2px_scaling, initial_eye_z=50 * self.mm2px_scaling)#眼睛模型
Hello, I would like to ask a question about initialization parameters. Because the test data I am using is synthetic image data, I am unable to determine the camera parameters.
Hi,
Really impressive work and many thanks for making it open-source. I was trying to replicate your model by re-training using another dataset, but I never reached comparable with your published pre-trained weights. While I understand that you might not be able to share your training data, could you please reveal some of the hyperparameters you used for training, e.g. learning rate, optimizer, batch size, epochs, regularization etc. (augmentation was kindly described in the paper).
Thank you
The demo videos can not be played, is there a way to play them?
Whenever I try to run it on Jetson nano, the device starts throttling and the program starts reporting about memory issue. What can I do to resolve this?
I've managed to run the code on the demo versions successfully. I then tried to use my own video (in .mp4 format) and receive this error:
ValueError: No way to determine width or height from video. Need
-sin
inputdict. Consult documentation on I/O.
I'm not sure whether my understanding of the documentation is wrong. I believe I have set the video size and sensor size correctly, and don't fully understand the above error. I have tried running the program using:
python -m deepvog --fit ./output.mp4 ./demo_eyeball_model.json -v ./demo_visualization_fitting.mp4 -b 32
as well as one command I found in the readme:
python -m deepvog --fit ./output.mp4 ./demo_eyeball_model.json --flen 12 -vs 300,400 -s 0.005,0.005 -b 32
Do you have any advice you could offer?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.