bptripp / fovnet Goto Github PK

Receptive fields in cortex generally get larger toward the periphery. We will get the same qualitative effect by convolving standard square kernels with retina-based remapping of the visual field. We should see how closely we can fit cortical RF data just by choosing kernel sizes.

The feature maps have dimensions of eccentricity and angle, rather than horizontal and vertical positions. So we do not have translation equivariance. Also, the image statistics may be different in the fovea and the periphery, due to different scales. These factors will work against the statistical efficiency of convolution. Other more subtle warping effects may cause problems as well. This should all be fixed.

Introducing rotation equivariance in the feature-map space might help to recover translation equivariance in image space. See https://arxiv.org/pdf/1711.07289.pdf re. rotation equivariace in CNNs via steerable filters.

Do deal with foveal-peripheral differences, it might help to use pairs of kernels that are averaged with different weights depending on eccentricity. It seems like this could go off the rails though depending on initialization. Maybe initialize the pair to the same values and let them diverge as necessary.

Integrate into a standard network like https://github.com/keras-team/keras/blob/master/keras/applications/resnet50.py

Choose foveation points using a human-foveation prediction network like https://deepgaze.bethgelab.org/

Probably extrapolate images a bit to allow wider range of foveation points without making images too small.

Blurred images in remap.ImageSampler should be downsampled to save filtering time.

Combine centres and surrounds using different opponent channels.