A list of papers I find interesting including a single sentece on main findings. On topics in NLP, DL and RL. Specifically, computer vision, knowledge graph embeddings, dialoge systems, reasoning using RL, and transformer-based architectures.
- Transporter Networks: Rearranging the Visual World for Robotic Manipulation: Transporter Network as a simple model architecture that rearranges deep features to infer spatial displacements from visual input for robotic manipulation
- Perceiver: General Perception with Iterative Attention: Introducing the Perceiver, a model that builds upon Transformers and hence makes few architectural assumptions about the relationship between its inputs, such as grid like structure in pictures for example.
- Rethinking Spatial Dimensions of Vision Transformers: Empirically show that such a spatial dimension reduction is beneficial to a transformer architecture as well, and propose a novel Pooling-based Vision Transformer (PiT)
- FNet: Mixing Tokens with Fourier Transforms: Fourier transformations applied to tokens replaces the self-attention layer with an resuting accuracy of 92% compared to BERT.
- Asymmetric self-play for automatic goal discovery in robotic manipulation: Trained a single, goal-conditioned policy that can solve many robotic manipulation tasks, including tasks with previously unseen goals and objects
- Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets: adjusting GPT-3 models with as little as 80 text samples to reduce toxicity, gender, race, and religious imbalances.
- Pangu-α: large-scale autoregressive pretrained chinese language models with auto-parallel computation:100 billion parameter model trained on 1.1TB Chinese data
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity: Hardrouting information simplifies the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs
- DALL-E: Creating Images form Text: GPT-3 model trained to generate images from text description
- CLIP: Connecting Text and Images: Great zero-shot performance
- VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding: Towards multi-modal video understanding
- A survey on knowledge graphs: Representation, acquisition and applications: Comprehensive review on state-of-the-art knowledge graph embedding models.
- An overview of embedding models of entities and relationships for knowledge base completion: Summary of different types of scoring functions
- A survey on empathetic dialogue systems: Literature review of empathetic dialogue systems, whose goal is to enhance the perception and expression of emotional states, personal preference, and knowledge.
- LITE TRANSFORMER WITH LONG-SHORT RANGE ATTENTIONLITE TRANSFORMER WITH LONG-SHORT RANGE ATTENTION: The key primitive is the Long-Short Range Attention (LSRA), where one group of heads specializes in the local context modeling (by convolution) while another group specializes in the long-distance relationship modeling (by attention).
- ReadNet: Towards Accurate ReID with Limited and Noisy Samples
- Bayesian Deep Learning and a Probabilistic Perspective of Generalization: The key distinguishing property of a bayesian approach is marginalization instead of optimization. Rather than use a single setting of parameters w, use all settings weighted by their posterior probabilities in a bayesian model average
- LambdaNetworks: Modeling long-range Interactions without Attention: Transforming image context into linar layers (Lambdas). LambdaResNets reach state-of-the-art accuracies on ImageNet while being ∼4.5x faster than the popular EfficientNets on modern machine learning accelerators.
- Every Model Learned by Gradient Descent Is Approximately a Kernel Machine: This paper challenges this prevailing view and suggest that rather than representing the data, deep neural networks store superpositions of the training data in their weights and act as kernel machines at inference time.
- Big Self-Supervised Models are Strong Semi-Supervised Learners: Proposes SimCLRv2 and shows that semi-supervised learning benefits a lot from self-supervised pre-training.
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale: A pure transformer applied directly to sequences of image patches can perform very well on image classification task
- Wifi-based human activity recognition using raspberry pi: A radio frequency-based approach to activity recoginition using Conv/LSTM classifier. Can be used in smart-homes homes. However, just yet not really relyable.
- Ecological Reinforcement Learning: Aim to study how the non-episodic RL problem is affected by particular properties of the training environments.
- Reformer: The efficient Tranformer: Introduce two techniques to imporve efficiency, such as, locality-sensitive hashing and reversible residual layers.
- 3D human pose estimation in video with temporal convolutions and semi-supervised training: Pretrained model for 3D pose estimation with very good results
- Off-Policy Evaluation via Off-Policy Classification: Incorporating Graph Attention Mechanism into Knowledge Graph Reasoning Based on Deep Reinforcement Learning