Inspired by : GeoHot's TinyGrad
Learning how to make a custom deep learning framework using only numpy. Mostly, for now I will try to understand what George did in his work and recreate the same results. Hopefully, I will be able to add something new to it. If not, I will have at least learned something.
import numpy as np
from minigrad.tensor import Tensor
x = Tensor(np.eye(3))
y = Tensor(np.array([[2.0,0,-2.0]]))
z = y.dot(x).sum()
z.backward()
print(x.grad) # dz/dx
print(y.grad) # dz/dy
import torch
x = torch.eye(3, requires_grad=True)
y = torch.tensor([[2.0,0,-2.0]], requires_grad=True)
z = y.matmul(x).sum()
z.backward()
print(x.grad) # dz/dx
print(y.grad) # dz/dy
It turns out, a decent autograd tensor library is 90% of what you need for neural networks. Add an optimizer (SGD and Adam implemented) from minigrad.optim, write some boilerplate minibatching code, and you have all you need.
- Look into Adam Optimizer
- Implement convolutions