Implementations of KAN variations.

pip install .

To run mnist select config in main.py and run main.py.

To view charts, run tensorboard --logdir .experiments/

See examples/ (in future)

Performance (rtx 2060 mobile, mnist):

MLP (31.8M parameters) - 51 it/s

KANLinear (32.3 M parameters) - 4.3 it/s

KANLinear2 (31M parameters) - 36.5 it/s

• update_grid on cuda raise error (torch.linalg.lstsq assume full rank on cuda, only one algorithm) - solved temporary, moved calculating lstsq to cpu
• update_grid_from_samples in original KAN run model multiple times, is it necessary?
• parameters counting, is grid parameter or not?
• MLP training is almost instant, but KAN train slow on start

• Base structure
• KAN simple implementation
• KAN trainer
• train KAN on test dataset
• remove unnecessary dependencies in requirements.txt
• test update_grid and "Other possibilities are: (a) the grid is learnable with gradient descent" from paper.
• Regularization
• Compare with MLP
• Grid extension
• MNIST
• CIFAR10
• KAN ResNet?
• KAN as CNN filter?
• KAN in VIT?
• Fourier KAN?
• pruning
• test continual learning
• docs and examples - write notebooks like in KAN repo.
• KAN vs MLP in "LLM" - test?
• CUDA kernel for b_splines?
• unit tests?

@misc{liu2024kan,
      title={KAN: Kolmogorov-Arnold Networks}, 
      author={Ziming Liu and Yixuan Wang and Sachin Vaidya and Fabian Ruehle and James Halverson and Marin Soljačić and Thomas Y. Hou and Max Tegmark},
      year={2024},
      eprint={2404.19756},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

Original KAN repo - base idea

efficient-kan - KANLinear2 and optimizations