tracel-ai / burn Goto Github PK

Feature description

A user should be able to specify if the model data should be saved in half precision, and tell if the data should be loaded back to the full precision

Feature motivation

Often the performance of model inference does not degrade much if f32 precision is truncated in half but it may matter during training. To reduce the size of the exported model, it would be beneficial to save in bf16 (truncated version of f32) and later to load back in f32 (If the target architecture supports only f32 and up).

(Optional) Suggest a Solution

Use half library to convert f32 to bf16. half also supports serde for (de)serialization. User should be able to specify target data (f32).

@NathanielSimard -> @nath_simard

Describe the bug

I copy pasted everything out of the MNIST src into my own project because I wanted to try making a test NN.

It does not like one of the types in data.rs yet it compiles fine in the project itself?

  --> TicTacToeMaster\src\data.rs:29:42
   |
29 |             .map(|tensor| tensor.reshape([1,784]))
   |                                  ------- ^^^^^^^ expected struct `Shape`, found array `[{integer}; 2]`
   |                                  |
   |                                  arguments to this function are incorrect
   |
   = note: expected struct `Shape<_>`
               found array `[{integer}; 2]`
note: associated function defined here
  --> C:\Users\itscrabs\.cargo\registry\src\github.com-1ecc6299db9ec823\burn-tensor-0.2.3\src\tensor\base.rs:31:12
   |
31 |     pub fn reshape<const D2: usize>(&self, shape: Shape<D2>) -> Tensor<B, D2> {
   |            ^^^^^^^
help: try wrapping the expression in `burn::tensor::Shape`
   |
29 |             .map(|tensor| tensor.reshape(burn::tensor::Shape { dims: [1,784] }))
   |                                          +++++++++++++++++++++++++++         +

For more information about this error, try `rustc --explain E0308`.
error: could not compile `TicTacToeMaster` due to previous error

To Reproduce

Steps to reproduce the behavior:

1. Create new cargo project
2. Copy everything in examples/mnist to the project
3. add serde and burn to Cargo,toml
4. cargo run
5. See error

Expected behavior
Example to run

Additional context
I'm not so great with GitHub but just forcing it into the correct Burn shape using new() seems to fix it:
at examples/mnist/data.rs: line 29 should become:
.map(|tensor| tensor.reshape(burn::tensor::Shape::new([1,784])))

Feature description

Dataset package should provide with basic normalizer's, such standard score normal.

Feature motivation

Data normalization is a common operation and should be part of the library for convenience and less code writing for everyone.

(Optional) Suggest a Solution

1. For the starter, should add traits for data normalization.
2. Add standard score normalizer (across batch, features or the entire dataset).

And example how torch does: https://stackoverflow.com/questions/63746182/correct-way-of-normalizing-and-scaling-the-mnist-dataset

Feature description

Similar to LearnerBuilder, there should be builders for NN modules, such as Conv2d.

Feature motivation

Currently there two separate steps for defining nn module (config and construction). It would be easier if the user could use one step without passing complex config parameters.

(Optional) Suggest a Solution

It would great if the user could do this:

let conv = Conv2dBuilder()
        .withChannels([1,16])
        .withKernel([3,3])
        .build();

// OR

let builder = Conv2dBuilder()
              .withKernel([3,3])
              .withBias(false);


let conv1 = builder.withChannels([1,16]).build();
let conv2 = builder.withChannels([16,32]).build();
let conv3 = builder.withChannels([32,48]).build();

Action Item

• Update examples/mnist-inference-web to remove random initialization

Describe the bug

The plots are not clear and have too many dots.

To Reproduce
Run training of examples/mnist

Expected behavior
It would be if a different markers are used to plot.

Screenshots

Desktop (please complete the following information):

[~]$ uname -a
Darwin MacBook-Pro.local 22.3.0 Darwin Kernel Version 22.3.0: 
Mon Jan 30 20:38:37 PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T6000 arm64
[~]$ 

Feature description

Add conv1d layer.

Describe the bug

tch 0.8 is dependent on torch-sys ^0.8, and torch-sys 0.8 is dependent on libtorch v1.12.0, see here

If you've already installed libtorch before, you should check the version first, or else a build error will occur.

You could download the specific version of libtorch through:

gpu : https://download.pytorch.org/libtorch/cu113/libtorch-shared-with-deps-1.12.0%2Bcu113.zip # (CUDA 11.3)
cpu : https://download.pytorch.org/libtorch/cpu/libtorch-shared-with-deps-1.12.0%2Bcpu.zip # (CPU)

see here

Another thing, you'd better update the rustc to the latest stable version. It's easy to do that, just run: rustup update stable.

To Reproduce

Expected behavior

Screenshots

Desktop (please complete the following information):

• OS: [e.g. iOS] Mac Monterey v12.0.1
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Additional context

Describe the bug
I wanted to play around with the provided MNIST example a bit and replaced the NdArray backend with the Tch one by adapting a few lines in the main.rs file from the example folder.
The training runs considerably faster (no shit it runs in parallel now) but after running for two epochs, the code panics out.

To Reproduce

1. Clone the current master branch of the repository 0110ccc96471a404479c5e67801f22309bcf7682
2. Replace the relevant lines in the main.rs file, i.e.:
   let device = TchDevice::Cpu; training::run::<TchADBackend<f32>>(device);
   and add the corresponding imports.
3. Build and run the example
4. Observe the code panic after two epochs.

Expected behavior
I would expect the training to succeed in the same way as with the NdArray backend. I would definitely not expect a panic to happen :(

Screenshots
I don't think a screenshot is necessary, the stack trace is included in an appended file.
crash_dump.txt

Desktop (please complete the following information):

• OS: Arch Linux (zen kernel 6.0.6), fully up to date

Additional context
Error/bug reporting is a completely new thing for me so I hope I am doing everything right here. If you need any additional information or I can help in any other way, I would be very interested :)
crash_dump.txt

Feature description

burn-core, burn-tensor, and burn-ndarray should run with no_std attribute.

Feature motivation

Making inference packages (burn-core, burn-tensor, and burn-ndarray) no_std compatible gives developers ability to deploy models for inference mode to wasm, embedded systems and mobile (or any other system where OS might not be available.

(Optional) Suggest a Solution

General steps on how are outlined in this book's chapter on no_std: https://www.lurklurk.org/effective-rust/no-std.html.

Specific steps on the current solution will be updated here (TBD).

Feature description

The compiler or runtime should explain in a clear language what is wrong and offer a solution, similar what Rust compiler does.

Feature motivation

It's very common to miscalculate/misconfigure NN with incorrect shapes. Sometimes it's part of the process. However, the Burn's error messages are inconsistent across backends and NN modules. Different backends bubble up errors differently. For example, torch tells what size is expected vs given, but ndarray just says wrong size. Additionally, it's very hard to know exactly which line is causing the problem. All these errors will be especially difficult for beginners.

(Optional) Suggest a Solution

Compiler does an excellent job for dimension mismatches. It would be great if compiler could identify the issue early. And if it's possible, then the runtime should check the inputs before passing to the backends, perhaps in debug mode or always.

Feature description

Implement std::fmt::Display for Tensor and BoolTensor the same way for each Backend with basic metadata.
This could look something like the following, but it's just a draft.

Tensor {
  data: [[0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         ...,
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.]],
  shape:   [30, 30],
  device:  Cpu,
  backend: ndarray,
  dtype:   f32,
}

Feature description

Currently there is BatchNorm2d but BatchNorm1d module is missing.

Feature motivation

Lots of sound NN operations use 1D modules, such as Conv1D. So having BatchNorm1d is necessary for training. I attempted to use BatchNorm2d by reshaping the input data, however, I got [1,1,1,1] type of error.

(Optional) Suggest a Solution

It might be useful to come up (or refactor from the existing code) with a universal BatchNorm function, similar to PyTorch's: https://pytorch.org/docs/stable/generated/torch.nn.functional.batch_norm.html and use for all 3 different dimensions.

Feature description

Add the adam optimizer.

Feature description

Support train deep learning models on Apple Silicon GPU? M1 and M2

Feature motivation

Faster

(Optional) Suggest a Solution

Feature description

Bring your own datamodel (BYOD) will be a great slogan for Burn. It would be great if one could export a model with weights from Pytorch and load it in Burn.

Feature motivation

Many ML practitioners use tools, such as PyTorch, and they would like to try out Burn, perhaps, use burn's inference for the start and they transition to the training. Being able to import models quickly will reduce friction for Burn's adoption. Also one can feel safe to take out her/his models elsewhere if needed.

Describe the bug
Build/test fails on burn-tch: cargo test on Mac M1/M2 and aarch64 linux.

To Reproduce
on Mac M1/M2 or aarch64 linux run cargo test on burn-tch

OS Information

bash: uname -a
Darwin MacBook-Pro.local 22.3.0 Darwin Kernel Version 22.3.0: 
Mon Jan 30 20:38:37 PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T6000 arm64

[burn]$ uname -a
Linux e80e5571e6de 5.15.49-linuxkit #1 SMP PREEMPT Tue Sep 13 07:51:32 UTC 2022
 aarch64 aarch64 aarch64 GNU/Linux

Feature description

Add BatchNorm layer

Hello!

I just found your project, this is very exciting work! In my spare time, I played around with AD in Rust and came to similar design decisions as you did. I noticed that you are using a a PyTorch style approach for the modules. One thing I wanted to make you aware of is the functional style used by JAX. In this design, modules are not stateful, instead they only have pure functions, and the user maintains all state. Many ML researcher found this style to be superior. It also leads to a more modular design of the library ecosystem.

Here is an example of how this could look like (code is copied from my snippets so does not match Burn exactly):

pub trait Module {
    type Params<B: Backend>;
    type Input<B: Backend>;
    type Output<B: Backend>;

    fn params<B: Backend>(&self) -> Self::Params<B>;

    fn forward<B: Backend>(&self, params: &Self::Params<B>, input: &Self::Input<B>) -> Self::Output<B>;
}

pub struct LinearParams<B: Backend> {
    w: Tensor2<B>,
    b: Tensor1<B>,
}

pub struct Linear {
    inputs: usize,
    hiddens: usize,
}

impl Linear {
    pub fn new(inputs: usize, hiddens: usize) -> Self {
        Self { inputs, hiddens }
    }
}

impl Module for Linear {
    type Params<B: Backend> = LinearParams<B>;
    type Input<B: Backend> = Tensor2<B>;

    fn params<B: Backend>(&self) -> Self::Params<B> {
        LinearParams {
            w: Tensor::ones((self.inputs, self.hiddens)),
            b: Tensor::ones(self.hiddens),
        }
    }

    fn forward<B: Backend>(&self, params: &Self::Params<B>, input: &Self::Output<B>) -> Self::Output<B> {
        params.w.dot(&input).add(&params.b)
    }
}

And then user code to run this looks as follows:

fn main() {
    let x = Tensor2::<NdArrayBackend>::ones((8, 10));
    let h = Linear::new(10, 30);
    let p = h.params();
    let y = h.forward(&p, &x);
    println!("{:?}", y);
}

with current design of TensorOps, TensorPrivimitive is once initialized, never mutated,
not allowing Buffer of passed TensorPrimitive.
(well, while it's possible with interior mutability that wouldn't be an expected behavior)

some operations may be able to just use same buffer(ex. reshape)
or write result into same buffer(ex. sqrt, relu)
this behavior will prevent allocating new buffer for result, possibly cutting memory usage in half,
but makes user to clone the Tensor before passing if planning to use original value of passed Tensor.

i'm suggesting to support both current behavior and new "append" behavior
by adding both functions to TensorOps trait and making current behavior implementation optional(using trait default behavior)

trait Foo {
    fn foo_append(t: &mut TensorPrimitive);
    fn foo(t: &TensorPrimitive) -> TensorPrimitive {
        let t = t.clone();
        Self::foo_append(&mut t);
        t
    }
}

i'm a newbie so correct me 😓

Feature description

Support strides in convolution (1D and 2D).

The forward pass has already been implemented, but the backward pass is currently missing. It may be necessary to also support transposed convolution.

Feature description

Add MaxPool1D layers.

Replace unwrap with clear expect explaining why it's OK to ignore a result.

Ubuntu 20.04 LTS

On my machine when running the mnist example I get this message:

thread 'main' panicked at 'called Result::unwrap() on an Err value: FailToDownloadPythonDependencies("pillow, numpy | error: No such file or directory (os error 2)")', burn-dataset/src/source/huggingface/mnist.rs:40:14
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace`

I tracked this down to https://github.com/burn-rs/burn/blob/0110ccc96471a404479c5e67801f22309bcf7682/burn-dataset/src/source/huggingface/downloader.rs#L196 which calls the command python when I believe it requires python3

The fix for me was to apt install python-is-python3

I then had problems with pandas requring a higher version of numpy. The fix for that was python -m pip install --upgrade numpy

I believe both these are not problems with the library code but with the installation instructions (of which there appear to be none) and might be worth mentioning when those are created.

Ubuntu 20.04 LTS, NVIDIA 3070 GPU (Driver 510.85.02, CUDA Version 11.6)

I am able to run the example as is and it trains successfully but it is very slow and appears to not be fully utilizing all the cores on my cpu. However at what appears to be the end of Epoch 2 (Last progress printout reports Iteration 80 Epoch 2/6, with 2 full bars) it crashes with this message:

thread 'main' panicked at 'called Result::unwrap() on an Err value: SendError { .. }', burn/burn/src/train/checkpoint/async_checkpoint.rs:68:40

I changed the example to use the Tch backend by changing main to this:

fn main() {
    use burn::tensor::backend::TchADBackend;

    let device = TchDevice::Cpu;
    training::run::<TchADBackend<f32>>(device);
    println!("Done.");
}

Which appeares to train using my full Cpu at a great speeds but then crashed both tries in 2 different ways. The first is the same message as above and upon using the vscode debugger it crashed in a different way:

thread '' panicked at 'attempt to subtract with overflow', burn/burn/src/train/checkpoint/file.rs:41:60

In that case epoch was 1 and self.num_keep was 2

I changed the example main as follows to try to use my GPU:

fn main() {
    use burn::tensor::backend::TchADBackend;

    let device = TchDevice::Cuda(0);
    training::run::<TchADBackend<f32>>(device);
    println!("Done.");
}

My first question is what does the magic number in TchDevice::Cuda(XXX) represent?

Then even with various numbers for that value (0, 1, 1024) the application crashes on the line model.to_device(device);
I always get this error message which I have been unable to solve:

thread 'main' panicked at 'called Result::unwrap() on an Err value: Torch("Could not run 'aten::empty_strided' with arguments from the 'CUDA' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::empty_strided' is only available for these backends: [Dense, Conjugate, Negative, UNKNOWN_TENSOR_TYPE_ID, QuantizedXPU, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, SparseCPU, SparseCUDA, SparseHIP, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, SparseXPU, UNKNOWN_TENSOR_TYPE_ID, SparseVE, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, NestedTensorCUDA, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID].\n\nCPU: registered at aten/src/ATen/RegisterCPU.cpp:37386 [kernel]\nMeta: registered at aten/src/ATen/RegisterMeta.cpp:31637 [kernel]\nQuantizedCPU: registered at aten/src/ATen/RegisterQuantizedCPU.cpp:1294 [kernel]\nBackendSelect: registered at aten/src/ATen/RegisterBackendSelect.cpp:726 [kernel]\nPython: registered at ../aten/src/ATen/core/PythonFallbackKernel.cpp:133 [backend fallback]\nNamed: registered at ../aten/src/ATen/core/NamedRegistrations.cpp:7 [backend fallback]\nConjugate: fallthrough registered at ../aten/src/ATen/ConjugateFallback.cpp:22 [kernel]\nNegative: fallthrough registered at ../aten/src/ATen/native/NegateFallback.cpp:22 [kernel]\nZeroTensor: fallthrough registered at ../aten/src/ATen/ZeroTensorFallback.cpp:90 [kernel]\nADInplaceOrView: fallthrough registered at ../aten/src/ATen/core/VariableFallbackKernel.cpp:64 [backend fallback]\nAutogradOther: registered at ../torch/csrc/autograd/generated/VariableType_2.cpp:14210 [autograd kernel]\nAutogradCPU: registered at ../torch/csrc/autograd/generated/VariableType_2.cpp:14210 [autograd kernel]\nAutogradCUDA: registered at ../torch/csrc/autograd/generated/VariableType_2.cpp:14210 [autograd kernel]\nUNKNOWN_TENSOR_TYPE_ID: registered at ../torch/csrc/autograd/generated/VariableType_2.cpp:14210 [autograd kernel]\nAutogradXLA: registered at ../torch/csrc/autograd/generated/VariableType_2.cpp:14210 [autograd kernel]\nAutogradMPS: registered at ../torch/csrc/autograd/generated/VariableType_2.cpp:14210 [autograd kernel]\nAutogradIPU: registered at ../torch/csrc/autograd/generated/VariableType_2.cpp:14210 [autograd kernel]\nAutogradXPU: registered at ../torch/csrc/autograd/generated/VariableType_2.cpp:14210 [autograd kernel]\nAutogradHPU: registered at ../torch/csrc/autograd/generated/VariableType_2.cpp:14210 [autograd kernel]\nUNKNOWN_TENSOR_TYPE_ID: registered at ../torch/csrc/autograd/generated/VariableType_2.cpp:14210 [autograd kernel]\nAutogradLazy: registered at ../torch/csrc/autograd/generated/VariableType_2.cpp:14210 [autograd kernel]\nAutogradPrivateUse1: registered at ../torch/csrc/autograd/generated/VariableType_2.cpp:14210 [autograd kernel]\nAutogradPrivateUse2: registered at ../torch/csrc/autograd/generated/VariableType_2.cpp:14210 [autograd kernel]\nAutogradPrivateUse3: registered at ../torch/csrc/autograd/generated/VariableType_2.cpp:14210 [autograd kernel]\nTracer: registered at ../torch/csrc/autograd/generated/TraceType_2.cpp:14069 [kernel]\nAutocastCPU: fallthrough registered at ../aten/src/ATen/autocast_mode.cpp:481 [backend fallback]\nAutocast: fallthrough registered at ../aten/src/ATen/autocast_mode.cpp:324 [backend fallback]\nBatched: registered at ../aten/src/ATen/BatchingRegistrations.cpp:1064 [backend fallback]\nVmapMode: fallthrough registered at ../aten/src/ATen/VmapModeRegistrations.cpp:33 [backend fallback]\nFunctionalize: registered at ../aten/src/ATen/FunctionalizeFallbackKernel.cpp:89 [backend fallback]\nPythonTLSSnapshot: registered at ../aten/src/ATen/core/PythonFallbackKernel.cpp:137 [backend fallback]\n\nException raised from reportError at ../aten/src/ATen/core/dispatch/OperatorEntry.cpp:447 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x6b (0x7f95aa2a79cb in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/build/torch-sys-d8a9710e31a4996b/out/libtorch/libtorch/lib/libc10.so)\nframe #1: c10::impl::OperatorEntry::reportError(c10::DispatchKey) const + 0x36b (0x7f95ab5e252b in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/build/torch-sys-d8a9710e31a4996b/out/libtorch/libtorch/lib/libtorch_cpu.so)\nframe #2: + 0x1b4df9b (0x7f95abe40f9b in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/build/torch-sys-d8a9710e31a4996b/out/libtorch/libtorch/lib/libtorch_cpu.so)\nframe #3: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef, c10::ArrayRef, c10::optionalc10::ScalarType, c10::optionalc10::Layout, c10::optionalc10::Device, c10::optional) + 0xac (0x7f95ac011e6c in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/build/torch-sys-d8a9710e31a4996b/out/libtorch/libtorch/lib/libtorch_cpu.so)\nframe #4: + 0x1fac735 (0x7f95ac29f735 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/build/torch-sys-d8a9710e31a4996b/out/libtorch/libtorch/lib/libtorch_cpu.so)\nframe #5: at::_ops::empty_strided::call(c10::ArrayRef, c10::ArrayRef, c10::optionalc10::ScalarType, c10::optionalc10::Layout, c10::optionalc10::Device, c10::optional) + 0x174 (0x7f95ac054114 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/build/torch-sys-d8a9710e31a4996b/out/libtorch/libtorch/lib/libtorch_cpu.so)\nframe #6: at::empty_strided(c10::ArrayRef, c10::ArrayRef, c10::TensorOptions) + 0xd8 (0x55f15452c2a8 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/mnist)\nframe #7: at::native::_to_copy(at::Tensor const&, c10::optionalc10::ScalarType, c10::optionalc10::Layout, c10::optionalc10::Device, c10::optional, bool, c10::optionalc10::MemoryFormat) + 0x1447 (0x7f95aba2cf97 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/build/torch-sys-d8a9710e31a4996b/out/libtorch/libtorch/lib/libtorch_cpu.so)\nframe #8: + 0x21479e3 (0x7f95ac43a9e3 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/build/torch-sys-d8a9710e31a4996b/out/libtorch/libtorch/lib/libtorch_cpu.so)\nframe #9: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optionalc10::ScalarType, c10::optionalc10::Layout, c10::optionalc10::Device, c10::optional, bool, c10::optionalc10::MemoryFormat) + 0x10d (0x7f95abd9d78d in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/build/torch-sys-d8a9710e31a4996b/out/libtorch/libtorch/lib/libtorch_cpu.so)\nframe #10: + 0x1faef51 (0x7f95ac2a1f51 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/build/torch-sys-d8a9710e31a4996b/out/libtorch/libtorch/lib/libtorch_cpu.so)\nframe #11: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optionalc10::ScalarType, c10::optionalc10::Layout, c10::optionalc10::Device, c10::optional, bool, c10::optionalc10::MemoryFormat) + 0x10d (0x7f95abd9d78d in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/build/torch-sys-d8a9710e31a4996b/out/libtorch/libtorch/lib/libtorch_cpu.so)\nframe #12: + 0x2fd82be (0x7f95ad2cb2be in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/build/torch-sys-d8a9710e31a4996b/out/libtorch/libtorch/lib/libtorch_cpu.so)\nframe #13: + 0x2fd883b (0x7f95ad2cb83b in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/build/torch-sys-d8a9710e31a4996b/out/libtorch/libtorch/lib/libtorch_cpu.so)\nframe #14: at::_ops::_to_copy::call(at::Tensor const&, c10::optionalc10::ScalarType, c10::optionalc10::Layout, c10::optionalc10::Device, c10::optional, bool, c10::optionalc10::MemoryFormat) + 0x202 (0x7f95abe1a1e2 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/build/torch-sys-d8a9710e31a4996b/out/libtorch/libtorch/lib/libtorch_cpu.so)\nframe #15: at::native::to(at::Tensor const&, c10::optionalc10::ScalarType, c10::optionalc10::Layout, c10::optionalc10::Device, c10::optional, bool, bool, c10::optionalc10::MemoryFormat) + 0x13e (0x7f95aba22dde in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/build/torch-sys-d8a9710e31a4996b/out/libtorch/libtorch/lib/libtorch_cpu.so)\nframe #16: + 0x2251799 (0x7f95ac544799 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/build/torch-sys-d8a9710e31a4996b/out/libtorch/libtorch/lib/libtorch_cpu.so)\nframe #17: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optionalc10::ScalarType, c10::optionalc10::Layout, c10::optionalc10::Device, c10::optional, bool, bool, c10::optionalc10::MemoryFormat) + 0x216 (0x7f95abf47b26 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/build/torch-sys-d8a9710e31a4996b/out/libtorch/libtorch/lib/libtorch_cpu.so)\nframe #18: at::Tensor::to(c10::TensorOptions, bool, bool, c10::optionalc10::MemoryFormat) const + 0xf0 (0x55f1545286e4 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/mnist)\nframe #19: + 0x247491 (0x55f154531491 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/mnist)\nframe #20: + 0x225035 (0x55f15450f035 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/mnist)\nframe #21: + 0x226137 (0x55f154510137 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/mnist)\nframe #22: + 0xdaf55 (0x55f1543c4f55 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/mnist)\nframe #23: + 0xaa848 (0x55f154394848 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/mnist)\nframe #24: + 0x9ec37 (0x55f154388c37 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/mnist)\nframe #25: + 0x15bf7e (0x55f154445f7e in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/mnist)\nframe #26: + 0x114627 (0x55f1543fe627 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/mnist)\nframe #27: + 0x15b097 (0x55f154445097 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/mnist)\nframe #28: + 0x1304b7 (0x55f15441a4b7 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/mnist)\nframe #29: + 0x15b81b (0x55f15444581b in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/mnist)\nframe #30: + 0x12f1b7 (0x55f1544191b7 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/mnist)\nframe #31: + 0x11c5d6 (0x55f1544065d6 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/mnist)\nframe #32: + 0xa0170 (0x55f15438a170 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/mnist)\nframe #33: + 0xb54cb (0x55f15439f4cb in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/mnist)\nframe #34: + 0x130afe (0x55f15441aafe in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/mnist)\nframe #35: + 0x133c81 (0x55f15441dc81 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/mnist)\nframe #36: + 0x34b21f (0x55f15463521f in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/mnist)\nframe #37: + 0x133c5a (0x55f15441dc5a in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/mnist)\nframe #38: + 0xa01d1 (0x55f15438a1d1 in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/mnist)\nframe #39: __libc_start_main + 0xf3 (0x7f95a9eb9083 in /lib/x86_64-linux-gnu/libc.so.6)\nframe #40: + 0x7962e (0x55f15436362e in /home/matthew/Projects/Rust/machine_learning/burn/target/debug/mnist)\n")', /home/matthew/.cargo/registry/src/github.com-1ecc6299db9ec823/tch-0.8.0/src/wrappers/tensor_generated.rs:12977:27

Feature description

Put in place a landing page for the framework and attach to it thorough doc/usage.

Feature motivation

It could help in growing our community.

Feature description

For an easy project setup, we should have Dockerfile with all necessary dependency prerequisites.

Feature motivation

With modern IDEs, such as vscode, it's super easy to open a project in Docker. This saves developer's time to setup Burn on their local machine.

(Optional) Suggest a Solution

Add Dockerfile to the root with all required packages and rust tools. Also look into a possibility for CI to use the same Dockerfile to run the tests (for consistency and efficiency).

Feature description

Support groups in convolutions (1D & 2D).

Feature description

Add the sigmoid activation function.

Feature description

Add an (idiomatic) way of computing (mixed) partial derivatives of models.

Feature motivation

Most of the automatic differentiation libraries provide a first-class way of computing gradient of models of form f(X) -> Y. With the introduction of Physics-Informed Neural Networks there is a need for efficient computation of (mixed) partial derivatives (eg. df/dxdy) of models of form f(x, y, z) -> u.

In currently most popular ML libraries, computation of partial derivatives comes with a significant performance overhead and sometimes requires some "hackery".

As far as I see burn does not provide a method for computing partial derivatives of models. I believe that implementing this feature in Rust can benefit from its high performance. And if implemented as a first-class feature it can provide burn a significant advantage over other ML libraries.

(Optional) Suggest a Solution

In terms of API, pytorch and tensorflow provide a method for computing general gradients of form grad(model_output, one_of_model_inputs) -> partial_derivative_of_model_output_wrt_input.

This API is convenient and easy to understand but requires multiple function calls for second and higher derivatives, which can introduce performance overhead - however in Rust this might not be an issue.

Describe the bug

thread 'web::tests::inference' panicked at '
State could not be loaded into model: LoadingError { message: "Can't load module conv1: 
Loading error: Can't load module conv: Loading error: Can't load module bias:
 Loading error: Can't load Option<Tensor> from NamedState" }', 
examples/mnist-inference-web/src/state.rs:37:10

To Reproduce
Here is trained model:
model-6.json.gz

Here is the code:

State loading:

use crate::model::Model;

use burn::module::Module;
use burn::module::State;
use burn_ndarray::NdArrayBackend;

pub type Backend = NdArrayBackend<f32>;

const MODEL_STATE_FILE_NAME: &str = "model-6.json.gz";

pub fn build_and_load_model() -> Model<Backend> {

    let model: Model<Backend> = Model::new();

    let state: State<f32> =
        State::load(MODEL_STATE_FILE_NAME).expect(concat!("Model JSON file could not be loaded"));

    model
        .load(&state)
        .expect("State could not be loaded into model")
}

And here is the model:

use alloc::{format, vec::Vec};

use burn::{
    module::{Module, Param},
    nn,
    tensor::{backend::Backend, Tensor},
};

#[derive(Module, Debug)]
pub struct Model<B: Backend> {
    conv1: Param<ConvBlock<B>>,
    conv2: Param<ConvBlock<B>>,
    conv3: Param<ConvBlock<B>>,
    dropout: nn::Dropout,
    fc1: Param<nn::Linear<B>>,
    fc2: Param<nn::Linear<B>>,
    activation: nn::GELU,
}

const NUM_CLASSES: usize = 10;

impl<B: Backend> Model<B> {
    pub fn new() -> Self {
        let conv1 = ConvBlock::new([1, 8], [3, 3]); // out: [Batch,1,26,26]
        let conv2 = ConvBlock::new([8, 16], [3, 3]); // out: [Batch,1,24x24]
        let conv3 = ConvBlock::new([16, 24], [3, 3]); // out: [Batch,1,22x22]

        let fc1 = nn::Linear::new(&nn::LinearConfig::new(24 * 22 * 22, 32).with_bias(false));
        let fc2 = nn::Linear::new(&nn::LinearConfig::new(32, NUM_CLASSES).with_bias(false));

        let dropout = nn::Dropout::new(&nn::DropoutConfig::new(0.3));

        Self {
            conv1: Param::from(conv1),
            conv2: Param::from(conv2),
            conv3: Param::from(conv3),
            fc1: Param::from(fc1),
            fc2: Param::from(fc2),
            dropout,
            activation: nn::GELU::new(),
        }
    }

    pub fn forward(&self, input: Tensor<B, 3>) -> Tensor<B, 2> {
        let [batch_size, heigth, width] = input.dims();

        let x = input.reshape([batch_size, 1, heigth, width]).detach();
        let x = self.conv1.forward(x);
        let x = self.conv2.forward(x);
        let x = self.conv3.forward(x);

        let x = x.reshape([batch_size, 24 * 22 * 22]);

        let x = self.fc1.forward(x);
        let x = self.activation.forward(x);
        let x = self.dropout.forward(x);

        let x = self.fc2.forward(x);
        x
    }
}

#[derive(Module, Debug)]
pub struct ConvBlock<B: Backend> {
    conv: Param<nn::conv::Conv2d<B>>,
    activation: nn::GELU,
}

impl<B: Backend> ConvBlock<B> {
    pub fn new(channels: [usize; 2], kernel_size: [usize; 2]) -> Self {
        let conv = nn::conv::Conv2d::new(
            &nn::conv::Conv2dConfig::new(channels, kernel_size).with_bias(false),
        );

        Self {
            conv: Param::from(conv),
            activation: nn::GELU::new(),
        }
    }

    pub fn forward(&self, input: Tensor<B, 4>) -> Tensor<B, 4> {
        let x = self.conv.forward(input);
        self.activation.forward(x)
    }
}

Expected behavior
No errors. It worked before recent changes.

Additional context

Describe the bug
Running the training gives NaN for loss:

[Metrics]
  - Train Accuracy: epoch 0.00 % - batch 0.00 %
  - Train Loss: epoch NaN - batch NaN
  - Valid Accuracy: epoch 0.00 % - batch 0.00 %
  - Valid Loss: epoch NaN - batch NaN

[Progress]
  - Iteration 1422 Epoch 2/6
  - iteration [############################>-----------------------------------------------------------------------] (84s)
  - epoch     [#################################>-------------------------------------------------------------------] (4m)

To Reproduce

1. Check out the latest commit: be96160
2. cd to examples/mnist
3. run: cargo run --example mnist --release --features tch-gpu

Expected behavior

Screenshots

Desktop (please complete the following information):

[mnist]$ uname -a
Darwin MacBook-Pro.local 22.3.0 Darwin Kernel Version 22.3.0: 
Mon Jan 30 20:38:37 PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T6000 arm64

Additional context
It worked on a commit before a few days.

Feature description

It would be useful to have "named dimensions" on tensors, i.e. a "Width" dimension for image-like tensors.

Feature motivation

Many bugs when writing ML models come from improperly mixing dimensions. Being able to name the dimensions, and disallowing mixing them improperly, might significantly reduce the risk of bugs.

Suggest a Solution

I haven't thought too much about this, but perhaps using tag structs for each dimension might be the way to go, maybe also with a special struct that is const-generic over a usize that represents dimensions known at compile time.

Feature description

Add Conv2d layer.

Add an example on how to save the trained weights and how to load them.
This should include saving and loading config as well.

Feature description

Support dilation in convolution layers.

Feature description

As part of CI build we should generate Source-based Code Coverage report.

Feature motivation

This is a great way to see if tests are missing for added code. This also will encourage more testing.

(Optional) Suggest a Solution

Use Source-based Code Coverage. Also there is GitHub action that could be used to read coverage reports: https://github.com/marketplace/actions/code-coverage-summary.

Example how a coverage is reported: bincode-org/bincode#619 (comment)

I believe Burn is more than a library or framework and it deserves documentation in the style of book, like Rust (https://doc.rust-lang.org/book/) or Serde (https://serde.rs/).

I would like to see these topic to be included:

1. Introduction and motivation (pretty much from the first blog post: https://burn-rs.github.io/blog/a-case-for-rust-in-deep-learning)
2. Overall design and concepts.
3. Design goals and tenets for the project (kinda a promise what burn will be or evolve, similar to Rust's zero cost abstraction)
4. Explanation of components (training, datasets , etc).
5. Examples (training, inference, etc).
6. Contributing, like how would I add a new backend or new tensor operations.
7. Future road maps.

I am sure there are more but I really would love to dive in and read this book.

Description

It would be beneficial to incorporate SQLite as a supported format for our dataset backend. This could enhance the speed and scalability of data reads.

Motivation

Currently, the data transformed from Huggingface is stored in JSON format. The main limitations of this approach are:

1. Data cannot be accessed by index directly, as it requires line-by-line reading.
2. Storing data as serialized JSON is not particularly efficient.

By supporting SQLite format, these challenges could be effectively addressed.

Feature description

Similar to Pytorch's vision utility, it'd be great if burn provides with interface and a few useful image augmentation utilities, such as random image rotation.

Feature motivation

Instead of having users coming up with their own utils, it would be beneficial for all if Burn's dataset supports data augmentation.

(Optional) Suggest a Solution

For image augmentation we could use these image backends: 1) https://lib.rs/gh/imazen/imageflow/imageflow_core or 2)https://github.com/image-rs/imageproc

Describe the bug

As is, cargo run --example shows burn-tensor/examples/autodiff_simple.rs as the only example. That fails on my M1 Mac, which initially made me think that burn wouldn't work on Apple Silicon.

$ cargo run --example mnist
error: no example target named `mnist`.
Available example targets:
    autodiff_simple

However, it looks like this is the wrong example; as is, the correct example, mnist, doesn't show up as an available example.

Expected behavior

This should work when run from the workspace root:

$ cargo run --example mnist

It takes a little reorganization to get this working. PR incoming.

Desktop (please complete the following information):

• OS: MacOS 13

Additional context

• rust-lang/cargo#7467

The easiest way I've found to create tensors so far is this:

let tensor: Tensor::<B, 2> = Tensor::from_data(Data::<f32 , 2>::from([[1.0f32, 2.0], [3.0, 4.0]]).convert());

This took quite a bit of digging to find out, and I'd expect this to be more straightforward.

Suggestion 1 - Add a method directly to Tensor

// Note: I think it should be possible to infer the type on the LHS here (assuming the backend can be inferred from context)
let tensor: Tensor::<B, 2> = Tensor::from_array([[1.0f32, 2.0], [3.0, 4.0]]);

Not sure if from_array is the best name here. We could also just use the From trait;

Implementation Details

This will require adding multiple impls like the following to Tensor:

impl<B: Backend>  Tensor<B, 1> {
    pub fn from_array<const D1: usize>(data: [B::Elem; D1]) -> Self {
        todo!()
    }
}

impl<B: Backend, >  Tensor<B, 2> {
    pub fn from_array<const D1: usize, const D2: usize>(data: [[B::Elem; D1]; D2]) -> Self {
        todo!()
    }
}

Suggestion 2 - Add a macro

// Note: Again, the LHS type should be inferrable
let tensor: Tensor::<B, 2> = tensor![[1.0f32, 2.0], [3.0, 4.0]];

Feature description

Support updating weight and bias in convolution (1D and 2D).

Feature motivation

Currently weights and biases attributes are initialized with random data. There are no public methods to override them.

(Optional) Suggest a Solution

Create public methods to override the weight and bias attributes. Even better option would be to provide with additional constructors that can take the weights and biases. This is highly useful for inferencing mode. Currently the model assumes the use case for training.

Feature description

Hey, can I implement the Mish activator (of course as well as its unit test) for burn?

Feature motivation

It does have better performance in our projects (implemented by Tensorflow in the past).

(Optional) Suggest a Solution

I wanna do this job myself. Could I:smile:?

Feature description

Introduce the learning rate scheduler concept to control how the learning rate evolves during training.

Feature description

Ability to configure the initial parameters for Conv, Linear and Embedding NN ops.

Feature motivation

Currently when Conv2d constructed, the parameters are initialized with random values. For the inference mode, we do not need randomize the weights. In fact, it's wasteful (especially for large models).

Another motivation is that the NN components needs to be flexible and configurable.

(Optional) Suggest a Solution

There should be universal initializer (probably enum) that could be set in NN configurations. The default should be the same init strategy as it's now implemented.

Feature description

Add MeanPool2D layers.

Feature description

blas-src has a support for Accelerate framework in Mac.

Feature motivation

Instead of only relying on CPU compute in ndarray for mac, Accelerate gives additional boost in training.

(Optional) Suggest a Solution

Enable "accelerate" flag in blas-src dependency.

Feature description

Save model data into more compact and efficient binary format, such as bincode or messagepack

Feature motivation

Saving large data into JSON is not efficient (large files and memory requirements). Additionally, saving float numbers to JSON and getting from is not precise because of some intermediate number type conversion (some may consider as data loss).

The efficiency deserialization (space and speed) becomes more important for applications that want to bundle model data into the code, using include_bytes! macro.

(Optional) Suggest a Solution

In our binary format, we should include format and version information the the output for future compatibility and support.

messagepack has more language support and could be used as our de-facto export format. Bincode seems to work on rust.

Action Item

Remove/modify examples/mnist-inference-web/build.rs once we have built-in facility to export data directly to binary format.

Describe the bug
You can reproduce this on crates.io 0.5, but it should also be reproducible on master.

To replicate the bug, try the following program

use burn::tensor::Distribution;
use burn::tensor::Tensor;
use burn_autodiff::ADBackendDecorator;
use burn_ndarray::NdArrayBackend;

type BackendAlias = NdArrayBackend<f32>;
type ADBackendAlias = ADBackendDecorator<BackendAlias>;

pub fn main() {
    let x: Tensor<ADBackendAlias, 2> = Tensor::random([2, 3], Distribution::Standard);
    let label: Tensor<ADBackendAlias, 2> = Tensor::random([2, 1], Distribution::Standard);

    let weights: Tensor<ADBackendAlias, 2> = Tensor::random([3, 1], Distribution::Standard);
    let bias: Tensor<ADBackendAlias, 2> = Tensor::random([1, 3], Distribution::Standard);

    let y = x.matmul(&weights).add(&bias);
    let loss = y.sub(&label).powf(2.0).sum();

    println!("y shape = {:?}", y.shape());
    println!("{:?}", loss.inner());
    let gradients = loss.backward();
    let weights_grad = weights.grad(&gradients);
    let bias_grad = bias.grad(&gradients);
    println!("{:?}\n{:?}", weights_grad, bias_grad);

}

When you run this, the forward pass works (it prints the shape of y and the value of the loss), but the backward pass produces a runtime error generating the error:

Tensor { value: NdArrayTensor { array: [15.016455], shape=[1], strides=[1], layout=CFcf (0xf), dynamic ndim=1 } }
thread 'main' panicked at 'ndarray: inputs 2 × 3 and 1 × 3 are not compatible for matrix multiplication', <path_omitted>/.cargo/registry/src/github.com-1ecc6299db9ec823/ndarray-0.15.6/src/linalg/impl_linalg.rs:299:5

Expected behavior
The expected behavior is if a forward pass is valid and runs without runtime error, a backward pass should also run without a runtime error.

Additional context
Note that this is not a typical model definition. This looks like an affine function, but it's actually a little different with an unexpected implicit broadcast in the add operation.

That is, for a proper Affine function, we should expect the bias term to have shape [1, 1] (or be reshaped to that). Instead, it has shape [1, 3]

This means the add operation in x.matmul(&weights).add(&bias) is adding tensors with shapes [2, 1] and shapes [1, 3] which produces a tensor with shape [2, 3] by implicitly broadcasting the first tensor along the second dimension to size 3, and implicitly broadcasting the second tensor along the first dimension to size 2.

Based on the error, I believe the issue is the backward pass is trying to push the head gradient of the add (with shape [2, 3]) onto the input with shape [1, 3], but the backward pass for add is not account for the fact that the broadcast could have happened. I think it would effectively have to inject a broadcast operation implicitly inside itself first and then the autodiff should work.