The cpp-micrograd from 10-zin

cpp-micrograd's People

Contributors

Stargazers

Watchers

cpp-micrograd's Issues

It is not possible to implement multiplication between a double type and a Value type, as well as compound operations like std::shared_ptr<Value> value1_2 = value1 + (value2 * 3.0);

more details are following:
std::shared_ptr value1 = std::make_shared(2.5);
std::shared_ptr value2 = std::make_shared(3.7);
std::shared_ptr value3 = std::make_shared(-3.0);
std::shared_ptr value4 = std::make_shared(1.7);

// Perform addition using the operator+
std::shared_ptr<Value> value1_2 = value1 + (value2 * 3.0);   # error

Memory leaks in c++ impl.

Following snippet highlights the issue:

int main() {
  {
    auto a = std::make_shared<Value>(1);
    auto b = std::make_shared<Value>(2);
    auto c = a + b;
  }
  // a, b and c are not destroyed ...
}

Issue seems to be with the captured variables of _backward lambda:

std::shared_ptr<Value> Value::operator+(const std::shared_ptr<Value>& other) {
    auto out_prev = std::unordered_set<std::shared_ptr<Value>>{shared_from_this(), other};

    auto out = std::make_shared<Value>(data + other->data, out_prev, "+");

    out->_backward = [this, other, out] { // We're capturing out, hence the lambda shares ownership of out!
        grad += out->grad;
        other->grad += out->grad;
    };
    return out;
}

This leads to a situation where the lambda inside out shares ownership of out. So, _backward is not destroyed until until out is destroyed and out is not destroyed until _backward is destroyed, hence nothing is destroyed.

To fix, the lambda should only take weak ownership.

std::shared_ptr<Value> Value::operator+(const std::shared_ptr<Value>& other) {
    auto out_prev = std::unordered_set<std::shared_ptr<Value>>{shared_from_this(), other};

    auto out = std::make_shared<Value>(data + other->data, out_prev, "+");
    Value* weak_ref = out.get();

    out->_backward = [this, other, weak_ref] {
        grad += weak_ref->grad;
        other->grad += weak_ref->grad;
    };
    return out;
}