Use a BLAS lib & CLBlast for CPU & GPU speedups. [Enhancement] about llama2.c HOT 4 CLOSED

Yeah... I definitely want to keep the simplicity of the repo. I'll take a look.

from llama2.c.

So I tried using cblas

#include <cblas.h>

void accum(float *a, float *b, int size) {
    cblas_saxpy(size, 1.0f, b, 1.0f, a, 1);
}


void rmsnorm(float* o, float* x, float* weight, int size) {
    float ss = cblas_sdot(size, x, 1.0f, x, 1.0f);
    ss /= size;
    ss += 1e-5f;
    ss = 1.0f / sqrt(ss);

    for (int j = 0; j < size; j++) {
        o[j] = weight[j] * (ss * x[j]);
    }
}

void matmul(float* xout, float* x, float* w, int n, int d) {
    cblas_sgemv(CblasRowMajor, CblasNoTrans, d, n, 1.0f, w, n, x, 1, 0.0f, xout, 1);
}

Gives a decent speedup

from llama2.c.

If CPU(BLAS)&GPU(CLBlast) speedups were applied, I am very interested in benchmarking the program with different quantization model by different edge devices, such as RK3588，Nvidia Jetson Orin Nano, even Android mobile(like Qualcomm Snapdragon 7/8).
As karpathy said , " in this repo we focus on more narrow applications ", I believe deploying the program on edge devices with CPU&GPU speedup can have significant value for many interesting and specific scenarios.

from llama2.c.

Available in a separate fork. Thanks for the discussion. Closing.

from llama2.c.