Giter VIP home page Giter VIP logo

raybnn_dataloader's Introduction

RayBNN_DataLoader

Data Loader for RayBNN

Read CSV, numpy, and binary files to Rust vectors of f16, f32, f64, u8, u16, u32, u64, i8, i16, i32, i64

Read CSV, numpy, and binary files to Arrayfire GPU arrays of f16, f32, f64, u8, u16, u32, u64, i8, i16, i32, i64

Install Arrayfire

Install the Arrayfire 3.9.0 binaries at https://arrayfire.com/binaries/

or build from source https://github.com/arrayfire/arrayfire/wiki/Getting-ArrayFire

Add to Cargo.toml

arrayfire = { version = "3.8.1", package = "arrayfire_fork" }
rayon = "1.10.0"
num = "0.4.3"
num-traits = "0.2.19"
half = { version = "2.4.1" , features = ["num-traits"] }
npyz = "0.8.3"
RayBNN_DataLoader = "2.0.3"

List of Examples

Read a CSV file to a floating point 64 bit CPU Vector

The vector is completely flat

let (mut cpu_vector,metadata) = RayBNN_DataLoader::Dataset::CSV::file_to_vec_cpu::<f64>(
    "./test_data/read_test.dat"
);

Read a CSV file to a integer 64 bit CPU Vector

The vector is completely flat

let (mut cpu_vector,metadata) = RayBNN_DataLoader::Dataset::CSV::file_to_vec_cpu::<i64>(
    "./test_data/read_test2.dat"
);

Read a CSV file to a floating point 64 bit arrayfire

The array is 2D existing in GPU or OpenCL

let read_test = RayBNN_DataLoader::Dataset::CSV::file_to_arrayfire::<f64>(
    "./test_data/read_test.dat"
);

Read a CSV file to a floating point 64 bit HashMap

let hashdata = RayBNN_DataLoader::Dataset::CSV::file_to_hash_cpu::<f64>(
    "./test_data/dataloader.csv",
    7,
    5
);

Read a CSV file to a floating point 64 bit HashMap with Arrayfire

let arr_dims = arrayfire::Dim4::new(&[5, 7, 1, 1]);
let (hashdata,metadata) = RayBNN_DataLoader::Dataset::CSV::file_to_hash_arrayfire::<f64>(
    "./test_data/dataloader.csv",
    arr_dims,
);

Write a float 32 bit CPU vector to CSV file

let mut metadata: HashMap<String,u64> = HashMap::new();

metadata.insert("dim0", 11);
metadata.insert("dim1", 3);

RayBNN_DataLoader::Dataset::CSV::write_vec_cpu_to_csv::<f32>(
    "./randvec2.csv",
    &randvec,
    &metadata
);

Write a float 64 bit arrayfire to CSV file

RayBNN_DataLoader::Dataset::CSV::write_arrayfire_to_csv::<f64>(
    "./randvec.csv",
    &arr
);

Read YAML Model Information

let mut modeldata_string:  HashMap<String, String> = HashMap::new();
let mut modeldata_float:  HashMap<String, f64> = HashMap::new();
let mut modeldata_int:  HashMap<String, u64> = HashMap::new();

RayBNN_DataLoader::Model::YAML::read(
    "./test_data/test.yaml",

    &mut modeldata_string,
    &mut modeldata_float,
    &mut modeldata_int,
);

Write YAML Model Information

RayBNN_DataLoader::Model::YAML::write(
    "./sample.yaml",

    &modeldata_string,
    &modeldata_float,
    &modeldata_int,
);

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.