Giter VIP home page Giter VIP logo

plex's People

Contributors

acashmoney avatar adamgoyer avatar alabdao avatar developerfred avatar flowscience avatar hevans66 avatar jeffmerrick avatar lilyhg avatar niklastr avatar phiflei avatar saisamarth123 avatar supraja-968 avatar thetechnocrat-dev avatar winternewt avatar zorlin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

plex's Issues

GCP GPU Access

Waiting to hear back from GCP re: quota increase by EOD 23 Jan

client dependent bafy CIDs for web3.storage vs. kubo

when using the web3.storage CLI I am receiving different bafys for a given file when compared to kubo. We can try this with this repo here:

labdao % w3 put ganglia/README.md 
# Packed 1 file (0.0MB)
# bafybeiaj7fh6mh2yfzlpis4aozddm2lcsicthekdav4b3lwvfjpvtn2dty
⁂ Stored 1 file
⁂ https://dweb.link/ipfs/bafybeiaj7fh6mh2yfzlpis4aozddm2lcsicthekdav4b3lwvfjpvtn2dty
labdao % ipfs add -r --cid-version=1 ganglia/README.md
added bafkreibhdu66zohpk33uraa4p5u4zfwbv2rqq7edt43xnhke4wwaulwmoq README.md
 1.46 KiB / 1.46 KiB [===========================================================================================================================] 100.00%

labdao % ipfs pin remote add --background --service=pinata-api bafkreibhdu66zohpk33uraa4p5u4zfwbv2rqq7edt43xnhke4wwaulwmoq
CID:    bafkreibhdu66zohpk33uraa4p5u4zfwbv2rqq7edt43xnhke4wwaulwmoq
Name:   
Status: queued

create compute infrastructure repo

This repo will hold code for managing and deploying our compute node infrastructure.

This will most likely end up including code for:

  • installing nodes
  • deploying nodes
  • whatever queue/pub-sub system we come up with for distributing jobs amongst a cluster of nodes
  • base docker containers or wrapper scripts for standardizing running of jobs
  • workflows for chaining jobs

The first code will be the install.sh script defined in ticket 17.

Possible names;

  • labdao/compute-infra
  • labdao/ganglia (decentralized "mini-brains" in an octopus or more generally clusters of neurons in peripherally nervous systems)

nvidia-container-cli: initialization error: nvml error: driver/library version mismatch: unknown

testing on gpu-prototyping instance

Testing Nvidia Container Toolkit Install
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: nvml error: driver/library version mismatch: unknown.
ERRO[0000] error waiting for container: context canceled 
````

modify putfile unit test for random file generation

The following code is in draft. It generates a random file. Then it calculates the expected cid and compares this with the output of putFile, which returns a cid upon execution. The cid comparison outputs an error, indicating the expected and actual cids differ.

package main

import (
	"crypto/rand"
	"fmt"
	"io/ioutil"
	"os"
	"testing"

	cid "github.com/ipfs/go-cid"
	mh "github.com/multiformats/go-multihash"
	"github.com/web3-storage/go-w3s-client"
)

func TestPutFile(t *testing.T) {
	client, err := w3s.NewClient(
		w3s.WithEndpoint("https://api.web3.storage"),
		w3s.WithToken(testToken),
	)
	if err != nil {
		t.Fatalf("error creating client: %v", err)
	}

	fileContent := make([]byte, 1024)
	if _, err := rand.Read(fileContent); err != nil {
		t.Fatalf("error generating random bytes: %v", err)
	}

	file, err := ioutil.TempFile("", "")
	if err != nil {
		t.Fatalf("error creating temp file: %v", err)
	}
	defer file.Close()
	defer os.Remove(file.Name())

	if _, err := file.Write(fileContent); err != nil {
		t.Fatalf("error writing to file: %v", err)
	}

	fileBytes, err := ioutil.ReadFile(file.Name())
	if err != nil {
		t.Fatalf("error reading file: %v", err)
	}

	// calculate predicted CID
	// codec 0x70 is dag-pb
	pref := cid.Prefix{
		Version:  1,
		Codec:    uint64(0x70),
		MhType:   mh.SHA2_256,
		MhLength: -1,
	}
	calculatedCid, err := pref.Sum(fileContent)
	if err != nil {
		t.Fatalf("error calculating CID: %v", err)
	}

	// validation that file content and file bytes are the same
	fmt.Printf("fileContent length: %d, first few elements: % x\n", len(fileContent), fileContent[:10])
	fmt.Printf("file length: %d, first few elements: % x\n", len(fileBytes), fileBytes[:10])

	// putFile takes in args of type fs.File, NOT the bytes of the file
	actualCid := putFile(client, file)

	if !calculatedCid.Equals(actualCid) {
		t.Fatalf(`unmatching cids
			expected CID: %s
			actual CID: %s`, calculatedCid, actualCid,
		)
	}
}

Example error logs

fileContent length: 1024, first few elements: 27 25 bc be 8c f5 66 15 a8 6b
file length: 1024, first few elements: 27 25 bc be 8c f5 66 15 a8 6b
Uploading to IPFS via web3.storage... 
CID: bafybeidk2alhga7fztbtiyult7hsawnfhawtspy5a6xzzvjtaxavvfn2fm
--- FAIL: TestPutFile (1.09s)
    ipfs_test.go:77: unmatching cids
                                expected CID: bafybeieqzyhpmacugzcn7mrh6pth6xakrxxudvbn3l5kvdtderadua27wy
                                actual CID: bafybeidk2alhga7fztbtiyult7hsawnfhawtspy5a6xzzvjtaxavvfn2fm

Current hypothesis is that w3s.Put function modifies the file before pinning. CID references below, where the Multihash digests differ in spite of the file contents being identical as shown above.

Look into CAR wrapping and generating manually. (lib from Web3S https://github.com/web3-storage/ipfs-car)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.