Giter VIP home page Giter VIP logo

deoss's Introduction

DeOSS
GitHub license Go Reference build Go Report Card

DeOSS ( Decentralized Object Storage Service ) is a decentralized object-based mass storage service that provides low-cost, secure and scalable distributed data storage services for the web3 domain.

πŸ“ Reporting a Vulnerability

If you find any system errors or you have better suggestions, please submit an issue or PR, or join the CESS discord to communicate with us.

πŸ“’ Announcement

CESS test network rpc endpoints

wss://testnet-rpc.cess.cloud/ws/

CESS test network bootstrap node

_dnsaddr.boot-miner-testnet.cess.cloud

CESS test network public gateway

Address http://deoss-pub-gateway.cess.cloud/
Account cXhwBytXqrZLr1qM5NHJhCzEMckSTzNKw17ci2aHft6ETSQm9

🚰 CESS test network faucet

https://testnet-faucet.cess.cloud/

⚠️ Attention

The following commands are executed with root privileges, if the prompt Permission denied appears, you need to switch to root privileges, or add sudo at the top of these commands.

βš™ System configuration

System requirements

  • Linux 64-bit Intel/AMD

Install application tools

For the Debian and ubuntu families of linux systems:

apt install git curl wget vim util-linux -y

For the Fedora, RedHat and CentOS families of linux systems:

yum install git curl wget vim util-linux -y

Firewall configuration

By default, DeOSS uses port 8080 to listen for incoming connections and internally uses port 4001 for p2p communication, if your platform blocks these two ports by default, you may need to enable access to these port.

ufw

For hosts with ufw enabled (Debian, Ubuntu, etc.), you can use the ufw command to allow traffic to flow to specific ports. Use the following command to allow access to a port:

ufw allow 8080
ufw allow 4001

firewall-cmd

For hosts with firewall-cmd enabled (CentOS), you can use the firewall-cmd command to allow traffic on specific ports. Use the following command to allow access to a port:

firewall-cmd --get-active-zones

This command gets the active zone(s). Now, apply port rules to the relevant zones returned above. For example if the zone is public, use

firewall-cmd --zone=public --add-port=4001/tcp --permanent
firewall-cmd --zone=public --add-port=8080/tcp --permanent

Note that permanent makes sure the rules are persistent across firewall start, restart or reload. Finally reload the firewall for changes to take effect.

firewall-cmd --reload

iptables

For hosts with iptables enabled (RHEL, CentOS, etc.), you can use the iptables command to enable all traffic to a specific port. Use the following command to allow access to a port:

iptables -A INPUT -p tcp --dport 4001 -j ACCEPT
iptables -A INPUT -p tcp --dport 8080 -j ACCEPT
service iptables restart

πŸ— Get the binary program

Method one

Download the latest release of the binary application directly at:

wget https://github.com/CESSProject/DeOSS/releases/download/v0.3.6/DeOSS0.3.6.linux-amd64.tar.gz

Method two

Compile the binary program from the DeOSS source code and follow the process as follows:

1) install go

DeOSS requires Go 1.21, See the official Golang installation instructions.

Open go mod mode:

go env -w GO111MODULE="on"

Users in China can add go proxy to speed up the download:

go env -w GOPROXY="https://goproxy.cn,direct"

2) clone code

git clone https://github.com/CESSProject/DeOSS.git

3) compile code

cd DeOSS/
go build -o deoss cmd/main.go

4) Grant execute permission

chmod +x deoss

πŸ’° Configure Wallet

Step 1: create a wallet account

The wallet is your unique identity in the cess network, it allows you to do transactions with the cess chain, provided that you have some balance in your wallet.

Please refer to Create-CESS-Wallet to create your cess wallet.

Step 2: Recharge your wallet

If you are using the test network, Please join the CESS discord to get it for free. If you are using the official network, please buy CESS tokens.

Configuration file

Use deoss to generate configuration file templates directly in the current directory:

./deoss config

The contents of the configuration file template are as follows. The contents inside are the defaults and you will need to modify them as appropriate. By default, deoss uses conf.yaml in the current directory as the runtime configuration file. You can use -c or -config to specify the location of the configuration file.

# RPC endpoint of the chain node
Rpc:
  # test network
  - "wss://testnet-rpc.cess.cloud/ws/"
# bootstrap nodes
Boot:
  # test network
  - "_dnsaddr.boot-miner-testnet.cess.cloud"
# signature account mnemonic
Mnemonic: "xxx ... xxx"
# service workspace
Workspace: /
# P2P communication port
P2P_Port: 4001
# service listening port
HTTP_Port: 8080
# Access mode: public / private
# In public mode, only users in Accounts can't access it. 
# In private mode, only users in Accounts can access it.
Access: public
# Account black/white list
Accounts:
# If you want to expose your oss service, please configure its domain name
Domain:

# User Files Cacher config
# File cache size, default 512G, (unit is byte)
CacheSize: 10
# File cache expiration time, default 3 hour (unit is minutes)
Expiration:
# Directory to store file cache, default path: Workspace/filecache/
CacheDir:

# Storage Node Selector config
# Used to find better storage node partners for DeOSS to upload or download files
# Two strategies for using your specified storage nodes, "priority" or "fixed", default is "priority"
SelectStrategy: 
# JSON file used to specify the storage node. If it does not exist, it will be automatically created.
# You can configure which storage nodes to use or not use in this file.
NodeFilePath:
# Maximum number of storage nodes allowed for long-term cooperation, default 120
MaxNodeNum:
# Maximum tolerable TTL for communication with storage nodes, default 500 ms (unit is milliseconds)
MaxTTL:
# Available storage node list refresh time, default 4 hours (unit is hours)
RefreshTime:

🟒 Usage for DeOSS

start deoss service

Backend operation mode (the default configuration file is in the current directory):

nohup ./deoss run 2>&1 &

view deoss status

./deoss stat
+-------------------+------------------------------------------------------+
| role              | deoss                                                |
| peer id           | 12D3KooWFAcDpT7vTtbsS361P14z8LpgxPMRywQr19sAdNfdDBYE |
| signature account | cXhwBytXqrZLr1qM5NHJhCzEMckSTzNKw17ci2aHft6ETSQm9    |
| domain name       | http://deoss-pub-gateway.cess.cloud/                 |
+-------------------+------------------------------------------------------+

exit the cess network

It is generally not recommended to use this command:

./deoss exit

πŸ“– Usage for API

The public API endpoint URL of DeOSS is the server you deploy, All endpoints described in this document should be made relative to this root URL,The following example uses URL instead.

Before using DeOSS, you must authorize it as follows:

  1. Create a wallet account and fund it, refer to Configure Wallet

  2. Purchase a territory

  3. Authorize the use right to DeOSS:Authorize

Identity signature

Calling some APIs requires authentication of your identity. In web3, your wallet is your identity. Generate your signature data in the block browser, and then add your signature information in the API request header to authenticate your identity. Please refer to the signature method.

The authentication information you need to add in the header:

Key Description Example
Account wallet account cX...
Message signed message ...
Signature signature 0x...

Create a bucket

PUT /bucket

The put bucket interface is used to create a bucket. When uploading files, the bucket must be specified for storage.

  • Request Header
key value
Bucket created bucket name

Identity signature required: yes

  • Request example
# curl -X PUT URL/ -H "Bucket: bucket_name" -H "Account: cX..." -H "Message: ..." -H "Signature: 0x..."

Upload a file

PUT /file

The put file interface is used to upload files to the cess system. You need to submit the file as form data and use provide the specific field. If the upload is successful, you will get the fid of the file. If you want to encrypt your file, you can specify the cipher field in the header and enter your password (the length cannot exceed 32 characters), and the system will automatically encrypt it.

  • Request Header
key description
Bucket bucket name
Territory territory name
Cipher(optional) cipher

Identity signature required: yes

  • Request Body
key value
file file[binary]
  • Request example
# curl -X PUT URL/file -F '[email protected];type=application/octet-stream' -H "Bucket: bucket_name" -H "Territory: territory_name" -H "Account: cX..." -H "Message: ..." -H "Signature: 0x..."

Upload an object

PUT /object

This interface is used to upload an object, you can write what you want to store directly in the body instead of specifying a file. If the upload is successful, you will get the fid of the object. if you want to encrypt the object, you can specify the "Cipher" field in the header of the request and enter a password (the length can not be more than 32 characters), the system will encrypt it automatically.

  • Request Header
key description
Bucket bucket name
Territory territory name
Cipher(optional) cipher

Identity signature required: yes

  • Request Body

[content]

  • Request example
# curl -X PUT URL/object --data "content" -H "Bucket: bucket_name" -H "Territory: territory_name" -H "Account: cX..." -H "Message: ..." -H "Signature: 0x..."

Chunked upload

PUT /chunks

Compared with uploading the entire file directly, resumable upload has some more parameter requirements, but has the same return result. At the same time, the uploaded file can also be encrypted.

  • Request Header
key description
Bucket stored bucket name
Territory territory name
Cipher(optional) your cipher
FileName file name or alias
BlockNumber The number of chunks the file is to be divided into
BlockIndex index of chunk to be uploaded, [0,BlockNumber)
TotalSize the byte size of the file, the sum of the sizes of all chunks

Identity signature required: yes

  • Request Body
key value
file file[binary]
  • Request example
# curl -X PUT URL/chunks -F 'file=@test-chunk0;type=application/octet-stream' -H "Bucket: bucket_name" -H "Territory: territory_name" -H "Account: cX..." -H "Message: ..." -H "Signature: 0x... -H FileName: test.log -H BlockNumber: 5 -H BlockIndex: 0 -H TotalSize: 1000"

Download a file

GET /download/{fid}

This interface is used to download a file with a specified fid. If you encrypted the file when you uploaded it, you also need to tell the gateway your cipher to decrypt your file.

  • Request Header
key value
Cipher(optional) cipher
  • Request example
# curl -X GET -o <save_file> URL/download/<fid>

Preview a file

GET /open/{fid}

This interface is used to preview a file, it has two prerequisites: one is that the file is not encrypted, and the other is that the file format supports preview.

  • Request example

Open in browser: URL/open/

Delete a file

The delete file interface is used for delete a file.

DELETE /file/{fid}
  • Request Header

Identity signature required: yes

  • Request example
# curl -X DELETE URL/file/<fid> -H "Account: cX..." -H "Message: ..." -H "Signature: 0x..."

Delete a bucket

The delete bucket interface is used for delete a bucket, all files in the bucket will also be deleted together.

DELETE /bucket/{bucket_name}
  • Request Header

Identity signature required: yes

  • Request example
# curl -X DELETE URL/bucket/<bucket_name> -H "Account: cX..." -H "Message: ..." -H "Signature: 0x..."

View bucket info

GET /bucket

This interface is used to view bucket information, including the number of stored files and file IDs.

  • Request Header
key value
Account cX...
Bucket bucket_name
  • Request example
# curl -X GET URL/bucket -H "Account: cX..." -H "Bucket: bucket_name"

View bucket list

GET /bucket
  • Request Header
key value
Account cX...

This interface is used to view all buckets.

  • Request example
# curl -X GET URL/bucket -H "Account: cX..."

View file metadata

GET /metadata/{fid}

This interface is used to view the basic information of a file.

  • Request example
# curl -X GET URL/metadata/<fid>

View version

GET /version

This interface is used to view the version number of the gateway.

  • Request example
# curl -X GET URL/version

License

Licensed under Apache 2.0

deoss's People

Contributors

astafrode avatar jiuquxzy avatar sunway910 avatar tehsunnliu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

deoss's Issues

"rpc err: connection failed" While running a local DeOSS server.

Hi, we have our DeOSS gateway setup locally where we have our CESS rpc node running locally at ws://localhost:9944/ on the same machine.
Whenever we upload a file the DeOSS throws error

[GIN] 2024/05/13 - 04:49:20 | 200 |  1.703779904s |   122.179.90.62 | PUT      "/"
XX 2024-05-13 04:49:21 [ws://localhost:9944/] rpc err: connection failed
XX 2024-05-13 04:49:21 rpc err: connection failed
XX 2024-05-13 04:49:31 [] rpc err: connection failed
++ 2024-05-13 04:49:32 [ws://localhost:9944/] rpc reconnection successful
image

The DeOSS returns a file ID to the user however, the file doesn't get distributed to the miners, it can't be queried in CESS explorer
"fileBank > file" nor in "fileBank > dealmap".

The solution to the problem was to add some tokens to the DeOSS gateway wallet account address. Once enough token was added the file started showing up in deamap and also in the file upon successful distribution to the miners.

My concern is it will be helpful if a correct error message is reported by either DeOSS or the rpc "chain" logs stating that the DeOSS gateway doesn't have enough TCESS instead of RPC connection dropping every time.

Looking forward for this to be fixed in the upcoming release.
Thank you.

DeOSS not uploading file to the miners that is already in the track directory.

Recently, I tried to upload a 1GiB file to my self-hosted DeOSS gateway. The gateway tried to upload the file segments to the miners for a while and then stopped. The file can not be found in the dealMap either.
I can see that the file is still in the /track directory with FID: 34b9a4787bb567035f99add437938cbbf89087da4f8d6f9ba3844645349fcec0

image

When I restart the DeOSS, it resumes uploading other files but doesn't try to upload 34b9a4787bb567035f99add437938cbbf89087da4f8d6f9ba3844645349fcec0 this particular file.

Timeout when download file

Cannot download file from http://deoss-pub-gateway.cess.cloud/ these days, it was good before, please help to check

    const res = await axios.get('http://deoss-pub-gateway.cess.cloud/75a3d8fad3d5d2d26b4530c4aefeca8dc7f871ed61175834da285b581d44307f', {
        headers: {
            Operation: "download",
            Account: "cXhwBytXqrZLr1qM5NHJhCzEMckSTzNKw17ci2aHft6ETSQm9",
        },
        responseType: "stream",
        onDownloadProgress: (progressEvent) => {
            if (progressEvent === undefined || progressEvent.total === undefined) return
            const percentCompleted = Math.round((progressEvent.loaded * 100) / progressEvent.total);
            console.log(`progress: ${percentCompleted}%`);
        },
    })
    const outS = fs.createWriteStream('./temp.mp4')
    res.data.pipe(outS)
    console.log(res.data)

Response:

esponse: {
    status: 504,
    statusText: 'Gateway Time-out',
    headers: Object [AxiosHeaders] {
      server: 'nginx/1.22.1',
      date: 'Wed, 15 May 2024 09:30:46 GMT',
      'content-type': 'text/html',
      'content-length': '4020',
      connection: 'close',
      etag: '"639e5807-fb4"'
    },
...

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.