arweaveteam / arweave Goto Github PK

View Code? Open in Web Editor NEW

897.0 62.0 185.0 107.41 MB

The Arweave server and App Developer Toolkit.

Home Page: https://www.arweave.org

License: GNU General Public License v2.0

Erlang 94.10% Makefile 0.14% Shell 1.17% HTML 0.22% C 1.99% C++ 1.31% Nix 1.08%

arweave's Introduction

Arweave Server

This is the repository for the official Erlang implementation of the Arweave protocol and a gateway implementation.

Arweave is a distributed, cryptographically verified permanent archive built on a cryptocurrency that aims to, for the first time, provide feasible data permanence. By leveraging our novel Blockweave datastructure, data is stored in a decentralised, peer-to-peer manner where miners are incentivised to store rare data.

Getting Started

Download and extract the latest archive for your platform on the release page, then run the included bin/start script to get started.

For more information, refer to the mining guide.

Building from source

Requirements

The full arweave node functionality is only supported on Linux, but it is possible to run a VDF Server on MacOS. Refer to the mining VDF guide for more information on running your own VDF server.

General requirements:

OpenSSL 1.1.1+
OpenSSL development headers
GCC or Clang (GCC 8+ recommended)
Erlang OTP v24, with OpenSSL support
GNU Make
CMake (CMake version > 3.10.0)
SQLite3 header
GNU MP

To install the dependencies on Ubuntu 22 (recommended):

sudo apt install libssl-dev libgmp-dev libsqlite3-dev make cmake gcc g++

On some systems you might need to install libncurses-dev.

To install the dependencies on MacOS:

Install Homebrew
Install dependencies

brew install gmp [email protected] erlang cmake pkg-config

Notes:

This process has only been tested on a fresh install of MacOS Ventura running on a Mac Mini M2. It may or may not work on other configurations.
We have not validated mining or packing on MacOS, but as of May, 2024 the M2 provides the fastest known VDF implementation and so makes a good candidate for VDF Servers.

Download the repo:

$ git clone --recursive https://github.com/ArweaveTeam/arweave.git
$ cd arweave

Increase the open file limits.

Run in the development mode:

./arweave-server \
peer ams-1.eu-central-1.arweave.net \
peer fra-1.eu-central-2.arweave.net \
peer sgp-1.ap-central-2.arweave.net \
peer blr-1.ap-central-1.arweave.net \
peer sfo-1.na-west-1.arweave.net

Make a production build:

$ ./rebar3 as prod tar

You will then find the gzipped tarball at _build/prod/rel/arweave/arweave-x.y.z.tar.gz.

Testnet

To make a testnet build, run:

$ ./rebar3 as testnet tar

The tarball is created at _build/testnet/rel/arweave/arweave-x.y.z.tar.gz.

You can join the public testnet now:

./bin/start peer testnet-1.arweave.net peer testnet-2.arweave.net peer testnet-3.arweave.net

We recommend you do not use your mainnet mining address on testnet. Also, do not join the testnet from the mainnet machine.

Starting New Weave

To start a new weave, create a new data directory

mkdir -p localnet_data_dir

, create a wallet:

./bin/create-wallet localnet_data_dir

, and run:

$ ./bin/start-localnet init data_dir <your-data-dir> mining_addr <your-mining-addr>
storage_module 0,<your-mining-addr> mine

The given address (if none is specified, one will be generated for you) will be assigned 1_000_000_000_000 AR in the new weave.

The network name will be arweave.localnet. You can not start the same node again with the init option unless you clean the data directory - you need to either restart with the start_from_block_index option or specify a peer from the same Arweave network via peer <peer>. Note that the state is only persisted every 50 blocks so if you restart the node without peers via start_from_block_index before reaching the height 50, it will go back to the genesis block.

As with mainnet peers, each peer must be run in its own physical or virtual environment (e.g. on its own machine or in its own container or virtual machine). If you try to run two nodes within the same environment you will get an error like Protocol 'inet_tcp': the name [email protected] seems to be in use by another Erlang node

When POST'ing transactions to your localnet make sure to include the X-Network: arweave.localnet header. If the header is omitted, the mainnet network will be assumed and the request will fail.

Configuring localnet

See the localnet section in rebar.config for instructions on changing network constants for your localnet.

Contributing

Make sure to have the build requirements installed.

Clone the repo and initialize the Git submodules:

$ git clone --recursive https://github.com/ArweaveTeam/arweave.git

Running the tests

$ bin/test

Running a shell

$ bin/shell

bin/test and bin/shell launch two connected Erlang VMs in distributed mode. The master VM runs an HTTP server on the port 1984. The slave VM uses the port 1983. The data folders are data_test_master and data_test_slave respectively. The tests that do not depend on two VMs are run against the master VM.

Run a specific test (the shell offers autocompletion):

([email protected])1> eunit:test(ar_fork_recovery_tests:height_plus_one_fork_recovery_test_()).

If it fails, the nodes keep running so you can inspect them through Erlang shell or HTTP API. The logs from both nodes are collected in logs/. They are rotated so you probably want to consult the latest modified [email protected].* and [email protected].* files first - ls -lat logs/.

See CONTRIBUTING.md for more information.

HTTP API

You can find documentation regarding our HTTP interface here.

Contact

If you have questions or comments about Arweave you can get in touch by finding us on Twitter, Reddit, Discord or by emailing us at [email protected].

For more information about the Arweave project visit https://www.arweave.org or have a look at our yellow paper.

License

The Arweave project is released under GNU General Public License v2.0. See LICENSE for full license conditions.

Arweave Bug Bounty Program

Arweave core team has initiated an Arweave bug bounty program, with a maximum bounty of up to USD 1,000,000. The program is focused on discovering potential technical vulnerabilities and strengthening Arweave core protocol security.

The Arweave core team puts security as its top priority and has dedicated resources to ensure high incentives to attract the community at large to evaluate and safeguard the ecosystem. Whilst building Arweave, the team has engaged with industry-leading cybersecurity audit firms specializing in Blockchain Security to help secure the codebase of Arweave protocol.

We encourage developers, whitehat hackers to participate, evaluate the code base and hunt for bugs, especially on issues that could potentially put users’ funds or data at risk. In exchange for a responsibly disclosed bug, the bug bounty program will reward up to USD 1,000,000 (paid in $AR tokens) based on the vulnerability severity level, at the discretion of the Arweave team. Please email us at [email protected] to get in touch.

arweave's People

Contributors

Stargazers

Watchers

Forkers

soros1321 hengestone enilsen16 redeyespb jasp83 toknapp shashank-ezdiagno bioharz majastanislawska samcamwilliams zhaohaijun dev43 foxfromabyss arweave-kyle antonshevchenko84 iscry eartharchive raphexion maxmellen dwaas mdaiter mkg20001 wenlong0913 demshine mazzz-zzz tukudev ayanda-d garrettian robertdigital ltlmaps yue169 sherlock-shi-x edescourtis koder-abc123 pyaesone1933 phunk87 sidneyy domhaobaobao doytsujin crcrcr999 theloneronin goomario onesk cromatikap alexander-morris ecolog djwhitt strogo andychenw liuchengxu 7e4 creativewebmatrixsolutions artdgn luisriverag allchain spingreekgod strongpool netluxe harrisin2037 virdpool baldur10 parvinderparashar singularitypostman libreearth daonow daocommunity leowutao 0xawy hlolli nextbtc arweave-sophie michaelbyt3s opusderic vporton jumpersdevice mediafoundation samonh petedram kenman79 deepanshujha eizperchain 0xsirah 15012700225 tomzhang edwardt strategist922 shenzhen-cloudatawalk-technology-co-ltd rosebush qizhou z0rlord qxvge domaintulen blokzinciri douganderson444 alexfreska ndeadcoin ocean00 wangwangdogzi daotlresearch crypto-forks

arweave's Issues

make clean doesn't remove object files

To recompile properly, you have to remove object files manually.

Elixir example

Hello!

I really think this project is really great. I have an application in mind, is there any documentation pointing to the elixir way of doing things?

Kind regards!

Error handling when sending transaction is confusing

When sending an incorrect transaction to the network, the information why the transaction is denied is extremely misleading. Example:

Using the arweave-go SDK, i create a transaction with the owner field being doubly base64url encoded the response is:

tx_too_cheap tag_field_illegally_specified tx_id_not_valid which is not true, it's simply the owner field that is not valid.

This led me to try to minimize all these errors thinking I was getting closer to the right encoding of the fields, but turns that it was the opposite (the more errors the better!!!!) wich is super unintuitive.

Print mining reward when block is accepted

Current mining log gives feedback on block generation/acceptance

src/ar_miner_log.erl

log("[Stage 3/3] Your block ~s was accepted by the network!", [ar_util:encode(BH)]).

Would be nice if this also explicitly printed the reward the miner gained for this block.

GET /price/SizeInBytes[/Addr] should catch non-integer SizeInBytes

Both endpoints currently return 500. eg http://167.99.249.43:1984/price/121x23/qweasd

See

https://github.com/ArweaveTeam/arweave/blob/master/src/ar_http_iface.erl#L325-L343

blocksize

I'm mining ar for few days(mining address: qFzYCl02V2De6b0KVvBKThRI01Xw84r_XVZ1_0XXt98) and I have noticed that the blocks are full with ALL ar addresses. Seriously? That doesn't scale at all. I can easily spam the hole network with creating new low balance Addresses.
If nobody write me back, I will stress test the network the next few days.
{"network":"arweave.N.1","version":3,"height":14590,"current":"iEP_8U9N4wSvjt59AFVZeWAqIYEhNsu7KDc5CYdFxcYuHv5AhqaC94PYSDqLh_60","blocks":15842,"peers":83,"queue_length":0}
block 14596_WJMN57VvUQII2CXPqGFPY0KX8HaRcDVZIvqox58RIm2Lm_DXS9x_7OJYWz7dWIxm has a size of:
1247002 bite
that equals 1.2MB!
for one block!
Nobody is using AR and we already creating 1.2MB blocks?
every 120 sec?
the blockchain size is already over 10GB!

GET /blablabla...(at least 43 chars) returns 500

The handler that assumes a long string is a tx hash further assumes that either the string is exactly 43 characters, or the 44th character is "." and there is at least one character after the dot. Otherwise the server crashes.

The case clause at lines 639-42 should have a catchall to return 4xx instead of 500.

Add log rotation to session log

Today new sessions logs get created when starting a node and grow during uptime. During long uptimes this can lead to quite large and unusable logs.

This has to be changed into configurable log rotation intervals. While the active and the first archived log should be uncompressed all other ones have to be compressed.

Configuration arguments:

Log rotation interval
Log rotation size (to be discussed: rotate already when size is reached before end of interval)
Number or archived logs before getting deleted or moved

Invalid JSON issue on join

Occasionally miners will receive the following error when getting a block from peers:

=ERROR REPORT==== 23-Oct-2018::08:15:23 ===
Error in process <0.951.0> with exit value:
{{nocatch,{error,{1,invalid_json}}},
 [{jiffy,decode,2,[{file,"lib/jiffy/src/jiffy.erl"},{line,71}]},
  {ar_serialize,json_struct_to_block,1,
                [{file,"src/ar_serialize.erl"},{line,146}]},
  {ar_storage,do_read_block,2,[{file,"src/ar_storage.erl"},{line,227}]},
  {ar_node_utils,get_full_block,3,[{file,"src/ar_node_utils.erl"},{line,41}]},
  {ar_join,get_block_and_trail,4,[{file,"src/ar_join.erl"},{line,159}]},
  {ar_join,do_join,3,[{file,"src/ar_join.erl"},{line,66}]}]}

HTTP 206 returned on tx/{id}/data.html for pending transaction

When requesting the data for a pending transaction, a 206 status code is returned with an empty body, instead of HTTP 202. The 206 response is consistent so not an issue only when under load etc.

E.g.

Request:
Request URL: http://wallet-1.nodes.arweave.org:1984/tx/SDy5z6Cg33uHC-o-ewYpJzLXyzI3PjjTtIIGzL4DQLE/data.html
Request Method: GET

Response:
Status Code: 206 Partial Content
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 0
Content-Range: bytes 0--1/5946

Joining on old peers can lead to wallet list retrieval issues

One user encountered this issue on joining with 1.5.0.0-RC-4:

    'NodeWorkerERROR': {{case_clause,
                            {ok,{{<<"421">>,<<"Misdirected Request">>},
                                 [{<<"content-length">>,<<"49">>},
                                  {<<"connection">>,<<"keep-alive">>}],
                                 <<"Subfield block querying is disabled on this node.">>,
                                 129,654747}}},
                        [{ar_http_iface,get_wallet_list,2,
                             [{file,"src/ar_http_iface.erl"},{line,986}]},
                         {ar_http_iface,get_full_block,3,
                             [{file,"src/ar_http_iface.erl"},{line,953}]},
                         {ar_node_utils,'-get_full_block/3-fun-0-',4,
                             [{file,"src/ar_node_utils.erl"},{line,49}]},
                         {lists,foldl,3,[{file,"lists.erl"},{line,1263}]},
                         {ar_node_utils,start_mining,2,
                             [{file,"src/ar_node_utils.erl"},{line,198}]},
                         {ar_node_worker,recovered_from_fork,2,
                             [{file,"src/ar_node_worker.erl"},{line,615}]},
                         {ar_node_worker,handle,2,
                             [{file,"src/ar_node_worker.erl"},{line,129}]},
                         {ar_node_worker,server,2,
                             [{file,"src/ar_node_worker.erl"},{line,56}]}]}

Allow for a subset of fields to be requested on tx/{id}

Currently we have /tx/{id} which will return the full transaction including the data field, this can often be quite large and isn't always needed depending on the use case and /tx/{id}/{field} only returns one field at a time.

It would be useful to allow for a subset of fields to be specified by the client not the request, for example

GET /tx/{id}?fields=last_tx,target

could return

{
  "last_tx": "9t981ng4b04gnmants1sge307k933od9rbednpxl1f3",
  "target": "rwa8ddz5b2ax27pgqrsx98u242has4wcpf6nwph1wue"
}

[build] Error 127

Hi I am using Mac (latest update) and facing following problem by starting a mining process:

git submodule update --init Submodule 'lib/accept' (https://github.com/deadtrickster/accept.git) registered for path 'lib/accept' Submodule 'lib/prometheus' (https://github.com/deadtrickster/prometheus.erl.git) registered for path 'lib/prometheus' Submodule 'lib/prometheus_process_collector' (https://github.com/deadtrickster/prometheus_process_collector.git) registered for path 'lib/prometheus_process_collector' Cloning into '/Users/Konstantin/arweave/lib/accept'... Cloning into '/Users/Konstantin/arweave/lib/prometheus'... Cloning into '/Users/Konstantin/arweave/lib/prometheus_process_collector'... Submodule path 'lib/accept': checked out '9b52b6c5e23f32feccca9bb187913a4fe3ca7865' Submodule path 'lib/prometheus': checked out 'e7744268131c63d666cf1413cabf4f620b918ba9' Submodule path 'lib/prometheus_process_collector': checked out 'bad06f50e36f246db170f84b73a5d97960a75f26' rm -rf priv rm -rf data/mnesia cd lib/jiffy && ./rebar compile && cd ../.. && mv lib/jiffy/priv ./ env: escript: No such file or directory make: *** [build] Error 127

Previously I got a following message, don't know if this is an issue
HEAD is now at 87fccde - Verify full block download before adding txs in ar_http_iface:get_full_block.

Not enough disk space reported on startup

The following error intermittently gets reported on startup which stops the node loading any further.

=ERROR REPORT==== 15-Aug-2018::21:25:25 ===
Error in process <0.884.0> with exit value:
{badarith,[{ar_storage,enough_space,1,
                       [{file,"src/ar_storage.erl"},{line,391}]}

Nodes in question have plenty of space available so this is not accurate, restarting them again works fine, sometime it may take 1 or 2 restarts before it continues.

Debugged by @themue, issue seems to be async calls to ar_node_db:*(used_space) on startup. Occasionally we seem to get a race condition where the value hasn't been set, while elsewhere in the application we're trying to read that value which returns not_found as it hasn't been set yet, this seems to trigger the error above.

Making sure this value is set before any process accesses this should resolve the issue.

Transaction encoding error when retrieving block from peer

Version 1.5 expects lists of transaction IDs, older peers not analysing the X-Version header deliver blocks containing full transactions. Leads to decode error:

=ERROR REPORT==== 24-Sep-2018::20:53:23 ===
Error in process <0.8559.12> with exit value:
{function_clause,[{ar_util,decode,
                           [{[{<<"id">>,
                               <<"ki2Zf_RQ-i3zP3giEEIR3pi4CIfekyIxc_EUOiy0HXM">>},
                              {<<"last_tx">>,<<>>},
                              {<<"owner">>,
                               <<"l_Q0ZukzXaxOWK7j2zoeuX2AR__HcxJCYLKb2tLFu33fEMUyR_O2u73vWqyGOu-ZrCZqXH0AcNQoyIkRZ9uOexoxoI7KKzPBhHbyELOcS8NStvM_UgmQXb4EGf1pq5rQkkLlMaUNq2q1Z-CdYoIo91V1HdSwB6rcRpf3cOtEgSYQSLvLMlXE2xfA_xSeuWqgMUaYybAgmVUCLDayuP-UwrGDXLaqVox_Q53nDZV19_43UCmlRXzd2SGby2vxzqSnBWBIJUN73JBnWXTtRvlpxMWbnrKDrHig0fuEyLl5Zh0v6F-22lA7-ZJGE86pzCyIAMjJ20EpjTIM12tC9z9wCTG6iAVYLoICgci7fvXM21mewMcHcxZmZdc0xmngesuMNl0u2SBfEaantApVjbGRwRWDcNjB-6F4eNpHIgIlgMdws23gjW0hwrFvDpa6oldBQXDXcvpE-XYDRCH6Upn4Ao7iUhdydRWxWBQSWvyyPKfFZX9iVh_KF1czAC0ruV4oQE18I1tUe9N5HSiD6DBK7WI_oU1yMdtB8ZHDom0EjZdKD-q6epDvMCEJ2lxoqCMUo1VYgr97Kc780g2VvroOJ1dOyecu4ZX11CQDMH2EqePra6Y50v3WxDbmZTwvXYX8tk76MPcHZ0Zp2rfYau6vMYDFMCA78nD2_FTcPoHmDzU">>},
                              {<<"tags">>,[]},
                              {<<"target">>,
                               <<"DIQS78-NxbLplF7OgJFXg7BNYM28FZeujIMvU2ljN7o">>},
                              {<<"quantity">>,<<"338090000000000">>},
                              {<<"data">>,<<>>},
                              {<<"reward">>,<<"250321179212">>},
                              {<<"signature">>,
                               <<"GigyuiLLvMSCTRPwiL8EmR7crysCUHsZppociPqE3Ai2HvvpnFoEzEEHmW_UWGIxK1z0xTfxuyyxBb4eE82Jcwm9P3gBIijA5BM3N-ZV1zs2FjfAeRr3oYlUbhzClvH2hmcAoSA5b9-XJz3-dn3RQNAKAFqwMsuEPaMEuEfIXrHfvraeOOLL5kNUH-WXtq7EIpJCddd7U7Lpl5nnDuB1vuKthXY-3EuPR3sUcvJXdzCd5ys26AbGvnIXIkEVfv7IZn-6qsr7LQCUx4nLCLPadQQT2WgayvAkk9F37TYSq2TJwBiJBgR5ItQR8Z3i3BCmDFzbwunS9mFI75ki_ya-aqqpgLD7m4fNDAypS-5jn6zk5Zahx8AYrooQUFKslyPbFx5q6zfGiMbybmNSueWUFG_0dBCNLWRB8Ubawl2gddolenaH5tRfSji4cJAdNlDp1d09czy7YKXhHTc6pKnTG8Azys_-UoXV2AbalAxOZZsP9fHDCpeF0ykQs8kC5xz_a1h6C68Q87G7ZDr0J0YmK9RN2RkogbS_3KZNHvx80lbcCObWWgXz_o2StjqwDBFo3hFARBLuqO0dd2CclYqP0Uj2mnYwZCYcMOPkGMmEVmPooBHjenDOpaS6nA9V8nEO4UXg-5nwCUyGkf5LH4cddm3SmDbBJc1Uz7X_Lad-1sk">>}]}],
                           [{file,"src/ar_util.erl"},{line,43}]},
                  {lists,map,2,[{file,"lists.erl"},{line,1239}]},
                  {ar_serialize,json_struct_to_block,1,
                                [{file,"src/ar_serialize.erl"},{line,168}]},
                  {ar_http_iface,get_full_block,3,
                                 [{file,"src/ar_http_iface.erl"},{line,863}]},
                  {ar_node_utils,'-get_full_block/3-fun-0-',4,
                                 [{file,"src/ar_node_utils.erl"},{line,49}]},
                  {lists,foldl,3,[{file,"lists.erl"},{line,1263}]},
                  {ar_fork_recovery,server,1,
                                    [{file,"src/ar_fork_recovery.erl"},
                                     {line,155}]}]}

Individual GET /block/hash/<hash> fail

Individual calls of GET /block/hash/<hash> fail during generation of hash list:

    elli_event: request_error
    args: [{req,'GET',undefined,undefined,undefined,
               [<<"block">>,<<"hash">>,
                <<"QPLJABTTZVE_7O5jUL19QJ67jW3NwDWF_NzIVbk8044xO1j-FHIe5O0_mHmQx1cu">>],
               [],
               <<"/block/hash/QPLJABTTZVE_7O5jUL19QJ67jW3NwDWF_NzIVbk8044xO1j-FHIe5O0_mHmQx1cu">>,
               {1,1},
               [{<<"Content-Length">>,<<"0">>},
                {<<"Host">>,<<"62.75.236.171:1984">>}],
               <<>>,<0.29204.0>,
               {plain,#Port<0.25735>},
               {elli_middleware,
                   [{mods,
                        [{ar_blacklist,[]},
                         {ar_metrics,[]},
                         {ar_http_iface,[]}]}]}},
           function_clause,
           [{ar_block,do_generate_hash_list_for_block,
                [<<64,242,201,0,20,211,101,81,63,236,238,99,80,189,125,64,158,
                   187,141,109,205,192,53,133,252,220,200,85,185,60,211,142,49,
                   59,88,254,20,114,30,228,237,63,152,121,144,199,87,46>>,
                 []],
                [{file,"src/ar_block.erl"},{line,38}]},
            {ar_storage,do_read_block,2,
                [{file,"src/ar_storage.erl"},{line,223}]},
            {ar_http_iface,handle,3,
                [{file,"src/ar_http_iface.erl"},{line,493}]},
            {ar_http_iface,handle,2,
                [{file,"src/ar_http_iface.erl"},{line,95}]},
            {elli_middleware,process,2,
                [{file,"lib/elli/src/elli_middleware.erl"},{line,108}]},
            {elli_middleware,handle,2,
                [{file,"lib/elli/src/elli_middleware.erl"},{line,64}]},
            {elli_http,execute_callback,1,
                [{file,"lib/elli/src/elli_http.erl"},{line,287}]},
            {elli_http,handle_request,4,
                [{file,"lib/elli/src/elli_http.erl"},{line,104}]}]]
    config: []

Add support for fetching new transactions in the polling mode

Nodes in the polling mode only poll for blocks, not transactions. So every other block with transactions fails to be validated. Usually, it results in a failed nonce validation:

=INFO REPORT==== 28-Nov-2018::11:49:26 ===
    invalid_nonce: <<235,1,8,239,186,29,240,142,126,190,76,209,204,17,116,231,
                     94,57,182,169,222,100,108,44,251,37,206,128,122,241,59,
                     102,221,126,38,244,116,225,179,58,171,211,113,217,66,245,
                     111,37>>

=INFO REPORT==== 28-Nov-2018::11:49:26 ===
invalid_dependent_hash

=INFO REPORT==== 28-Nov-2018::11:49:26 ===
    could_not_validate_new_block: <<"u565HMidLeZ82dRBZujgyxcQmz_R7SsoPMc42osftKI20lsIG4Fibtfcg3v1RoZG">>

Shortly after, the node successfully validates the block inside the fork recovery.

Enable CORS preflight responses and headers

Currently no CORS headers are enabled which limits dynamic embedding of Arweave hosted content.

For example, if I wanted to make an AJAX call to an Arweave node from a browser to fetch the current block height, or get some transaction data, this would fail as nodes don't currently don't respond to preflight requests at all so would be blocked by browsers.

If a webpage with AJAX content is served from the same node the AJAX request is going back to, then this should be allowed under the same origin policy, so I guess this is only really an issue if the webpage is hosted outside of the Arweave network.

I think nodes should respond to preflight OPTIONS requests with Access-Control-Allow-Origin: * and Access-Control-Allow-Methods: GET across all GET endpoints at a minimum, although maybe it makes sense to enable it across all endpoints as there are scenarios where will might be useful to be able to post transactions from a browser which is currently only possible using forms.

Configuration via file

Today configuration of a node is done via command line arguments. Depending on the number of non-default arguments as well as initial peers the list ist long, inconvenient, and error prone.

Defining them in a configuration file improves this. Additionally templates for own configurations can be provided.

The format is to be discussed:

Erlang/OTP terms
Different standard formats like JSON, YAML, or TOML
Propriatary formats

Joining the net when provided with many offline IPs is slow.

The node should check the online status of the peers before joining the network, to avoid long timeouts.

Mining config generator needs styling

The Arweave mining configuration generator (found here: https://github.com/ArweaveTeam/arweave/blob/feature/config-generator/src/apps/config_generator.html ) is helpful, but is currently poorly styled.

There is up to a 200 AR bounty available for anyone that would like to give this a nicer UI!

Arweave build failure

I believe the Arweave dockerfile build is failing. Any Erlang heroes out there who could help me understand this?

cd lib/jiffy && ./rebar compile && cd ../.. && mv lib/jiffy/priv ./
{"init terminating in do_boot",{load_failed,[erl_lint,ets,lists,gen_server,erl_eval,gen,filename,erl_parse,proc_lib,gen_event,supervisor]}}
init terminating in do_boot ({load_failed,[erl_lint,ets,lists,gen_server,erl_eval,gen,filename,erl_parse,proc_lib,gen_event,supervisor]})

Crash dump is being written to: erl_crash.dump...done
make: *** [Makefile:39: all] Error 1
ERROR: Service 'arweave' failed to build: The command '/bin/sh -c make all' returned a non-zero code: 2

Change all instances of list_to_integer(binary_to_list(B)) to binary_to_integer(B)

easier to read;
infinitesimally faster (binary_to_integer is a bif).

GET /wallet/Address/last_tx should have a node unavailable clause

... like the one in GET /wallet/Address/balance.

See

https://github.com/ArweaveTeam/arweave/blob/master/src/ar_http_iface.erl#L409-L435

Mining config generator should elegantly handle missing data

The mining config generator web page ( https://github.com/ArweaveTeam/arweave/blob/feature/config-generator/src/apps/config_generator.html ) currently does not handle missing input very well.

For example, if you do not provide a trusted peer, the output command will include ./arweave-server peer mining_addr which is not a valid argument set.

100 AR for anyone that can add the appropriate error state handling!

Configuration of data directories

Today the directories for blocks, transactions, wallets, data, and logs are relative to project root. Other locations are possible by exchanging these manually by symlinks to other locations.

Configuration should be extended to allow individual configuration of these directories. If not configured location is relative to project root like today.

POST /arql will 500 as it tries to dejsonify a list not a binary

Replace binary_to_list(QueryJson) with just QueryJson (ar_http_iface.erl line 201).

Make Docker implementation restartable

Today the Docker implementation builds once and then runs. The signal of a new release would stop implementation and so also the container.

Change the implementation to act like the full node one with rebuilding the system after the according signal.

BadRecord, block error in NOVLNS

Bad record, block line 69, ar_fork_recovery

Timezone mismatch in arweave-server log

arweave-server uses erlang.system_time() to show its log. I think os.system_time() is more appropriate to use in this case.

Version management for peers needed

With block format (and also other data formats in future) changing versions posting blocks in the new format to peers running an older version leads to errors there. So our peer management and the bridge need to retrieve the version of a peer when joining and manage it.

In case of an incompatible too old peer blocks cannot be sent. Only to peers which already have updated. Also old peers have to be polled periodically if they already have updated to a new version.

Harmonise use of bignum strings across TX and block JSON.

At the moment TXs use strings to represent (potentially) very large numbers, keeping compatibility with 32 bit programming environments. Block JSON representations do not follow the same standard and should be converted.

Prefer Rebar3 to makefile

I've noticed a few things that I think rebar3 could help solve.

No built tarbars of the testnet, this would be pretty easy to do with rebar3 and then you could just drop the tarball and your keys on a server.
Manually bring in dependencies. I noticed the current dependencies are actually source code. Pulling these from hex would make updating/removing etc much easier.

What do you think?

Some EUnit Tests are Non-Deterministic

ar_http_iface:find_block/3 calls binary_to_integer on integers

This causes 500s on urls like GET /block/height/123/wallet_list or GET /block/height/123/<field>.

The first clause of find_block (for <<"height">>) is always called with an integer as the RawHeight argument, but line 1449 calls binary_to_integer(RawHeight):

https://github.com/ArweaveTeam/arweave/blob/master/src/ar_http_iface.erl#L1446-L1451

POST /block must check if block has already been processed first

Before any other check block has to be unmarshalled and checked, if it already has been processed to avoid a gossiping.

Move documents into directory ./docs

Move all documents but readme and license into directory ./docs
Harmonise naming and internal styles
Add index
All documents are markdown

Document issue handling

Also change hint on naming in version-control.md.

Block should contain the time that the block was mined.

The AR block contains a 'timestamp' member, but it appears to be the time that mining started not the time that mining succeeded.

The Timestamp value returned by server in ar_mine appears to be the Timestamp evaluated before the server was spawned. This is then the valued used when ar_weave:add is called in ar_node.

Whether this is intentional or not, I think the block should contain the time it was mined, as an additional member if necessary.

If the existing member were modified to show the mined time then, that timestamp should probably still be strictly monotonic increasing during migration - to avoid zero or negative durations. I think it would be so long as any clock variation between mines is less than the time it takes for a new block to propagate. But we would see some timestamps differing by just a few seconds and others differing by the time taken to mine the two previous blocks.

GET /block/Type/ID errors messages are switched

Request with invalid hash returns "invalid height";
Request with invalid height returns "invalid hash".

See

https://github.com/ArweaveTeam/arweave/blob/master/src/ar_http_iface.erl#L476-L480

Don't join peers until the block trail is downloaded

In ar_join.erl right now join_peers() is called before the block trail is downloaded completely. This may lead to forks.

EUnit Tests timing out.

Some EUnit tests are timing out. To try to mitigate I have begun writing ar_unit_testing that generates tests with timeout param. Might be too hacky though.

Jiffy error in N.1.5 candidate

Discovered the following error in logs of a candidate 1.5 node running over the weekend:

Sep 14 10:14:30 aws-oregon-1 session_2018-09-13_18-24-56.log:  =ERROR REPORT==== 14-Sep-2018::08:14:30 ===
Sep 14 10:14:30 aws-oregon-1 session_2018-09-13_18-24-56.log:  Error in process <0.3600.2> with exit value:
Sep 14 10:14:30 aws-oregon-1 session_2018-09-13_18-24-56.log:  {{nocatch,{error,{790609,invalid_string}}},
Sep 14 10:14:30 aws-oregon-1 session_2018-09-13_18-24-56.log:   [{jiffy,decode,2,[{file,"lib/jiffy/src/jiffy.erl"},{line,71}]},
Sep 14 10:14:30 aws-oregon-1 session_2018-09-13_18-24-56.log:    {ar_serialize,json_struct_to_block,1,
Sep 14 10:14:30 aws-oregon-1 session_2018-09-13_18-24-56.log:                  [{file,"src/ar_serialize.erl"},{line,146}]},
Sep 14 10:14:30 aws-oregon-1 session_2018-09-13_18-24-56.log:    {ar_http_iface,get_full_block,3,[{file,"src/ar_http_iface.erl"},{line,990}]},
Sep 14 10:14:30 aws-oregon-1 session_2018-09-13_18-24-56.log:    {ar_node_utils,'-get_full_block/3-fun-0-',4,
Sep 14 10:14:30 aws-oregon-1 session_2018-09-13_18-24-56.log:                   [{file,"src/ar_node_utils.erl"},{line,49}]},
Sep 14 10:14:30 aws-oregon-1 session_2018-09-13_18-24-56.log:    {lists,foldl,3,[{file,"lists.erl"},{line,1263}]},
Sep 14 10:14:30 aws-oregon-1 session_2018-09-13_18-24-56.log:    {ar_fork_recovery,server,1,[{file,"src/ar_fork_recovery.erl"},{line,152}]}]}

Sporadic errors reading wallet list from disk

Errors are followed by writings of blocks to disk:

=INFO REPORT==== 18-Sep-2018::05:44:19 ===
    error_reading_wallet_list_from_disk: <<"1_M5TofT7b__-s3RO-mSn8v6spSL8ZQADvjfiFORg91siIUeSiVXKRQCsgVfsvr0">>
    type: enoent

=INFO REPORT==== 18-Sep-2018::05:44:35 ===
    writing_block_to_disk: <<"1_M5TofT7b__-s3RO-mSn8v6spSL8ZQADvjfiFORg91siIUeSiVXKRQCsgVfsvr0">>

GET /services 500s if there is no http_service_node

Suggest a more informative 5xx or 404 is returned.

Encoding error when starting mining

Sometimes start of mining fails due to direct encoding of a transaction record.

=INFO REPORT==== 18-Sep-2018::08:54:09 ===
    'NodeWorkerERROR': {function_clause,
                           [{base64url,encode,
                                [{tx,<<58,26,87,204,41,186,170,151,252,122,177,
                                       215,85,138,143,87,97,6,74,91,131,161,
                                       226,22,115,224,46,137,137,39,43,89>>,
                                     <<104,13,107,126,156,7,12,22,122,108,218,
                                       178,119,53,114,192,237,78,24,118,8,46,
                                       165,68,80,38,178,101,135,57,168,122>>,
                                     <<209,85,176,128,234,61,227,107,122,129,
                                       106,152,147,239,82,143,30,3,226,120,212,
                                       191,32,169,146,79,175,83,229,241,249,7,
                                       68,80,56,79,171,255,68,66,240,190,182,
                                       14,184,34,188,78,62,231,119,39,140,153,
                                       46,177,207,76,20,196,39,122,19,241,209,
                                       4,239,104,196,229,92,34,171,117,88,187,
                                       2,89,209,156,14,28,222,53,192,90,155,98,
                                       255,160,241,250,82,67,35,225,1,28,224,
                                       130,13,238,53,242,163,217,16,242,133,
                                       136,8,220,70,30,203,244,218,178,65,151,
                                       96,177,42,148,249,148,28,91,74,189,81,
                                       11,20,45,198,20,63,254,231,40,146,219,
                                       222,83,7,42,20,154,152,43,242,185,212,
                                       250,191,182,27,126,163,35,88,248,165,14,
                                       148,179,46,136,187,4,57,103,234,92,48,
                                       51,146,91,224,19,241,122,99,172,166,172,
                                       20,135,51,101,147,81,31,39,211,103,58,
                                       12,135,241,178,110,242,184,93,112,176,
                                       184,244,148,148,31,139,17,66,221,63,150,
                                       76,222,149,51,250,100,196,180,53,57,200,
                                       234,241,80,98,136,58,195,248,115,176,81,
                                       200,150,25,6,254,201,112,235,20,58,37,
                                       173,141,186,161,57,89,124,151,206,143,
                                       67,187,211,184,79,40,83,166,229,69,61,
                                       148,42,210,170,137,84,235,2,126,185,148,
                                       68,169,52,188,159,24,190,120,154,225,65,
                                       17,67,252,175,91,2,174,42,171,213,248,
                                       50,227,219,163,15,246,233,213,188,172,
                                       72,232,27,14,19,150,216,177,14,234,17,1,
                                       236,65,187,148,159,63,153,202,12,26,32,
                                       214,80,117,114,160,216,17,110,76,33,238,
                                       149,71,177,92,228,229,58,235,92,124,115,
                                       181,11,41,44,118,201,1,97,226,145,234,
                                       109,141,88,146,87,180,251,152,252,81,6,
                                       80,59,207,77,171,72,73,84,198,207,153,
                                       16,145,82,43,145,73,246,229,64,152,64,
                                       194,9,188,124,151,90,78,61,51,182,231,
                                       173,200,172,53,119,40,107,77,52,85,151,
                                       90,205,196,214,49,109,93,142,250,53,189,
                                       7,255,119,58,24,130,161,209,96,217,19,
                                       125,229,243,120,27,245,175,219,113,237,
                                       40,25,222,39,248,166,22,254,89,241,37,
                                       86,179,125,14,16,20,211,133,96,13,76,78,
                                       101,208,196,56,174,6,92,214,88,70,94,56,
                                       56,62,127,215,32,154,75,14,7,51,182,239,
                                       63,144,30,249,26,162,41>>,
                                     [],
                                     <<171,175,213,161,197,45,34,200,143,56,88,
                                       0,249,247,20,11,105,56,77,24,97,117,242,
                                       195,47,110,99,85,37,76,153,54>>,
                                     33750000000000000,<<>>,
                                     <<208,180,203,185,200,234,202,143,227,172,
                                       61,60,127,109,29,133,138,106,255,79,225,
                                       244,41,114,70,100,130,201,197,163,113,
                                       179,111,211,77,47,99,37,175,0,233,210,
                                       110,200,164,87,21,158,157,190,4,235,133,
                                       160,134,46,200,213,116,130,249,236,66,
                                       189,114,210,220,65,15,8,85,227,3,89,76,
                                       241,196,28,4,80,163,105,14,134,0,22,118,
                                       43,90,28,46,10,249,170,0,225,33,84,174,
                                       144,19,102,194,28,145,212,128,242,56,76,
                                       240,3,4,168,77,102,217,192,92,93,149,98,
                                       37,77,109,137,66,42,16,99,33,38,152,74,
                                       12,149,181,198,220,131,45,244,255,171,
                                       238,55,132,224,96,154,1,40,198,59,112,
                                       93,99,61,172,127,241,92,88,14,21,215,36,
                                       80,146,236,121,227,144,121,112,138,41,
                                       37,82,249,158,14,93,189,195,186,242,132,
                                       64,15,182,31,177,15,20,163,179,245,146,
                                       233,242,4,49,106,176,109,211,20,146,196,
                                       239,68,240,80,102,145,252,120,10,126,
                                       120,95,88,203,177,181,88,148,177,198,54,
                                       47,182,172,186,33,184,165,117,36,40,134,
                                       58,120,176,4,110,98,71,97,10,221,72,162,
                                       40,95,122,69,231,204,41,82,170,126,19,
                                       197,146,178,34,253,139,26,207,209,241,
                                       109,90,182,154,232,131,40,235,2,184,248,
                                       106,159,130,145,13,123,178,182,220,18,
                                       81,73,131,162,170,198,219,143,14,199,
                                       221,248,29,42,96,158,235,246,71,47,64,
                                       172,167,141,46,3,10,165,234,18,53,28,
                                       150,248,26,43,87,243,201,50,239,38,228,
                                       60,1,148,197,194,76,14,17,191,87,100,55,
                                       64,247,154,57,14,240,205,58,60,62,3,196,
                                       182,200,31,44,95,72,255,97,22,68,84,111,
                                       10,88,9,150,193,231,136,198,227,224,166,
                                       13,238,252,19,8,178,40,116,121,14,44,
                                       103,183,123,61,182,67,211,214,28,217,7,
                                       158,229,154,131,9,68,112,36,6,148,148,
                                       215,149,45,84,176,125,231,138,178,79,
                                       143,153,237,44,71,246,50,51,196,237,140,
                                       10,99,240,24,91,231,192,104,217,166,243,
                                       49,194,149,117,162,132,217,207,231,39,0,
                                       118,2,68,36,239,92,241,209,140,36,191,
                                       242,250,160,230,183,155,118,11,56,192,
                                       238,57,51,160,174,32,133,49,145,69,230,
                                       132,100,139,43,159,170,236,114,183,39,
                                       249,99,240,12,58,118,14,232>>,
                                     521472182}],
                                [{file,"lib/base64url/src/base64url.erl"},
                                 {line,22}]},
                            {lists,map,2,[{file,"lists.erl"},{line,1239}]},
                            {ar_weave,indep_hash,1,
                                [{file,"src/ar_weave.erl"},{line,252}]},
                            {ar_weave,verify_indep,2,
                                [{file,"src/ar_weave.erl"},{line,196}]},
                            {ar_node_utils,start_mining,2,
                                [{file,"src/ar_node_utils.erl"},{line,203}]},
                            {ar_node_worker,process_new_block,6,
                                [{file,"src/ar_node_worker.erl"},{line,371}]},
                            {ar_node_worker,handle_gossip,2,
                                [{file,"src/ar_node_worker.erl"},{line,196}]},
                            {ar_node_worker,server,2,
                                [{file,"src/ar_node_worker.erl"},{line,56}]}]}

base64url:encode/1 expects a binary or a list of integers and binaries which will be converted to a binary as argument.

Infrequent POST /block errors

Tests showed infrequent errors when posting blocks:

Sep 13 08:22:59 aws-oregon-1 session_2018-09-12_16-27-13.log:  =ERROR REPORT==== 13-Sep-2018::06:22:59 ===
Sep 13 08:22:59 aws-oregon-1 session_2018-09-12_16-27-13.log:  Error in process <0.32711.1> with exit value:
Sep 13 08:22:59 aws-oregon-1 session_2018-09-12_16-27-13.log:  {function_clause,[{lists,last,[undefined],[{file,"lists.erl"},{line,228}]},
Sep 13 08:22:59 aws-oregon-1 session_2018-09-12_16-27-13.log:                    {ar_block,'-generate_block_from_shadow/2-fun-2-',2,
Sep 13 08:22:59 aws-oregon-1 session_2018-09-12_16-27-13.log:                              [{file,"src/ar_block.erl"},{line,498}]},
Sep 13 08:22:59 aws-oregon-1 session_2018-09-12_16-27-13.log:                    {lists,dropwhile,2,[{file,"lists.erl"},{line,1396}]},
Sep 13 08:22:59 aws-oregon-1 session_2018-09-12_16-27-13.log:                    {ar_block,generate_block_from_shadow,2,
Sep 13 08:22:59 aws-oregon-1 session_2018-09-12_16-27-13.log:                              [{file,"src/ar_block.erl"},{line,497}]},
Sep 13 08:22:59 aws-oregon-1 session_2018-09-12_16-27-13.log:                    {ar_http_iface,'-handle/3-fun-1-',4,
Sep 13 08:22:59 aws-oregon-1 session_2018-09-12_16-27-13.log:                                   [{file,"src/ar_http_iface.erl"},
Sep 13 08:22:59 aws-oregon-1 session_2018-09-12_16-27-13.log:                                    {line,230}]}]}

Spelling mistakes in command line options lead to confusing error message.

Can we make the below user experience clearer? The 'Unknown argument' message is present, but it is hard to differentiate from the Erlang termination info.

[sam@mymachine arweave]$ ./arweave-server peer 84.54.153.83 polling ming_addr SamLK5LDq3CXPf1BJthISaMg2eVnV4fXWRSmiqCp7iQ mine
git submodule update --init
...
Build complete!
Unknown argument: ming_addr. Terminating.{"init terminating in do_boot",{function_clause,[{ar,start,[ok],[{file,"src/ar.erl"},{line,178}]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}
init terminating in do_boot ({function_clause,[{ar,start,[ok],[{_},{_}]},{init,start_em,1,[]},{init,do_boot,3,[]}]})

Crash dump is being written to: erl_crash.dump...done
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
[os_mon] memory supervisor port (memsup): Erlang has closed
Arweave Heartbeat: The Arweave server has terminated. It will restart in 15 seconds.
Arweave Heartbeat: If you would like to avoid this, press control+c to kill the server.

Exceeding disk_space not handled gracefully

A few users have reported errors when using the disk_space option to limit the amount of disk space used by the node, it doesn't appear to handle this limit gracefully, and can in some instances cause a node to not join as it doesn't reserve/clear enough space to download the current tail on join.

Example log

=INFO REPORT==== 30-Jul-2018::02:39:08 ===
    'NodeERROR': {{badrecord,block},
                  [{ar_node,server,1,[{file,"src/ar_node.erl"},{line,1115}]},
                   {ar_node,server,1,[{file,"src/ar_node.erl"},{line,965}]},
                   {ar_node,server,1,[{file,"src/ar_node.erl"},{line,968}]},
                   {ar_node,server,1,[{file,"src/ar_node.erl"},{line,1026}]},
                   {ar_node,server,1,[{file,"src/ar_node.erl"},{line,968}]},
                   {ar_node,server,1,[{file,"src/ar_node.erl"},{line,818}]},
                   {ar_node,server,1,[{file,"src/ar_node.erl"},{line,965}]},
                   {ar_node,server,1,[{file,"src/ar_node.erl"},{line,968}]}]}

=INFO REPORT==== 30-Jul-2018::02:39:08 ===
    node_not_joining
    reason: cannot_get_full_block_from_peer
    received_instead: unavailable

=INFO REPORT==== 30-Jul-2018::02:39:33 ===
    writing_block_to_disk: <<"eK7WmQC5H-jsDayVIQ7I0HtwSYTLJ6_S6U-pUW-Qc7BDuMA1_5CevwCbv5EfrUxi">>

=INFO REPORT==== 30-Jul-2018::02:39:34 ===
    {not_enough_space_to_write_block}
    {block_not_written}

Looback addresses should not appear in the HTTP API peers list.

When accessing "/peers" through the HTTP API the returned list of peer addresses sometimes includes loopback addresses such as 127.0.0.9 and 127.0.0.1.

Loopback, Private, Link-Local and other reserved address should probably not be returned through the public HTTP API. (RFC 5735 has a succinct table of IPv4 special use addresses). Perhaps the server should filter out these addresses even if they are configured?

Perhaps there needs to be a distinction between "public" and "private" peers (assuming folk want their own servers talking directly to each other)?