arionum / node Goto Github PK

if you import a sql dump of the database from another node, the "config" table has the other nodes "hostname" in it. Unless you know this is in there and what it does it will configure your peer with wrong hostname.

I have a fix in my bash script for rebuilding by importing sql file but to others doing this manually its an issue.

After tinkering with this for hours, I am getting timeouts from simple api calls to 127.0.0.1 after an import of pxgamers dump. I updated config hostname to my ip. It syncs fine but cant make any API calls without it hanging constantly. Cant figure out why.

I think what we need is a util.php tool to dump and restore the block chain data. What data tables can we dump and copy from node to node without problems?

util.php ->clean
DELETE is slooooooooooow.
Use TRUNCATE TABLE instead.

You will need to set... "SET foreign_key_checks=0;" before then "SET foreign_key_checks=1;" after.
Must be done in same session or setting is lost.

Just did a fresh install and seem to be stuck at block 10801.
Nothing in php/nginx error logs.
Syncing fine until it hit 10801

Ubuntu 16, Nginx, Php-fpm, MariaDB. 1GB RAM, 1 CPU VPS.

Not sure but Is issue that server is too small? If so, there needs to be some sort of alert/log telling user so. I have a second node thats been running for a while that is synced. It has 2GB of RAM. 1 CPU.

If minimum requirements are 2GB of RAM then in my bash setup script I am working on I can check for 2GB of RAM and tell user not enough if less. Thoughts on how to present this requirement to a new user?

Maybe on homepage of node, check for minimum requirements and notify user. Instead of saying "System Check complete" give a breif diagnostic summary. I'd like to see more information on the node homepage personally. This page is first thing new node runners see. It should blow them away. Personally, I love this simple clean design but add more stats to it. https://aro.pxgamer.xyz/

According to the README, it looks like this node build has many sections for developers.

It's not easy for developers to do this and I think we can add the Dockerfile or docker-compose.yml files in this repository.

If this issue is accepted, I can help this repository to complete the Docker support :).

Right now with all cryptocurrencies, all you need is the private key. And because it is impractical to memorize private keys, specially if you have many private keys, you need to store them somewhere. If this place of storage gets into the wrong hands, your investment is doomed.

So this raises the question, can a password be used in conjunction to a private key to authorize a transaction?

The basis of this is that you would never store your password except in your mind and it would be a difficult one.

For example, if your name was Eve Adam, you would use a password such as:

eV3AD@m31/11/1990

Code:
$bad_peers=array("127.0.0.1","localhost","10.0.0","192.168.0");

Fix:
Needs to be expanded to cover at least...
127., 10., 172.16.0.0 to 172.31.255.255, 192.168.*

I was able to get peers to pickup 127.0.0.2 as the hostname with many failures.
For full reserved list see see wiki: https://en.wikipedia.org/wiki/Reserved_IP_addresses

How I discovered this:

At some point in beginning after install if i run $ curl http://127.0.0.2/ to load schema in config it had saved hostname in "config" table as 127.0.0.2 instead of my server ip address. Then block updates fail since peers are trying to connect to 127.0.0.2 instead of my ip/hostname (might have been a typo in my code initially but it accepted 127.0.0.2 as localhost so bug is valid). I should be able to run $ curl http://127.0.0.1/ to initiate schema (NOTE: after second look i think 127.0.0.1 works OK but 127.0.0.2 or any 127.0.0.* causes failure since its not in $bad_peers. Peers pick up the bad address and cause flood of fails)

Additional Thoughts:
Also what if someone creates a local host file and points their domain to their server then loads site this way the first time. Seems like it would make them an invalid peer as well since hostname wont resolve (node operator error but still).

Also what happens if someone loads initial page through a cache layer or proxy? Like maybe cloudflare or something. What IP address does it pick up?

i should be able to run this from console...

php -f /var/www/mynode.com/util.php current

it doesnt even error. It just outputs nothing. You have to be in the webroot directory to run util.php.

The problem is include paths are relative. A full path to web directory should be determined in a bootstrap and loaded properly via full path.

require_once("include/init.inc.php");

TYPO: "Could nto connect to arionum.com" in sanity.php

(sorry, moving to the correct tab hopefully)

PROBLEM:
If any of the services go down (php fpm, nginx/apache/mysql), restart, upgrade, or just crash... Maybe even from a reboot or unatended upgrades... it is possible that the sanity-lock file can be orphaned, preventing sanity runs to keep node in sync.

SOLUTION:
IDEA: maybe check the modified date using filemtime() of sanity-lock file. If its older than 3X the "sanity_interval" from config then do an additional check to see if php sanity.php process is running (by looking at the process list) on the system. If it is not running then unlink( sanity-lock ).

It appears cant sync node past block 10801 unless you have 2GB of RAM. Please add minimum and recommended server requirements to README.md.

This is the first thing new users read. Please add more technical details to the README. A brief description of how the system works would be helpful, who/what its for (PHP Developers to Build Blockchain applications natively).

And it should give basic overview of how to access and read the /doc/ it's currently a hidden easter egg since no mention of it anywhere. The /doc/ content is good. It should be pointed out as a priority for developers to read first thing. In my bash script I am mentioning this to console on install.

If doing a sql dump, it fails on re-import with this error.

ERROR 1005 (HY000) at line 3: Can't create table aronode.accounts (errno: 150 "Foreign key constraint is incorrectly formed")

Process is...

1. stop nginx/php
2. delete database
3. recreate database
4. import dump
5. restart services
6. refresh git, delete sanity lock, and load site to refresh sync

Fix appears to be SET foreign_key_checks=0;
before import then set back to 1 after import.
I just appended commands to beginning and end of sql dump.

Something like so...

dump.sql
SET FOREIGN_KEY_CHECKS = 0;
..... SQL DUMP ....
SET FOREIGN_KEY_CHECKS = 1;

it defaults to 1 on basic mysql install so others have and will run ito this problem. Any ideas on how to fix/prevent this issue?

I suggest that we should add tags for versions. It would be easier to manage and allow downloading releases at a specific version for scripts, etc.

I've listed the latest commits for each version below.

This should be fine to do using:

git tag -s v0.1a 50ed36734dc2a2702c1ea9d1283f6dc99c866882
git tag -s v0.2b 66dee6516a17daa0872b835a45f5a7ac4282f84a
git tag -s v0.3.0 3a4aaf973c6085c17c0fc193b653a055bf80245e
git push --tags

Which gives you a list like the following (link):

If you have a GPG key, I've added the usage of -s for signing. Which is advised for tags.

When doing a git pull instead of relying on .gitignore make a file named config-sample.inc.php. This file stays the same no matter what happens and contains all the recommended defaults.

Then in code check if config.inc.php exists and if it doesnt tell user to copy the sample and enter in their db details. Then you can always have a copy of the default file for reference. This also allows for user to install with a zip file of the package without using git. This is how wordpress and other frameworks do it. A dummy sample config.

PROBLEM:
sanity.php ...
$f = file("https://www.arionum.com/peers.txt");

This makes the peers list dependent on a central domain arionum.com. The nodes should not be dependent on any calls to arionum.com.

SOLUTION:
Create an array in the main config file to allow someone to enter a list of known/custom peers. The sample config would have whats listed in https://www.arionum.com/peers.txt but would allow a user to change them.

CAVEATS:
Since users configs are not overwritten by updates, i think it does make sense to use https://www.arionum.com/peers.txt for now as the backup for a user that does not have this new config array set. The default should be to use the central domain but allow user to override it in config with custom list.

While many do, not all of the SQL statements which take potentially untrusted input being executed, such as data from connecting clients, are using using prepared statements and binding variables before forming and executing the query (separating data from code, preventing injection of other characters).

Some of the input from clients is being filtered beforehand using filter_var() which is good, but not perfect and there may be bypasses there. But sanitizing input is paramount in many places because data is often being passed to concatenate sql queries or to system() (which is dangerous if not used very, very carefully). peer.php needs special attention here.

Recommendations would be to...

• Explicitly ensure input looks as expected by whitelisting variables to be alphanumeric only (such as you're doing in many places already with san(), nice one), or containing numbers and dots only for IP addresses, etc.

• Use http://php.net/manual/en/function.escapeshellcmd.php to sanitize all variables before passing any to system()

• Bind parameters with prepared statements before every SQL statement concatenation and execution

• Go through the code and sanitize variables as soon as they come from the client so you don't need to worry about them later on, all it takes in one assumption later on to create a vulnerability that could easily compromise the node

PROBLEM: The use of select count() in mysql is extremely inefficient and will not scale well with tables full of millions+ or even billions+ of rows.

PROOF:
sql: EXPLAIN SELECT COUNT(1) FROM transactions; # this shows that it did a full index/table scan of every row in the table. 1.5M rows. Full index/table scans are bad. They are SLOOOW and will not scale well.

sql: EXPLAIN SELECT COUNT(id) FROM transactions; # still full scan 1.5M rows

sql: EXPLAIN SELECT id FROM transactions ORDER BY id LIMIT 0,1; # still full scan 1.5M rows

sql: EXPLAIN SELECT id FROM transactions ORDER BY id LIMIT 1; # scanned ONE row.

sql: SHOW TABLE STATUS LIKE 'transactions'; # this took 80 milliseconds

RESULT: The difference is 2+ seconds vs 80 milliseconds. This is huge and the 2+ seconds will increase over time as data grows. The 80 milliseconds will not.

SOLUTION:

1. If you need to check if at least one result is returned use ORDER BY id LIMIT 0,1; instead of SELECT COUNT. Avoid using COUNT if at all possible. This will not scale.
2. SQL queries in code should be tested with EXPLAIN. If they are doing full table scans, they need to be removed completely from code and other methods should be used that can scale to billions of rows.
3. NOTE: InnoDB (MyISAM is dead) do not store row counts and will count() all rows each time.
4. if you need table counts try "SHOW TABLE STATUS LIKE ..." NOTE this is an approximation but should be fine for large tables when you need to do some sort of paging.
5. This should also speed up PAGING on the block explorer. I am seeing that the explorer is starting to get slower. This is one reason why.

I have been writing a bash script to manage a node and I think what is happening occasionally is if node is syncing and nginx or php-fpm or mysql on my node go down OR are restarted, tmp/sanity-lock file remains which prevents my node from syncing further. It gets stuck at whatever block it was on before the service went down. I havent dove deep into code so I might be mis-understading the full flow.

If i am missing something here, is there an easy way to reset the lock/sync/queue?
I was deleting lock file and resetting sanity_last config in database to 0 after i restart services (nginx, php-fpm, mysql) which seems to work ok and in my bash script its just a simple...
# bash aronode mainnet restart

Recommend changing "date_default_timezone_set("Europe/Amsterdam");" to "UTC".