loda-lang / loda-cpp Goto Github PK

The build works already, but the runtime needs fixes

OeisManager and MultiGenerator should be reloaded like every 24h. We should be able to remove the restarts in mine_parallel.sh.

Provide a loda submit <file.asm> <A-number> for check and submitting programs. Remove the optional argument for loda mine.

Implement a new program generator. It should load a random existing program and mutate it.

The goal is to simplify the usage as much as possible. Users should just download the binary and then run loda setup to configure their environment. LODA should automatically check for updates and inform the user. There should be a new command loda update to install the latest version.

The user should have an option to enter his/her name in the setup to get credit for his/her mining effort. Found programs should then contain a comment "Mined by XYZ". Of course this is optional.

This error occurred directly (for all processes) after mine_parallel.sh had restarted all processes:

2021-09-24 16:39:04|INFO |Using LODA home directory /home/jon/loda/
2021-09-24 16:39:04|ERROR|Error parsing line: A166616: 
2021-09-24 16:39:04|INFO |Starting LODA v21.9.21. See https://loda-lang.org
terminate called after throwing an instance of 'std::runtime_error'
2021-09-24 16:39:04|INFO   what():  |Using LODA home directory /home/jon/loda/
Error parsing line: A166616: 
2021-09-24 16:39:04|INFO |Starting LODA v21.9.21. See https://loda-lang.org
2021-09-24 16:39:04|INFO |Using LODA home directory /home/jon/loda/
2021-09-24 16:39:04|ERROR|Error parsing line: A166616: 
terminate called after throwing an instance of 'std::runtime_error'
  what():  Error parsing line: A166616: 
2021-09-24 16:39:04|ERROR|Error parsing line: A166616: 
terminate called after throwing an instance of 'std::runtime_error'
  what():  Error parsing line: A166616: 
...

All processes complained at A166616.

And finally:

2021-09-24 22:39:05|ERROR|Error parsing line: A166616: 
terminate called after throwing an instance of 'std::runtime_error'
  what():  Error parsing line: A166616: 
./mine_parallel.sh: line 93: 2229830 Aborted                 (core dumped) loda mine -l ${log_level} -i ${i} $@
./mine_parallel.sh: line 93: 2229831 Aborted                 (core dumped) loda mine -l ${log_level} -i ${i} $@
./mine_parallel.sh: line 93: 2229832 Aborted                 (core dumped) loda mine -l ${log_level} -i ${i} $@
./mine_parallel.sh: line 93: 2229833 Aborted                 (core dumped) loda mine -l ${log_level} -i ${i} $@
./mine_parallel.sh: line 93: 2229834 Aborted                 (core dumped) loda mine -l ${log_level} -i ${i} $@
./mine_parallel.sh: line 93: 2229835 Aborted                 (core dumped) loda mine -l ${log_level} -i ${i} $@
./mine_parallel.sh: line 93: 2229836 Aborted                 (core dumped) loda mine -l ${log_level} -i ${i} $@
...

Speed up gcd operation using https://en.wikipedia.org/wiki/Binary_GCD_algorithm ?

While running the loda mine client, the following message appeared. The file exists in the path and it didn't look corrupted to the naked eye (I did not look carefully or specifically around the given index).

2021-09-22 15:25:12|INFO |Fetched https://oeis.org/A163960/b163960.txt
2021-09-22 15:25:13|WARN |Unexpected index 9675 in b-file /home/jon/loda/oeis/b/123/b123329.txt
2021-09-22 15:25:13|ERROR|Error loading b-file /home/jon/loda/oeis/b/123/b123329.txt
terminate called after throwing an instance of 'std::runtime_error'
  what():  Error loading b-file /home/jon/loda/oeis/b/123/b123329.txt
2021-09-22 15:25:14|INFO |Generated 4096 programs

The existing folder lock mechanism works only on Linux/Mac:

https://github.com/loda-lang/loda-cpp/blob/main/src/file.cpp#L196-L230

We need to add an implementation for Windows (#ifdef _WIN64 ...). Here is a useful link:

https://aljensencprogramming.wordpress.com/2015/05/06/file-locking-with-win32/

The current script for parallel mining is implemented in bash:

https://github.com/loda-lang/loda-cpp/blob/main/mine_parallel.sh

So, it does not (natively) work on windows. Therefore we want to add another script (Batch or PowerShell) for parallel mining on windows. It should behave the same as the existing bash script.

The default miner config should be updated together with the programs repo update. We must not throw an error if a generator version or a matcher is unknown.

2021-11-13 09:31:25|ALERT|First program for A142938: Primes congruent to 29 mod 64. Terms: 29,157,349,541,733,797,1117,1181,1373,1693. Submitted by Jon Maiga
2021-11-13 09:31:25|ALERT|Faster program for A137421: Decimal expansion of growth constant in random Fibonacci sequence. Terms: 1,2,0,5,5,6,9,4,3,0. Submitted by Jon Maiga
curl: (22) The requested URL returned error: 400 Bad Request
2021-11-13 09:31:26|ERROR|Error submitting program to miner API
terminate called after throwing an instance of 'std::runtime_error'
  what():  Error submitting program to miner API
2021-11-13 09:31:29|INFO |Generated 2614001 programs

Looks like the miner (binary) sent a bad request and was terminated.

The miner considers the following program for A007088 as faster, but it is actually incorrect:

mov $1,4086
mov $2,$0
lpb $2
  lpb $2
    mul $1,10
    mul $2,3
    dif $2,6
  lpe
  add $0,$1
  sub $2,1
lpe
div $0,4086

Move the existing benchmark from tests to a dedicated class and command. Add new benchmarks that are stable when we increase the bigint size. Uses common sequences and measure:

1. Maximum number of terms that can be computed
2. Execution time for computing first 100 terms

Results should be printed as markdown table and get included in change log.

Relevant sequences:

• Fibonacci
• primes
• Decimal expansions of e, pi, sqrt 2

On my ubuntu machine I get this error 100% of the time (tried restarting the miner too). It's connected to the internet. Works fine on my Windows machine (via WSL).

Starting LODA v21.12.1. See https://loda-lang.org/
...
2021-12-13 08:09:33|ALERT|Faster program for A062011: a(n) = 2*tau(n) = 2*A000005(n). Terms: 2,4,4,6,4,8,4,8,6,8. Submitted by Jon Maiga
2021-12-13 08:09:34|ERROR|Error submitting program to API server
2021-12-13 08:09:39|ALERT|Faster program for A114003: Rows sums of triangle A114002. Terms: 1,3,3,5,3,7,3,7,5,7. Submitted by Jon Maiga
2021-12-13 08:09:39|ERROR|Error submitting program to API server
2021-12-13 08:09:45|ALERT|Faster program for A334954: a(n) is 1 plus the number of divisors of n. Terms: 2,3,3,4,3,5,3,5,4,5. Submitted by Jon Maiga
2021-12-13 08:09:45|ERROR|Error submitting program to API server
2021-12-13 08:09:47|ALERT|Faster program for A062011: a(n) = 2*tau(n) = 2*A000005(n). Terms: 2,4,4,6,4,8,4,8,6,8. Submitted by Jon Maiga
2021-12-13 08:09:47|ERROR|Error submitting program to API server
2021-12-13 08:09:55|ALERT|Faster program for A065091: Odd primes. Terms: 3,5,7,11,13,17,19,23,29,31. Submitted by Jon Maiga
2021-12-13 08:09:55|ERROR|Error submitting program to API server
2021-12-13 08:09:58|ALERT|Faster program for A334954: a(n) is 1 plus the number of divisors of n. Terms: 2,3,3,4,3,5,3,5,4,5. Submitted by Jon Maiga
2021-12-13 08:09:58|ERROR|Error submitting program to API server
2021-12-13 08:10:16|ALERT|Faster program for A014675: The infinite Fibonacci word (start with 1, apply 1->2, 2->21, take limit). Terms: 2,1,2,2,1,2,1,2,2,1. Submitted by Jon Maiga
2021-12-13 08:10:16|ERROR|Error submitting program to API server
2021-12-13 08:10:18|ALERT|Faster program for A165851: Totally multiplicative sequence with a(p) = 30. Terms: 1,30,30,900,30,900,30,27000,900,900. Submitted by Jon Maiga
2021-12-13 08:10:18|ERROR|Error submitting program to API server

In this example, the optimizer could replace $6 by the constant 2.

mov $2,-2
bin $2,$0
pow $2,100
mov $6,2
lpb $1,9
  div $2,2
  add $3,$6
lpe
mov $0,$3
sub $0,2
div $0,2

2021-09-23 16:44:09|ERROR|Program evaluated to different sequence after optimization

gcd $0,2
mul $0,$0
pow $0,0
max $0,$0
mod $$2,2
clr $4,1
div $4,$$1
trn $0,1
add $3,2
mul $4,2
gcd $4,$3
clr $2,$0
clr $$3,2
bin $4,$3
bin $$2,$3
gcd $4,$2
mov $26,$2
cmp $26,0
add $2,$26
div $$3,$2
mul $4,2
max $0,$3
mod $4,11

This also allows us to reuse the miner code for matching.

Idea: when setup downloads the miners.default.json file, it injects the current version number. When run again, it compares its own version number with the one in the file. Also: use the miners.default.json from the version tag!!

Followed the instructions as per https://loda-lang.org/install/

Press return to download the default programs repository:
[https://github.com/loda-lang/loda-programs.git]
fatal: Too many arguments.

If it helps, my user directory has a space.

The code is in miner.cpp

C:\Users\user name>md %USERPROFILE%\loda\bin
A subdirectory or file C:\Users\user already exists.

(Works with md "%USERPROFILE%\loda\bin")

for sequences with only ~50 terms, the faster/better check can yield incorrect results and overwrite actually better programs. see the prime related functions in protect.txt

We currently create copies of the v1 generator for every template including the propability distributions. We should refactor the v1 generator such that it uses a vector of templates instead of just one (use round-robin). Then we don't need to create copies of the generator and probabiliyt distributions and save memory.

The util function for checking memory usage is disabled for MacOS at the moment, because it reports wrong numbers:

https://github.com/loda-lang/loda-cpp/blob/main/src/file.cpp#L159-L167

We need to get the "private" memory usage reported by the Activity Monitor (same as in top).

It shall be possible to mine programs without creating PRs to loda-programs. Found programs shall be pushed to a central server and automatically validated and added. It shall be possible to automatically add your name to the mined programs, like "Mined by XYZ..."

Metrics for generated and found programs are incorrectly assigned to the wrong generator on the server. There is no simple solution for this. For now, we removed dashboards that report these details.

The goal is to get rid of the mine_parallel.sh script and enable parallel mining on Windows. Instead of using mine_parallel.sh, we add a new option to the loda executable to toggle parallel mining. Specifically, loda mine -p would start multiple processes and mine in parallel. We would add a setup config to control the maximum number of processes.

The main benefits are:

• enable parallel mining on Windows
• get rid of separate script

We would need to start and stop processes in the C++ code. Here some examples for POSIX + Windows: http://www.cplusplus.com/forum/lounge/17684/

We want to reduce the traffic on the OEIS website for fetching stripped.gz, names.gz and the b-files. Miners should not fetch the files directly from OEIS, but from a LODA proxy which caches the files.

We would implement a simple proxy server that caches the files locally. For scalability: we add a look-up endpoint to the (central) API server. Every miner calls the look-up API once during start up. It fetches a list of servers (and their capabilities). It randomly selects one of the available servers and uses it for its lifetime (probably change after 12h or so). That would allow us to scale, also for submission of programs. It would also allow us blue/green deployments.

This will take some effort, but it will be faster, allow bigger values and enable binary gcd.

port to windows: https://github.com/loda-lang/loda-cpp/blob/main/src/file.cpp#L162-L185

Sometimes I think I have crafted a new program that corresponds to an OEIS sequence.

I can manually see that the first 60 terms is a match. However I'm not sure how many terms match. Is it 800 terms until the first mismatch, or is it the 61 term.

I think I have used -l debug in the past, but it seems to output the same.

PROMPT> loda check A123456
error
PROMPT> loda check A123456 -l debug
error
PROMPT> 

Perhaps a message like this:

PROMPT> loda check A123456
error, 61 matching terms followed by a mismatch.
PROMPT> 

We currently use a base-10 encoding for big number words. Concretely: we use the range 0..(10^18)-1 for storing one word. We should instead use the full 64 bit. This will have multiple advantages:

• allows us to store larger numbers with the same amount of bits
• should be significantly faster because we can replace the "mod 10^..." by a simple bit shift
• enables us to implement faster gcd: #23

https://github.com/loda-lang/loda-cpp/blob/main/src/include/big_number.hpp
https://github.com/loda-lang/loda-cpp/blob/main/src/big_number.cpp

It should implement the Generator interface. It should not use an AdaptiveScheduler but simply use the generators in round-robin style.

Support alternative eval modes:

(1) Rational Number Sequences

Use $0 as input. Two output cells: $0 is the numerator and $1 is the denominator. So we can evaluate a program to a sequence of rationals.

(2) Irrational Numbers

(2.1) Use (1) to generate a sequence of rationals. Then sum up all terms and take the limit. Convergence not guaranteed. What about products like in Horner's method?

(2.2) Use (1) to generate two sequences a(n) and b(n). Use these two sequences in a continued fraction (cf. http://www.ramanujanmachine.com/ ).

In theory we should be able to do mining for irrational numbers, like roots, e, pi, and zeros of the Riemann zeta function: http://www.dtc.umn.edu/~odlyzko/zeta_tables/index.html

Copied from ckrause/loda#113

It would be nice to have a metric to estimate how many LODA processes are currently running. It needs to work also in client/server mode.

It would be much more consumer-friendly if we had prebuilt binaries for different platforms available. We would need to think about how to deliver the programs in this case though.

Possible cross compile toolchain: https://hub.docker.com/r/multiarch/crossbuild

Copied from ckrause/loda#127

We currently call the wget and curl command-line tools to fetch/put files from the Internet. It would be better to integrate libcurl instead: https://curl.se/libcurl/

Blocker at the moment: we currently use make for building. If we start consuming libraries, we'll probably need to switch to cmake instead.

Copied from: ckrause/loda#126

When running in local or client mode, regularly run git fetch + rebase in the programs clone.

The check for "better" programs should be enhanced in this way: if the first non-decreasing term is past the know terms, discard the update.

see https://docs.microsoft.com/en-us/troubleshoot/windows-client/admin-development/create-desktop-shortcut-with-wsh

There should be a configurable retention policy (maybe default to 14 days?)

I forgot I could just run loda setup and curled home the latest first.

===== Welcome to LODA v21.12.1! =====

This command will guide you through its setup.

Enter the directory where LODA should store its files.
Note that non-default locations require manually adding
the LODA_HOME environment variable to your computer.
Press return for the default location (see below).
[C:\Users\Jon Maiga\loda]

LODA v21.12.1 is available!
Do you want to install the update? (Y/n) Y
curl: (22) The requested URL returned error: 404 Not Found
2021-12-02 13:38:46|ERROR|Error fetching https://github.com/loda-lang/loda-cpp/releases/download/v21.12.1/loda-windows-19.exe

Perhaps it should check if we already downloaded (this is maybe not only for Windows).

We currently have random generators distributed over the source code. It would be good to use one central generator instead (maybe in util).