but from Gekko's config.js also and specify strategy parameters itself.
It will be convenient if we can set the priority of the place where to get the strategy as well.

Is it possible to generate a random number of combinations (like 10,000) instead of just computing all possible combinations?

It's nice to have the possibility to test all possible combinations, but when the range is broad or you have too many variables in the strategy, there is no way to get node to manage the memory usage even when use the max-old-space-size option when running node..

Or you can maybe optimize the memory usage when the combinations are calculated?

The overall use case would be to randomly brute force a broad range of combinations and then reduce to a small range where all combinations can be tested..

I have a running Gekko on DigitalOcean Ubuntu 16.04 droplet.
Tried different URLs for api url in config.js of gekko-batcher without luck: exact ip, 0.0.0.0, localhost. All of them on 3000 port — the same as defined in UIconfig.js in Gekko.

Any suggestion?

Search for the 'best' parameters found by brute force method.

Hi. Great job. Much faster and more stable than the same BacktestTool.

For better usability, it’s worth adding a few features:

1. Ability to set the range of tested parameters
2. Output amount of time to optimize
3. Output to Exel or similar

Thanks.

It would be useful if we can see estimated time remaining to complete all tests.

Hi,
I can't seem to figure out why I have different results in the Gekko UI when I run seemingly the same test with bruteforce. If the profit percentages were off, that could be explained by the papertrader settings but the problem is already in a huge difference in the amount of trades.

The red boxes show the results/values of the bruteforce.csv. This shows 4 trades by the bruteforce tool and 73 by Gekko UI.

batcher config.js

const config = {};

// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
//                          GENERAL SETTINGS
// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

config.gekkoPath = '../gekko/';

config.apiUrl = "http://localhost:3000";

config.parallelQueries = 4;

config.candleSizes = [45, 60, 75];

config.historySizes = [10, 15];

// Format: [exchange, currency, asset]
config.tradingPairs = [
    ["binance", "usdt", "btc"],
];

config.daterange = {
    from: '2018-11-19T00:00:00Z',
    to: '2018-12-24T00:00:00Z'
};

// Where to get method's settings.
// The first has high priority. Then second, if there's no settings in first place and so on.

// batcher – strategy settings below here
// gekko – gekko's config.js
// toml.js – gekko's toml files
config.configPriorityLocations = ['batcher', 'gekko', 'gekko-toml'];

// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
//                          BACKTEST BATCHER
// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

config.methods = ['MACD'];

// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
//                          BRUTEFORCE SEARCHER
// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

config.method = 'MACD';

// Shuffle generated combinations of method's configs
config.shuffle = true;

// Generate all possible combinations of set of settings with given ranges
// Format for range: 'start:step:end'
config.ranges = {
    short: '8:1:10',
    long: '17:1:24',
    signal: '7:1:9',
    thresholds: {
        down: '-0.025',
        up: '0.025',
        persistence: 1
    }
};

// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
//                          STRATEGY SETTINGS
// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

config.MACD = {
    short: '8:1:10',
    long: '17:1:24', 
    signal: '7:1:9',
    thresholds: {
        down: '-0.025',
        up: '0.025',
        persistence: 1
    }
};

module.exports = config;

Any idea what I'm missing here?

Please let me know which gekko config you would like to see.

EDIT: added batcher's config.js

Hi,

When running bruteforce, I receive this message during the console output:

Complete: RSI_BULL_BEAR USDT/BTC 5/1000 Binance
Started: RSI_BULL_BEAR USDT/BTC 5/1000 Binance
(node:22519) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 201 unhandledRejection listeners added. Use emitter.setMaxListeners() to increase limit
Complete: RSI_BULL_BEAR USDT/BTC 5/1000 Binance
Started: RSI_BULL_BEAR USDT/BTC 5/1000 Binance
Complete: RSI_BULL_BEAR USDT/BTC 5/1000 Binance

The bruteforce backtest does seem to continue to run properly.

Batcher's config.js:

const config = {};

// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
//                          GENERAL SETTINGS
// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

config.gekkoPath = '../gekko/';

config.apiUrl = "http://localhost:3000";

config.parallelQueries = 3;

config.candleSizes = [5];

config.historySizes = [1000];

// Format: [exchange, currency, asset]
config.tradingPairs = [
    ["binance", "usdt", "btc"],
];

config.daterange = {
    from: '2018-11-18T00:00:00Z',
    to: '2019-01-08T00:00:00Z'
};

// Initial balance, fees and slippage/spread
config.paperTrader = {
    simulationBalance: {
        currency: 1,
        asset: 1
    },
    feeMaker: 0.05,
    feeTaker: 0.05,
    feeUsing: 'maker',
    slippage: 0.05,
}



// Where to get method's settings.
// The first has high priority. Then second, if there's no settings in first place and so on.

// batcher – strategy settings below here
// gekko – gekko's config.js
// toml.js – gekko's toml files
config.configPriorityLocations = ['batcher', 'gekko', 'gekko-toml'];

// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
//                          BACKTEST BATCHER
// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

config.methods = ['RSI_BULL_BEAR_ADX'];

// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
//                          BRUTEFORCE SEARCHER
// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

config.method = 'RSI_BULL_BEAR';

// Shuffle generated combinations of method's configs
config.shuffle = true;

// Generate all possible combinations of set of settings with given ranges
// Format for range: 'start:step:end'
config.ranges = {
    SMA_long: '300:200:700',
    SMA_short: '30:20:70',
    BULL_RSI: '10:5:20',
    BULL_RSI_high: '70:10:90',
    BULL_RSI_low: '50:10:70',
    BEAR_RSI: '10:5:20',
    BEAR_RSI_high: '30:20:70',
    BEAR_RSI_low: '10:10:30'
};

// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
//                          STRATEGY SETTINGS
// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

config.RSI_BULL_BEAR = {
    SMA_long: '300:200:700',
    SMA_short: '30:20:70',
    BULL_RSI: '10:5:20',
    BULL_RSI_high: '70:10:90',
    BULL_RSI_low: '50:10:70',
    BEAR_RSI: '10:5:20',
    BEAR_RSI_high: '30:20:70',
    BEAR_RSI_low: '10:10:30'
};

module.exports = config;

Please let me know if you need any more feedback.

There must be something I'm missing. I keep getting "ERROR: can't find strategy" when trying to run batch. I've tried installing and setting up numerous times on MacOS and Ubuntu without success. I've limited the batch to neuralnet_v2. The log shows the following:
uncaughtException TypeError: Cannot read property 'learning_rate' of undefined
at Base.init (/home/dbw/gekko/strategies/neuralnet_v2.js:70:36)

The address shown is correct for the strategy, but the error suggests that it's looking for the toml. I've tried setting it up to read the parameters directly from the batcher config, but I get the same errors.
I think I've followed the install and setup instuctions exactly, so I'm stumped. Can you help?

$node batcher
140 combinations
(node:1001) UnhandledPromiseRejectionWarning: TypeError: _.startCase is not a function
at runBacktest (/usr/src/app/batcher.js:213:227)
at /usr/src/app/batcher.js:177:16
at run (/usr/src/app/node_modules/promise-limit/index.js:32:30)
at semaphore (/usr/src/app/node_modules/promise-limit/index.js:49:14)
at Promise.all.allConfigs.map (/usr/src/app/batcher.js:176:12)
at Array.map ()
at Object. (/usr/src/app/batcher.js:175:24)
at Module._compile (module.js:653:30)
at Object.Module._extensions..js (module.js:664:10)
at Module.load (module.js:566:32)
(node:1001) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 141)
(node:1001) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

--config.js
const config = {};

// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
// GENERAL SETTINGS
// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

config.gekkoPath = '/usr/src/app/';

config.apiUrl = "http://localhost:3001";

config.parallelQueries = 4;

config.candleSizes = [5, 10, 20, 30, 40, 50, 60];

config.historySizes = [10, 20,30,40,50,60,70,80,90,100];

// Format: [exchange, currency, asset]
config.tradingPairs = [
["binance", "usdt", "btc"],
["binance", "usdt", "eth"]
];

config.daterange = {
from: '2018-01-18T00:00:00Z',
to: '2018-12-31T23:59:59Z'
};

// Initial balance, fees and slippage/spread
config.paperTrader = {
simulationBalance: {
currency: 50,
asset: 0
},
feeMaker: 0.25,
feeTaker: 0.25,
feeUsing: 'maker',
slippage: 0.05,
}

// Where to get method's settings.
// The first has high priority. Then second, if there's no settings in first place and so on.

// batcher – strategy settings below here
// gekko – gekko's config.js
// toml.js – gekko's toml files
config.configPriorityLocations = ['batcher', 'gekko', 'gekko-toml'];

// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
// BACKTEST BATCHER
// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

config.methods = ['neuralnet_v2'];

// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
// BRUTEFORCE SEARCHER
// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

config.method = 'neuralnet_v2';

// Shuffle generated combinations of method's configs
config.shuffle = true;

// Generate all possible combinations of set of settings with given ranges
// Format for range: 'start:step:end'
config.ranges = {
threshold_buy : '1.0:0.5:5',
threshold_sell : '-0.1:-0.1:-1',
learning_rate : '0.01:0.01:0.1',
momentum : '0.1:0.1:1',
decay : '0.01:0.01:0.01',
stoploss_enabled : true,
stoploss_threshold : '0.9:0.02:0.98',
hodl_threshold : '1:1:5',
price_buffer_len : 100,
min_predictions : 1000
};

// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
// STRATEGY SETTINGS
// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

config.neuralnet_v2 = {
threshold_buy : 4.0,
threshold_sell : -0.5,
learning_rate : 0.01,
momentum : 0.1,
decay : 0.01,
stoploss_enabled : true,
stoploss_threshold : 0.96,
hodl_threshold : 2,
price_buffer_len : 100,
min_predictions : 1000
};

module.exports = config;

execute 'node bruteforce' and exeucte javascript memory error
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

run 'npm install -g increase-memory-limit'
and
execute 'node --max_old_space_size=60000 bruteforce'

and still error
FATAL ERROR: invalid array length Allocation failed - JavaScript heap out of memory
1:
<--- Last few GCs --->

[10373:0x37350c0] 260033 ms: Scavenge 22917.5 (23406.3) -> 22914.4 (23406.3) MB, 59.7 / 0.0 ms allocation failure
[10373:0x37350c0] 260114 ms: Scavenge 22923.6 (23406.3) -> 22920.5 (23406.3) MB, 58.2 / 0.0 ms allocation failure
[10373:0x37350c0] 260198 ms: Scavenge 22929.8 (23406.3) -> 22926.7 (23406.3) MB, 60.3 / 0.0 ms allocation failure
[10373:0x37350c0] 260282 ms: Scavenge 22935.9 (23406.3) -> 22932.8 (23406.3) MB, 60.8 / 0.0 ms allocation failure

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x3939cd8a58b9
1: push(this=0x23b63ee0e7d1 <JSArray[112813858]>)
2: generateValueCombinations(aka generateValueCombinations) [/opt/gekko-batcher/node_modules/combos/lib/utils.js:41] [bytecode=0xe3e0d9b19e9 offset=191](this=0x3e608f5822d1 ,_ref=0x23b63ee0e811 <JSArray[8]>)
3: generateValueCombinations(aka generateValueCombinations) [/opt/gekko-batcher/node_modules/combos/lib/utils.js:35] ...

node::Abort() [node]
2: 0x8ccf9c [node]
3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
5: v8::internal::Heap::AllocateRawFixedArray(int, v8::internal::PretenureFlag) [node]
6: v8::internal::Heap::AllocateFixedArrayWithFiller(int, v8::internal::PretenureFlag, v8::internal::Object*) [node]
7: v8::internal::Factory::NewUninitializedFixedArray(int) [node]
8: 0xd81a1f [node]
9: 0xd95b55 [node]

==config.js==
const config = {};

config.gekkoPath = '/usr/src/app/';

config.apiUrl = "http://localhost:3001";

config.parallelQueries = 10;

config.candleSizes = [10,30,60,90,120,180];

config.historySizes = [5,10,15,20,50,100,150];

config.tradingPairs = [
["binance", "usdt", "btc"],
["binance", "usdt", "eth"],
["poloniex", "usdt", "xrp"]

];

config.daterange = {
from: '2018-06-01T00:00:00Z',
to: '2019-01-15T23:59:59Z'
};

config.paperTrader = {
simulationBalance: {
currency: 50,
asset: 0
},
feeMaker: 0.25,
feeTaker: 0.25,
feeUsing: 'maker',
slippage: 0.05,
}

config.configPriorityLocations = ['batcher', 'gekko', 'gekko-toml'];

config.methods = ['MK_RSI_BULL_BEAR'];

config.method = 'MK_RSI_BULL_BEAR';

config.shuffle = true;

config.ranges = {
SMA_long : '200:50:800',
SMA_short : '20:10:50',
BULL_RSI : 10,
BULL_FAST_RSI : 15,
BULL_SLOW_RSI : 14,
BULL_RSI_HIGH : '80:1:90,
BULL_RSI_LOW : '55:1:65',
BEAR_RSI : 15,
BEAR_FAST_RSI : 5,
BEAR_SLOW_RSI : 14,
BEAR_RSI_HIGH : '45:1:55',
BEAR_RSI_LOW : '15:1:25'
};

config.MK_RSI_BULL_BEAR = {
SMA_long : 350,
SMA_short : 20,
BULL_RSI : 10,
BULL_FAST_RSI : 5,
BULL_SLOW_RSI : 14,
BULL_RSI_HIGH : 70,
BULL_RSI_LOW : 50,
BEAR_RSI : 15,
BEAR_FAST_RSI : 3,
BEAR_SLOW_RSI : 10,
BEAR_RSI_HIGH : 45,
BEAR_RSI_LOW : 15
};

module.exports = config;

SMA_long : 200,
SMA_short : 20,
BULL_RSI : 10,
BULL_FAST_RSI : 5,
BULL_SLOW_RSI : 14,
BULL_RSI_HIGH : 70,
BULL_RSI_LOW : 50,
BEAR_RSI : 15,
BEAR_FAST_RSI : 3,
BEAR_SLOW_RSI : 10,
BEAR_RSI_HIGH : 45,
BEAR_RSI_LOW : 15

SMA Trends

MK 200 worked better - all other changes were worse

SMA_long = 1000

SMA_long = 350
SMA_short = 20

BULL

BULL_RSI = 10
BULL_FAST_RSI = 5
BULL_SLOW_RSI = 14
BULL_RSI_HIGH=85
BULL_RSI_LOW=60

BEAR

BEAR_RSI = 15
BEAR_FAST_RSI = 5
BEAR_SLOW_RSI = 14
BEAR_RSI_HIGH=50
BEAR_RSI_LOW=20

BULL/BEAR is defined by the longer SMA trends

if SHORT over LONG = BULL

if SHORT under LONG = BEAR

any suggesstions?