smartcontractkit / chainlink Goto Github PK

chainlink create {...} should create a new job spec with the parameters passed in

The next nonce for the TxManager is currently pulled from the mempool of the Ethereum node you're connected to.

https://www.pivotaltracker.com/story/show/157213214

• runat initiator w/o time
• startat time after endat
• cron initiator w/o schedule
• runAt created before startat or after endat

Hi. I tried to configure prettier/eslint with standardjs options but I couldn't get it to produce results consistent with the codebase. Can we add tips on how to achieve this with available formating tools?

Unfortunately, after my Prettier formats the code, yarn lint isn't enough to keep the commit clean from unnecessary additions/deletions.

cc @rupurt

displays account address, link balance, eth balance, unconfirmed transactions

Following instructions again. $ chainlink n produces the following error:

Is this because of not running geth first? Even if I run geth first and then chainlink node, I still get the same error.

System Information

• Network (Main net, Ropsten, etc.): Ropsten
• Ethereum client (Geth/Parity) and version: Parity 1.11.10
• Go version: 1.10
• Host OS: Amazon Linux
• Commit (INFO line when starting the node): aa0a1c6

Environment Variables

LOG_LEVEL: debug
ROOT: <redacted>
CHAINLINK_PORT: 6688
ETH_URL: <redacted>
ETH_CHAIN_ID: 3
CLIENT_NODE_URL: http://localhost:6688
TX_MIN_CONFIRMATIONS: 12
TASK_MIN_CONFIRMATIONS: 0
ETH_GAS_BUMP_THRESHOLD: 12
ETH_GAS_BUMP_WEI: 5000000000
ETH_GAS_PRICE_DEFAULT: 20000000000
LINK_CONTRACT_ADDRESS: 0x514910771AF9Ca656af840dff83E8264EcF986CA
MINIMUM_CONTRACT_PAYMENT: 1000000000000000000
ORACLE_CONTRACT_ADDRESS:
DATABASE_POLL_INTERVAL: 500ms
ALLOW_ORIGINS: http://localhost:3000,http://localhost:6688
CHAINLINK_DEV: true

Additional Information

The load I've been sending at the nodes on ropsten has been at a maximum of 20 an hour per node, I've recently doubled that and the maximum now can be 40 an hour. I'm sending 0-10 new runs to the node every 15 minutes.

To firstly show the graph of job run statuses, this has been over the last 3 days:

As seen, the amount of jobs either in_progress or pending_confirmations steadily rises, then a huge amount are confirmed and then it rises again. With the gap in metrics over last night and this morning, the stats service couldn't scrape metrics as it was timing out at the load-balancer, which is set to a minute.

Important to note: This is only happening on one node, the other node in this period was completely fine.

On that one node which is seeing this issue, these errors are in the logs every minute or two:

2018-09-17T18:06:46Z [ERROR] Application Run Channel Executor: error executing run services/job_runner.go:128 status=in_progress error=ExecuteRun: Job#: id field must not be a zero value stacktrace=github.com/smartcontractkit/chainlink/logger.Errorw
2018-09-17T18:07:17Z [ERROR] Application Run Channel Executor: error finding run services/job_runner.go:122 error=not found stacktrace=github.com/smartcontractkit/chainlink/logger.Errorw
2018-09-17T18:07:17Z [ERROR] Application Run Channel Executor: error executing run services/job_runner.go:128 stacktrace=github.com/smartcontractkit/chainlink/logger.Errorw
/go/src/github.com/smartcontractkit/chainlink/services/job_runner.go:105 status=in_progress error=ExecuteRun: Job#: id field must not be a zero value

node_in_progress_error.log

I've also attached a subset of the logs from one of the jobs that is stuck in pending_confirmations. The logs is the output of a search of the run id f314c9616c25432493aff486fcf88b3b. When filtering this by the day, the subset of logs attached can be seen outputted all throughout the day, pretty much every minute.

I've double checked that our Parity instances on Ropsten haven't fallen out of sync, and they're fine and still syncing new blocks in sub 100ms on average.

During this time, there was also responses back from the node to the Oracle contract:
https://ropsten.etherscan.io/txs?a=0xd9675316cf0eb95000c0c57e003217199f19ec80&p=10

A job spec needs at least one initiator to start, and at least one task to do anything useful. On creation, validate that a job spec has at least one initiator and at least one task.

There is already some precedent for how to do validation.

We currently display a node's Ether balance on startup, but we also need to know the node's Link on startup.

This has a couple steps:

• add a LINK_TOKEN_CONTRACT field to the config which . (needs to be configurable, as the address will need to change on testnet, devnet...) Defaults to 0x514910771AF9Ca656af840dff83E8264EcF986CA.
• Call the Link token contract's balanceOf method to get the number of LINK with the node's address, using Ethereum's eth_call
• log the result so it is visible when first running chainlink node, adjusting LINK for decimal places(n/10^18)

Hi. There are 65 branches, most of which no loner serve any purpose. Why not clean them up?

The multiplier adapter currently only accepts integers for the times field, but should accept strings as well.

On Windows the test produces a different output because Windows executable have an .exe file extension which makes the expected test output differ from the actual output.

got:

NAME:
chainlink.test.exe - CLI for Chainlink

want:

NAME:
chainlink.test - CLI for Chainlinks

Failure:

github.com/smartcontractkit/chainlink/services_test
services\application_test.go:22:2: undefined: syscall.Kill
--- FAIL: ExampleRun (0.00s)

Cause:

syscall.Kill(syscall.Getpid(), syscall.SIGTERM)

eslintrc
package

Why is there no package.json script to eslint chainlink/gui?

https://www.pivotaltracker.com/story/show/156404056

When starting a node in debug mode, the node currently prints the API endpoints. The goal here is to print the values in config.go to output for each environment variable as well.

https://www.pivotaltracker.com/story/show/156939369

Any contributions welcome.

I followed instruction steps, but installation failed during make install.

System Information

• Network (Main net, Ropsten, etc.): Ropsten
• Ethereum client (Geth/Parity) and version: Geth
• Go version: 1.10
• Host OS: Linux
• Commit (INFO line when starting the node): 84b2e23

I am trying to point my node to my GETH node. Do I have to find and replace "ws://localhost:8546" with "ws://:8546" or is there a single config file?

Remove job from API results

(Only jobs past their end at can be archived?)

should handle error like

buf, err := ioutil.ReadAll(c.Request.Body)
if err != nil {
     logger.Error(err.Error())
}

Hello. Running geth and node now and tried yarn run start and yarn run serve. Operator UI doesn't connect to the node. Attaching picture:

System Information

• Network (Main net, Ropsten, etc.): Ropsten
• Ethereum client (Geth/Parity) and version: Parity 1.11.10
• Go version: 1.10
• Host OS: Amazon Linux
• Commit (INFO line when starting the node): aa0a1c6

Environment Variables

LOG_LEVEL: debug
ROOT: <redacted>
CHAINLINK_PORT: 6688
ETH_URL: <redacted>
ETH_CHAIN_ID: 3
CLIENT_NODE_URL: http://localhost:6688
TX_MIN_CONFIRMATIONS: 12
TASK_MIN_CONFIRMATIONS: 0
ETH_GAS_BUMP_THRESHOLD: 12
ETH_GAS_BUMP_WEI: 5000000000
ETH_GAS_PRICE_DEFAULT: 20000000000
LINK_CONTRACT_ADDRESS: 0x514910771AF9Ca656af840dff83E8264EcF986CA
MINIMUM_CONTRACT_PAYMENT: 1000000000000000000
ORACLE_CONTRACT_ADDRESS:
DATABASE_POLL_INTERVAL: 500ms
ALLOW_ORIGINS: http://localhost:3000,http://localhost:6688
CHAINLINK_DEV: true

Additional Information

Off the back of #576.

I restarted the nodes to get them back monitored, and this is absolutely spammed throughout the logs constantly:

2018-09-18T13:11:24Z [DEBUG] Woke upe52572ccafca43f9b29dcdb3e4232740worker to process 4065007 services/job_runner.go:125
2018-09-18T13:11:24Z [ERROR] Application Run Channel Executor: error executing run services/job_runner.go:128 error=ExecuteRun: Job#6ff0cfaabcf1470bb2a93fec6b53bae3: Unable to start with status in_progress job=6ff0cfaabcf1470bb2a93fec6b53bae3 run=e52572ccafca43f9b29dcdb3e4232740 stacktrace=github.com/smartcontractkit/chainlink/logger.Errorw
/go/src/github.com/smartcontractkit/chainlink/logger/logger.go:105
github.com/smartcontractkit/chainlink/services.(*jobRunner).workerLoop
/go/src/github.com/smartcontractkit/chainlink/services/job_runner.go:128
github.com/smartcontractkit/chainlink/services.(*jobRunner).channelForRun.func1
/go/src/github.com/smartcontractkit/chainlink/services/job_runner.go:105 status=in_progress

Presuming this is due to all the jobs left in a pending/in progress state when I restarted the node. The job runner doesn't just attempt to wake these runs once, rather it keeps trying which completely fills the logs.

This is more an enhancement rather than an issue. I was speaking to @thodges-gh about this, but I think it'd be best to raise an issue to discuss it.

Problem:
Upon testing and thinking of ways to set-up nodes in a high-availability fashion, due to BoltDB, it seems the only feasible way is to use shared low-latency network disks that multiple nodes mount at the same time.

When this is implemented and both nodes are booted, one opens the BoltDB file whereas the other node gets into a reboot loop where it boots, outputs fatal over and over.

Since I'm building this in a highly containerised fashion, if any node exits, AWS recovers it in around 6-7 seconds from my testing. To me that's good, but not ready for the prime-time especially when off-chain computing is fully implemented.

In the current state, running two concurrently doesn't really benefit as the second node would just keep rebooting and see failover times up to around 6-7 seconds, like as if it wasn't there at all.

Solution:
From looking at the code, it seems this error check is causing the exit:
https://github.com/smartcontractkit/chainlink/blob/master/store/models/orm.go#L32

My suggestion is rather than exiting, to output a message saying that the BoltDB is currently locked, and that it will poll every X ms (configurable) to then grab the lock if it becomes available.

With this, I could configure the second node to poll every 500ms or so, and see drastically reduced failover times which would provide a production ready HA solution.

Thoughts?

Hi there,

I am on MacOS High Sierra 10.13.3 with Go 1.10 (last version from today)

I successfully installed the required dependencies and make install the node;

Unfortunately I could not locate the executable so I manually ran the main.go file.

Here is the fatal error I got:

EtherBook:chainlink sylvain$ go run main.go node
{"level":"info","ts":1520104145.9259455,"caller":"logger/logger.go:71","msg":"Starting Chainlink Node 0.2.0 at commit unset"}
{"level":"fatal","ts":1520104145.932296,"caller":"logger/logger.go:119","msg":"bad status","stacktrace":"github.com/smartcontractkit/chainlink/logger.Fatal\n\t/usr/local/go/bin/src/github.com/smartcontractkit/chainlink/logger/logger.go:119\ngithub.com/smartcontractkit/chainlink/store.NewStore\n\t/usr/local/go/bin/src/github.com/smartcontractkit/chainlink/store/store.go:41\ngithub.com/smartcontractkit/chainlink/services.NewApplication\n\t/usr/local/go/bin/src/github.com/smartcontractkit/chainlink/services/application.go:31\ngithub.com/smartcontractkit/chainlink/cmd.ChainlinkAppFactory.NewApplication\n\t/usr/local/go/bin/src/github.com/smartcontractkit/chainlink/cmd/client.go:121\ngithub.com/smartcontractkit/chainlink/cmd.(*Client).RunNode\n\t/usr/local/go/bin/src/github.com/smartcontractkit/chainlink/cmd/client.go:37\ngithub.com/smartcontractkit/chainlink/cmd.(*Client).RunNode-fm\n\t/usr/local/go/bin/src/github.com/smartcontractkit/chainlink/main.go:46\ngithub.com/smartcontractkit/chainlink/vendor/github.com/urfave/cli.HandleAction\n\t/usr/local/go/bin/src/github.com/smartcontractkit/chainlink/vendor/github.com/urfave/cli/app.go:490\ngithub.com/smartcontractkit/chainlink/vendor/github.com/urfave/cli.Command.Run\n\t/usr/local/go/bin/src/github.com/smartcontractkit/chainlink/vendor/github.com/urfave/cli/command.go:210\ngithub.com/smartcontractkit/chainlink/vendor/github.com/urfave/cli.(*App).Run\n\t/usr/local/go/bin/src/github.com/smartcontractkit/chainlink/vendor/github.com/urfave/cli/app.go:255\nmain.Run\n\t/usr/local/go/bin/src/github.com/smartcontractkit/chainlink/main.go:61\nmain.main\n\t/usr/local/go/bin/src/github.com/smartcontractkit/chainlink/main.go:12\nruntime.main\n\t/usr/local/Cellar/go/1.10/libexec/src/runtime/proc.go:198"}
exit status 1

Can't finish install.

RUN:
admin@LINKnode1:~/go/src/github.com/smartcontractkit/chainlink$ make install

RESULT:
dep ensure -vendor-only
make: dep: Command not found
Makefile:27: recipe for target 'godep' failed
make: *** [godep] Error 127

Go version = go1.10.3
HOST = Ubuntu 16.04.4
dep installed via brew

Hello and thank you for your interest in Chainlink!

This issue tracker may be used to submit bug reports and feature requests. If you have a question about running a node, feel free to ask us on Gitter first, and we'll be happy to assist you.

If you came across an error while running the node, be sure to look at our Troubleshooting page first to make sure we don't already have a documented fix. If you don't see the error listed there, please fill out as much as possible below so that we can help.

System Information

• Network (Main net, Ropsten, etc.):
• Ethereum client (Geth/Parity) and version:
• Go version:
• Host OS:
• Commit (INFO line when starting the node):

Environment Variables

(Paste the output of the environment variables when running the node in debug mode here)

Additional Information

(Only what's applicable)

• Spec:

{}

• Oracle contract address:
• Transaction ID:

System Information

• Network (Main net, Ropsten, etc.): Ropsten
• Ethereum client (Geth/Parity) and version: Parity 1.11.7
• Go version: 1.10
• Host OS: Amazon Linux
• Commit (INFO line when starting the node): 7946cd2

Environment Variables

LOG_LEVEL: debug
ROOT: <redacted>
CHAINLINK_PORT: 6688
ETH_URL: <redacted>
ETH_CHAIN_ID: 3
CLIENT_NODE_URL: http://localhost:6688
TX_MIN_CONFIRMATIONS: 12
TASK_MIN_CONFIRMATIONS: 0
ETH_GAS_BUMP_THRESHOLD: 12
ETH_GAS_BUMP_WEI: 5000000000
ETH_GAS_PRICE_DEFAULT: 20000000000
LINK_CONTRACT_ADDRESS: 0x514910771AF9Ca656af840dff83E8264EcF986CA
MINIMUM_CONTRACT_PAYMENT: 1000000000000000000
ORACLE_CONTRACT_ADDRESS:
DATABASE_POLL_INTERVAL: 500ms
ALLOW_ORIGINS: http://localhost:3000,http://localhost:6688
CHAINLINK_DEV: true

Additional Information

When a new node is booted and it attempts to respond on chain to a new spec run, it gets the following error:

2018-08-07T10:56:07Z [DEBUG] Task ethtx services/job_runner.go:250 taskrun=1cdf614700084e1585c028e95ee0eda1 task=4 result=TxManager CreateTX authentication needed: password or unlock type=ethtx params=
{
    "address": "0xB8486925b76b6a151aAe205d4CcCAdBD829e2B60",
    "confirmations": 0,
    "dataPrefix": "0x62b026a353c54cd24c0ffade0ce8255794e1a45fe5ff7e61f7bf864e7859b1a2",
    "functionSelector": "0x76005c26",
    "path": [
        "USD"
    ],
    "type": "ethtx",
    "url": "https://min-api.cryptocompare.com/data/price?fsym=ETH&tsyms=USD,EUR,JPY"
}

This causes the spec run status to be errored once this error is shown. Just to confirm, this error is only shown when the wallet is a new account, as this is shown:
There are no accounts, creating a new account with the specified password

Once I reboot the node, this error is never shown again and it completes spec runs fine.

https://www.pivotaltracker.com/story/show/156857908

It’s brittle, makes tests fail offline, and makes Little Snitch complain. Replace with instances of httptestserver/or cutest.MockServer.

Should the Amount validation:

if input.Amount != nil &&
		store.Config.MinimumContractPayment.Cmp(input.Amount) > 0 {
		msg := fmt.Sprintf(
			"Rejecting job %s with payment %s below minimum threshold (%s)",
			job.ID,
			input.Amount,
			store.Config.MinimumContractPayment.Text(10))
		run = run.ApplyResult(input.WithError(errors.New(msg)))
	}

Be moved to the ValidateRunLog function?

Right now when running go vet the errors are mostly of one type right now: ... composite literal uses unkeyed fields. To fix this, I believe you have to change object initialization from using order based parameters to named parameters. For example:

t := models.Time{now}

becomes

t := models.Time{Time: now}

There may be a few other errors, which we definitely want to eliminate as well, but this should be the vast majority of them. Eliminating this current set of errors will enable us to turn on automated checks, and get alerts when new, possibly more serious, vet errors come in.

System Information

• Network (Main net, Ropsten, etc.): Ropsten
• Ethereum client (Geth/Parity) and version: Parity 1.11.7
• Go version: 1.10
• Host OS: Amazon Linux
• Commit (INFO line when starting the node): 7946cd2

Environment Variables

LOG_LEVEL: debug
ROOT: <redacted>
CHAINLINK_PORT: 6688
ETH_URL: <redacted>
ETH_CHAIN_ID: 3
CLIENT_NODE_URL: http://localhost:6688
TX_MIN_CONFIRMATIONS: 12
TASK_MIN_CONFIRMATIONS: 0
ETH_GAS_BUMP_THRESHOLD: 12
ETH_GAS_BUMP_WEI: 5000000000
ETH_GAS_PRICE_DEFAULT: 20000000000
LINK_CONTRACT_ADDRESS: 0x514910771AF9Ca656af840dff83E8264EcF986CA
MINIMUM_CONTRACT_PAYMENT: 1000000000000000000
ORACLE_CONTRACT_ADDRESS:
DATABASE_POLL_INTERVAL: 500ms
ALLOW_ORIGINS: http://localhost:3000,http://localhost:6688
CHAINLINK_DEV: true

Additional Information

The spec I'm using is from the repo for Consumer in examples, the one requesting:
https://min-api.cryptocompare.com/data/price?fsym=ETH&tsyms=USD,EUR,JPY

This error is outputted when executing EthTx in the spec:

2018-08-04T15:56:56Z [ERROR] EthTx Adapter Perform Resuming: json: cannot unmarshal non-string into Go struct field TxReceipt.blockNumber of type *hexutil.Big adapters/eth_tx.go:67 stacktrace=github.com/smartcontractkit/chainlink/logger.Error
/go/src/github.com/smartcontractkit/chainlink/logger/logger.go:135
github.com/smartcontractkit/chainlink/adapters.ensureTxRunResult
/go/src/github.com/smartcontractkit/chainlink/adapters/eth_tx.go:67
github.com/smartcontractkit/chainlink/adapters.(*EthTx).Perform
/go/src/github.com/smartcontractkit/chainlink/adapters/eth_tx.go:27
github.com/smartcontractkit/chainlink/services.startTask
/go/src/github.com/smartcontractkit/chainlink/services/job_runner.go:283
github.com/smartcontractkit/chainlink/services.ExecuteRunAtBlock
/go/src/github.com/smartcontractkit/chainlink/services/job_runner.go:231
github.com/smartcontractkit/chainlink/services.(*jobRunner).workerLoop
/go/src/github.com/smartcontractkit/chainlink/services/job_runner.go:111
github.com/smartcontractkit/chainlink/services.(*jobRunner).channelForRun.func1
/go/src/github.com/smartcontractkit/chainlink/services/job_runner.go:89

The node actually fired the transaction, as seen here:
https://ropsten.etherscan.io/tx/0xb071411150d7366bd4ba3a25fce044584bcce7a2fd13c1a9c7cd5c6e3305f683

But the node seems to loop waiting for confirmations and always getting 0:

2018-08-04T16:01:22Z [DEBUG] Task ethtx pending_confirmations services/job_runner.go:250 type=ethtx status=pending_confirmations task=0 params=
{
    "address": "0x763741cAa59cF41823b9cED24679eA7FF2EFC9c4",
    "confirmations": 0,
    "dataPrefix": "0xb3d16f5ad3d927bcaed569016c5f5729e4e29b7acc1307c2bc26c64a4e688dd6",
    "functionSelector": "0x76005c26",
    "path": [
        "USD"
    ],
    "type": "ethtx",
    "url": "https://min-api.cryptocompare.com/data/price?fsym=ETH&tsyms=USD,EUR,JPY",
    "value": "0xb071411150d7366bd4ba3a25fce044584bcce7a2fd13c1a9c7cd5c6e3305f683"
}
taskrun=9cda070ade1441c4b769239d2206e2e7

At the point of that log entry, it had around 33 confirmations.

From what I've followed in the logs, it marks it as complete after a while on 0 confirmations.

Upon further investigation, the status of the spec run is marked as complete rather than errored also.

git fetch
git reset --hard origin/master
go test ./...

{"level":"warn","ts":1522402636.138564,"caller":"logger/logger.go:71","msg":"Unable to update latest block header","err":"EthMock: Method eth_getBlockByNumber not registered"}
{"level":"info","ts":1522402636.1555693,"caller":"logger/logger.go:61","msg":"Web request","method":"POST","status":200,"path":"/v2/specs","query":"","body":"{\r\n "initiators": [{ "type": "web" }],\r\n "tasks": [\r\n { "type": "HttpGet", "url": "https://bitstamp.net/api/ticker/" },\r\n { "type": "JsonParse", "path": ["last"] },\r\n { "type": "EthBytes32" },\r\n {\r\n "type": "EthTx",\r\n "address": "0x356a04bce728ba4c62a30294a55e6a8600a320b3",\r\n "functionSelector": "0x609ff1bd"\r\n }\r\n ]\r\n}\r\n","clientIP":"127.0.0.1","comment":"","servedAt":"2018/03/30 - 02:37:16","latency":"9.5024ms"}
{"level":"info","ts":1522402636.1575668,"caller":"logger/logger.go:61","msg":"Web request","method":"POST","status":200,"path":"/v2/specs/0b8ed3251eb644af815ec3b1954a290f/runs","query":"","body":"","clientIP":"127.0.0.1","comment":"","servedAt":"2018/03/30 - 02:37:16","latency":"0s"}
{"level":"info","ts":1522402636.1675646,"caller":"logger/logger.go:61","msg":"Starting job","job":"0b8ed3251eb644af815ec3b1954a290f","run":"f222724624a445f980ef010474dcb568","status":"in progress"}
{"level":"info","ts":1522402636.960366,"caller":"logger/logger.go:61","msg":"Finished current job run execution","job":"0b8ed3251eb644af815ec3b1954a290f","run":"f222724624a445f980ef010474dcb568","status":"pending_external"}
{"level":"info","ts":1522402637.000395,"caller":"logger/logger.go:61","msg":"Starting job","job":"0b8ed3251eb644af815ec3b1954a290f","run":"f222724624a445f980ef010474dcb568","status":"in progress"}
{"level":"info","ts":1522402637.0078933,"caller":"logger/logger.go:61","msg":"Bumping gas to 25000000000 for transaction 0xa08c9a66509581074abe162d68386fcabcb9f1a2381c2b6c2b7cdf853b02343c","txat":{"Hash":"0xa08c9a66509581074abe162d68386fcabcb9f1a2381c2b6c2b7cdf853b02343c","TxID":1,"GasPrice":25000000000,"Confirmed":false,"Hex":"0xf88b8201008505d21dba008307a12094356a04bce728ba4c62a30294a55e6a8600a320b380a4609ff1bd31303538332e37350000000000000000000000000000000000000000000000002aa04a26a6f79c846d29d9f9617d5bf05930202fd9280733322f640f08efd0725bfaa021e06980788e2be9215b39f220e894f6ae84a1486f0f1d8a0a9ba720ec0ed751","SentAt":23459}}
{"level":"info","ts":1522402637.0158913,"caller":"logger/logger.go:61","msg":"Finished current job run execution","job":"0b8ed3251eb644af815ec3b1954a290f","run":"f222724624a445f980ef010474dcb568","status":"pending_external"}
{"level":"info","ts":1522402637.0469034,"caller":"logger/logger.go:61","msg":"Starting job","job":"0b8ed3251eb644af815ec3b1954a290f","run":"f222724624a445f980ef010474dcb568","status":"in progress"}
{"level":"info","ts":1522402637.055407,"caller":"logger/logger.go:61","msg":"Finished current job run execution","job":"0b8ed3251eb644af815ec3b1954a290f","run":"f222724624a445f980ef010474dcb568","status":"pending_external"}
{"level":"info","ts":1522402637.0784159,"caller":"logger/logger.go:61","msg":"Starting job","job":"0b8ed3251eb644af815ec3b1954a290f","run":"f222724624a445f980ef010474dcb568","status":"in progress"}
{"level":"info","ts":1522402637.0904086,"caller":"logger/logger.go:61","msg":"Bumping gas to 30000000000 for transaction 0x978fea8979bd837e81dd7e6387d058ebc86c14ffa2493b8dd8ea089943a37b8a","txat":{"Hash":"0x978fea8979bd837e81dd7e6387d058ebc86c14ffa2493b8dd8ea089943a37b8a","TxID":1,"GasPrice":30000000000,"Confirmed":false,"Hex":"0xf88b8201008506fc23ac008307a12094356a04bce728ba4c62a30294a55e6a8600a320b380a4609ff1bd31303538332e37350000000000000000000000000000000000000000000000002aa02b643f9c0641d4b3bb1d712da28ebba523faf5e0afd6fa5f5f11585983bf4039a036f4e6ff8487413c2b77fa4c91d78de7bda9c0ffe1b80cd16d23e5133e4dbc54","SentAt":23466}}
{"level":"info","ts":1522402637.1011639,"caller":"logger/logger.go:61","msg":"Confirmed tx 0xb7862c896a6ba2711bccc0410184e46d793ea83b3e05470f1d359ea276d16bb5","txat":{"Hash":"0xb7862c896a6ba2711bccc0410184e46d793ea83b3e05470f1d359ea276d16bb5","TxID":1,"GasPrice":20000000000,"Confirmed":true,"Hex":"0xf88b8201008504a817c8008307a12094356a04bce728ba4c62a30294a55e6a8600a320b380a4609ff1bd31303538332e373500000000000000000000000000000000000000000000000029a053ccecb237a486eea77b0c16445a9fca8ec5805ca27ad2801c9e0c9db83faa7ea075b68922e8b925d63762aee1ebdef15fa0fe08c70ca31e319d6e3847fc480471","SentAt":23456},"receipt":{"blockNumber":"0x5ba4","transactionHash":"0xb7862c896a6ba2711bccc0410184e46d793ea83b3e05470f1d359ea276d16bb5"}}
{"level":"info","ts":1522402637.1091592,"caller":"logger/logger.go:61","msg":"Finished current job run execution","job":"0b8ed3251eb644af815ec3b1954a290f","run":"f222724624a445f980ef010474dcb568","status":"completed"}
{"level":"info","ts":1522402637.125895,"caller":"logger/logger.go:87","msg":"Gracefully exiting..."}
{"level":"warn","ts":1522402637.2403963,"caller":"logger/logger.go:71","msg":"Unable to update latest block header","err":"EthMock: Method eth_getBlockByNumber not registered"}
{"level":"info","ts":1522402637.2563999,"caller":"logger/logger.go:61","msg":"Listening for runlog from block #0 (0x0) for address [all] for job a1d3b09672594a3d9eef020ed19bb731"}
{"level":"info","ts":1522402637.2563999,"caller":"logger/logger.go:61","msg":"Web request","method":"POST","status":200,"path":"/v2/specs","query":"","body":"{\r\n "initiators": [{"type": "runLog"}],\r\n "tasks": [{"type": "NoOp","confirmations": 100}]\r\n}\r\n","clientIP":"127.0.0.1","comment":"","servedAt":"2018/03/30 - 02:37:17","latency":"8.505ms"}
{"level":"info","ts":1522402637.2729003,"caller":"logger/logger.go:61","msg":"Starting job","job":"a1d3b09672594a3d9eef020ed19bb731","run":"f7795ac87ccc4606b1a52424b8d37037","status":"in progress"}
{"level":"info","ts":1522402637.2798994,"caller":"logger/logger.go:61","msg":"Finished current job run execution","job":"a1d3b09672594a3d9eef020ed19bb731","run":"f7795ac87ccc4606b1a52424b8d37037","status":"pending_confirmations"}
{"level":"info","ts":1522402637.3078997,"caller":"logger/logger.go:61","msg":"Starting job","job":"a1d3b09672594a3d9eef020ed19bb731","run":"f7795ac87ccc4606b1a52424b8d37037","status":"in progress"}
{"level":"info","ts":1522402637.3103962,"caller":"logger/logger.go:87","msg":"Gracefully exiting..."}
{"level":"info","ts":1522402637.3143966,"caller":"logger/logger.go:61","msg":"Finished current job run execution","job":"a1d3b09672594a3d9eef020ed19bb731","run":"f7795ac87ccc4606b1a52424b8d37037","status":"pending_confirmations"}
--- FAIL: TestIntegration_RunLog (0.19s)
testing_t_support.go:22:
C:/GoPath/src/github.com/smartcontractkit/chainlink/vendor/github.com/onsi/gomega/internal/asyncassertion/async_assertion.go:154 +0x6f8
github.com/smartcontractkit/chainlink/vendor/github.com/onsi/gomega/internal/asyncassertion.(*AsyncAssertion).Should(0xc042454600, 0xd34700, 0xc0421a9250, 0x0, 0x0, 0x0, 0xd31a80)
C:/GoPath/src/github.com/smartcontractkit/chainlink/vendor/github.com/onsi/gomega/internal/asyncassertion/async_assertion.go:48 +0x69
github.com/smartcontractkit/chainlink/internal/cltest.JobRunStays(0xc04224aff0, 0xc042102f00, 0xc052359940, 0x20, 0xc052359980, 0x20, 0x0, 0x0, 0x5, 0xc0523c2f18, ...)
C:/GoPath/src/github.com/smartcontractkit/chainlink/internal/cltest/cltest.go:472 +0x223
github.com/smartcontractkit/chainlink/internal/cltest.JobRunStaysPendingConfirmations(0xc04224aff0, 0xc042102f00, 0xc052359940, 0x20, 0xc052359980, 0x20, 0x0, 0x0, 0x5, 0xc0523c2f18, ...)
C:/GoPath/src/github.com/smartcontractkit/chainlink/internal/cltest/cltest.go:481 +0xb4
github.com/smartcontractkit/chainlink/web_test.TestIntegration_RunLog(0xc04224aff0)
C:/GoPath/src/github.com/smartcontractkit/chainlink/web/integration_test.go:182 +0x6c4
testing.tRunner(0xc04224aff0, 0xcbec80)
C:/Go/src/testing/testing.go:777 +0xd7
created by testing.(*T).Run
C:/Go/src/testing/testing.go:824 +0x2e7
Failed after 0.021s.
Expected
<models.RunStatus>: in progress
to equal
<models.RunStatus>: pending_confirmations

Get the following error while trying to install on MacOS High Sierra 10.13.3

# github.com/smartcontractkit/chainlink/utils
go/src/github.com/smartcontractkit/chainlink/utils/utils.go:115:35: multiple-value uuid.NewV4() in single-value context

If we take down devnet and bring it back up, we shouldn't have to restart Chainlink just to reconnect eth subscriptions.

This includes refactoring subscriptions to truly be concurrent (w their own logs channel) and not collapse everything into nl.logs.

System Information

• Network (Main net, Ropsten, etc.): Ropsten
• Ethereum client (Geth/Parity) and version: Parity//v1.11.7-stable-085035f-20180717/x86_64-linux-gnu/rustc1.27.1
• Go version: 1.10
• Host OS: Amazon Linux AMI release 2018.03
• Commit (INFO line when starting the node): 6e9f542

Environment Variables

CHAINLINK_PORT: 6688
USERNAME: <redacted>
ETH_URL: ws://parity-ws.ropsten.linkpool.io
ETH_CHAIN_ID: 3
CLIENT_NODE_URL: http://localhost:6688
TX_MIN_CONFIRMATIONS: 12
TASK_MIN_CONFIRMATIONS: 0
ETH_GAS_BUMP_THRESHOLD: 12
ETH_GAS_BUMP_WEI: 5000000000
ETH_GAS_PRICE_DEFAULT: 20000000000
LINK_CONTRACT_ADDRESS: 0x514910771AF9Ca656af840dff83E8264EcF986CA
MINIMUM_CONTRACT_PAYMENT: 1000000000000000000
ORACLE_CONTRACT_ADDRESS:
DATABASE_POLL_INTERVAL: 500ms
ALLOW_ORIGINS: http://localhost:3000,http://localhost:6688

Additional Information

To give some background, I've recently been working on a metrics gatherer for nodes allowing me to graph data and provide a WS api to our front-end.

Part of the metric gathering is to retrieve all the spec runs to collate all of the statuses, how many there's been and what adaptors are used. The service that gathers metrics fires each request in its own goroutine, resulting in 80+ API calls for each spec within 1s~. I understand this is very inefficient, but this is a good reason to create an API endpoint that outputs overall statistics, rather than needing to call each spec run individually to gather data. An example of the gathered data:

{
  "0xC491f0d6fB2b4F0853a2DE78aaCC91849729ebf4": {
    "totalSpecs": 41,
    "totalSpecRuns": 41,
    "totalBridges": 1,
    "address": "0xC491f0d6fB2b4F0853a2DE78aaCC91849729ebf4",
    "ethBalance": "3400000000000000000",
    "linkBalance": "0",
    "specTaskCounts": {
      "ethbytes32": 3,
      "ethint256": 1,
      "ethtx": 41,
      "ethuint256": 37,
      "httpget": 41,
      "jsonparse": 41,
      "multiply": 38,
      "sleep": 1
    },
    "specRunStatusCounts": {
      "errored": 41
    }
  }
}

Anyway, after firing all these requests off I'm seeing large degradation of performance, with response times often exceeding 1m. To give some examples:

[INFO] Web request web/router.go:143 clientIP=10.0.92.207 servedAt=2018/07/27 - 11:05:58 method=GET status=200 path=/v2/specs/ff9148d5597847e7b7f679ff01187a38/runs latency=1m3.307433551s
[INFO] Web request web/router.go:143 method=GET status=200 clientIP=10.0.92.207 path=/v2/specs/dc458dd7cc1f4cf98223d6bb770fa634/runs servedAt=2018/07/27 - 11:06:29 latency=1m18.816509244s

The highest I've seen is over 2m.

Meanwhile, the CPU/RAM usage of the node while responses are this high aren't extraordinarily high. Seeing averages of around 12% CPU, 3% RAM usage. Disk usage isn't the bottleneck either.

Maybe the node is waiting on bucket locks or something? If this is something that can be improved, I don't mind picking up the work for this.

OneTime.AddJob is only tested through integration tests, making it difficult for us to confidently refactor. Add unit tests, similar to Recurring.AddJob, for OneTime.AddJob.

Fired up an AWS AMI instance, installed Go 1.10.
Then,
$ go get -u github.com/smartcontractkit/chainlink

Immediately see this error:
Projects/src/github.com/smartcontractkit/chainlink/store/models/eth.go:33:5: cannot use tx.GasLimit (type *big.Int) as type uint64 in argument to types.NewTransaction

I've been testing on Ropsten with my own node so I can re-deploy the contracts and use my own specs, one of them being using the assetPrice external adaptor.

Here's the job spec:

{
   "initiators":[
      {
         "type":"runlog",
         "address": ""
      }
   ],
   "tasks":[
      {
         "type":"assetPrice"
      },
      {
         "type":"jsonParse"
      },
      {
         "type":"ethuint256"
      },
      {
         "type":"ethtx"
      }
   ]
}

Consumer contract:

pragma solidity ^0.4.23;

import "../Chainlinked.sol";
import "../lib/strings.sol";
import "openzeppelin-solidity/contracts/ownership/Ownable.sol";

contract AssetPriceConsumer is Chainlinked, Ownable {
  bytes32 internal jobId;
 
  mapping(bytes32 => uint256) public assetPrices;
  mapping(bytes32 => bytes32) public requests;

  using strings for string;
  using strings for strings.slice;

  event RequestFulfilled(
    bytes32 indexed requestId,
    uint256 indexed price
  );

  constructor(address _link, address _oracle, bytes32 _jobId) public {
    setLinkToken(_link);
    setOracle(_oracle);
    jobId = _jobId;
  }

  function getAssetPrice(string base, string quote) public {
    ChainlinkLib.Run memory run = newRun(jobId, this, "fulfill(bytes32,bytes32)");
    run.add("base", base);
    run.add("quote", quote);
    string[] memory path = new string[](1);
    path[0] = "price";
    run.addStringArray("path", path);
    bytes32 requestId = chainlinkRequest(run, LINK(1));
    requests[requestId] = keccak256(base, quote);
  }

  function cancelRequest(uint256 _requestId)
    public
    onlyOwner
  {
    oracle.cancel(_requestId);
  }

  function fulfill(bytes32 _requestId, uint256 _price)
    public
    onlyOracle
    checkRequestId(_requestId)
  {
    emit RequestFulfilled(_requestId, _price);
    assetPrices[requests[_requestId]] = _price;
  }

  modifier checkRequestId(bytes32 _requestId) {
    require(requests[_requestId] != keccak256(0));
    _;
  }

}

When I fire off a tx for getAssetPrice, I can see the node starting to execute the tasks:

2018-05-06T14:27:08Z [INFO] Starting job services/job_runner.go:94 job=7077c72ac18e4de58621245800ccf999 run=46a2a9f361d1498f9f846e46a9be56d3 status=in_progress
2018-05-06T14:27:09Z [DEBUG] Produced task run services/job_runner.go:132 taskRun=TaskRun(c84fc923819e48d184592dc53ba7c89a,assetpricenew,completed,)

Logs from the bridge type:

06/May/2018:14:27:08 +000 200 351630μs "POST /price HTTP/1.1" - "Go-http-client/1.1"

JSON from the assetPrice bridge type:

{
  "jobRunId": "1234",
  "data": {
    "base": "LINK",
    "quote": "ETH",
    "id": "LINK-ETH",
    "price": "0.00067659",
    "volume": "760207",
    "exchanges": [
      "Binance"
    ]
  },
  "status": "",
  "error": null,
  "pending": false
}

Error from JSONParse:

2018-05-06T14:27:09Z [DEBUG] Task jsonparse services/job_runner.go:133 task=1 type=jsonparse params=
{
    "address": "0xF790026Adf478AAE540c36c2De6f3DF7A4f1c7a8",
    "base": "LINK",
    "dataPrefix": "0x0000000000000000000000000000000000000000000000000000000000000009",
    "functionSelector": "76005c26",
    "path": [
        "price"
    ],
    "quote": "ETH",
    "type": "jsonParse"
}
taskrun=b54529fedc2847b89625fe4e2f1beccf result=non string value

Originally when I was getting this, the bridge type was returning the price as a float value and on refactoring so it returns a string, got the same error.

It was my understanding that anything within data in the JSON is merged throughout the tasks so it's persisted through to the end, but it seems it's not being included here?

Similar to the CLI command chainlink create, add a CLI command chainlink run that takes a job ID and JSON optional parameters. The run command posts to the /jobs/$JOB_ID/runs endpoint, which has the same permissions as that endpoint, so creating a job run is only allowed if the job spec has a web initiator.

Hey guys,

Found several security issues at the website of yours. These issues could have a big impact on the business of ChainLink itself. Because I don't want to make these issues public available, I'll just describe the title(s) of these issues.

1.Fill the whole the database storage capacity
2.User Enumeration/Sensitive data exposure(on the website)
3.Malicious File upload
4.Broken access control

And on another domain:

1.Sensitive data exposure

For more information just contact me. Have in mind that a reward linked to this would be nice :-)

Kind regards,

Quikko

As of now, we have two approaches:

1. Standard approach of creating a docker image with devnet or geth preconfigured with unlocked accounts to communicate with chainlink via JSONRPC over HTTP and ws.
2. Novel approach of staying entirely in the golang ecosystem and testing chainlink against geth inside goroutines in the same process, still using JSONRPC over HTTP and ws.

Let's discuss the pros and cons of each and perhaps come up with other ideas.

bytes32 and uint256 is currently supported, but not int256.

Bridge Types are how a node operator defines the custom task types, bridging a task to an separate service, known as an External Adapter. Bridge Types can be created via a POST to the API /bridge_types. On creation, make sure that the new bridge type name does not collide with any other adapter, including the core adapters.

There is already some precedent for how to handle validation.

The chainlink create CLI command currently accepts JSON. It should optionally accept a file path where it reads the JSON from instead.

We're open to ideas about the exact API, open to suggestions. Ideally any argument that doesn't start with a left curly bracket would be identified as a file path which curl tries to look up and read. The no curly left bracket to start a file name seems like a reasonable assumption(you should probably be starting your paths with /..., ./..., or ~/... anyway). If there's a problem with that, passing a flag or starting with "@" like curl does seems like a good way forward.

Open to feedback.