Comments (7)
pkittenis
commented on June 18, 2024
The docker container is not supposed to connect to localhost:8086 - it connects to dockerhost:8086 instead which is the host ip running the container and also running influx. That's done so you do not have give containers access to the hosts network.
See config.
curl dockerhost:8086/ping should work from the container.
For your question, index build is probably taking longer than gunicorn worker timeout, will need to increase it in that case.
Turn on influxgraph debug logging and will see what it is doing when gunicorn thinks it's timing out. It will not respond to queries before initial index build as it is required for queries to work. Might need a flag to disable this though for such cases.
from influxgraph.
benmathews
commented on June 18, 2024
The dockerhost isn't getting set right. From within the container.
root@27a2e97459dc:/# grep dockerhost /etc/hosts
172.17.0.1 dockerhost
root@27a2e97459dc:/# curl -v dockerhost:8086/ping
* Trying 172.17.0.1...
* connect to 172.17.0.1 port 8086 failed: Connection refused
* Failed to connect to dockerhost port 8086: Connection refused
* Closing connection 0
curl: (7) Failed to connect to dockerhost port 8086: Connection refused
I created my own container w/
finders:
- influxgraph.InfluxDBFinder
influxdb:
host: 172.31.13.13
port: 4001
host: localhost
log_level: debug
memcache:
host: localhost
And launched it w/ --network=host. The log is filled with errors like
[DEBUG] 2017-05-09 16:03:26,775 - finder.__init__() - Configured aggregation functions - {<SRE_Pattern object at 0x00000000015d1f60>: 'min', <SRE_Pattern object at 0x0000000001676020>: 'max', <SRE_Pattern object at 0x0000000001676060>: 'last', <SRE_Pattern object at 0x00000000016760a0>: 'sum'}
[INFO] 2017-05-09 16:03:26,776 - finder._start_loader() - Starting initial series list load - this may take several minutes on databases with a large number of series..
[DEBUG] 2017-05-09 16:03:26,777 - finder._get_series() - Series loader calling influxdb with query - SHOW SERIES LIMIT 10000 OFFSET 0
[ERROR] 2017-05-09 16:03:26,783 - finder._start_loader() - Error calling InfluxDB from initial series and field list load -
[INFO] 2017-05-09 16:03:26,784 - finder._series_loader() - Starting background series loader with interval 900
[INFO] 2017-05-09 16:03:26,785 - finder.build_index() - Starting index build
[DEBUG] 2017-05-09 16:03:26,785 - finder._get_series() - Series loader calling influxdb with query - SHOW SERIES LIMIT 10000 OFFSET 0
[ERROR] 2017-05-09 16:03:26,791 - finder.build_index() - Error getting series list from InfluxDB - -Retrying after 30sec..
[DEBUG] 2017-05-09 16:03:26,799 - finder.__init__() - Configured aggregation functions - {<SRE_Pattern object at 0x000000000181e460>: 'min', <SRE_Pattern object at 0x000000000181e4a0>: 'max', <SRE_Pattern object at 0x000000000181e4e0>: 'last', <SRE_Pattern object at 0x000000000181e520>: 'sum'}
[DEBUG] 2017-05-09 16:03:26,800 - finder._start_loader() - Series loader mutex exists influxgraph_series_loader - skipping series load
[INFO] 2017-05-09 16:03:26,801 - finder.build_index() - Starting index build
[INFO] 2017-05-09 16:03:26,801 - finder._series_loader() - Starting background series loader with interval 900
[DEBUG] 2017-05-09 16:03:26,802 - finder._get_series() - Series loader calling influxdb with query - SHOW SERIES LIMIT 10000 OFFSET 0
[ERROR] 2017-05-09 16:03:26,808 - finder.build_index() - Error getting series list from InfluxDB - -Retrying after 30sec..
[INFO] 2017-05-09 16:03:56,793 - finder.build_index() - Starting index build
[DEBUG] 2017-05-09 16:03:56,795 - finder._get_series() - Series loader calling influxdb with query - SHOW SERIES LIMIT 10000 OFFSET 0
[ERROR] 2017-05-09 16:03:56,821 - finder.build_index() - Error getting series list from InfluxDB - -Retrying after 30sec..
[INFO] 2017-05-09 16:03:56,837 - finder.build_index() - Starting index build
[DEBUG] 2017-05-09 16:03:56,838 - finder._get_series() - Series loader calling influxdb with query - SHOW SERIES LIMIT 10000 OFFSET 0
[ERROR] 2017-05-09 16:03:56,848 - finder.build_index() - Error getting series list from InfluxDB - -Retrying after 30sec..
[INFO] 2017-05-09 16:04:26,842 - finder.build_index() - Starting index build
[DEBUG] 2017-05-09 16:04:26,844 - finder._get_series() - Series loader calling influxdb with query - SHOW SERIES LIMIT 10000 OFFSET 0
[ERROR] 2017-05-09 16:04:26,871 - finder.build_index() - Error getting series list from InfluxDB - -Retrying after 30sec..
[INFO] 2017-05-09 16:04:26,876 - finder.build_index() - Starting index build
[DEBUG] 2017-05-09 16:04:26,878 - finder._get_series() - Series loader calling influxdb with query - SHOW SERIES LIMIT 10000 OFFSET 0
[ERROR] 2017-05-09 16:04:26,889 - finder.build_index() - Error getting series list from InfluxDB - -Retrying after 30sec..
[INFO] 2017-05-09 16:04:56,901 - finder.build_index() - Starting index build
[DEBUG] 2017-05-09 16:04:56,903 - finder._get_series() - Series loader calling influxdb with query - SHOW SERIES LIMIT 10000 OFFSET 0
[INFO] 2017-05-09 16:04:56,917 - finder.build_index() - Starting index build
[DEBUG] 2017-05-09 16:04:56,919 - finder._get_series() - Series loader calling influxdb with query - SHOW SERIES LIMIT 10000 OFFSET 0
[ERROR] 2017-05-09 16:04:56,925 - finder.build_index() - Error getting series list from InfluxDB - -Retrying after 30sec..
[ERROR] 2017-05-09 16:04:56,934 - finder.build_index() - Error getting series list from InfluxDB - -Retrying after 30sec..
[2017-05-09 16:05:26 +0000] [26] [CRITICAL] WORKER TIMEOUT (pid:47)
[2017-05-09 16:05:26 +0000] [26] [CRITICAL] WORKER TIMEOUT (pid:48)
[2017-05-09 16:05:26 +0000] [75] [INFO] Booting worker with pid: 75
However, from within the container, I could readily query the db.
# curl 'localhost:8086/query?pretty=true&db=graphite' --data-urlencode "q=show series limit 10000 offset 0"|head
{
"results": [
{
"statement_id": 0,
"series": [
{
"columns": [
"key"
],
"values": [
# curl 'localhost:8086/query?pretty=true&db=graphite' --data-urlencode "q=show series limit 10000 offset 0"|wc
30016 30021 1557234
Any ideas what might be going wrong?
from influxgraph.
pkittenis
commented on June 18, 2024
The config has two host entries and port looks wrong. The logs are saying it can't connect to influx.
host: 172.31.13.13
port: 4001
host: localhost
If from within the container, influxdb is at localhost:8086 then use the following config:
finders:
- influxgraph.InfluxDBFinder
influxdb:
host: localhost
port: 8086
log_level: debug
memcache:
host: localhost
The DB seems to be empty though so queries might not return any data. Still, they should not time out. Can remove the debug line once it works.
from influxgraph.
benmathews
commented on June 18, 2024
I think we just have a scale issue. I put some debugging code in and it took about 30 minutes to work through all the SHOW SERIES steps. I finished with
Series loader calling influxdb with query - SHOW SERIES LIMIT 10000 OFFSET 680000
Hitting the localhost:8000 endpoint still times out.
If there is some more information that I could provide, I'd be glad to help.
FYI, We have a large graphite installation that has over a million series in it and takes in about 50K points/10 seconds. I've been forking this data and pushing it into an influxdb for the last couple days.
from influxgraph.
pkittenis
commented on June 18, 2024
Sounds like you will need to use templates to split those metrics into tags.
If you disable gunicorn's timeout and let it run it will eventually load and index all those series, it will just take a while. It doesn't sound like the influxdb installation is particularly fast in your case.
By using templates the number of series in the DB is limited and performance of both the DB and influxgraph improves. See influx and influxgraph configuration for examples.
Can also try increasing loader limit configuration to see if that speeds up series loading at all.
See also sizing guidelines in influxdb documentation.
It is expected that the webapp will not respond before it can load and index metrics in the DB.
from influxgraph.
benmathews
commented on June 18, 2024
Thanks for the pointers. I'll work through this and reply back. It might take me some time to get templates set up.
from influxgraph.
pkittenis
commented on June 18, 2024
Closing as no code issue that I can see. Can re-open if one is shown.
from influxgraph.
Related Issues (20)
- Umm. am I an idiot?! HOT 7
- Fields for measurement appear in all metric paths that contain measurement
- 504 Gateway Timeout with Influx Cloud Integration HOT 4
- multiple target query fails with IndexError HOT 8
- Auto-conversion of tag values containing dots HOT 1
- Per metric path configurable fill parameters
- Build _manylinux_ binary wheels of releases HOT 1
- Double consolidation/aggregation of data points by graphite-api
- Use modified graphite-api package HOT 2
- Make block webapp on startup while building index behaviour configurable
- Support for multiple databases as prefix HOT 3
- Building and refreshing of index happens in every worker process putting high load on Influx HOT 1
- Improve InfluxDB response parsing performance
- Allow configurable prefix for retention policies
- TypeError: object of type 'NoneType' has no len() HOT 5
- Docker: make memcache max value configurable HOT 6
- Segfaults on building the index HOT 4
- Updates of Docker Image HOT 4
- Python 3.7 wheel build failing
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from influxgraph.