docker cannot perform about haipproxy HOT 6 CLOSED

I found this

from haipproxy.

daneel@daneel:~/PycharmProjects/haipproxy$ docker-compose up
Creating haipproxy_splash_1
Creating haipproxy_redis_1
Creating haipproxy_haipproxy_1
Attaching to haipproxy_splash_1, haipproxy_redis_1, haipproxy_haipproxy_1
splash_1 | 2018-03-12 15:15:53+0000 [-] Log opened.
splash_1 | 2018-03-12 15:15:53.390628 [-] Splash version: 3.2
splash_1 | 2018-03-12 15:15:53.394522 [-] Qt 5.9.1, PyQt 5.9, WebKit 602.1, sip 4.19.3, Twisted 16.1.1, Lua 5.2
splash_1 | 2018-03-12 15:15:53.394628 [-] Python 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609]
splash_1 | 2018-03-12 15:15:53.394736 [-] Open files limit: 1048576
splash_1 | 2018-03-12 15:15:53.394831 [-] Can't bump open files limit
splash_1 | 2018-03-12 15:15:53.501162 [-] Xvfb is started: ['Xvfb', ':108054654', '-screen', '0', '1024x768x24', '-nolisten', 'tcp']
splash_1 | QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
splash_1 | 2018-03-12 15:15:53.861056 [-] proxy profiles support is enabled, proxy profiles path: /etc/splash/proxy-profiles
splash_1 | 2018-03-12 15:15:54.390841 [-] verbosity=1
splash_1 | 2018-03-12 15:15:54.391009 [-] slots=50
splash_1 | 2018-03-12 15:15:54.391138 [-] argument_cache_max_entries=500
splash_1 | 2018-03-12 15:15:54.391566 [-] Web UI: enabled, Lua: enabled (sandbox: enabled)
splash_1 | 2018-03-12 15:15:54.391671 [-] Server listening on 0.0.0.0:8050
redis_1 | 1:C 12 Mar 15:15:53.852 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1 | 1:C 12 Mar 15:15:53.853 # Redis version=4.0.8, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1 | 1:C 12 Mar 15:15:53.853 # Configuration loaded
redis_1 | 1:M 12 Mar 15:15:53.855 * Running mode=standalone, port=6379.
redis_1 | 1:M 12 Mar 15:15:53.855 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1 | 1:M 12 Mar 15:15:53.855 # Server initialized
redis_1 | 1:M 12 Mar 15:15:53.855 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1 | 1:M 12 Mar 15:15:53.855 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1 | 1:M 12 Mar 15:15:53.855 * Ready to accept connections
splash_1 | 2018-03-12 15:15:54.392523 [-] Site starting on 8050
splash_1 | 2018-03-12 15:15:54.392665 [-] Starting factory <twisted.web.server.Site object at 0x7f7658d957f0>
haipproxy_1 | 2018/03/12 15:15:54| Set Current Directory to /var/spool/squid
haipproxy_1 | 2018/03/12 15:15:54| Starting Squid Cache version 3.5.12 for x86_64-pc-linux-gnu...
haipproxy_1 | 2018/03/12 15:15:54| Service Name: squid
haipproxy_1 | 2018/03/12 15:15:54| Process ID 13
haipproxy_1 | 2018/03/12 15:15:54| Process Roles: master worker
haipproxy_1 | 2018/03/12 15:15:54| With 1048576 file descriptors available
haipproxy_1 | 2018/03/12 15:15:54| Initializing IP Cache...
haipproxy_1 | 2018/03/12 15:15:54| DNS Socket created at [::], FD 8
haipproxy_1 | 2018/03/12 15:15:54| DNS Socket created at 0.0.0.0, FD 9
haipproxy_1 | 2018/03/12 15:15:54| Adding nameserver 127.0.0.11 from /etc/resolv.conf
haipproxy_1 | 2018/03/12 15:15:54| Adding ndots 1 from /etc/resolv.conf
haipproxy_1 | 2018/03/12 15:15:54| Logfile: opening log daemon:/var/log/squid/access.log
haipproxy_1 | 2018/03/12 15:15:54| Logfile Daemon: opening log /var/log/squid/access.log
haipproxy_1 | 2018/03/12 15:15:54| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
haipproxy_1 | 2018/03/12 15:15:54| Store logging disabled
haipproxy_1 | 2018/03/12 15:15:54| Swap maxSize 0 + 262144 KB, estimated 20164 objects
haipproxy_1 | 2018/03/12 15:15:54| Target number of buckets: 1008
haipproxy_1 | 2018/03/12 15:15:54| Using 8192 Store buckets
haipproxy_1 | 2018/03/12 15:15:54| Max Mem size: 262144 KB
haipproxy_1 | 2018/03/12 15:15:54| Max Swap size: 0 KB
haipproxy_1 | 2018/03/12 15:15:54| Using Least Load store dir selection
haipproxy_1 | 2018/03/12 15:15:54| Set Current Directory to /var/spool/squid
haipproxy_1 | 2018/03/12 15:15:55| Finished loading MIME types and icons.
haipproxy_1 | 2018/03/12 15:15:55| HTCP Disabled.
haipproxy_1 | 2018/03/12 15:15:55| commBind: Cannot bind socket FD 13 to [::1]: (99) Cannot assign requested address
haipproxy_1 | 2018/03/12 15:15:55| commBind: Cannot bind socket FD 14 to [::1]: (99) Cannot assign requested address
haipproxy_1 | 2018/03/12 15:15:55| ERROR: Failed to create helper child read FD: UDP[::1]
haipproxy_1 | 2018/03/12 15:15:55| Squid plugin modules loaded: 0
haipproxy_1 | 2018/03/12 15:15:55| Adaptation support is off.
haipproxy_1 | 2018/03/12 15:15:55| Accepting HTTP Socket connections at local=[::]:3128 remote=[::] FD 12 flags=9
haipproxy_1 | 2018/03/12 15:15:55| storeLateRelease: released 0 objects
haipproxy_1 | 2018/03/12 15:15:57| Reconfiguring Squid Cache (version 3.5.12)...
haipproxy_1 | 2018/03/12 15:15:57| Closing HTTP port [::]:3128
haipproxy_1 | 2018/03/12 15:15:57| Logfile: closing log daemon:/var/log/squid/access.log
haipproxy_1 | 2018/03/12 15:15:57| Logfile Daemon: closing log daemon:/var/log/squid/access.log
haipproxy_1 | 2018/03/12 15:15:57| Startup: Initializing Authentication Schemes ...
haipproxy_1 | 2018/03/12 15:15:57| Startup: Initialized Authentication Scheme 'basic'
haipproxy_1 | 2018/03/12 15:15:57| Startup: Initialized Authentication Scheme 'digest'
haipproxy_1 | 2018/03/12 15:15:57| Startup: Initialized Authentication Scheme 'negotiate'
haipproxy_1 | 2018/03/12 15:15:57| Startup: Initialized Authentication Scheme 'ntlm'
haipproxy_1 | 2018/03/12 15:15:57| Startup: Initialized Authentication.
haipproxy_1 | 2018/03/12 15:15:57| Processing Configuration File: /etc/squid/squid.conf (depth 0)
haipproxy_1 | 2018/03/12 15:15:57| Logfile: opening log daemon:/var/log/squid/access.log
haipproxy_1 | 2018/03/12 15:15:57| Logfile Daemon: opening log /var/log/squid/access.log
haipproxy_1 | 2018/03/12 15:15:57| Squid plugin modules loaded: 0
haipproxy_1 | 2018/03/12 15:15:57| Adaptation support is off.
haipproxy_1 | 2018/03/12 15:15:57| Store logging disabled
haipproxy_1 | 2018/03/12 15:15:57| DNS Socket created at [::], FD 10
haipproxy_1 | 2018/03/12 15:15:57| DNS Socket created at 0.0.0.0, FD 11
haipproxy_1 | 2018/03/12 15:15:57| Adding nameserver 127.0.0.11 from /etc/resolv.conf
haipproxy_1 | 2018/03/12 15:15:57| Adding ndots 1 from /etc/resolv.conf
haipproxy_1 | 2018/03/12 15:15:57| HTCP Disabled.
haipproxy_1 | 2018/03/12 15:15:57| commBind: Cannot bind socket FD 13 to [::1]: (99) Cannot assign requested address
haipproxy_1 | 2018/03/12 15:15:57| commBind: Cannot bind socket FD 14 to [::1]: (99) Cannot assign requested address
haipproxy_1 | 2018/03/12 15:15:57| ERROR: Failed to create helper child read FD: UDP[::1]
haipproxy_1 | 2018/03/12 15:15:57| Finished loading MIME types and icons.
haipproxy_1 | 2018/03/12 15:15:57| Accepting HTTP Socket connections at local=[::]:3128 remote=[::] FD 12 flags=9

from haipproxy.

every time I run docker-compose up again, it show different error?
can you help me ?

from haipproxy.

Can you send requests with squid? if you can, just leave the errors alone.It doesn't matter

from haipproxy.

connection refused

from haipproxy.

It seems there's no problem.You should connect to squid after a while, about 5 minutes later after all the docker containers are running. Because the scrapy worker will scrape some proxies and then write them to squid conf, not right now when all the containers started up.If there is no proxy in squid conf,
requests will be fail.

On my mac using haipproxy, this is the log

haipproxy_1  | 2018/03/13 06:02:12| Startup: Initializing Authentication Schemes ...
haipproxy_1  | 2018/03/13 06:02:12| Startup: Initialized Authentication Scheme 'basic'
haipproxy_1  | 2018/03/13 06:02:12| Startup: Initialized Authentication Scheme 'digest'
haipproxy_1  | 2018/03/13 06:02:12| Startup: Initialized Authentication Scheme 'negotiate'
haipproxy_1  | 2018/03/13 06:02:12| Startup: Initialized Authentication Scheme 'ntlm'
haipproxy_1  | 2018/03/13 06:02:12| Startup: Initialized Authentication.
haipproxy_1  | 2018/03/13 06:02:12| Processing Configuration File: /etc/squid/squid.conf (depth 0)
haipproxy_1  | 2018/03/13 06:02:12| Logfile: opening log daemon:/var/log/squid/access.log
haipproxy_1  | 2018/03/13 06:02:12| Logfile Daemon: opening log /var/log/squid/access.log
haipproxy_1  | 2018/03/13 06:02:12| Squid plugin modules loaded: 0
haipproxy_1  | 2018/03/13 06:02:12| Adaptation support is off.
haipproxy_1  | 2018/03/13 06:02:12| Store logging disabled
haipproxy_1  | 2018/03/13 06:02:12| DNS Socket created at [::], FD 10
haipproxy_1  | 2018/03/13 06:02:12| DNS Socket created at 0.0.0.0, FD 11
haipproxy_1  | 2018/03/13 06:02:12| Adding nameserver 127.0.0.11 from /etc/resolv.conf
haipproxy_1  | 2018/03/13 06:02:12| Adding ndots 1 from /etc/resolv.conf
haipproxy_1  | 2018/03/13 06:02:12| HTCP Disabled.
haipproxy_1  | 2018/03/13 06:02:12| commBind: Cannot bind socket FD 13 to [::1]: (99) Cannot assign requested address
haipproxy_1  | 2018/03/13 06:02:12| commBind: Cannot bind socket FD 14 to [::1]: (99) Cannot assign requested address
haipproxy_1  | 2018/03/13 06:02:12| ERROR: Failed to create helper child read FD: UDP[::1]
haipproxy_1  | 2018/03/13 06:02:12| Configuring Parent 119.82.241.156/53281/0
haipproxy_1  | 2018/03/13 06:02:12| Configuring Parent 117.202.20.70/555/0
haipproxy_1  | 2018/03/13 06:02:12| Configuring Parent 118.114.77.47/8080/0
haipproxy_1  | 2018/03/13 06:02:12| Configuring Parent 116.11.254.37/80/0
haipproxy_1  | 2018/03/13 06:02:12| Configuring Parent 103.90.229.26/1997/0
haipproxy_1  | 2018/03/13 06:02:12| Configuring Parent 103.238.227.208/3128/0
haipproxy_1  | 2018/03/13 06:02:12| Configuring Parent 103.54.100.65/8080/0
haipproxy_1  | 2018/03/13 06:02:12| Configuring Parent 52.22.218.51/80/0
haipproxy_1  | 2018/03/13 06:02:12| Configuring Parent 159.89.195.153/8118/0
haipproxy_1  | 2018/03/13 06:02:12| Configuring Parent 216.177.233.181/8080/0
haipproxy_1  | 2018/03/13 06:02:12| Configuring Parent 34.231.147.235/3128/0
haipproxy_1  | 2018/03/13 06:02:12| Finished loading MIME types and icons.
haipproxy_1  | 2018/03/13 06:02:12| Accepting HTTP Socket connections at local=[::]:3128 remote=[::] FD 12 flags=9

You can see this above

haipproxy_1  | 2018/03/13 06:02:12| Configuring Parent 159.89.195.153/8118/0
haipproxy_1  | 2018/03/13 06:02:12| Configuring Parent 216.177.233.181/8080/0
haipproxy_1  | 2018/03/13 06:02:12| Configuring Parent 34.231.147.235/3128/0

You should send requests after this happens.Sometimes the request will fail because the proxy ip is not good.But most time it will succeed.So you should have retry in your code and handle the exceptions may happen during the request.Nobody can be sure all the proxies are stable all the time!

BTW,I recommend you use py_cli or other clients you implement in production.

from haipproxy.