crawlergo_x_XRAY-master>python launcher_new.py
File "launcher_new.py", line 29
cmd = ["./crawlergo", "-c", "C:\Users\gkd\Desktop\chrome-win\chrome.exe","-t", "5","-f","smart","--fuzz-path", "--push-to-proxy", "http://127.0.0.1:7888/", "--push-pool-max", "10","--output-mode", "json" , target]
^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape

C:\Users\gkd\Desktop\crawlergo_x_XRAY-master>python launcher_new.py
Traceback (most recent call last):
File "launcher_new.py", line 50, in
main(data1)
File "launcher_new.py", line 30, in main
rsp = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
File "C:\Users\gkd\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py", line 854, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\gkd\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py", line 1307, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
OSError: [WinError 193] %1 不是有效的 Win32 应用程序。

老哥您好，问下，我都配置好了，crawlergo能爬取到结果，但是xray一直不进行扫描，是什么原因造成的呢？求解


是xray接收的问题吗？

HTTP请求头字段会泄露Spider-Name: crawlergo， 如何去除？

运行后就这样子没动了，不管多久都是这样，然后看了下进程发现占了不少cpu，是全部扫完之后才会输出还是我运行失败了？

header或cookie的支持

运行提示FileNotFoundError: [WinError 2] 系统找不到指定的文件。
查了路径是没有问题的
报错信息如下：
Traceback (most recent call last):
File "launcher.py", line 89, in
main(data1)
File "launcher.py", line 59, in main
rsp = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
File "H:\python\python3.7\lib\subprocess.py", line 775, in init
restore_signals, start_new_session)
File "H:\python\python3.7\lib\subprocess.py", line 1178, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] 系统找不到指定的文件。

运行
python3 launcher_new.py
等一会儿就自动结束了。没有任何报错，也不知道这是咋回事。

C:\Users\Public\crawlergo_x_XRAY>python3 launcher.py
Traceback (most recent call last):
File "launcher.py", line 83, in
main(data1)
File "launcher.py", line 59, in main
rsp = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
File "C:\Program Files\Python37\lib\subprocess.py", line 800, in init
restore_signals, start_new_session)
File "C:\Program Files\Python37\lib\subprocess.py", line 1207, in _execute_child
startupinfo)

邮件发你

没有装ca证书也能用代理模式抓https？

大佬，咋自定义cookie运行啊

环境python3 Windows server16

在targets中的url，爬虫显示爬取到了。x详见下图

但是在xray缺看不到扫描，扫描的都是一些别的网址

看了下，是单线程的，每一次循环都会阻塞一段时间，效率太低。另外这里搞的队列感觉也没啥意义，没有发挥出多线程的魅力。改为多线程后，效率提升显著。

launcher_new.py使用了fake-useragent可能会出现报错：
fake_useragent.errors.FakeUserAgentError: Maximum amount of retries reached

本人尝试了如下解决方案：

先：
pip3 install --upgrade fake_useragent

1.修改代码launcher_new.py第十一行
ua = UserAgent(use_cache_server=False,verify_ssl=False,cache=False)

2.利用临时文件
下载https://fake-useragent.herokuapp.com/browsers/0.1.11 //好像国内无法访问
改名为fake_useragent_0.1.11.json
放入临时文件夹(如/tmp)

import tempfile
tempfile.gettempdir()

都无法解决

查看代码site-packages/fake_useragent/settings.py
发现要请求BROWSERS_STATS_PAGE = 'https://www.w3schools.com/browsers/default.asp'
但是该url好像国内无法访问所以报该错误

所以建议使用国外vps，或者弃用fake-useragent库。

fake_useragent.errors.FakeUserAgentError: Maximum amount of retries reached

0.1.13版本报错：fake_useragent.errors.FakeUserAgentError: Error occurred during getting browser: random
解决：pip3 install fake_useragent==0.1.11

subprocess方法的communicate()函数会卡住。

command命令在powershell能成功执行，但是放到subprocess的communicate()就会卡死。
我自己写了个test，用command命令会卡死，用dir/b命令就不会卡。

#!/usr/bin/python3
\# coding: utf-8
import simplejson
import threading
import subprocess
import requests
import warnings
warnings.filterwarnings(action='ignore')

def runcmd():
    command = ["./crawlergo", "-c", "D:\tools\crawlergo_x_XRAY\chromium\chrome.exe","-t", "20","-f","smart","--fuzz-path", "--output-mode", "json", "http://testphp.vulnweb.com/"]
    #command = ['dir','/b']
    #print(command)
    ret = subprocess.Popen(command, stdout=subprocess.PIPE,stderr=subprocess.PIPE)
    output, error = ret.communicate()
    print(output.decode('gbk'))
    print(error.decode('gbk'))

#runcmd(["dir","/b"])#序列参数
#runcmd("exit 1")#字符串参数

runcmd()

请问这个是这个函数的bug吗?还是我8g内存太小了？
有没有解决方法？

Traceback (most recent call last):
File "launcher.py", line 83, in
main(data1)
File "launcher.py", line 59, in main
rsp = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
File "C:\Users\DELL\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 854, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\DELL\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 1307, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
PermissionError: [WinError 5] 拒绝访问。

请问这个情况怎么解决