rad's People
Forkers
a0xpg rassec gh-jy bu1xuan2 yut0u samueltt antime guaguala raystyle selfevo kiang70 zlzhangv007 aixueyou canghailongheng 6m03sr standj7 omiter rgzn-aiyun fdlucifer gaecom leegojames fatmeat 3as0n bigbigx fadinglr imzgforever yangh1368 zhaoshiling1017 halftion leovon hatchetxuexi isgasho xiashuijun zhouclay smallbluefox lazylee007 chaosophia pyking krystianli skypoc avgirl kopherhhj j5s empty2081 wyyloveyyl gysf666 gilsy12333 lhtest429 hkxiaoyao lzb960827 fzpixzj90h7baqieoop5hg mytfx tai-rex ddostest123 shenliehuozhi snnifer liupanxuexi happyqingye normannoldi mountain-cow qq529952515 lifansama hillee-dev sobinge luyouli lowliness9 crackercat fucksos lztcode dixonhadden shimxx test2504 cleanmgr112 gordenbrown51 tank2737 thisismyusernamehh meiwanle1 zhanghuixi zmdprogrom lands69 brucetyler0 nocomp jost4r fgz00 viplurker resenhacyber fostane lehendsec unr0i istoliving jaygith dontian122 solitary321 pupulit e-jarvis tz0385 xiaoqin00 int2ecall chingru chztrad's Issues
建议增加忽略证书校验的功能
在命令行或者配置文件中加入是否忽略证书校验的选项,这样被动扫描的时候就不用倒入扫描器的证书了,更方便一些。
只要在启动chrome的时候 添加 --ignore-certificate-errors 参数即可
配置文件中设置 user-agent 无效
添加--http-proxy 参数后爬取结果异常
如题,添加--http-proxy 参数后爬取结果异常
PS C:\Users\Admin\Downloads> .\rad_windows_amd64.exe -t http://testphp.vulnweb.com
██████╗ █████╗ ██████╗
██╔══██╗██╔══██╗██╔══██╗
██████╔╝███████║██║ ██║
██╔══██╗██╔══██║██║ ██║
██║ ██║██║ ██║██████╔╝
╚═╝ ╚═╝╚═╝ ╚═╝╚═════╝
[Rad 0.2/638f30fd]
Build: [2020-09-10] [linux/amd64] [RELEASE/__unknown__]
Compiler Version: go version go1.14.4 linux/amd64
[INFO] 2020-09-10 19:32:53 [collector:mitm.go:226] loading cert from ./ca.crt
[INFO] 2020-09-10 19:32:53 [collector:mitm.go:227] loading cert key from ./ca.key
[INFO] 2020-09-10 19:32:54 [collector:mitm.go:55] no global filter config found, use default
[INFO] 2020-09-10 19:32:54 [collector:mitm.go:108] no global mitm config found, use default
[INFO] 2020-09-10 19:32:54 [collector:mitm.go:264] starting mitm server at 127.0.0.1:64253
GET http://testphp.vulnweb.com/
POST http://testphp.vulnweb.com/search.php?test=query
GET http://testphp.vulnweb.com/index.php
GET http://testphp.vulnweb.com/categories.php
GET http://testphp.vulnweb.com/artists.php
GET http://testphp.vulnweb.com/disclaimer.php
GET http://testphp.vulnweb.com/AJAX/index.php
GET http://testphp.vulnweb.com/guestbook.php
GET http://testphp.vulnweb.com/userinfo.php
GET http://testphp.vulnweb.com/login.php
GET http://testphp.vulnweb.com/privacy.php
GET http://testphp.vulnweb.com/Mod_Rewrite_Shop/
GET http://testphp.vulnweb.com/AJAX/artists.php
GET http://testphp.vulnweb.com/hpp/
POST http://testphp.vulnweb.com/userinfo.php
GET http://testphp.vulnweb.com/AJAX/infoartist.php?id=1
GET http://testphp.vulnweb.com/Mod_Rewrite_Shop/Details/network-attached-storage-dlink/1/
GET http://testphp.vulnweb.com/Mod_Rewrite_Shop/Details/web-camera-a4tech/2/
GET http://testphp.vulnweb.com/AJAX/categories.php
GET http://testphp.vulnweb.com/AJAX/titles.php
GET http://testphp.vulnweb.com/listproducts.php?cat=1
GET http://testphp.vulnweb.com/signup.php
POST http://testphp.vulnweb.com/guestbook.php
GET http://testphp.vulnweb.com/hpp/?pp=12
GET http://testphp.vulnweb.com/Mod_Rewrite_Shop/BuyProduct-1/
POST http://testphp.vulnweb.com/AJAX/showxml.php
GET http://testphp.vulnweb.com/Mod_Rewrite_Shop/RateProduct-1.html
GET http://testphp.vulnweb.com/Mod_Rewrite_Shop/BuyProduct-3/
GET http://testphp.vulnweb.com/Mod_Rewrite_Shop/RateProduct-3.html
GET http://testphp.vulnweb.com/hpp/params.php?aaaa%2F=%26%2325552%3B%26%2320132%3B
GET http://testphp.vulnweb.com/artists.php?artist=1
GET http://testphp.vulnweb.com/comment.php?aid=1
GET http://testphp.vulnweb.com/Mod_Rewrite_Shop/BuyProduct-2/
GET http://testphp.vulnweb.com/Mod_Rewrite_Shop/RateProduct-2.html
GET http://testphp.vulnweb.com/hpp/params.php?p=valid&pp=12
POST http://testphp.vulnweb.com/secured/newuser.php
GET http://testphp.vulnweb.com/listproducts.php?artist=1
POST http://testphp.vulnweb.com/comment.php
GET http://testphp.vulnweb.com/showimage.php?file=./pictures/1.jpg
GET http://testphp.vulnweb.com/comment.php?pid=1
GET http://testphp.vulnweb.com/showimage.php?file=./pictures/1.jpg&size=160
GET http://testphp.vulnweb.com/product.php?pic=1
PS C:\Users\Admin\Downloads> .\rad_windows_amd64.exe -t http://testphp.vulnweb.com --http-proxy http://127.0.0.1:7777
██████╗ █████╗ ██████╗
██╔══██╗██╔══██╗██╔══██╗
██████╔╝███████║██║ ██║
██╔══██╗██╔══██║██║ ██║
██║ ██║██║ ██║██████╔╝
╚═╝ ╚═╝╚═╝ ╚═╝╚═════╝
[Rad 0.2/638f30fd]
Build: [2020-09-10] [linux/amd64] [RELEASE/__unknown__]
Compiler Version: go version go1.14.4 linux/amd64
GET http://testphp.vulnweb.com/
PS C:\Users\Admin\Downloads>
除了 http://testphp.vulnweb.com 这个站点之外,尝试过其他站点也存在这种情况,由于授权的原因,无法提交示例
功能建议
增加批量
支持基于会话爬虫
如题
报错提示:panic: runtime error: invalid memory address or nil pointer dereference
.\rad_windows_amd64.exe --http-proxy 127.0.0.1:8080 -t https://xxxx/ -text-output result.txt
报错:exec: "google-chrome": executable file not found in %PATH%
如何更改默认的浏览器路径
exec_path: "离谱"
新增功能建议
-
看起来只能针对某个单一的域名进行爬取,建议增加一个批量读取的URL的
- 如:本地新建txt文本,把需要爬取的目标放到里面,然后批量爬取
-
另外只能针对高级版进行深度融合吗?社区办支持吗?文档里面没有说明~
websocket url timeout reached这个是什么意思
websocket url timeout reached这个是什么意思
xray高级版+rad的融合有问题
xray webscan --browser-crawler https://www.域名.com --html-output vuln.html 不正常工作,表现为调用rad后直接显示爬完
xray webscan --browser-crawler https://192.111.11.111 --html-output vuln.html 正常工作
目标为域名 rad不正常工作,目标为IP rad正常工作
希望增加timeout参数
如题 发现有的时候会卡死
设置不允许url关键字不起作用
新弹出的浏览器窗口里的内容和流量无法抓包
0.4版本报错
root@iZt4n1sswjid2nrd8b0cetZ:~/testrad# ./rad_linux_amd64 -t https://example.com
██████╗ █████╗ ██████╗
██╔══██╗██╔══██╗██╔══██╗
██████╔╝███████║██║ ██║
██╔══██╗██╔══██║██║ ██║
██║ ██║██║ ██║██████╔╝
╚═╝ ╚═╝╚═╝ ╚═╝╚═════╝
[Rad 0.4/ebbb3ce3]
Build: [2021-04-30] [linux/amd64] [RELEASE/unknown]
Compiler Version: go version go1.15.6 linux/amd64
[ERRO] 2021-05-01 02:16:40 [rad:radium.go:209] [launcher] Failed to get the debug url: [0501/021639.748738:ERROR:zygote_host_impl_linux.cc(90)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
[launcher] Failed to get the debug url: [0501/021639.748738:ERROR:zygote_host_impl_linux.cc(90)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
加上捕获ctrl+c比较方便中止
功能建议
- 在爬取之前进行目录扫描,增加入口点(可加入选项)
- http隐藏参数爆破
- json格式输出(post包要带data)(最好可直接打印出来和存成文件)可参考crawlergo输出形式
exec: "google-chrome": executable file not found in %PATH%错误怎么解决
我是第一名嘛
我是第一名嘛
无法爬取window.open的url
使用默认的参数运行 ./rad_linux_amd64 -t http://testphp.vulnweb.com/ --text-output testphp.vulnweb.com.txt
发现如http://testphp.vulnweb.com/artists.php 这个页面中的 http://testphp.vulnweb.com/comment.php?aid=1 无法抓取到. 不知道是程序的抓取方法没匹配到还是说要设置什么参数
已解决
zz
爬虫时遇到验证码图片就不断递归爬验证码
新增功能建议
1、单个标签页的timeout时间设定
2、单个域名爬取时的最大时间设定
功能建议
我看到rad是爬了一些接口,可以单独把js爬出来么🌹。
例如网页为
<script src="/js/jquery-1.11.0.min.js" charset="gbk"></script>
--
| <script src="http://target1.com/new/pc/js/https_util.min.js?v=20180607"></script>
<script src="http://target2.com/new/pc/js/https_util.min.js?v=20180607"></script>
<script src="http://target3.com/new/pc/js/https_util.min.js?v=20180607"></script>
提取结果
``
无法爬取window.location.assign
比如这个url里的
https://adsl-99-126-65-113.dsl.akrnoh.sbcglobal.net/
onclick="window.location.assign('/hr/')
chrome failed to start
好像不支持input自动填充?
parse query error invalid URL escape "%fG"
[ERRO] 2021-09-08 21:27:56 [default:request.go:166] parse query error invalid URL escape "%fG"
max-concurrent 最大页面并发配置修改不生效
希望增加这个功能,可以导入txt里面的目标,而不是每次去扫一个url
--wait-login与-http-proxy联合使用问题
命令:./rad_darwin_amd64 -t https://xxx.com/login --wait-login -http-proxy 127.0.0.1:7777
--wait-login命令如何与-http-proxy一起使用?谢谢。
--wait-login打开一个新的chrome,终端监听了地址端口,新的chrome装不了插件。
增加push-to-proxy功能,将爬虫结果推送到代理
全部走上游代理的话流量太大
http-equiv="refresh" 跳转跟随
rad是否可以提供这个功能,在遇到 http-equiv="refresh"
时,爬虫可以跟随到 /admin
路径下
<head>
<meta http-equiv="refresh" content="1;URL='/admin'"/>
</head>
配置文件中设置了headers后,会覆盖cookies
支持socks代理吗?
是只能支持http代理吗?这边环境需要socks代理,怎么操作。
rad_config.yml
如果把rad添加到环境变量,rad_config.yml这个配置文件会生成在cmd运行的位置,默认是C:\User\user,找了半天才找到
#爬虫爬着爬着就自己停了
#爬虫爬着爬着就自己停了
运行rad报错
chrome-linux版本为79
[ERRO] 2021-10-28 15:05:48 [rad:radium.go:209] [launcher] Failed to get the debug url: [1028/150547.437603:ERROR:zygote_host_impl_linux.cc(89)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
[launcher] Failed to get the debug url: [1028/150547.437603:ERROR:zygote_host_impl_linux.cc(89)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
建议增加一个爬虫结束字段或者在爬虫结束之后增加一个例如:爬取完毕的字样
建议增加一个爬虫结束字段或者在爬虫结束之后增加一个例如:爬取完毕的字样,这样方便进行进程的监控,不然都不知道爬取完毕了没有
invalid memory address or nil pointer dereference
对webgoat进行登录爬取,并且在disallowed-urls里配置了logout的排除,但是没爬多少数据就报错了
环境:
Rad 0.4/ebbb3ce3
配置文件:
enable-image-display: false # 启用图片显示,适用于需要验证码登录的情况,启用wait-login自动开启
load-wait: 0 # 页面加载完毕后的等待时间,单位秒,网速不佳时可尝试调大该值
exec-path: "" # 启动chrome的路径
disable-headless: false # 禁用无头模式
request-config: # 请求头配置
user-agent: "" # 请求user-agent配置
headers: # 请求header配置
- key: "" # header的key
value: "" # header的value
cookies: # 请求cookie配置
- name: "" # cookie的name
value: "" # cookie的value
restrictions-on-urls: # 对爬取的URL的一些限制项
disallowed-suffix: [] # 不允许的文件后缀
disallowed-keywords-in-path-and-query: [] # 不允许的URL关键字
disallowed-domain: [] # 不允许的域名
disallowed-urls: ["[a-z/]*logout[a-z/]*"] # 不允许的URL(正则)
allowed-domains: [] # 允许的域名,起始目标会被自动加入其中
allowed-urls: [] # 允许的URL(正则)
restrictions-on-requests: # 对请求行为的一些限制项
max-concurrent: 10 # 最大页面并发(不大于10)
max-depth: 5 # 最大页面深度限制
max-click-depth: 5 # 一个页面中最大点击深度限制
max-count-of-page: 1000 # 最多爬取的页面数量限制
max-click-or-event-trigger: 1000 # 单个页面中最大点击或事件触发次数(不大于10000)
click-or-event-interval: 1000 # 点击间隔,单位毫秒
rad运行会停止
- 我把rad命令使用
;``或
&&`连接起来形成类似这样的语句的时候: ./rad_linux_amd64 -t http://xxxx.com --http-proxy 127.0.0.1:7777 &&./rad_linux_amd64 -t http://xxasdaxx.com --http-proxy 127.0.0.1:7777 &&./rad_linux_amd64 -t https://xxasxxdaxx.com --http-proxy 127.0.0.1:7777 &&./rad_linux_amd64 -t https://xxxxasdaxx.com --http-proxy 127.0.0.1:7777
(当然比现在的还要长,大概上千个)- 会出现爬到某一个网站的某一个页面的时候直接停掉,就停在那里不动,也不进行下一个目标的爬取,也不结束程序,直到人为操控,这是咋回事呢?
- 应该并不是另一边xray那边队列长度的限制,因为xray那边都扫描完了,这边的还是不动
希望增加--push-to-proxy功能
希望增加--push-to-proxy功能
因为节点可能不是本机 所以建议支持此功能
rad建议
enhancement:
- fuzz 入口发现
- 针对各种延时/超时设置
- 启动参数中指定config文件绝对路径
- 隐藏参数发现
bug:
以下站点爬取不全
https://www.mapmyfitness.com/
http://www.cloudways.com
https://cs.money/
以下站点网络不稳定情况爬取不到:
https://hsrd.yahoo.com/
https://mobile.line.me
m1 support
建议添加Method URL 去Method选项
只想要url
能否加一个批量的功能,只能扫描一个url,感觉有些鸡肋
说好的darwin_amd64呢
登录功能不可用
rad -t http://example.com -wait-login
使用该命令,命令提示“进行手动登录”,但是登录界面未加载,浏览器为空白页,任意键后浏览器加载目标页面,直接进入扫描,手动输入功能不可用。
建议支持url列表功能
建议支持url列表功能。类似xray 的 --url-file参数
希望增加配置文件自定义以及浏览器自定义的参数
希望增加配置文件自定义以及浏览器自定义的参数
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.