webraybtl / codeqlpy Goto Github PK

View Code? Open in Web Editor NEW

729.0 729.0 83.0 158.08 MB

CodeQLpy是一款基于CodeQL实现的半自动化代码审计工具，目前仅支持java语言。实现从源码反编译，数据库生成，脆弱性发现的全过程，可以辅助代码审计人员快速定位源码可能存在的漏洞。

Python 26.29% CodeQL 73.71%

code-analysis codeqlpy java

codeqlpy's People

Contributors

Stargazers

Watchers

codeqlpy's Issues

jar包无法反编译的问题

使用的是https://github.com/j3ers3/Hello-Java-Sec这个项目里面的jar包，无法反编译

springboot的jar包支持jar包内部的jar进行扫描

像ruoyi这种打包好的jar包，lib下面还是有jar包是网站代码

check.py ERROR database or codeql is error. qlpath error,check it at config/config.ini or close the Visual Studio Code.

2023/04/22 16:56:53 check.py ERROR database or codeql is error.
qlpath error,check it at config/config.ini or close the Visual Studio Code.

arch -x86_64 codeql database create out/database/XXEDemo --language=java --command="/bin/bash -c /Users/code/python/CodeQLpy/out/decode/run.sh" --overwrite

Initializing database at /Users/code/python/CodeQLpy/out/database/XXEDemo.
Running build command: [/bin/bash, -c, /Users/code/python/CodeQLpy/out/decode/run.sh]
[2023-01-30 16:36:17] [build-stdout] [INFO] Scanning for projects...
[2023-01-30 16:36:18] [build-stdout] [INFO]
[2023-01-30 16:36:18] [build-stdout] [INFO] ---------------------< io.github.talelin:latticy >----------------------
[2023-01-30 16:36:18] [build-stdout] [INFO] Building latticy 0.2.1-RELEASE
[2023-01-30 16:36:18] [build-stdout] [INFO] --------------------------------[ jar ]---------------------------------
[2023-01-30 16:36:19] [build-stdout] [INFO]
[2023-01-30 16:36:19] [build-stdout] [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ latticy ---
[2023-01-30 16:36:19] [build-stdout] [INFO] Deleting /Users/code/python/CodeQLpy/target
[2023-01-30 16:36:19] [build-stdout] [INFO]
[2023-01-30 16:36:19] [build-stdout] [INFO] --- maven-resources-plugin:3.2.0:resources (default-resources) @ latticy ---
[2023-01-30 16:36:19] [build-stdout] [INFO] Using 'UTF-8' encoding to copy filtered resources.
[2023-01-30 16:36:19] [build-stdout] [INFO] Using 'UTF-8' encoding to copy filtered properties files.
[2023-01-30 16:36:19] [build-stdout] [INFO] skip non existing resourceDirectory /Users/code/python/CodeQLpy/src/main/resources
[2023-01-30 16:36:19] [build-stdout] [INFO] skip non existing resourceDirectory /Users/code/python/CodeQLpy/src/main/java
[2023-01-30 16:36:19] [build-stdout] [INFO]
[2023-01-30 16:36:19] [build-stdout] [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ latticy ---
[2023-01-30 16:36:20] [build-stdout] [INFO] No sources to compile
[2023-01-30 16:36:20] [build-stdout] [INFO]
[2023-01-30 16:36:20] [build-stdout] [INFO] --- maven-resources-plugin:3.2.0:testResources (default-testResources) @ latticy ---
[2023-01-30 16:36:20] [build-stdout] [INFO] Using 'UTF-8' encoding to copy filtered resources.
[2023-01-30 16:36:20] [build-stdout] [INFO] Using 'UTF-8' encoding to copy filtered properties files.
[2023-01-30 16:36:20] [build-stdout] [INFO] skip non existing resourceDirectory /Users/code/python/CodeQLpy/src/test/resources
[2023-01-30 16:36:20] [build-stdout] [INFO]
[2023-01-30 16:36:20] [build-stdout] [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ latticy ---
[2023-01-30 16:36:20] [build-stdout] [INFO] No sources to compile
[2023-01-30 16:36:20] [build-stdout] [INFO]
[2023-01-30 16:36:20] [build-stdout] [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ latticy ---
[2023-01-30 16:36:20] [build-stdout] [INFO] Tests are skipped.
[2023-01-30 16:36:20] [build-stdout] [INFO]
[2023-01-30 16:36:20] [build-stdout] [INFO] --- maven-jar-plugin:3.2.0:jar (default-jar) @ latticy ---
[2023-01-30 16:36:20] [build-stdout] [WARNING] JAR will be empty - no content was marked for inclusion!
[2023-01-30 16:36:20] [build-stdout] [INFO] Building jar: /Users/code/python/CodeQLpy/target/latticy-0.2.1-RELEASE.jar
[2023-01-30 16:36:20] [build-stdout] [INFO]
[2023-01-30 16:36:20] [build-stdout] [INFO] --- spring-boot-maven-plugin:2.5.2:repackage (repackage) @ latticy ---
[2023-01-30 16:36:21] [build-stdout] [INFO] ------------------------------------------------------------------------
[2023-01-30 16:36:21] [build-stdout] [INFO] BUILD FAILURE
[2023-01-30 16:36:21] [build-stdout] [INFO] ------------------------------------------------------------------------
[2023-01-30 16:36:21] [build-stdout] [INFO] Total time: 3.989 s
[2023-01-30 16:36:21] [build-stdout] [INFO] Finished at: 2023-01-30T16:36:21+08:00
[2023-01-30 16:36:21] [build-stdout] [INFO] ------------------------------------------------------------------------
[2023-01-30 16:36:21] [build-stdout] [ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:2.5.2:repackage (repackage) on project latticy: Execution repackage of goal org.springframework.boot:spring-boot-maven-plugin:2.5.2:repackage failed: Unable to find main class -> [Help 1]
[2023-01-30 16:36:21] [build-stdout] [ERROR]
[2023-01-30 16:36:21] [build-stdout] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[2023-01-30 16:36:21] [build-stdout] [ERROR] Re-run Maven using the -X switch to enable full debug logging.
[2023-01-30 16:36:21] [build-stdout] [ERROR]
[2023-01-30 16:36:21] [build-stdout] [ERROR] For more information about the errors and possible solutions, please read the following articles:
[2023-01-30 16:36:21] [build-stdout] [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[2023-01-30 16:36:22] [build-stdout] Eclipse Compiler for Java(TM) v20160829-0950, 3.12.1

十个里面十个运行不起来

插件问题

在config.ini中配置qlpath后，是不是程序会同时使用ql 库中的插件和程序目录plugins中的插件进行扫描。

结果实时写入

扫个项目等待的时间太长了，能否把结果实时写入到文件
（windows写入的时候打开貌似会占用导致报错，linux和mac不会）

javax.xml.stream.FactoryConfigurationError

大部分文件都是这个报错，最后无法生成数据库初始化

报错functions.py WARNING

你好，我在大项目中使用，出现了如下问题
database.py INFO Found 2047 jsp files to decode
后面就一直 functions.py WARNING
最终结果就是
database.py WARNING Decode jsp file 0/2047 success
database.py ERROR Auto decompiler error, no java file found.
请问这个错误要怎么解决？

插件目录中插件的来源是哪里？

作者您好！感谢您卓越的工作！

我查看了插件目录中的部分 java 插件，发现与 ql 库中的插件不完全一致，请问您是从哪里收集到的这些插件？插件的作用经过您的验证了吗？

爆codeql路径不在

上个月还可以使用

2024/03/25 15:15:07 database.py ERROR Target SourceCode doesn't have WEB-INF/web.xml file,exit

javasec.jar 案例错误

windows下不能用吗?

FileNotFoundError: [WinError 3] 系统找不到指定的路径。: '/Users/pang0lin/CodeQL/ql/java/ql/test/tmpbstpho93'

编译报错

PS E:\codeql\CodeQLpy>  codeql database create out/database/webroot --language=java --command="E:\codeql\CodeQLpy\out\decode/run.cmd" --overwrite
Initializing database at E:\codeql\CodeQLpy\out\database\webroot.
Running build command: [E:\codeql\CodeQLpy\out\decode/run.cmd]
[2023-03-14 19:10:13] [build-stdout] offjava -jar E:\codeql\CodeQLpy\lib\ecj-4.6.1.jar   -extdirs "E:\codeql\CodeQLpy\out\decode\lib;lib/spring_mvc_lib;lib/common_lib;lib/java8_lib;lib/tomcat_lib" -sourcepath E:\codeql\CodeQLpy\out\decode\classes -encoding UTF-8 -8 -warn:none -proceedOnError -noExit @E:\codeql\CodeQLpy\out\decode/file.txt
Finalizing database at E:\codeql\CodeQLpy\out\database\webroot.
CodeQL detected code written in Java but could not process any of it. This can occur if the specified build commands failed to compile or process any code.
 - Confirm that there is some source code for the specified language in the project.
 - For codebases written in Go, JavaScript, TypeScript, and Python, do not specify
   an explicit --command.
 - For other languages, the --command must specify a "clean" build which compiles
   all the source code files without reusing existing build artefacts.

文件名、目录名或卷标语法不正确。

如图所示

pei配置如下

目录如下，跟着配置不知道哪里有些问题，望指教

运行报错

运行后提示/out/decode/lib 目录不存在，尝试过手动创建

使用问题

纯class文件或者java文件跑不起来，得需要web.xml? 或者是poc.xml，建议加个判断

java堆设置的问题

在创建数据库的时候会出现堆内存设置过小，我去找这个codeql的堆内存设置的地方，但是没找到，师傅遇到过吗

这个超时问题是因为jdk版本吗？

timed out after 120 seconds

能不能扫springboot的jar包里面的lib的jar包，有些漏洞点在那些lib的jar包下面

如何添加自己编写的qll库呢？

插件的里的ql，如何包含自己编写的qll库？

是否可以将结果导入到VSCode？

因为想查看具体的每一步的节点

运行codeql --version时出现的Bug

如下图所示

本地确认codeql version在系统环境变量之下

师傅你好，mac 兼容吗

看到很多关于windows得bug修复，不知道Mac能否正常兼容

yaml语法错误,qlpack.yml这个文件使用的默认的内容，需要重新生成吗

2023/03/30 10:46:44 functions.py WARNING
2023/03/30 10:46:44 functions.py WARNING
2023/03/30 10:46:46 functions.py WARNING A fatal error occurred: Error reading /Users/xxxxx/tools/codeqldemo/ql/java/ql/test/qlpack.yml: Unexpected '$'.
${workspace}
^ (through reference chain: com.semmle.frontend.packs.QlPack["dependencies"]->java.util.TreeMap["codeql/java-all"])

2023/03/30 10:46:46 check.py ERROR database or codeql is error.
qlpath error,check it at config/config.ini or close the Visual Studio Code.

创建数据库时遇到以下情况

�win10
mvn3.9.2

创建数据库提示java.io.EOFException: Unexpected end of ZLIB input stream

当我创建数据库时，遇到了那种大型项目提示上面的报错

我看到师傅你也去codeql官方那里反馈了，但是他说他解决不了，请问师傅后来是怎么解决的

这种目录结构如何运行？

师傅求教一下

WEB-INF/lib为jar包，关键代码都在jar里面，jsp文件只有几个，没啥用主要文件都在几个lib文件夹的jar里，我该怎么执行命令呢？
python main.py -t C:\Users\john\Desktop\oa\ -c -j oa-*?.jar 失败了
错误信息：
`2023/09/25 17:02:43 database.py WARNING Decode jsp file 0/11 success

2023/09/25 17:02:43 database.py ERROR Auto decompiler error, no java file found.
`

Target SourceCode doesn't have WEB-INF/web.xml file,exit

2023/02/16 12:44:02 database.py ERROR Target SourceCode doesn't have WEB-INF/web.xml file,exit

有些项目就没有jsp

2023/01/22 00:19:06 database.py ERROR Target SourceCode doesn't found any jsp fi
le,exit

There should probably be a qlpack.yml file declaring dependencies

Could not resolve module java. There should probably be a qlpack.yml file declaring dependencies

其他语言支持

太好用了！！希望支持codeql所支持的其他语言

无法创建CodeQLpy/out/decode/lib

2023/01/20 00:47:40 functions.py WARNING
2023/01/20 00:47:40 functions.py WARNING
Traceback (most recent call last):
File "/Users/xxx/CodeQL/CodeQLpy/main.py", line 71, in
createDB(parse_args.target, parse_args.compiled, version, parse_args.jar)
File "/Users/xxx/CodeQL/CodeQLpy/compiler/database.py", line 402, in createDB
return createDir(source, compiled, version, jars)
File "/Users/xxxx/CodeQL/CodeQLpy/compiler/database.py", line 139, in createDir
compile_cmd = ecjcompile(qlConfig("decode_savedir"), source)
File "/Users/xxx/CodeQL/CodeQLpy/compiler/ecjcompiler.py", line 98, in ecjcompile
jar_libs = dirFiles(os.path.join(save_path, "lib"))
File "/Users/xxx/CodeQL/CodeQLpy/utils/functions.py", line 25, in dirFiles
for filename in os.listdir(dirpath):
FileNotFoundError: [Errno 2] No such file or directory: '/Users/xxx/CodeQL/CodeQLpy/out/decode/lib'

初始化数据库错误

python3 main.py -t ***.jar -c

decoding e.jar...
Command '/Library/Java/JavaVirtualMachines/jdk-11.0.15.jdk/Contents/Home/bin/java -jar lib/jd-cli.jar --outputDir out/decode/xboot-module.jar /.jar' timed out after 240 seconds
Traceback (most recent call last):
File "/Users//Security/Web/CodeQLpy/main.py", line 75, in
createDB(parse_args.target, parse_args.compiled, version, parse_args.jar, parse_args.root)
File "/Users//Security/Web/CodeQLpy/compiler/database.py", line 408, in createDB
return createJar(source, compiled, version)
File "/Users//Security/Web/CodeQLpy/compiler/database.py", line 46, in createJar
copyJavaFile(java_file, os.path.join(qlConfig("decode_savedir"), "classes"))
File "/Users//Security/Web/CodeQLpy/utils/functions.py", line 77, in copyJavaFile
with open(srcpath, 'rb') as r:
IsADirectoryError: [Errno 21] Is a directory: 'out/decode/**.jar/META-INF/maven/com.github.spullara.mustache.java'

substring not found

Traceback (most recent call last):
File "C:\code\CodeQLpy\main.py", line 71, in
createDB(parse_args.target, parse_args.compiled, version, parse_args.jar)
File "C:\code\CodeQLpy\compiler\database.py", line 398, in createDB
return createDir(source, compiled, version, jars)
File "C:\code\CodeQLpy\compiler\database.py", line 139, in createDir
compile_cmd = ecjcompile(qlConfig("decode_savedir"), source)
File "C:\code\CodeQLpy\compiler\ecjcompiler.py", line 103, in ecjcompile
source_java_path = getSourcePath(source_path)
File "C:\code\CodeQLpy\compiler\ecjcompiler.py", line 78, in getSourcePath
pack_loc = java_file.index(packname.replace(b".", b"/").decode("utf-8"))
ValueError: substring not found

加载Config.ini 中文字符报错

UnicodeDecodeError: 'gbk' codec can't decode byte 0xae in position 38: illegal multibyte sequence

utils/option.py
line8替换成：
conn.read("config/config.ini",encoding="utf-8")

qlpath 问题

qlpath是什么路径呢？是指存放了codeql.exe 的目录吗？

反编译后的代码无法再次编译

扫描问题

有一些项目是war或者jar包，但是里面没有web.xml，进行第一步时会报找不到web.xml，这种怎么办

可以设置要跑的漏洞类别吗

所有漏洞都跑一遍时间太长了，能不能单独设置跑哪些漏洞

win无法使用？

请问是否支持离线审计？

扫描数据库就会提示缺少qlpack.yml文件

python3 main.py -d /Users/xxxxxxxx/tools/CodeQL/CodeQLpy/out/database/ruoyi_demo
Welcome to
.. _________ . ________ .
_ __ | | _ ___ \ ____ | /_ _____ \ | |
| // | | / \ / / _ \ / __ |/ __ \ / / \ | |
| | _ | \ _( <> ) // \ // _/. \ |
|| / | _ /_/_ |__ >_____\ _/_______ / / / / __> /

2023/03/29 14:57:26 functions.py WARNING
2023/03/29 14:57:26 functions.py WARNING
2023/03/29 14:57:28 functions.py WARNING Compiling query plan for /Users/xxxxxx/tools/CodeQL/tmpjfw8i5b8/4a24a317f6b24508ac45368607e75799.ql.
ERROR: Could not resolve module java. There should probably be a qlpack.yml file declaring dependencies in /Users/xxxxxx/tools/CodeQL/tmpjfw8i5b8 or an enclosing directory. (/Users/xxxxxxx/tools/CodeQL/tmpjfw8i5b8/4a24a317f6b24508ac45368607e75799.ql:1,8-12)

2023/03/29 14:57:28 check.py ERROR database or codeql is error.
qlpath error,check it at config/config.ini or close the Visual Studio Code.

能否支持自定义扩展

https://redteamrecipe.com/Finding-0-day-vulnerabilities-in-apps-using-the-Red-Team-approach/

用ecj编译java文件的时候提示The import com.xxxx collides with another import statement

在eclipse的bug记录中有一条这种记录https://bugs.eclipse.org/bugs/show_bug.cgi?id=424839，这里就是两个类名相同，但是在不同的package中会导致编译错误，官方提供了https://git.eclipse.org/r/20293这个，但是已经访问不到了，师傅能解决一下吗

运行卡住

startscan: UnsafeTlsVersion，运行到这一步就会一直卡住，不再执行后续

有想法和大模型结合起来么

error=7, Argument list too long

ecj使用@符号读取生成的file.txt（由于项目太大）作为参数展开导致传递给codeql创建数据库会触发标题错误

-- Initializing database at /home/john/Desktop/codeql_analysis/out/databases/seeyon.
Running build command: [/usr/lib/jvm/java-8-openjdk-amd64/bin/java, -jar, /home/john/Desktop/codeql_analysis/plugins/ecj-4.20.jar, -extdirs, /home/john/Desktop/codeql_analysis/tmp/decompiler/dir/seeyon/lib:/home/john/Desktop/codeql_analysis/plugins/java8_lib, -sourcepath, /home/john/Desktop/codeql_analysis/tmp/decompiler/dir/seeyon/classes, -encoding, UTF-8, -8, -warn:none, -proceedOnError, -noExit, @/home/john/Desktop/codeql_analysis/tmp/decompiler/dir/seeyon/file.txt]
A fatal error occurred: IOException while executing process with args: [/home/john/CodeQL/codeql/tools/linux64/preload_tracer, /usr/lib/jvm/java-8-openjdk-amd64/bin/java, -jar, /home/john/Desktop/codeql_analysis/plugins/ecj-4.20.jar, -extdirs, /home/john/Desktop/codeql_analysis/tmp/decompiler/dir/seeyon/lib:/home/john/Desktop/codeql_analysis/plugins/java8_lib, -sourcepath, /home/john/Desktop/codeql_analysis/tmp/decompiler/dir/seeyon/classes, -encoding, UTF-8, -8, -warn:none, -proceedOnError, -noExit, /home/john/Desktop/codeql_analysis/tmp/decompiler/dir/seeyon/classes/code2/www/seeyon/com/system/auth/SeeyonDog.java, /home/john/Desktop/codeql_analysis/tmp/decompiler/dir/seeyon/classes/com/alibaba/fastjson/IOUtils.java, /home/john/Desktop/codeql_analysis/tmp/decompiler/dir/seeyon/classes/com/alibaba/fastjson/JSONArray.java, /home/john/Desktop/codeql_analysis/tmp/decompiler/dir/seeyon/classes/com/alibaba/fastjson/JSONException.java, /home/john/Desktop/codeql_analysis/tmp/decompiler/dir/seeyon/classes/com/alibaba/fastjson/JSONObject.java, /home/john/Desktop/codeql_analysis/tmp/decompiler/dir/seeyon/classes/com/alibaba/fastjson/TypeReference.java, /home/john/Desktop/codeql_analysis/tmp/decompiler/dir/seeyon/classes/com/alibaba/fastjson/TypeUtils.java, ...]

webraybtl / codeqlpy Goto Github PK

codeqlpy's People

Contributors

Stargazers

Watchers

Forkers

codeqlpy's Issues

arch -x86_64 codeql database create out/database/XXEDemo --language=java --command="/bin/bash -c /Users/code/python/CodeQLpy/out/decode/run.sh" --overwrite

Recommend Projects

Recommend Topics

Recommend Org