github / codeql-go Goto Github PK
View Code? Open in Web Editor NEWThe CodeQL extractor and libraries for Go.
License: MIT License
The CodeQL extractor and libraries for Go.
License: MIT License
Hi I get an error when trying to run go queries
Running queries.
ERROR: Referenced pack 'codeql/suite-helpers' not found. (C:\codeql-home\codeql-go\ql\src\qlpack.yml:1,1-1)
A fatal error occurred: Could not resolve library path for C:\codeql-home\codeql-go\ql\src
It would be nice to move to comment-based tests as seen in the callgraph test and its corresponding source.
@smowton I think it would be nice to have a framework for doing data-flow and taint tests, so while doing this for #285 so ideally that would be somewhat general.
There's probably no need to do this all at once; since this is mostly nice for review I think we can just do the transition as the tests change.
I was setting up everything to run codeql cli for golang and was able to create the database, but when running the command to analyze the database, an error was returned:
Command I run: codeql database analyze ./database go1.ql --format=csv --output=output.csv
Output of the terminal:
Running queries.
Compiling query plan for /Users//projects/<repository_name>/go1.ql.
ERROR: Could not resolve module go. There should probably be a qlpack.yml file declaring dependencies in /Users//projects/<repository_name> or an enclosing directory. (/Users//projects/<repository_name>/go1.ql:1,8-10)
The content of the go1.ql file is the following:
import go
select "hello world"
The steps I followed for installing codeql and codeql-go were the following:
Pages I have read to understand the problem that the terminal returned:
After all this research, I couldn't fix it and make this work in the terminal. Has anyone had the same problem and could solve the issue? Thanks in advance.
Assumption A is a struct with fields {birthday, name, id}, i got instanceA from unmarshal input data.
I want to TaintTracking the field "instanceA.birthday",
How do I write the ql to location the source node?
This line triggered a CodeQL warning in the cli/cli
repo:
var placeholderRE = regexp.MustCompile(`\:(owner|repo|branch)\b`)
The warning says:
A regular expression used here contains a literal backspace
\b
; did you mean\b
, a word boundary?
And in its "recommendation" section, it says:
Alternatively, you can use backtick-delimited raw string literals. For example, the
\b
inregexp.Compile(
hello\bworld)
matches a word boundary, not a backspace character, as within backticks\b
is not an escape sequence.
Well, we are using backtick-delimited raw string literal in the "offending" code, and it should be trivial for CodeQL to detect that. I think that CodeQL is convinced that we've passed a double-quoted string literal as argument to regexp.MustCompile()
, which could be indicative of a deeper, more serious problem in CodeQL for Go.
When investigating a failure in one of our pipelines, I noticed I was unable to satisfy the CodeQL query for CWE-326 / InsufficientKeySize the way I'd expect to be able to. I believe this is because it doesn't take into account some combination of switch/if
statements that restrict valid key sizes.
I've been able to reproduce this with the test suite (ql/test/query-tests/Security/CWE-326/InsufficientKeySize.go
) embedded in the codeql-go
repository as follows.
Presently our code looks (to CodeQL) like the simplified:
func foo6() {
foo8("rsa", 1024) // BAD
}
func foo7() {
foo8("rsa", 2048) // GOOD
}
func foo8(keyType string, keyBits int) {
switch keyType {
case "rsa":
rsa.GenerateKey(rand.Reader, keyBits)
default:
return
}
}
...except 1024
/2048
key sizes are dynamically chosen by the API caller, and additional restrictions are placed on the inputs via other function calls...
I was hoping to solve this as follows:
func foo9() {
foo11("rsa", 1024) // GOOD
}
func foo10() {
foo11("rsa", 2048) // GOOD
}
func foo11(keyType string, keyBits int) {
switch keyType {
case "rsa":
if keyBits < 2048 || keyBits > 8192 {
return
}
rsa.GenerateKey(rand.Reader, keyBits)
default:
return
}
}
Here, we explicitly guard, at rsa.GenerateKey
call-site, that the key sizes are within a valid, acceptable range. Note that, if foo11
is called as foo11("rsa" 1024)
, the key will not be generated, due to the if
statement's early return in the rsa
case branch.
However, when running this test, I generate the following:
[cipherboy@xps15 codeql-go]$ codeql test run ql/test/query-tests/Security/CWE-326/ --search-path . --consistency-queries ql/test/consistency
Executing 2 tests in 1 directories.
Extracting test database in /home/cipherboy/GitHub/github/codeql-go/ql/test/query-tests/Security/CWE-326.
Compiling queries in /home/cipherboy/GitHub/github/codeql-go/ql/test/query-tests/Security/CWE-326.
Executing tests in /home/cipherboy/GitHub/github/codeql-go/ql/test/query-tests/Security/CWE-326.
--- expected
+++ actual
@@ -2,6 +2,10 @@
| InsufficientKeySize.go:13:10:13:13 | 1024 : int | InsufficientKeySize.go:14:31:14:34 | size |
| InsufficientKeySize.go:18:7:18:10 | 1024 : int | InsufficientKeySize.go:25:11:25:14 | definition of size : int |
| InsufficientKeySize.go:25:11:25:14 | definition of size : int | InsufficientKeySize.go:26:31:26:34 | size |
+| InsufficientKeySize.go:30:14:30:17 | 1024 : int | InsufficientKeySize.go:37:27:37:33 | definition of keyBits : int |
+| InsufficientKeySize.go:37:27:37:33 | definition of keyBits : int | InsufficientKeySize.go:40:32:40:38 | keyBits |
+| InsufficientKeySize.go:47:15:47:18 | 1024 : int | InsufficientKeySize.go:54:28:54:34 | definition of keyBits : int |
+| InsufficientKeySize.go:54:28:54:34 | definition of keyBits : int | InsufficientKeySize.go:61:32:61:38 | keyBits |
nodes
| InsufficientKeySize.go:9:31:9:34 | 1024 | semmle.label | 1024 |
| InsufficientKeySize.go:13:10:13:13 | 1024 : int | semmle.label | 1024 : int |
@@ -9,7 +13,15 @@
| InsufficientKeySize.go:18:7:18:10 | 1024 : int | semmle.label | 1024 : int |
| InsufficientKeySize.go:25:11:25:14 | definition of size : int | semmle.label | definition of size : int |
| InsufficientKeySize.go:26:31:26:34 | size | semmle.label | size |
+| InsufficientKeySize.go:30:14:30:17 | 1024 : int | semmle.label | 1024 : int |
+| InsufficientKeySize.go:37:27:37:33 | definition of keyBits : int | semmle.label | definition of keyBits : int |
+| InsufficientKeySize.go:40:32:40:38 | keyBits | semmle.label | keyBits |
+| InsufficientKeySize.go:47:15:47:18 | 1024 : int | semmle.label | 1024 : int |
+| InsufficientKeySize.go:54:28:54:34 | definition of keyBits : int | semmle.label | definition of keyBits : int |
+| InsufficientKeySize.go:61:32:61:38 | keyBits | semmle.label | keyBits |
#select
| InsufficientKeySize.go:9:31:9:34 | 1024 | InsufficientKeySize.go:9:31:9:34 | 1024 | InsufficientKeySize.go:9:31:9:34 | 1024 | The size of this RSA key should be at least 2048 bits. |
| InsufficientKeySize.go:14:31:14:34 | size | InsufficientKeySize.go:13:10:13:13 | 1024 : int | InsufficientKeySize.go:14:31:14:34 | size | The size of this RSA key should be at least 2048 bits. |
| InsufficientKeySize.go:26:31:26:34 | size | InsufficientKeySize.go:18:7:18:10 | 1024 : int | InsufficientKeySize.go:26:31:26:34 | size | The size of this RSA key should be at least 2048 bits. |
+| InsufficientKeySize.go:40:32:40:38 | keyBits | InsufficientKeySize.go:30:14:30:17 | 1024 : int | InsufficientKeySize.go:40:32:40:38 | keyBits | The size of this RSA key should be at least 2048 bits. |
+| InsufficientKeySize.go:61:32:61:38 | keyBits | InsufficientKeySize.go:47:15:47:18 | 1024 : int | InsufficientKeySize.go:61:32:61:38 | keyBits | The size of this RSA key should be at least 2048 bits. |
[1/2 eval 1.7s] FAILED(RESULT) /home/cipherboy/GitHub/github/codeql-go/ql/test/query-tests/Security/CWE-326/InsufficientKeySize.qlref
[2/2 eval 7ms] PASSED /home/cipherboy/GitHub/github/codeql-go/ql/test/query-tests/Security/CWE-326/CONSISTENCY/UnexpectedFrontendErrors.ql
1 tests passed; 1 tests failed:
FAILED: /home/cipherboy/GitHub/github/codeql-go/ql/test/query-tests/Security/CWE-326/InsufficientKeySize.qlref
My understanding is this last line states that from foo9
(on line 47 when both excerpts foo6...foo11
are added to the test suite), rsa.GenerateKey(..., 1024)
can be reached, which is a false-positive.
I've tried various combinations of:
if keyBits < 2048 {
return
}
... code ...
if keyBits >= 2048 {
... code ...
} else {
return
}
if keyBits < 2048 {
return
} else {
... code ...
}
and similar, but was unable to make the CodeQL false-positive go away. Note that lines 50...52 (foo10
) are absent from the report, because they satisfy the original source condition. Let me know if you need any more information, thank you!
/cc @mickael-hc
Given the following Go code:
package main
func source() interface{} {
return nil
}
type context struct{}
//
func (this *context) Blob(_ int, ct string, body []byte) {}
const constContentType = "application/const"
const constApplication = "application/"
func main() {
// case 1:
{
bodyByte839a := source().([]byte)
var rece context
// The argument is a string literal:
rece.Blob(200, "application/string-literal", bodyByte839a)
}
// case 2:
{
bodyByte839b := source().([]byte)
var rece context
ct := "application/variable"
// The argument is a variable:
rece.Blob(200, ct, bodyByte839b)
}
// case 3:
{
bodyByte839b := source().([]byte)
var rece context
// The argument is a constant:
rece.Blob(200, constContentType, bodyByte839b)
}
// case 4:
{
bodyByte839b := source().([]byte)
var rece context
// The argument is a concatenation of string literals:
rece.Blob(200, "application/"+"concat", bodyByte839b)
}
// case 5:
{
bodyByte839b := source().([]byte)
var rece context
ct := "application/"
// The argument is a concatenation of a variable + a string literal:
rece.Blob(200, ct+"concat", bodyByte839b)
}
// case 6:
{
bodyByte839b := source().([]byte)
var rece context
// The argument is a concatenation of a constant + a string literal:
rece.Blob(200, constApplication+"concat", bodyByte839b)
}
// case 7:
{
bodyByte839b := source().([]byte)
var rece context
ct := "variable-concat"
// The argument is a concatenation of a constant + variable:
rece.Blob(200, constApplication+ct, bodyByte839b)
}
}
.. the following CodeQL query is not able to select all instances of the desired parameter's string value via getStringValue()
:
/**
*/
import go
from Method m, DataFlow::CallNode bodySetterCall
where m.hasQualifiedName(_, "context", "Blob") and bodySetterCall = m.getACall()
select bodySetterCall.getArgument(1).getStringValue(), bodySetterCall
Hello.
I tried to create a CodeQL database for kubernetes-1.9.0 (which is an old version) through LGTM.com.
It was built successfully, but I found a warning in the Golang 'Extraction' log, as follows.
...
[build] Detected go version: go version go1.15 linux/amd64.
[build] Kubernetes requires go1.9.1 or greater.
[build] Please install go1.9.1 or later.
...
As far as I know, I can specify the version through lgtm.yml
for languages such as Python, C#, Java.
However, in this document, I could not find any words to specify the version of Golang.
Is there any way to specify the Golang version when creating a CodeQL database?
My LGTM project is https://lgtm.com/projects/g/donghyunlee00/v2-kubernetes-1.9.0, exactly same code with kubernetes-1.9.0 release code
Best Regards.
Code:
package main
import (
"database/sql"
"net/http"
"fmt"
)
type RequestStruct struct {
Id int64 `db:"id"`
Category []string `db:"category"`
}
func handler(db *sql.DB, req *http.Request) {
RequestData := &RequestStruct{
Id: 1,
Category: req.URL.Query()["category"],
}
q := fmt.Sprintf("SELECT ITEM,PRICE FROM PRODUCT WHERE ITEM_CATEGORY='%s' ORDER BY PRICE",
RequestData.Category)
db.Query(q)
q2 := fmt.Sprintf("SELECT ITEM,PRICE FROM PRODUCT WHERE ITEM_CATEGORY='%s' ORDER BY PRICE",
req.URL.Query()["category"])
db.Query(q2)
}
Expected result:
This query depends on a user-provided value . test.go:26:11
This query depends on a user-provided value . test.go:30:11
Actual result:
This query depends on a user-provided value . test.go:30:11
In #260 and #264, the printAST
query was created and fixed up. There are still a few smaller issues. I'm breaking them into a new issue since this shouldn't prevent merging and closing of previous issues. I'm not even sure if everything here should be fixed:
Method declaration receivers are not in parameter list (or anywhere in the ast)
The FunctionName node is child of the FuncDecl/MethodDecl, though it seems more appropriate for it to be a child of the FuncTypeExpr since its source location is fully contained in that node.
There is a File ast node, which has no source locations and no children. For the purposes of the AST viewer, it's not very interesting. Would be nice if we could exclude this.
There is no Package statement node, but the identifier of the package statement is a node. So in here:
package json
there is no node for package json
, but there is one for json
, which is a direct decedent of the root node. And this is causing ordering issues in the AST viewer.
codeql test run \
--search-path=/home/laptop/vscode-codeql-starter/ \
/home/laptop/vscode-codeql-starter/codeql-go/ql/test/experimental/Unsafe
Executing 1 tests in 1 directories.
Extracting test database in /home/laptop/vscode-codeql-starter/codeql-go/ql/test/experimental/Unsafe.
Could not extract a dataset in /home/laptop/vscode-codeql-starter/codeql-go/ql/test/experimental/Unsafe: Extraction command /home/laptop/vscode-codeql-starter/codeql-go/build/codeql-extractor-go/tools/index.sh failed with status 127
Extraction command /home/laptop/vscode-codeql-starter/codeql-go/build/codeql-extractor-go/tools/index.sh failed with status 127
[1/1] FAILED(EXTRACTION) /home/laptop/vscode-codeql-starter/codeql-go/ql/test/experimental/Unsafe/WrongUsageOfUnsafe.qlref
0 tests passed; 1 tests failed:
FAILED: /home/laptop/vscode-codeql-starter/codeql-go/ql/test/experimental/Unsafe/WrongUsageOfUnsafe.qlref
codeql test run \
--search-path=/home/laptop/vscode-codeql-starter/ \
/home/laptop/vscode-codeql-starter/codeql-go/ql/test/library-tests/semmle/go/frameworks/StdlibTaintFlow
Executing 1 tests in 1 directories.
Extracting test database in /home/laptop/vscode-codeql-starter/codeql-go/ql/test/library-tests/semmle/go/frameworks/StdlibTaintFlow.
Could not extract a dataset in /home/laptop/vscode-codeql-starter/codeql-go/ql/test/library-tests/semmle/go/frameworks/StdlibTaintFlow: Extraction command /home/laptop/vscode-codeql-starter/codeql-go/build/codeql-extractor-go/tools/index.sh failed with status 127
Extraction command /home/laptop/vscode-codeql-starter/codeql-go/build/codeql-extractor-go/tools/index.sh failed with status 127
[1/1] FAILED(EXTRACTION) /home/laptop/vscode-codeql-starter/codeql-go/ql/test/library-tests/semmle/go/frameworks/StdlibTaintFlow/StdlibTaintFlow.ql
0 tests passed; 1 tests failed:
FAILED: /home/laptop/vscode-codeql-starter/codeql-go/ql/test/library-tests/semmle/go/frameworks/StdlibTaintFlow/StdlibTaintFlow.ql
codeql test run \
--search-path=/home/laptop/vscode-codeql-starter/ \
/home/laptop/vscode-codeql-starter/codeql-go/ql/test/library-tests/semmle/go/frameworks/TaintSteps
Executing 1 tests in 1 directories.
Extracting test database in /home/laptop/vscode-codeql-starter/codeql-go/ql/test/library-tests/semmle/go/frameworks/TaintSteps.
Could not extract a dataset in /home/laptop/vscode-codeql-starter/codeql-go/ql/test/library-tests/semmle/go/frameworks/TaintSteps: Extraction command /home/laptop/vscode-codeql-starter/codeql-go/build/codeql-extractor-go/tools/index.sh failed with status 127
Extraction command /home/laptop/vscode-codeql-starter/codeql-go/build/codeql-extractor-go/tools/index.sh failed with status 127
[1/1] FAILED(EXTRACTION) /home/laptop/vscode-codeql-starter/codeql-go/ql/test/library-tests/semmle/go/frameworks/TaintSteps/TaintStep.ql
0 tests passed; 1 tests failed:
FAILED: /home/laptop/vscode-codeql-starter/codeql-go/ql/test/library-tests/semmle/go/frameworks/TaintSteps/TaintStep.ql
$ codeql version
CodeQL command-line toolchain.
Version: 2.2.4.
Copyright (C) 2019-2020 GitHub, Inc.
Unpacked in: /home/laptop/.config/Code/User/globalStorage/github.vscode-codeql/distribution13/codeql
Analysis results depend critically on separately distributed query and
extractor modules. To list modules that are visible to the toolchain,
use 'codeql resolve qlpacks' and 'codeql resolve languages'.
Given this test code:
package main
func main() {
typ := getAType()
_ = typ
ptr := getAPointer()
_ = ptr
}
type Something struct {
}
func getAType() Something {
return Something{}
}
func getAPointer() *Something {
return &Something{}
}
... the following codeql query will only select instances of typ
, and ignore ptr
:
/**
* @kind problem
*/
import go
class Something extends ValueEntity {
Something() {this.getType().hasQualifiedName(_, "Something") }
}
select any(Something c).getAReference()
To write a class that would select all ValueEntity
instances of a type regardless of whether it's the base type or a pointer to that type, I need to do this:
/**
* @kind problem
*/
import go
class Something extends ValueEntity {
Something() { getBaseType(this.getType()).hasQualifiedName(_, "Something") }
}
Type getBaseType(Type typ) {
result = getBaseType*(typ.(PointerType).getBaseType())
or
result = typ
}
select any(Something c).getAReference()
Is this a feature, or is this a bug?
Thanks
For the following Go code:
regexp.MustCompile(`\Achezmoi\.io\z`)
CodeQL generates the warning "Missing regular expression anchor". However, the regular expression is anchored with the strong anchors \A
and \z
(which match the beginning and end of the text), not the weaker anchors ^
and $
(which match the start and end of a line anywhere in the text).
To fix this, the regular expression in isInterestingUnanchoredRegexpString
needs to be updated to detect the anchors \A
and \z
here:
Let me know if you'd like a PR, but it's probably a quick fix for a current contributor.
A megaproject example is modelling a few dozens web frameworks or the standard library :)
package main
import (
"database/sql"
"net/http"
"fmt"
"encoding/json"
"io/ioutil"
)
type RequestStruct struct {
Id int64 `db:"id"`
Category []string `db:"category"`
}
func handler(db *sql.DB, req *http.Request) {
....
// read data from request body and unmarshal to a indeterminacy struct
// POST: {"a": "b", "category": "test"}
var RequestDataFromJson map[string]interface{}
b, _ := ioutil.ReadAll(req.Body)
json.Unmarshal(b, &RequestDataFromJson)
q3 := fmt.Sprintf("SELECT ITEM,PRICE FROM PRODUCT WHERE ITEM_CATEGORY='%s' ORDER BY PRICE",
RequestDataFromJson["category"])
db.Query(q3)
// read data from request body and unmarshal to a determined struct
// POST: {"id": "1", "category": "test"}
var RequestDataFromJson2 RequestStruct
b2, _ := ioutil.ReadAll(req.Body)
json.Unmarshal(b2, &RequestDataFromJson2)
q4 := fmt.Sprintf("SELECT ITEM,PRICE FROM PRODUCT WHERE ITEM_CATEGORY='%s' ORDER BY PRICE",
RequestDataFromJson2.Category)
db.Query(q4)
// read json data from a url parameter
// GET: ?json={"id": 1, "category": "test"}
var RequestDataFromJson3 RequestStruct
json.Unmarshal([]byte(req.URL.Query()["json"][0]), &RequestDataFromJson3)
q5 := fmt.Sprintf("SELECT ITEM,PRICE FROM PRODUCT WHERE ITEM_CATEGORY='%s' ORDER BY PRICE",
RequestDataFromJson3.Category)
db.Query(q5)
}
Running the GitHub CodeQL scanning actions on the following code reports what I believe is a false positive CWE-190 / CWE-681:
package main
import (
"os"
"strconv"
)
func main() {
for _, arg := range os.Args[1:] {
n, err := strconv.ParseInt(arg, 10, 0)
if err == nil {
var itf interface{} = n
i32 := itf.(int32)
println(i32)
}
}
}
On line 13 (i32 := itf.(int32)
) it reports:
Incorrect conversion of an integer with architecture-dependent bit size from to a lower bit size type int32 without an upper bound check.
However this is not a type conversion, this is a type assertion. When itf
holds a type other than int32, the above code panics. (Perhaps that should be a different error.) Neither the logically equivalent type switch nor the two-valued type assertion report this.
var i32 int32
switch x := itf.(type) {
case int32:
i32 = x
default:
panic("bad type")
}
println(i32)
i32, ok := itf.(int32)
println(i32, ok)
The code from which this was extracted is an implementation of an interface similar to sql.Scanner, hence the indirection through an empty interface.
I found a note of mine from a long time ago that mentions HTTP::HeaderRead
, but I cannot find the model.
Was it ever modeled?
Should it be modeled?
How would I access the key used when reading an element of a map?
For example,
id := vars["id"]
If I traced the dataflow to vars
how would I output "id"
?
Comparing
codeql-go/ql/src/semmle/go/frameworks/XPath.qll
Lines 8 to 16 in d31b4d2
codeql-go/ql/src/semmle/go/frameworks/SQL.qll
Lines 8 to 29 in d31b4d2
If this is indeed wrong, feel free to fix it as I currently have no time for fixing it.
When I use codeql-go to regexpMatch url like "http://github.com". I write this ql file.
import go
import semmle.go.security.RequestForgery::RequestForgery
from CallExpr call, DataFlow::Expr arg,string reg
where call.getCalleeName().regexpMatch(".*print.*") and
reg = "/((https|http)?:\/\/[a-zA-Z0-9\.\?\/%-_]*)/gm" and
call.getArgument(0).getStringValue().regexpMatch(reg) and
arg = call.getArgument(0)
select call, arg.getStringValue()
But I get a token recognition error ,because this var reg .
And I remove "\" to slove this error,but this regexp is not work.
So, how can I make this regexp string work?
My codeql version is codeql-cli v2.6.0.
Conextual queries for find references and definitions are not returning any results for go in VS Code. Currently using 2ba9bbf in the latest release of VSCode extension for codeql and master of vscode-codeql-starter workspace.
To reproduce:
src/opt/src/cmd/verifydependencies/verifydependencies.go
collector
right-click -> find referencesBUG
Expected: References to that symbol are found on line 186 and elsewhere
Actual: no references
Similarly, select the collector
symbol on line 186 and invoke Go to Definition. No results are found.
See also github/codeql#4351 for a similar problem in C#.
It'd be really nice to have a guide on how to use this repo correctly. Most of the CLIs bundled inside of it have little or no usage info. I think I managed to create a database correctly using go-autobuilder
, but I'm really not sure. I had to set a couple environment variables (the purpose of which I could only guess). I imagine that such docs would help both random interested parties like me and new members of your team.
"XORM is a Simple & Powerful ORM Framework for Go Programming Language"
Link: https://xorm.io/
If there is a need for this, I will open a pull request later.
A taint-tracking class like this:
private class ReaderReset extends TaintTracking::FunctionModel, Method {
// signature: func (*Reader).Reset(r io.Reader)
ReaderReset() { this.(Method).hasQualifiedName("bufio", "Reader", "Reset") }
override predicate hasTaintFlow(FunctionInput inp, FunctionOutput outp) {
inp.isParameter(0) and outp.isReceiver()
}
}
will not taint-track code like this:
func TaintStepTest_BufioReaderReset(r io.Reader) {
// The flow is from `r` into `reader`.
// Declare `reader` variable:
var reader bufio.Reader
// Call the method that **should** transfer the taint
// from the parameter `r` to the receiver `reader`
// (`reader` should now be tainted).
reader.Reset(r)
// Sink the tainted `reader`:
sink(reader)
}
This problem was brought up before in #118 (comment)
The underlying issue was solved/patched before (I believe in 0546c52)
Thanks!
The scripts/install-reps.sh file is missing on the v1.27.0 tag. I'm guessing we never merged that file into the lgtm branch. At this point, since codeql-go is being merged into codeql, this may not need to be fixed, but raising it anyway to make sure we are aware of the missing file.
Originally reported in github/codeql#8763
The CodeQL Go extractor and libraries are moving to a new home!
github/codeql-go
repositorygithub/codeql
repository, specifically https://github.com/github/codeql/tree/main/go.The github/codeql-go
repository currently contains the open-source CodeQL queries, libraries, and extractor for analyzing codebases written in the Go programming language with GitHub's CodeQL code analysis tools. These contents will be moved to the github/codeql
repository, to live alongside similar libraries for all other programming languages supported by CodeQL.
CodeQL support of the Go programming language will stay the same, except that the code and the development will continue in github/codeql
.
Once the move completes, GitHub will stop using the github/codeql-go
repository and replace its contents with a placeholder announcement.
This change is scheduled for 23 May 2022.
query
or queries
configuration directives in a CodeQL query suite.@id
in the metadata section at the top of the query file.codeql/go-queries
, rather than absolute paths.Examples:
Before | After |
---|---|
- query: path/to/checkout/of/codeql-go/ql/src/Security/CWE-079/ReflectedXss.ql |
- queries: . OR - query: Security/CWE-079/ReflectedXss.ql |
- queries: path/to/checkout/of/codeql-go/ql/src/path/to/subdirectory |
- queries: path/to/subdirectory |
- queries: path/to/subdirectory |
No change needed. |
- queries: path/to/subdirectory |
No change needed. |
- qlpack: codeql-go |
No change needed. |
- qlpack: codeql/go-queries |
No change needed. |
queries
field in a code scanning YAML configurationqueries
field in your code scanning YAML configuration file directly references a query or directory of queries in the github/codeql-go
repository.packs:
field and the corresponding path within the codeql/go-queries
query pack.
name: "My CodeQL config"
queries:
- uses: github/codeql-go/ql/src/experimental/CWE-327@<ref>
name: "My CodeQL config"
packs:
- 'codeql/go-queries:experimental/CWE-327'
queries
field in the github/codeql-action/init
step of a GitHub Actions workflowqueries
field in your workflow file directly references queries in the github/codeql-go
repository.packs:
field and the corresponding path within the codeql/go-queries
query pack.
- name: Initialize CodeQL
uses: github/codeql-action/init@v2
with:
languages: go
queries: github/codeql-go/ql/src/experimental/CWE-327@<ref>
- name: Initialize CodeQL
uses: github/codeql-action/init@v2
with:
languages: go
packs: 'codeql/go-queries:experimental/CWE-327'
github/vscode-codeql-starter
repository or one of its forks, which contain the github/codeql
and github/codeql-go
repositories as submodules.git submodule update --remote
to keep the submodules up to date.ql
submodule and workspace folder, and delete the codeql-go
submodule and workspace folder. Up-to-date checkouts and VS Code workspaces will continue to work.github/codeql
and github/codeql-go
repositoriesgithub/codeql
and github/codeql-go
repositories in sibling directories for use with the CodeQL CLI, as recommended in the CodeQL CLI documentation.github/codeql-go
, please update and rebase them on github/codeql
instead.github/codeql
repository, and leave only placeholder data in the codeql-go
repository. Up-to-date checkouts will reflect this change. After this point you are free to delete the checkout of the github/codeql-go
repository.We expect no change to be required to your CodeQL or code scanning setup.
Please open an issue in github/codeql or file an issue with GitHub support if you encounter any difficulties with this change, and we will be happy to help.
Hi,
I can run a single rule without any problem by using Codeql CLI, but when I pass the codeql-go
rule folder, I get the following error.
...
Starting evaluation of codeql-go/RedundantCode/SelfAssignment.ql.
Starting evaluation of codeql-go/RedundantCode/RedundantRecover.ql.
Starting evaluation of codeql-go/localReferences.ql.
Starting evaluation of codeql-go/AlertSuppression.ql.
Starting evaluation of codeql-go/Metrics/FLinesOfCode.ql.
Starting evaluation of codeql-go/Metrics/FLinesOfComment.ql.
Starting evaluation of codeql-go/printAst.ql.
Oops! A fatal internal error occurred.
com.semmle.util.exception.CatastrophicError: Program depends on template extensional selectedSourceFile, but no value was supplied for it
at com.semmle.inmemory.caching.TemplateStore.<init>(TemplateStore.java:34)
at com.semmle.inmemory.MemoryBackend.executeQuery(MemoryBackend.java:324)
at com.semmle.cli2.ql.EvaluationCommon.lambda$executeQueryAsync$2(EvaluationCommon.java:181)
at com.semmle.util.concurrent.FutureUtils.futureAutoclose(FutureUtils.java:185)
at com.semmle.cli2.ql.EvaluationCommon.executeQueryAsync(EvaluationCommon.java:179)
at com.semmle.cli2.execute.ExecuteQueriesCommand.lambda$executeSubcommand$3(ExecuteQueriesCommand.java:296)
at com.semmle.cli2.ql.Paralleliser$1.lambda$next$0(Paralleliser.java:68)
at com.semmle.util.concurrent.FutureUtils.supplyCompose(FutureUtils.java:217)
at com.semmle.cli2.ql.Paralleliser.lambda$startMoreJobs$1(Paralleliser.java:98)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Could you please help me to solve the problem?
Thanks!
The example for finding calls to fmt.Println
seems not to work. Maybe things have changed since it was written. Is there some other way to find calls to a particular import?
I've put up a demonstration here.
https://github.com/dittyroma/codeql-go/tree/find-calls
ql/src/experimental/Test
contains example queries. Println.ql
is a copy of the example calltofunction.ql
. Functions.ql
drops the filter to find all function calls. Methods.ql
tries to find all method calls. All 3 queries find nothing on this program.
package main
import (
"fmt"
)
func main() {
fmt.Println("line")
}
What I'd like to do is find file system functions, eg for os.Create()
calls that create files.
from Function function, DataFlow::CallNode call
where
function.hasQualifiedName("os", "Create") and
call = function.getACall()
select call
This command reproduces on my system.
$ codeql test run "/home/user/codeql/sft/codeql-repo-go/ql/test/experimental/Test"
Executing 3 tests in 1 directories.
Extracting test database in /home/user/codeql/sft/codeql-repo-go/ql/test/experimental/Test.
Compiling queries in /home/user/codeql/sft/codeql-repo-go/ql/test/experimental/Test.
Executing tests in /home/user/codeql/sft/codeql-repo-go/ql/test/experimental/Test.
Cannot find Methods.expected file.
Query produced no results.
[1/3 eval 703ms] FAILED(RESULT) /home/user/codeql/sft/codeql-repo-go/ql/test/experimental/Test/Methods.qlref
Cannot find Println.expected file.
Query produced no results.
[2/3 eval 56ms] FAILED(RESULT) /home/user/codeql/sft/codeql-repo-go/ql/test/experimental/Test/Println.qlref
Cannot find Functions.expected file.
Query produced no results.
[3/3 eval 44ms] FAILED(RESULT) /home/user/codeql/sft/codeql-repo-go/ql/test/experimental/Test/Functions.qlref
0 tests passed; 3 tests failed:
FAILED: /home/user/codeql/sft/codeql-repo-go/ql/test/experimental/Test/Methods.qlref
FAILED: /home/user/codeql/sft/codeql-repo-go/ql/test/experimental/Test/Println.qlref
FAILED: /home/user/codeql/sft/codeql-repo-go/ql/test/experimental/Test/Functions.qlref
๐๐ป Hi, folks! I'm here on behalf of @github/frameworks-services. We're researching how and why teams are using the zap
module (go.uber.org/zap
) to help us in our development planning and inform the Go guidance we provide to internal teams.
From my research, it appears @github/codeql-go is using zap
in the following repository/repositories:
zap
) configured & enabled in all the environment(s) the repositories are used in?zap
instead of some other module? (ie was it an arbitrary choice, or did you have specific reason(s)?)go-log
module?
zap
and go-log
or other obstacles that would make it difficult to switch from zap
to go-log
?zap
in your team? (If yes, please list them)Query some repo with this CodeQL:
import go
from ForStmt f
select f
Expected: for
loops with range
statement are available in output
Seen: no such loops found in code, only other types of for
in output
Similar to #635.
The suggested mitigation for "Improper Output Neutralization for Logs" includes performing a string replacement of "\n"
characters using strings.Replace
.
The "%q"
formatting directive (double-quoted string safely escaped with Go syntax) is also a form of escaping user input, in particular for plain-text logs, as it escapes non-printable characters (including "\n"
, "\r"
, "\t"
, etc).
However, code using log.Printf("%q", ...)
is still considered unsafe by CodeQL scanning.
Here's an example run:
https://github.com/getsentry/sentry-go/pull/399/checks?check_run_id=4583764415
Would it be possible to teach CodeQL about "%q"
? There are also variations, such as "%+q"
, "%#q
, "%060q"
, % q
, %[2]q
, etc, with the complete parsing rules being non-trivial.
Perhaps use the fact that fmt.(*fmt).fmtQ
is called before writing to the io.Writer
where the log message is going.
It seems that the go-autobuilder
binary is not included in this repo , so my db creation is failed due to the binary not found .
codeql-cli command : codeql database create my_repo --language go
Am I missing something ?
If a function returns only one result, then FunctionOutput.isResult()
will work, and FunctionOutput.isResult(0)
will not work.
If a function returns two or more results, then FunctionOutput.isResult(0)
will work, and FunctionOutput.isResult()
will not work.
It would be a lot less error-prone and a lot more awesome to be able to write FunctionOutput.isResult(0)
regardless of whether there is one result or more.
It would be a lot harder to make errors (such as using FunctionOutput.isResult()
when in reality the function returns more than one result) if FunctionOutput.isResult(0)
and FunctionOutput.isResult()
were equivalent regardless of the number of results returned by the function.
I believe this would make things simpler to the end-developer that will be writing codeql.
This issue is about adding support to the CodeQL VS Code extension for viewing ASTs of Go files. See the parent issue github/vscode-codeql#492.
Transport.RoundTrip() in net/http will performs an HTTP request to a URL
import (
"bytes"
"fmt"
"net/http"
)
func request2(url string) {
req, _ := http.NewRequest("GET", url, bytes.NewBuffer(nil))
tr := &http.Transport{}
resp, _ := tr.RoundTrip(req)
fmt.Println(resp.StatusCode)
}
Given this scenario:
func someFunc(ptr uintptr) {
type SomeStruct struct {
Hello uintptr
}
evilStruct := &SomeStruct{}
evilStruct.Hello = ptr
}
Is it possible to propagate taint from evilStruct
to ptr
, so that a flow config with source evilStruct
and sink ptr
could succeed?
Option A:
// Package: github.com/revel/[email protected]
// Function: func ActionInvoker(c *Controller, _ []Filter)
exists(Function fn, DataFlow::CallNode call |
fn.hasQualifiedName("github.com/revel/revel", "ActionInvoker")
|
call = fn.getACall() and
this = FunctionOutput::parameter(0).getExitNode(call)
)
or
// Package: github.com/revel/[email protected]
// Function: func AddControllerType(moduleSource *Module, controllerType reflect.Type, methods []*MethodType) (newControllerType *ControllerType)
exists(Function fn, DataFlow::CallNode call |
fn.hasQualifiedName("github.com/revel/revel", "AddControllerType")
|
call = fn.getACall() and
this = FunctionOutput::parameter(1).getExitNode(call)
)
Option B:
// Package: github.com/revel/[email protected]
exists(Function fn, DataFlow::CallNode call |
// Function: func ActionInvoker(c *Controller, _ []Filter)
fn.hasQualifiedName("github.com/revel/revel", "ActionInvoker") and
this = FunctionOutput::parameter(0).getExitNode(call)
or
// Function: func AddControllerType(moduleSource *Module, controllerType reflect.Type, methods []*MethodType) (newControllerType *ControllerType)
fn.hasQualifiedName("github.com/revel/revel", "AddControllerType") and
this = FunctionOutput::parameter(1).getExitNode(call)
|
call = fn.getACall()
)
The example only shows 2 functions, but imagine that there are 20. Which is better? Is there any difference of some kind other than less code repetition? Thanks
I create a database for my repo with gomod, but all the dependencies are skipped, which result that, when I run a dataflow query, it can't detect the tainted actions and nodes in the library.
Is there any mistake in my creation or query?
logs:
[2021-10-11 12:22:22] [build-stderr] 2021/10/11 12:22:22 Skipping dependency package xx/gopkg/jsonx
let's say A is a struct with fields {userid, username}, B is untrusted source, A.userid = B
I expect that A'll be tainted, but actually not..
I check the source code in the function 'localAdditionalTaintStep', seems it doesn't cover this action..
So if I want to let A be tainted by B, what can I do in my overrided function 'isAdditionalTaintStep'?
Thanks very much!!
Project proposal: Taint-track the golang.org/x/* packages.
Here is a list of the top 100 golang.org/x/* packages sorted by how many times they were imported (taken from here)
A package is relevant if it has elements that can be put into a TaintTracking
model. Not all packages from the below list will be "relevant" at the end.
I will use this ticket to track the progress.
CWE-117: Improper Output Neutralization for Logs
CWE-117 is being reported by CodeQL in the following code:
func makeErrorForHTTPResponse(resp *http.Response) error {
bodyBytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
return err
}
url := resp.Request.URL.String()
safeURL := strings.Replace(url, "\n", "", -1)
safeURL = strings.Replace(safeURL, "\r", "", -1)
return fmt.Errorf("%s %s returned HTTP %s; \n\n %#q", resp.Request.Method, safeURL, resp.Status, bodyBytes)
}
Despite this code being near identical to the provided "good" example
func handlerGood(req *http.Request) {
username := req.URL.Query()["username"][0]
escapedUsername := strings.Replace(username, "\n", "", -1)
escapedUsername = strings.Replace(escapedUsername, "\r", "", -1)
log.Printf("user %s logged in.\n", escapedUsername)
}
here.
Here is a screen shot of the output logs for further clarification
It appears that CodeQL completely ignores the above two functions performing the string replacement.
CWE-190: Integer Overflow or Wraparound
CodeQL identifies CWE-190 in this code:
// parsePositiveInt parses value as an int. It returns an error if value cannot
// be parsed or is negative.
func parsePositiveInt(value string) (int, error) {
switch i64, err := strconv.ParseInt(value, 10, 64); {
case err != nil:
return 0, fmt.Errorf("unable to parse positive integer %q: %v", value, err)
case i64 < 0:
return 0, fmt.Errorf("unable to parse positive integer %q: negative value", value)
case i64 > math.MaxInt:
return 0, fmt.Errorf("unable to parse positive integer %q: overflow", value)
default:
return int(i64), nil
}
}
reporting:
Incorrect conversion of a 64-bit integer from to a lower bit size type int without an upper bound check.
for the line
return int(i64), nil
Clicking on "Show paths" in the GitHub Code Scanning Alert shows:
I believe this is a false positive because the switch
cases do include an upper bound check, but it looks like these are not taken into account by CodeQL.
I think plain go build
is equivalent to go build .
, but our extractor doesn't extract anything in the case where no arguments are provided. To reproduce: provide a Makefile that runs go build
without args and observe that nothing is extracted, even though go build
loads .go files.
Source-locations at least for non-go-modules projects (i.e., those which are extracted by moving the project into a fake GOPATH and then performing post-hoc path-rewriting to make the source-locations reflect the pre-move file locations) don't use the realpath
of the source filename. This means that if the source root isn't its own realpath (e.g., it's a symlink pointing elsewhere, such as /tmp -> /private/tmp
on OSX), codeql bqrs interpret
will discard the results because the source-location (which hasn't been realpath
'd) doesn't appear to fall within the source root (which has been realpath
'd by codeql database create
).
If I have an HTTP client, and I do a HTTP request to a remote server, does the response data count as UntrustedFlowSource
?
Hello codeql-go team,
I'm working on a new tool that will generate concrete implementations of codeql concepts, and I'm starting with UntrustedFlowSource
.
Below are examples of what the generated codeql might look like. Please let me know what you think about it. Thanks!
/**
* This is a doc.
*/
import go
private module SomeFramework {
/**
* This is a doc.
*/
private class Source extends UntrustedFlowSource::Range {
Source() {
// Example block 1: the type is a function.
// The source is either the result (one or more), or a parameter (one or more),
// or a mix of them.
(
// Function: github.com/example/package.GetSomething
exists(Function fn, DataFlow::CallNode call |
fn.hasQualifiedName("github.com/example/package", "GetSomething")
|
// The source is the result:
call = fn.getACall() and this = call.getResult()
)
or
// Function: github.com/example/package.ParseSomething
exists(Function fn, DataFlow::CallNode call |
fn.hasQualifiedName("github.com/example/package", "ParseSomething")
|
// The source is the 0th parameter:
call = fn.getACall() and this = FunctionOutput::parameter(0).getExitNode(call)
)
)
or
// Example block 2: the type is a struct.
// The source can be a method call (results or parameters of a call),
// or a field read.
(
// Struct: github.com/example/package.Context
exists(string typeName | typeName = "Context" |
// Method calls on `Context`:
exists(DataFlow::MethodCallNode call, string methodName |
call.getTarget().hasQualifiedName("github.com/example/package", typeName, methodName) and
(
methodName = "FullPath" and
(
// The source is the method call result #0:
this = call.getResult(0)
or
// The source is method call parameter #0:
this = FunctionOutput::parameter(0).getExitNode(call)
)
or
// The source is any result of the call?
methodName = "GetHeader"
)
)
or
// Field reads on `Context`:
exists(DataFlow::Field fld, string fieldName |
fld.hasQualifiedName("github.com/example/package", typeName, fieldName) and
// The source is one of these fields reads:
fieldName in ["Accepted", "Params"] and
this = fld.getARead()
)
)
or
// Struct: github.com/example/package.SomeStruct
exists(DataFlow::Field fld, string fieldName |
fld.hasQualifiedName("github.com/example/package", "SomeStruct", fieldName) and
// The source is one of these fields reads:
fieldName in ["Hello", "World"] and
this = fld.getARead()
)
)
or
// Example block 3: the type is some custom type.
// The source is a method call on that type (results or parameters of a call).
// Some type: github.com/example/package.Slice
exists(string typeName | typeName = "Slice" |
// Method calls:
exists(DataFlow::MethodCallNode call, string methodName |
call.getTarget().hasQualifiedName("github.com/example/package", typeName, methodName) and
(
methodName = "GetHeader" and
// The source is the method call result #0:
this = call.getResult(0)
or
methodName = "ParseHeader" and
// The source is method call parameter #1:
this = FunctionOutput::parameter(1).getExitNode(call)
)
)
)
or
// Example block 4: the type is an interface.
// The source is a method call on that interface (results or parameters of a call).
// Interface: github.com/example/package.SomeInterface
exists(string typeName | typeName = "SomeInterface" |
// Method calls:
exists(DataFlow::MethodCallNode call, string methodName |
call.getTarget().implements("github.com/example/package", typeName, methodName) and
(
methodName = "GetSomething" and
// The source is the method call result #0:
this = call.getResult(0)
or
methodName = "UnmarshalSomething" and
// The source is method call parameter #2:
this = FunctionOutput::parameter(2).getExitNode(call)
)
)
)
}
}
}
/**
* Module summary:
*
* UntrustedFlowSource:
* FUNCS:
* func (*clevergo.tech/clevergo.Context).BasicAuth() (username string, password string, ok bool)
* func (*clevergo.tech/clevergo.Context).Decode(v interface{}) (err error)
* func (*clevergo.tech/clevergo.Context).DefaultQuery(key string, defaultVlue string) string
* func (*clevergo.tech/clevergo.Context).FormValue(key string) string
* func (*clevergo.tech/clevergo.Context).GetHeader(name string) string
* func (*clevergo.tech/clevergo.Context).PostFormValue(key string) string
* func (*clevergo.tech/clevergo.Context).QueryParam(key string) string
* func (*clevergo.tech/clevergo.Context).QueryParams() net/url.Values
* func (*clevergo.tech/clevergo.Context).QueryString() string
* func (clevergo.tech/clevergo.Decoder).Decode(req *net/http.Request, v interface{}) error
* func (clevergo.tech/clevergo.Params).String(name string) string
* STRUCTS:
* clevergo.tech/clevergo.Context
* clevergo.tech/clevergo.Param
* TYPES:
* clevergo.tech/clevergo.Params
*
* ---
* TaintTracking:
* FUNCS:
* func (*clevergo.tech/clevergo.Application).RouteURL(name string, args ...string) (*net/url.URL, error)
* func (*clevergo.tech/clevergo.Context).Context() context.Context
* func (clevergo.tech/clevergo.Decoder).Decode(req *net/http.Request, v interface{}) error
* func (clevergo.tech/clevergo.Params).String(name string) string
* func (clevergo.tech/clevergo.Renderer).Render(w io.Writer, name string, data interface{}, c *clevergo.tech/clevergo.Context) error
* func clevergo.tech/clevergo.CleanPath(p string) string
*
* ---
* HTTP::Redirect:
* FUNCS:
* func (*clevergo.tech/clevergo.Context).Redirect(code int, url string) error
*
* ---
* HTTP::ResponseBody:
* FUNCS:
* func (*clevergo.tech/clevergo.Context).Blob(code int, contentType string, bs []byte) (err error)
* func (*clevergo.tech/clevergo.Context).Emit(code int, contentType string, body string) (err error)
* func (*clevergo.tech/clevergo.Context).Error(code int, msg string) error
* func (*clevergo.tech/clevergo.Context).HTML(code int, html string) error
* func (*clevergo.tech/clevergo.Context).HTMLBlob(code int, bs []byte) error
* func (*clevergo.tech/clevergo.Context).JSON(code int, data interface{}) error
* func (*clevergo.tech/clevergo.Context).JSONBlob(code int, bs []byte) error
* func (*clevergo.tech/clevergo.Context).JSONP(code int, data interface{}) error
* func (*clevergo.tech/clevergo.Context).JSONPBlob(code int, bs []byte) error
* func (*clevergo.tech/clevergo.Context).JSONPCallback(code int, callback string, data interface{}) error
* func (*clevergo.tech/clevergo.Context).JSONPCallbackBlob(code int, callback string, bs []byte) (err error)
* func (*clevergo.tech/clevergo.Context).SetContentType(v string)
* func (*clevergo.tech/clevergo.Context).SetContentTypeHTML()
* func (*clevergo.tech/clevergo.Context).SetContentTypeJSON()
* func (*clevergo.tech/clevergo.Context).SetContentTypeText()
* func (*clevergo.tech/clevergo.Context).SetContentTypeXML()
* func (*clevergo.tech/clevergo.Context).String(code int, s string) error
* func (*clevergo.tech/clevergo.Context).StringBlob(code int, bs []byte) error
* func (*clevergo.tech/clevergo.Context).Stringf(code int, format string, a ...interface{}) error
* func (*clevergo.tech/clevergo.Context).Write(data []byte) (int, error)
* func (*clevergo.tech/clevergo.Context).WriteString(data string) (int, error)
* func (*clevergo.tech/clevergo.Context).XML(code int, data interface{}) error
* func (*clevergo.tech/clevergo.Context).XMLBlob(code int, bs []byte) error
*
* ---
* HTTP::HeaderWrite:
* FUNCS:
* func (*clevergo.tech/clevergo.Context).SetHeader(key string, value string)
*/
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.