We need C wrappers in order to connect with other programs like Nginx and Apache.

We are going to copy the functions used by modsecurity but replacing msc_ with coraza_.

A sample would be:

#include "coraza/coraza.h"


char main_rule_uri[] = "basic_rules.conf";

int main (int argc, char **argv)
{
    Coraza *waf = NULL;
    Parser *parser = NULL
    Transaction *transaction = NULL;
    waf = coraza_init();

    parser = coraza_parser(waf);
    coraza_parser_read_file(main_rule_uri);

    transaction = coraza_new_transaction(waf);

    coraza_process_connection(transaction, "127.0.0.1");
    coraza_process_uri(transaction, "http://www.modsecurity.org/test?key1=value1&key2=value2&key3=value3&test=args&test=test");
    coraza_process_request_headers(transaction);
    coraza_process_request_body(transaction);
    coraza_process_response_headers(transaction);
    coraza_process_response_body(transaction);

    return 0;
}

Summary

Instead of writing @ipMatch 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 we could just type @ipMatch private

Basic example

SecRule REMOTE_ADDR "@ipMatch public" "id: 4, phase:1, log, deny, status:403,msg:'External access'"

Your's such great & useful software

Libinjection is not part of the main project and it must be removed, it should be cloned as a git submodule.

There are two ways it can be done, we could change

Read(reader io.Reader, mime string, storagePath string) error
// To
Read(reader io.Reader, mime string, storagePath string, fileMode fs.FileMode) error

Or we could create a new way to send parameters to the Read function, like

Read(options types.BodyBufferOptions) error

It must be solved before relasing v2

Required by many CRS tests...

	if (Code >= 0 && Code <= 65535)  {
		hmap = utils.UnicodeMapping.At(Code)
	}

It's already implemented but should we load unicode mappings like modsecurity from a file or maybe hardcode the data?

Rule variables are supposed to support regex but they are only supporting strings:

for example

SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/.*secret/ "..." "..."

See test 930121-3

ALSO ADD TESTS

Future versions of Coraza WAF must have a plugins architecture that implies a few changes to the current APIs, here is a list of the possible plugins controllers:

• Operators: Add new operators to the rules system
• Actions: Add new actions to the rule system
• Directives: Add new directives to the directive system
• Logging Engine: Add new logging technique to the logging engine, ex: Elastic Search
• Persistance Engine: Add new persistance engine for collections, ex: Memcached

Additional utilities

• WAF Extras map to store plugins data map[int]interface{}

Possible plugins code

type TestPlugin struct{}

func (p *TestPlugin) OnLoad(plugin *engine.Plugins){
	plugin.RegisterOperator("test", testOperator)
	plugin.RegisterAction("test", testAction)
	plugin.RegisterDirective("test", testDirective)
	plugin.RegisterLoggingEngine("elastic", elasticEngine)
	plugin.RegisterPersistanceEngine("memcached", memcachedEngine)
}

func (p *TestPlugin) OnExit(){

}

func (p *TestPlugin) Name() string{
	return "TestPlugin"
}

We could add more hooks like OnTransactionCreate but I believe it requires more analysis.

I will start working on Coraza v2, if you are only using the high level APIs (Create WAF, tx, seclang and transaction manipulation) your migration will be smooth, because only low level APIs will be rewritten.

The idea of v2 is to extend compatibility with modsecurity v2 and remove CGO dependencies.

• full v2 Seclang compatibility (not deprecated or removed modsecurity features and not PCRE): Modifications like the new SecAuditLog will be rolled back
• The audit engine will work just like the modsecurity engine, but it will support json2 for the Coraza format
• Debug logging will be like modsecurity but in a structured format

Other important features are:

• Remove dependency from Seclang, now it's easier to create rules however you want
• Many new native plugin interfaces
• Almost no dependencies

Coraza v2 will only support highly tested features, every other feature will be included as a plugin until it can be merged to the main project. Only plugins without dependencies will be merged, some exceptions might be made.

v2.0

• #102
• Better internal APIs (public APIs are going to be kept)
• @rx will use RE2 instead of PCRE and create a PCRE plugin
• Remove @detectSQLi and @detectXSS, replace them with libinjection plugin
• #101
• Complete logging rework (error)
• New API to add error callback for transactions (something like tx.AddLogCb(cb))
• Create a CRS compatibility plugin, which includes libinjection and pcre, maybe it can be a branch
• Extend body processors interface, maybe transform them to plugin infrastructure
• Rework the test engine, it has a lot of bugs
• #99
• #103
• #105
• Pass lint checks
• #76

v2.x

• Handle temporary files from multipart
• Replace persistent collections with an internal persistence engine for plugins
• Add windows support

A @jwt operator should be useful to enforce the usage of JWT token, it should support pre-shared keys and certificates.

Example:

# will be triggered if expired or invalid signature
SecRule REQUEST_HEADERS:token "@jwt secret-token-key" "id: 1, deny, phase: 1, log, msg:'Invalid or expired JWT token'"

According to some reports, some rule projects use case insensitive directives, we might do that to for compatibility.

For v2

Rules should be marshaled to json so developers can create their own compilers

I have no more information by the moment but it requires testing.

GET /?var=%22in%20\u0076\u0061l\u0075e\u004F\u0066%3d HTTP/1.0

The previous payload is being read as a literal string

Help is required to improve coverage to 80%

There are two ways to generate coverage in Coraza:

• _test.go files, add test cases to go tests
• add practical test cases to utils/tests/engine_test.go

A branch was created for this, improve-coverage

Log action was implemented for both audit log and rule log.

Two log variables were implemented, one for *Rule and another for *Transaction, in the case of rule, it tells a transaction whether to log the current rule if matched, meanwhile tx.Log will tell tx.ProcessLogging if the transaction should be a candidate for Audit Logging.

The variables were already added, now we have to test it is properly working.

• AUTH_TYPE
• DURATION
• ENV
• HIGHEST_SEVERITY
• INBOUND_DATA_ERROR
• MULTIPART_* (DEPRECATED)
• OUTBOUND_DATA_ERROR
• PERF_*
• REMOTE_USER
• STATUS_LINE
• TIME_*
• URLENCODED_ERROR
• WEBAPPID

Plugins are a features expected for later v1 releases, they should be able to register operators, actions, transformations, audit loggers and persistence engines.

We are going to use .so files compiled using go build -buildmode=plugin.

Plugins should be importable like:

SecPluginImport /path/to/plugin.so
# or downloadable from Coraza servers:
SecPluginImport user/plugin-name@version

# report updates available on startup logs
SecPluginCheckUpdates On

Important: Coraza plugins won't support CGO, even pcre and libinjection, that's because they must be easy to share and non dependent from libc and pcre versions.

Writing a plugin

package pluginname

import (
   engine"github.com/jptosso/coraza-waf"
)

type PluginName struct {}

func (PluginName) Name() string {
    return "pluginName"
}

func (pn * PluginName) GetOperators() map[string]engine.Operator {
    return map[string]Operator {
        "TestOperator": SomeOperatorStruct
    }
}

// and the same for actions, transformations, etc...

We could also use interfaces{} instead of multiple functions for each kind of plugin.

Another approach could be using a Waf instance to create objects:

package pluginname

import (
   engine"github.com/jptosso/coraza-waf"
)

type CorazaPlugin struct {}

func (CorazaPlugin) Name() string {
    return "pluginName"
}

func (pn * CorazaPlugin) Init(waf *engine.Waf) error {
    waf.RegisterOperator("operatorName", SomeOperator)
    waf.RegisterTransformation("someTransformation", SomeTransaformation)
    return nil
}

About compiling the plugins

I think compiling plugins may be hard for some system administrators, that's why a compiling tool should be developed.

Testing

A testing package must be provided to help developers test and benchmark their plugins.

Auto compilation and online repository

There might be a repository like https://plugins.coraza.io/ with a plugin search engine and the options to download the .so files or find the sourcecode.

Full testing coverage is required to assure code quality.

Description

I tried to include OWASP CRS but I am getting this error.
{"level":"error","ts":1637334926.4698696,"msg":"[Parser] [Line 0] Failed to compile rule (Invalid variable URI_PARSER_ERROR): URI_PARSER_ERROR \"!@eq 0\" \"id:'200006', phase:1,t:none,log,deny,status:400,msg:'Failed to parse request url.',logdata:'Failed to parse request url',severity:2\"","line":0} {"level":"error","ts":1637334926.4699538,"msg":"Invalid variable URI_PARSER_ERROR","path":"./rules/crs2.conf"}

Config Lines:

SecRule URI_PARSER_ERROR "!@eq 0" \
"id:'200006', phase:1,t:none,log,deny,status:400,msg:'Failed to parse request url.',logdata:'Failed to parse request url',severity:2"

Steps to reproduce

	waf := coraza.NewWaf()
	parser, _ := seclang.NewParser(waf)
	parser.FromFile("./rules/crs2.conf")

wget https://raw.githubusercontent.com/jptosso/coraza-waf/master/coraza.conf-recommended
git clone https://github.com/coreruleset/coreruleset
cat coraza.conf-recommended coreruleset/crs-setup.conf.example coreruleset/rules/*.conf > crs2.conf

JSON is now fully supported and tested.

{"a": "b", "b": [1,2,3], "c": {"a": "b"}}

The previous json will be translated as the following ARGS:

• json.a = "b"
• json.b = 3
• json.b.1 = 1
• json.b.2 = 2
• json.b.3 = 3
• json.c.a = "b"

Somehow tests sometime fails for golang 1.14: https://travis-ci.org/github/jptosso/coraza-waf/jobs/771953316
Can't figure the issue but the problem is with the actions.yaml tests

From: https://github.com/microcosm-cc/bluemonday

Implementation must be discussed, maybe a policy file or a policy string

https://github.com/SpiderLabs/ModSecurity/tree/v3/master/test/test-cases/regression

They should be transformed to go-ftw format

👋 I am currently making a coraza-waf extension for https://github.com/caddyserver/caddy, which has been basically completed, but found that all requests after the phase2 process failed to pass the inspection.

How should I enable CRS correctly?

c.ctx = ctx
c.logger = ctx.Logger(c)
defer c.logger.Sync()

c.Waf = engine.NewWaf()
c.Waf.Init()
c.Waf.InitLogger()

crs, err := crs.NewCrs(c.Waf)
if err != nil {
	return err
}
crs.TemplateDir = "/home/naiba/Desktop/coraza-waf/docs/crs/rules"
err = crs.Build()
if err != nil {
	return err
}
l := len(c.Waf.Rules.GetRules())

if l == 0 {
	return errors.New("No rules found")
}

if l < 500 {
	return fmt.Errorf("Not enough CRS rules, found %d", l)
}

func (h *HTTPHandler) ServeHTTP(w http.ResponseWriter, r *http.Request, next caddyhttp.Handler) error {
	tx := h.Waf.NewTransaction()

	err := tx.ParseRequestObjectHeaders(r)
	if err != nil {
		h.logger.Error("ParseRequestObjectHeaders", zap.Any("err", err))
		return err
	}
	if tx.ExecutePhase(1) {
		w.Write([]byte(tx.GetErrorPage()))
		return errors.New("phase1")
	}

	err = tx.ParseRequestObjectBody(r)
	if err != nil {
		h.logger.Error("ParseRequestObjectBody", zap.Any("err", err))
		return err
	}
	if tx.ExecutePhase(2) {
		h.logger.Info("MatchedRules", zap.Any("disrupted", tx.Disrupted), zap.Any("rules", tx.MatchedRules))
		w.Write([]byte(tx.GetErrorPage()))
		return errors.New("phase2")
	}

	// Continue down the handler stack
	if err := next.ServeHTTP(w, r); err != nil {
		return err
	}

	return nil
}

Coraza WAF should be easy to integrate with any existing technology or language, so best way to achieve this should be an RPC interface.

Methods

Core methods

• Get Transaction
• Create Transaction: Creates transaction with the provided fields, can trigger an execution phase
• Update Transaction: Update transaction with the provided fields, can trigger an execution phase
• Trigger Transaction Execution phase

Possible Methods

• Read Persistent Collection
• Update Persistent Collection
• Remove persistent Collection
• Statistics
• Evaluate directive

Package structure

JSON or binary? Golang does not support packing and it could be hard to achieve a binary structure, by the other hand, binary packages are hard to understand and less developer friendly. Maybe we could have both RPC interfaces.

Other considerations

• Existing transactions will live as a json structure in Redis
• There should be a garbage collector capable of ending unfinished transactions by forcing phase 5
• A transaction is destroyed after phase 5 is executed
• We could create setters and getters for each variable or provide direct access to the collection data structure

Benchmark report must contain comparison for two use cases using Coreruleset:

• A GET request with multiple arguments
• A Multipart POST request with files and variables

Both must be implemented as a library and the result must contain the ms average for 100.000 requests splitted between both use cases.

Internal wrappers were created in order to achieve this, for example:

type ruleActionParams struct {
	// The name of the action, used for logging
	Name string

	// The action to be executed
	Function RuleAction
}

Audit logger was splitted between:

• Logger: Main logging structure, it handles log writters
• Writter: Writes the log wherever required
• Formatter: Transform the auditlog struct into a byte array of some format, for example json

Writters and formatters can be extended as plugins

Recaptcha operator is an idea to add recaptcha support, the syntaxis would look like:

#Set recaptcha key
SecRecaptchaKey key12345
#Set recaptcha template
SecRecaptchaTemplate /etc/coraza-waf/templates/recaptcha.html

# Initialize ip collection
SecAction "id:1, initcol:ip=%{remote_addr}, phase: 1"
# Check if the captcha was
SecRule REQUEST_METHOD "POST"  "id:2, phase: 1, chain"
    SecRule IP:recaptcha "1" "chain"
    SecRule ARGS_POST:recaptcha_token "@recaptcha" "setvar:ip.recaptcha=0"
# If challenge failed, recaptcha will render the SecRecaptchaTamplate
SecRule IP:recaptcha "1" "id: 2, recaptcha, phase: 1"

nginx - > Coraza WAF -> Application

What are your plans with this integration?
Send NGINX traffic to a Coraza using a subrequest (njs or lua)?

There are many bypasses for ip addresses, like http://⑯⑨。②⑤④。⑯⑨｡②⑤④, there should be a method to detect if the string is an ip address, we could use transformations or an operator.

Must be discussed

In order to remove CGO from Coraza we have to remove two dependencies, libinjection and libpcre.

Libpcre

Libpcre can be replaced with golang regexp (re2). A wrapper has been created under utils/regex, it will pick between the CGO version (libpcre) and non CGO (RE) based on the flag CGO_ENABLED.

Libinjection

I'm currently working on a port, libinjection-go, in the meantime it will be a different repository but the final version is going to be embedded with Coraza as it won't be maintained as a importable package.

Coraza does actually support most features required by CRS, we need:
ctl action
jsdecode transformation
cssdecode transformation

OWASP CRS test suite will always fail as it relies on some behaviours provided by Apache and Nginx that the GOLANG reverse proxy fixes internally.

What's your plan about this?

I am the maintainer of ingress-nginx and one of the pain points is the WAF (ModSecurity) feature.
I'm always researching for alternatives that allow me to decouple the feature from the ingress controller and use a sidecar or maybe a socket (Shared Volume) in the same deployment?

Coverage went down for coraza v2, it must be brought back to 80. This issue is here to keep track of the coverage levels.

Something like SecGeoAutoload DATABASE APIKEY, it should connect to maxmind site and download the proper database.

We must check licenses first, maybe they don't want us to do that.

Coraza WAF can be exported to C using CGO exports. We are going to build an Apache module that makes uses of this features. The project will be considered ready once it is fully executing the OWASP CRS test suite using go-ftw and the Coraza Apache module.

• Export tx.Request|ResponseBodyBuffer.Write and a write from file function
• Export the 5 phase processing functions (ProcessRequestHeaders, ProcessRequestBody, ProcessResponseHeaders, ProcessResponseBody, ProcessLogging)
• Export a function to get the interruption from a transaction
• Create additional export functions to create waf instances, transactions, evaluate directives and import directives file
• Create a Interruption interface in C compatible with coraza, it shall hold the disruptive action, status and additional parameters like redirect url or proxy address
• The functions are compatible with libmodsecurity, so we must use the official libmodsecurity apache module
• Immitate the logging callback system from libmodsecurity and make it compatible with each logging level
• Create freeing functions for coraza transactions and waf instances, also check how to reconnect the garbage collector with the CGO pointers or manually delete everything
• Create tests for the CGO functions and C exported functions
• Create a whole test suite for the apache wrapper
• Create benchmarks for the wrapper and the apache module
• Compare the results with Modsecurity v2/v3 and Apache

I have created some tests under https://github.com/jptosso/coraza-cexport, apxs is required to compile the module

Status of unsupported rules:

• 920180 Not a problem with the WAF, a problem with Apache compatibility
• 920190 Might be an issue with macro expansion or captures
• 920240
• 920250
• 920270
• 920271
• 920272
• 920273
• 920274
• 920360
• 920370
• 920380
• 920390
• 920420
• 920440
• 920450
• 920460
• 920480
• 921130
• 921150
• 921160
• 931130
• 932100
• 932140
• 933110
• 933130
• 933180
• 933210
• 934100
• 941100
• 941110
• 941130
• 941140
• 941150
• 941160
• 941180
• 941190
• 941200
• 941210
• 941220
• 941280
• 941310
• 941330
• 941350
• 941370
• 942100
• 942101
• 942130
• 942150
• 942210
• 942260
• 942420
• 942421
• 942432
• 942480
• 942490
• 943100
• 943110
• 944100
• 944110
• 944120
• 944130
• 944200
• 944210

Rule:

SecRule REQUEST_BASENAME "@detectSQLi" "id:942101,phase:1,block,capture,t:none,t:utf8toUnicode,t:urlDecodeUni,t:removeNulls,msg:'SQL Injection Attack Detected via libinjection',logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}',tag:'application-multi',tag:'language-multi',tag:'platform-multi',tag:'attack-sqli',tag:'OWASP_CRS',tag:'capec/1000/152/248/66',tag:'PCI/6.5.2',tag:'paranoia-level/3',ver:'OWASP_CRS/3.4.0-dev',severity:'CRITICAL',setvar:'tx.sql_injection_score=+%{tx.critical_anomaly_score}',setvar:'tx.anomaly_score_pl3=+%{tx.critical_anomaly_score}'"

Test request:

POST /[email protected]"%20sleep(10.to_i)%20" HTTP/1.0
Host: localhost

Logs results:

{"level":"debug","ts":1637549753.886731,"msg":"Evaluate rule operator","txid":"zqs9Bq9IKMZFBO0bedK","rule":942101,"event":"EVALUATE_RULE_OPERATOR","operator":"@detectSQLi","data":"[email protected]\\\" sleep(10.to_i) \\","variable":"REQUEST_BASENAME","key":"","value":"[email protected]\\\" sleep(10.to_i) \\","result":false}

Result expected:

Match expected

The chain of transformations is transforming "%20sleep(10.to_i)%20" to \\\" sleep(10.to_i) \\ and libinjection is failing to test it, it requires more debug.

See test 942101-6

The issue: If you manage thousands of routes it's not resources effective to have thousands of WAF instances.

The solution: Make "recyclable" WAF instances.

How to achieve it:

Maybe we could use SecWebAppId to define an "instance ID" and register this waf instance in a hashmap ``map[string]*engine.Waf``` under the skipper package.

For example:

#routes.yaml
route1:
     Path("/1) ->
     corazaWAF("/etc/coraza-waf/profiles/default/rules.conf")
route2:
     Path("/2") ->
     corazaWAF("/etc/coraza-waf/profiles/default/rules.conf")

#/etc/coraza-waf/profiles/default/rules.conf
SecWebAppId instance1

In this case route1 and route2 would share the same instance because of their secwebappid.

This idea might work but requires more analysis

Coraza v1 has achieved 90%+ compatibility but a complete redesign on Coraza v2 broke most of the compatibility. In order to get 100% compatibility we must find the issues that are breaking CRS.

• Check which variables are failing to be created
• Find bugs in the Coraza testing package
• Find issues in algorithms like: rule phases, chains, rule flow (secmarks, skip, etc), ctl, tx variables, etc

Note: the coraza testsuite includes plugins for PCRE and libinjection, so please have them installed.

Create a CRS bundle:

1. Join the default coraza config file, the crs config file and the rules from coreruleset/rules/*.conf
2. Prepend the test configurations:

SecAction "id:900005,\
  phase:1,\
  nolog,\
  pass,\
  ctl:ruleEngine=DetectionOnly,\
  ctl:ruleRemoveById=910000,\
  setvar:tx.paranoia_level=4,\
  setvar:tx.crs_validate_utf8_encoding=1,\
  setvar:tx.arg_name_length=100,\
  setvar:tx.arg_length=400,\
  setvar:tx.combined_file_sizes=65535"

3. Copy the .data files from rules/ to your bundle's path

Clone the test suite

git clone https://github.com/jptosso/coraza-testsuite

Update the coraza version

go get github.com/jptosso/coraza-waf/v2@LAST_REVISION

Run the test suite

#Compile or run:
go run *.go run -d ../coreruleset/tests/regression -r ../coreruleset/rules/rules.conf

The current results are posted in the first comment.

Currently detected issues (with status):

• MATCHED_VARS weren't working as expected
• collection.AddUnique was not working, failing to create variables like REQUEST_COOKIES_NAMES, REQUEST_HEADERS_NAMES, RESPONSE_HEADERS_NAMES, ARGS_NAMES, etc
• The testing library doesn't support the no_magic flag which stops forcing the content-length based on the body size
• ForceRequestBodyVariable is working on tests but is not working on CRS for text/plain. Fix: Create a default empty variable for variables.ReqBodyProcessor
• ArgsNames was not being set for POST requests
• XML:/* and XML://@* were expected to return xml content and attribute values, both XPATH expressions were hardcoded
• ARGS_NAMES was being overwritten by request body processors, now it is concated
• #115
• #114
• #113
• #112

The following rules are being ignored because of URL encoding issues

• 920181-1
• 942490-17
• 942260-17
• 942260-6
• 942150-6
• 920240-1
• 920240-5
• 920240-6
• 941130-11
• 941130-2
• 941130-4
• 941130-6
• 941130-9
• 941130-10
• 941130-12
• 941130-14
• 941130-16