@devasmith Can you check what are the values for timeout connect, timeout queue & timeout server configured to in haproxy vs transaction_ttl_seconds in coraza-spoa?

from coraza-spoa.

@devasmith Can you check what are the values for timeout connect, timeout queue & timeout server configured to in haproxy vs transaction_ttl_seconds in coraza-spoa?

Hi, sorry for the late reply.

timeout connect         10s
timeout server          10m

timeout queue has not been configured.

We bumped transation_ttl_seconds from 60000 to 6000000.

It seems as the internal/response.go causes this error. Managed to do a workaround but we disabled the service due to a memory leak.

			txInterface, err := app.cache.Get(id)
			if err != nil {
				app.logger.Error("failed to get transaction from cache", zap.String("transaction_id", id), zap.String("error", err.Error()), zap.String("app", app.name))
				return nil, fmt.Errorf("failed to get transaction from cache")
			}

from coraza-spoa.

@devasmith Can you check what are the values for timeout connect, timeout queue & timeout server configured to in haproxy vs transaction_ttl_seconds in coraza-spoa?

Hi, sorry for the late reply.

timeout connect         10s
timeout server          10m

timeout queue has not been configured.

We bumped transation_ttl_seconds from 60000 to 6000000.

It seems as the internal/response.go causes this error. Managed to do a workaround but we disabled the service due to a memory leak.

			txInterface, err := app.cache.Get(id)
			if err != nil {
				app.logger.Error("failed to get transaction from cache", zap.String("transaction_id", id), zap.String("error", err.Error()), zap.String("app", app.name))
				return nil, fmt.Errorf("failed to get transaction from cache")
			}

If you use transaction_ttl_seconds, your coraza-spoa is old. Please update it and use transaction_ttl_ms. There was an inconsistency in session ID storage length units. Now it's fixed and the value should be 600000 (for 10 minutes). Please be aware that every session you store, takes some memoro, so be sure to have enough memory for your server.

from coraza-spoa.

@NullIsNot0

I copied the value from the documentation that might have been old, we are using transaction_ttl_ms and had set value to 6000000 ms rather than 600000 ms which might have been too much.

We temporarily disabled the service due to the memory leak we encountered and haven't had time to look further into it but we would like to test this out again.

from coraza-spoa.

coraza-spoa sometime show this warning:
INFO[0000] spoe session ending with: write frame: write tcp 127.0.0.1:9000->127.0.0.1:16858: write: broken pipe
WARN[0000] spoe: error handling connection: write frame: write tcp 127.0.0.1:9000->127.0.0.1:16858: write: broken pipe
INFO[0000] spoe session ending with: write frame: write tcp 127.0.0.1:9000->127.0.0.1:16860: write: broken pipe
WARN[0000] spoe: error handling connection: write frame: write tcp 127.0.0.1:9000->127.0.0.1:16860: write: broken pipe
INFO[0000] spoe session ending with: write frame: write tcp 127.0.0.1:9000->127.0.0.1:16864: write: broken pipe
WARN[0000] spoe: error handling connection: write frame: write tcp 127.0.0.1:9000->127.0.0.1:16864: write: broken pipe
INFO[0000] spoe session ending with: write frame: write tcp 127.0.0.1:9000->127.0.0.1:16866: write: broken pipe
WARN[0000] spoe: error handling connection: write frame: write tcp 127.0.0.1:9000->127.0.0.1:16866: write: broken pipe
INFO[0000] spoe session ending with: write frame: write tcp 127.0.0.1:9000->127.0.0.1:16878: write: broken pipe
WARN[0000] spoe: error handling connection: write frame: write tcp 127.0.0.1:9000->127.0.0.1:16878: write: broken pipe
INFO[0000] spoe session ending with: write frame: write tcp 127.0.0.1:9000->127.0.0.1:16880: write: broken pipe
WARN[0000] spoe: error handling connection: write frame: write tcp 127.0.0.1:9000->127.0.0.1:16880: write: broken pipe
INFO[0000] spoe session ending with: write frame: write tcp 127.0.0.1:9000->127.0.0.1:16886: write: broken pipe
WARN[0000] spoe: error handling connection: write frame: write tcp 127.0.0.1:9000->127.0.0.1:16886: write: broken pipe
INFO[0000] spoe session ending with: write frame: write tcp 127.0.0.1:9000->127.0.0.1:16892: write: broken pipe
WARN[0000] spoe: error handling connection: write frame: write tcp 127.0.0.1:9000->127.0.0.1:16892: write: broken pipe
INFO[0000] spoe session ending with: write frame: write tcp 127.0.0.1:9000->127.0.0.1:16896: write: broken pipe
WARN[0000] spoe: error handling connection: write frame: write tcp 127.0.0.1:9000->127.0.0.1:16896: write: broken pipe
INFO[0001] spoe session ending with: write frame: write tcp 127.0.0.1:9000->127.0.0.1:16898: write: broken pipe
WARN[0001] spoe: error handling connection: write frame: write tcp 127.0.0.1:9000->127.0.0.1:16898: write: broken pipe

How can I fix this?
Another proplem, coraza service took too high memory, over 80%. how can I reduce this?
One more question: How can I log audit log with time format, like this: "2023-03-01T15:05:33.579874622Z xxxx". My coraza just log audit log without time format: "{"level":"info","ts":1680854748.1908743,"msg":"....""
Thanks,

from coraza-spoa.

After running coraza-spoa, backend log shows proxy ip instead of public source IP. Can I change this? I prefer showing original public ip in backend log.

from coraza-spoa.

@bananasgroup should be this line in the config, which maps the source ip from haproxy to coraza: https://github.com/corazawaf/coraza-spoa/blob/main/docker/haproxy/coraza.cfg#L14 you might want to use http-request set-src in haproxy as well.

from coraza-spoa.

that does not solve my problem. my coraza.cfg is:

https://github.com/haproxy/haproxy/blob/master/doc/SPOE.txt

[coraza]
spoe-agent coraza-agent
messages coraza-req coraza-res
option var-prefix coraza
option set-on-error error
timeout hello 100ms
timeout idle 2m
timeout processing 500ms
use-backend coraza-spoa
log global

spoe-message coraza-req
args app=str(haproxy) id=unique-id src-ip=src src-port=src_port dst-ip=dst dst-port=dst_port method=method path=path query=query version=req.ver headers=req.hdrs body=req.body
event on-frontend-http-request

spoe-message coraza-res
args app=str(haproxy) id=unique-id version=res.ver status=status
event on-http-response

Backend log still display proxy LAN IP instead of sourc public IP.

PS: Haproxy show parsing [/etc/haproxy/coraza.cfg:18] : 'args': unknown fetch method 'res.hdrs' so I removed headers=res.hdrs and body=res.body.

from coraza-spoa.

Performance issues are probably related to Coraza version, I will create a PR to upgrade it. Other problem you might face, by design, is the transaction ttl cache. A good solution would be to add an option to completely disable response processing, therefor disabling transaction ttl cache. What do you think? CC @jcchavezs

Other option would be to move the transaction to a cache server, like redis. But it might take a lot of work.

from coraza-spoa.

Hello,

When testing this application I too get the error above if I try to reduce the memory used. So, I leave them default which matches my haproxy settings as suggested.

The problem is... the memory slowly grows until the 8GB on the server is exhausted after a couple of hours. It doesn't appear to ever free the memory even though it's supposed to be only 60 seconds of cache.

Do you have suggestions?

from coraza-spoa.

Can you try with this? #60 it seems to address many of the existing problems

from coraza-spoa.

I'm seeing the same issue after changing my HAProxy timeouts. I've changed the HAProxy settings back to what I thought are the defaults but still seeing the same issue.

Could someone clarify exactly which settings in HAProxy should correlate with which settings in the SPOA config?

I'm running in Docker and using an image built in the last week.

from coraza-spoa.

I've run the latest release since a week and I've not seen any more crashes. Maybe we can close this one?

from coraza-spoa.

from coraza-spoa.