dsc stopped writing reports with 2.3.0 about dsc HOT 18 CLOSED

Thanks for the report, there seem to be issues with the threading, signaling and forking. Working on it but hard to say when it can be ready for testing.

What distribution are you running?

You can try running with the -T flag that will disable the usage of threads.

from dsc.

Thanks for the quick reply :-).
We are running Debian Jessie x64 on both servers.

from dsc.

Can you give me the spec of the servers and the QPS your receiving?

from dsc.

Servers receive around 2 to 3 QPS.
They are LXC containers with 4 cpu @ 2.80GHz and 4GB RAM each.

from dsc.

Can you describe how it stopped? What did you see on the ps output? Did you do a strace?

from dsc.

ps showed that the process was still running even though reports stopped being written. We did not run strace :-(.

from dsc.

What was the last file in the data directory? An ...xml.XXX?

from dsc.

No, the last written file had the proper .xml / .json extension.

from dsc.

If you have the possibility could you setup another container and run the develop branch and generate some QPS to it?

It seems that the issue is related to threads and that the process forks when writing the files, this makes libraries very rarely lock up. I was lucky enough to catch this while stracing and I saw a lockup in the NSS library used to get the IP protocol name.

The latest commits to develop has been to ensure signal handlers are correct after fork and to use thread safe libc functions.

I am also setting up dsc on all our build VMs to run the latest develop branch build continuously, generating 10 QPS against them and will monitor file creation.

from dsc.

If you have not already please read my announcement regarding this:
https://lists.dns-oarc.net/pipermail/dsc/2017-January/000361.html

I have setup some test now for dsc, if you have any other ideas of tests please let me know:
https://dev.dns-oarc.net/jenkins/view/dsctest/

from dsc.

Are you able to test the latest develop branch?

from dsc.

Hi @jelu. We haven't tested the develop branch yet. But we might be able to do so next week.

from dsc.

We now have it running on a server.

from dsc.

from dsc.

fa191b0

from dsc.

Do you have any issue to report with the develop branch?

from dsc.

We had no issue with the develop branch so far.

from dsc.

Thanks @jelu! Good job. :-)

from dsc.