FreeBSD CI-builds are failing about dotnet-ci HOT 20 CLOSED

Yeah I was looking at this yesterday. I went to go create a new node, so I captured the old one and created two new versions. Either it wasn't really working that well before and we just got lucky, or something happened in the deprovisioning and capture, because the temporary drive stopped working within a few minutes.and started giving IO errors.

from dotnet-ci.

I've updated to the latest version of waagent, which has correct provisioning for the temporary disk. Should be up and running quite soon.

from dotnet-ci.

Looking at the latest build they still don't seem entirely stable:

• http://dotnet-ci.cloudapp.net/job/dotnet_coreclr_freebsd_release/
• http://dotnet-ci.cloudapp.net/job/dotnet_coreclr_freebsd_debug/

Odd that they would chug along nicely to begin with and then suddenly stop working...

from dotnet-ci.

@josteink I think we might actually be good here. The last few failures on the 30th were real failures, but @ajensenwaud fixed this last night. Woohoo!

from dotnet-ci.

Ooh right. Nice of him then :)

from dotnet-ci.

Actually looks like you're right about the debug one. It appears that for whatever reason, it didn't update it's workspace directory when I updated the value in Jenkins. It was still using the primary drive, which has very little space. I'll figure out what's going on.

from dotnet-ci.

Done any changes since last time? They're all looking good now. Feel free to close this issue at your own discretion.

from dotnet-ci.

We are all good now, and the build will fail on FreeBSD failures now!

from dotnet-ci.

@mmitche, @josteink, now that dotnet/coreclr#1030 is merged, should the tests be enabled for FreeBSD as well?

Related: https://github.com/dotnet/coreclr/issues/1010.

from dotnet-ci.

@jasonwilliams200OK Yep will do so today

from dotnet-ci.

Currently 4 of the tests are failing. Will enabling the tests on the CI-build cause all PRs from now to turn red until we get those fixed?

If so I would advice to delay enabling the tests until we have the 4 failing tests fixed.

(My 2 norwegian øre)

from dotnet-ci.

@josteink Yes. we should have them all passing before we enable them in the CI

from dotnet-ci.

👍
@joshfree, US 💰 for you! 😉

from dotnet-ci.

@mmitche I looked slightly deeper into this, and it may be that enabling tests are safeish.

Can you take a look at the findings and weight in your opinion?

https://github.com/dotnet/coreclr/issues/1010#issuecomment-104756763

from dotnet-ci.

@josteink I got two failures on an exploratory run:

http://dotnet-ci.cloudapp.net/job/dotnet_coreclr_freebsd_debug/lastFailedBuild/console

The following test(s) failed:
threading/WaitForSingleObject/WFSOExMutexTest/paltest_waitforsingleobject_wfsoexmutextest. Exit code: 1
threading/WaitForSingleObject/WFSOExThreadTest/paltest_waitforsingleobject_wfsoexthreadtest. Exit code: 1

PAL Test Results:
Passed: 819
Failed: 2

from dotnet-ci.

Ok. If so I still advice against enabling them. Was worth a shot though. :)

from dotnet-ci.

How about we exclude these two tests for FreeBSD target something what these guys have done: dotnet/llilc#598? I mean only 0.2% of tests are failing. I think it is a bad idea to hold back the majority.

from dotnet-ci.

Currently we're investigating the cause for those performing badly in #1044, but if those investigations turn up inconclusive or without any resolution, I'm inclined to agree with @jasonwilliams200OK .

from dotnet-ci.

@jasonwilliams200OK Yeah that seems reasonable. If it seems like the values in the tests are unreasonable (they were probably chosen somewhat arbitrarily in the ancient past) then we could also alter them.

from dotnet-ci.

@mmitche, 👍

A very valuable and yet interesting discussion on this subject is going on here https://github.com/dotnet/coreclr/issues/1044 between @josteink and @bsdjhb (FreeBSD kernel dev). It seems like there is more to just the timer values which need adjustment. Hopefully, the root issue will be identified soon.

from dotnet-ci.