[IMPROVEMENT] Improve environment_check script for NFS protocol bug and the host system self diagnosis about longhorn HOT 16 OPEN

I installed a .100 release tonight for Ubuntu. Seems to have resolved the issue on my end.

from longhorn.

Besides of the environment_check script, I'm considering adding a condition in node.status.conditions to check the environment, like the kernel version.
WDYT? @innobead @james-munson

from longhorn.

Is there currently a workaround?

I think it might just be either downgrading to the .92 kernel release or wait for the .100 release

@derekbit Is that accurate?

Yes, totally right!

from longhorn.

Here we same, .100 kernel solved it. Also the actual /new her-kernel (6.5) solves it.

from longhorn.

Sounds good to me @derekbit . Let's update the requirements of this request.

Updated. Thank you.

from longhorn.

@james-munson remember to update the zenhub status.

from longhorn.

Looking at the probable culprit change and its fix it appears that this is an NFS v4.1 issue, but that is our default.

from longhorn.

Sounds good to me @derekbit . Let's update the requirements of this request.

from longhorn.

Is there currently a workaround?

from longhorn.

Is there currently a workaround?

I think it might just be either downgrading to the .92 kernel release or wait for the .100 release

@derekbit Is that accurate?

from longhorn.

Thank you, downgrading indeed solved the problem. 👍

from longhorn.

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are
  Find a cluster or install a problematic kernel, such as 5.15.94 or 6.5.6. Create an RWX volume and mount it. A failing mount should log the host kernel release and OS distro in the CSI plugin logging.

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at: In the defect. Essentially, install a different kernel. (This does not fix it, but it makes it easier to detect on the fly.)

• Does the PR include the explanation for the fix or the feature?

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
  The PR for the YAML change is at:
  The PR for the chart change is at:

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at

• Which areas/issues this PR might have potential impacts on?
  Area
  Issues

• If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?
  The documentation issue/PR is at longhorn/website#873

from longhorn.

May need to update the fixed and broken versions. I see in #6857 (comment) that 6.5.8 is broken, but the fix came in or before 6.5.11.

from longhorn.

Note that this issue happens to me on 5.15.0-1047-raspi

from longhorn.

And also (reported in Slack) Ubuntu kernel 6.5.0-21-generic #21~22.04.1-Ubuntu

from longhorn.

And also (reported in Slack) Ubuntu kernel 6.5.0-21-generic #21~22.04.1-Ubuntu

I am tracking this same bug with NFS4 on vsphere-csi but can confirm that it exists in the current HWE kernels for 20.04, 22.04 and the current release kernels of 23.10 along with haphazardly most of the current versions of the variant kernels that I tested. Finding an affected kernel in any current Ubuntu release is like shooting fish in a barrel. The list extends to pretty much every variant they currently ship -- publishing a list of affected package versions without actually testing all the packaged kernels is going to lead to a lot more reports over the coming weeks as an Ubuntu system is more or less going to unavoidably upgrade to a broken kernel.

I do not see any fixes on launchpad for the HWE kernel 6.5.0 similar to those already in the pipeline for 5.15.0-100 so it's tough to understand if this is actually getting the proper attention.

from longhorn.