[BUG] Volume with v1.5.x engine not worked well in v1.6.0-rc1 about longhorn HOT 3 CLOSED

2024-01-12T03:42:27.585902863Z time="2024-01-12T03:42:27Z" level=warning msg="pvc-cdcb2543-4f88-4f94-a53a-57192ad9e38d-r-3e260f9c: time=\"2024-01-12T03:37:57Z\" level=fatal msg=\"Error running start replica command\" func=cmd.ReplicaCmd.func1 file=\"replica.go:79\" error=\"directory name is required\"" func="controller.(*InstanceHandler).printInstanceLogs" file="instance_handler.go:467"

v1.5.3's engine CLIAPI version is 9, so the version check in v1.6.0 should be if engineCLIAPIVersion >= 10 {...}
https://github.com/longhorn/longhorn-manager/blame/master/engineapi/instance_manager.go#L356

cc @FrankYang0529

from longhorn.

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?

1. Deploy longhorn v1.5.x
2. Create 2 volume with workloads, wite data into mount point
3. Upgrade to master-head, don't perform engine upgrade
4. Delete 1 replica in workload1, the replica try rebuilding, but not success
5. Scale workload2, make volume detach/attach, the volume kept in loop attaching -> faulted

• Is there a workaround for the issue? If so, where is it documented?

• Does the PR include the explanation for the fix or the feature? Yes

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at longhorn/longhorn-manager#2454

• Which areas/issues this PR might have potential impacts on?

• If labeled: require/LEP Has the Longhorn Enhancement Proposal PR submitted?

• If labeled: area/ui Has the UI issue filed or ready to be merged (including backport-needed/*)?

• If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)

• If labeled: require/manual-test-plan Has the manual test plan been documented?

• If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?

from longhorn.

Verified pass on longhorn master (longhorn a78c2a)

Upgrade form v1.5.x to master-head, volume with v1.5.x-engine image worked well in below actions:

• Replica rebuilding
• Reattach volume by scale down/up workload
• Data read/write
• Create backup
• Concurrent upgrade engine image to master-head

from longhorn.