Server 2019. Kubernetes 1.14.3. After applying Windows updates and rebooting Windows nodes, the SMB volumes will not remount.

kubectl describe pod-
MountVolume.SetUp failed for volume "xyz" : mount command failed, status: Failure, reason: Caught exception Multiple connections to a server or shared resource by the same user, using more than one user name, are not allowed. Disconnect all previous connections to the server or shared resource and try again. with stack

kubelet log-
E0617 16:59:58.368344 4408 driver-call.go:274] mount command failed, status: Failure, reason: Caught exception Multiple connections to a server or shared resource by the same user, using more than one user name, are not allowed. Disconnect all previous connections to the server or shared resource and try again. with stack
E0617 16:59:58.398345 4408 nestedpendingoperations.go:267] Operation for ""flexvolume-microsoft.com/smb.cmd/784c0a82-9142-11e9-8e91-0050569e2770-data" ("784c0a82-9142-11e9-8e91-0050569e2770")" failed. No retries permitted until 2019-06-17 16:59:58.8983454 -0400 EDT m=+183.786315501 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume "data" (UniqueName: "flexvolume-microsoft.com/smb.cmd/784c0a82-9142-11e9-8e91-0050569e2770-data") pod "xyz-1560805020-bbvzg" (UID: "784c0a82-9142-11e9-8e91-0050569e2770") : mount command failed, status: Failure, reason: Caught exception Multiple connections to a server or shared resource by the same user, using more than one user name, are not allowed. Disconnect all previous connections to the server or shared resource and try again. with stack "

SMB server is an EMC SAN array configured as SMB NAS server. Joined to AD domain. All was working well prior to host reboot. No errors seen on the EMC end.

Troubleshooting steps taken-
Revert Windows updates
Revert Kubernetes version -> 1.14.2 -> 1.14.1 -> back to 1.14.3
Delete all files/folders under \var\lib\kubelet\pods\pods
Change user
Delete / recreate deployment
Tested mounting share as drive with SMB secret creds. This worked when mounting from Windows Explorer, but same share with same creds don't work when using plugin

Tested using latest master and release versions

I've noticed that when I delete a workload that had an iSCSI Flexvolume mounted, it leaves the target and LUN connected on the host. Should it be automatically disconnecting that target after a workload has been removed?

Greetings,

I got an issue when using FelxVolume with Windows Server 2019 built-in iSCSI target and file sharing. (Any requirement for iSCSI target? Help needed.)

What happened:
Running dir c:\d failed with DirIOError

What you expected to happen:
Running dir c:\d should return without error.

How to reproduce it (as minimally and precisely as possible):
kubectl apply [iscsi|smb].yaml (using mcr.microsoft.com/powershell:6.2.0-nanoserver-1809 instead)

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): v1.14.1
• Cloud provider or hardware configuration: VirtualBox
• OS (e.g: cat /etc/os-release): CentOS 7.4.1708 + Windows Server 2019 Standard
• Kernel (e.g. uname -a): Linux: 3.10.0-693.el7.x86_64; Windows: 10.0.17763.475
• Install tools: kubeadm, microsoft/sdn flannel related scripts
• Network plugin and version (if this is a network-related bug): Flannel
• Others:

Screenshots

SMB volumes will not mount to Pod on Windows. Flexvolume driver doesn't load correctly and this leaves the Pods stuck in ContainerCreating

deployment config-
volumes:
- flexVolume:
driver: microsoft.com/smb.cmd
options:
source: //smb_server/share/folder
secretRef:
name: smb-secret
name: data
volumeMounts:
-mountPath: c:/data
name: data

kubectl describe pod output-
Warning FailedMount 5s (x4 over 16s) kubelet, bcvmw19-kb-w3 MountVolume.SetUp failed for volume "data" : mount command failed, status: Failure, reason: Caught exception Invalid switch - "/". with stack at MakeSymLink, C:\usr\libexec\kubernetes\kubelet-plugins\volume\exec\microsoft.comsmb.cmd\flexvolume.ps1: line 128
at mount_command, C:\usr\libexec\kubernetes\kubelet-plugins\volume\exec\microsoft.comsmb.cmd\smb.ps1: line 33
at RunFlexVolume, C:\usr\libexec\kubernetes\kubelet-plugins\volume\exec\microsoft.comsmb.cmd\flexvolume.ps1: line 273
at , C:\usr\libexec\kubernetes\kubelet-plugins\volume\exec\microsoft.comsmb.cmd\smb.ps1: line 52
at , : line 1

kubelet error log-
E0607 21:04:22.274366 8616 driver-call.go:274] mount command failed, status: Failure, reason: Caught exception Invalid switch - "/". with stack at MakeSymLink, C:\usr\libexec\kubernetes\kubelet-plugins\volume\exec\microsoft.comsmb.cmd\flexvolume.ps1: line 128
at mount_command, C:\usr\libexec\kubernetes\kubelet-plugins\volume\exec\microsoft.comsmb.cmd\smb.ps1: line 33
at RunFlexVolume, C:\usr\libexec\kubernetes\kubelet-plugins\volume\exec\microsoft.comsmb.cmd\flexvolume.ps1: line 273
at , C:\usr\libexec\kubernetes\kubelet-plugins\volume\exec\microsoft.comsmb.cmd\smb.ps1: line 52
at , : line 1
E0607 21:04:22.275363 8616 nestedpendingoperations.go:267] Operation for ""flexvolume-microsoft.com/smb.cmd/586edf71-8989-11e9-b665-0050569e2770-data" ("586edf71-8989-11e9-b665-0050569e2770")" failed. No retries permitted until 2019-06-07 21:04:22.7753634 -0400 EDT m=+8900.921073401 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume "data" (UniqueName: "flexvolume-microsoft.com/smb.cmd/586edf71-8989-11e9-b665-0050569e2770-data") pod "inhouse-staging-8654dd7f7d-z7vh7" (UID: "586edf71-8989-11e9-b665-0050569e2770") : mount command failed, status: Failure, reason: Caught exception Invalid switch - "/". with stack at MakeSymLink, C:\usr\libexec\kubernetes\kubelet-plugins\volume\exec\microsoft.comsmb.cmd\flexvolume.ps1: line 128\r\nat mount_command, C:\usr\libexec\kubernetes\kubelet-plugins\volume\exec\microsoft.comsmb.cmd\smb.ps1: line 33\r\nat RunFlexVolume, C:\usr\libexec\kubernetes\kubelet-plugins\volume\exec\microsoft.comsmb.cmd\flexvolume.ps1: line 273\r\nat , C:\usr\libexec\kubernetes\kubelet-plugins\volume\exec\microsoft.comsmb.cmd\smb.ps1: line 52\r\nat , : line 1"

Powershell execution policy set as follows-
set-executionpolicy -scope localmachine bypass

This was needed as the driver didn't load at all before that.

Am I doing something wrong?

This is not an issue report, it's a tip for some support engineers who come from the future.

Background story
I want to confirm the potential impact when some group policies are applied for domain-joined windows nodes in a Kubernetes cluster. I choose Quality of Service (QoS) Policy for this test. The motivation is very simple, I want to control the network speed when transferring large files from containers to a SMB server.

Business Scenario
A web application that allows user to upload large attachments (less than 100MB).

Test Result
It works as expected! The QoS policy is taking effect on Windows nodes in a Kubernetes cluster. WOW! That's the power and amazing part of Windows Server!
(util01.corp.contoso.com is my test box and file server. I am building this lab on a PC with 32GB memory.)

Potential issue symptoms in support tickets

1. As an end user, it takes a long time for me to upload attachment. Yet another bad system!
2. As a developer, I don't know what's happening on network. Windows container is blamed because it's a new technology.
3. As a network engineer, I only know the concept of overlay network. I swear that I have nothing to do with this slow network issue. I don't apply any QoS policy on any network device.
4. As a newbie of Windows Server administration, I am facing a bunch of documents. Kubernetes, overlay network, flannel, hybrid cluster, what's next?

PS
KBps in group policy and Kbps in resource monitor is not an issue anymore.

Some nights you're programming and you say "may God have mercy on my soul for this line of code." -Scott Hanselman

I have a node with 2 interfaces, one for "Server" traffic, and one for "Storage" traffic, in this case iscsi traffic. How do I force the iSCSI requests to use that interface? As it stands today, this config:

apiVersion: v1
kind: Pod
metadata:
  name: iscsi
  namespace: default
spec:
  restartPolicy: Never
  containers:
  - name: iscsi
    image: microsoft/nanoserver
    command: ["C:\\Windows\\System32\\cmd.exe"]
    args: ["/C", "echo", "0", ">", "c:\\d\\iscsi_test.txt"]
    volumeMounts:
    - name: iscsi-volume
      mountPath: /d
  nodeName: k8s-w19-c1b4
  volumes:
  - name: iscsi-volume
    flexVolume:
      driver: "microsoft.com/iscsi.cmd"
      fsType: "NTFS"
      secretRef:
        name: "iscsi-secret"
      readOnly: false
      options:
        chapAuthDiscovery:  "false"
        chapAuthSession:  "true"
        targetPortal:  "172.16.15.10"
        iqn:  "iqn.2010-01.com.solidfire:1q2x.k8s-windows-testvol.1109"
        lun:  "0"
        authType:  "ONEWAYCHAP"

Fails with the following error:

FailedMount 	MountVolume.SetUp failed for volume "iscsi-volume" : mount command failed,

status: Failure, reason: Caught exception did not find target iqn.2010-01.com.solidfire:1q2x.k8s-windows-testvol.1109 with stack at GetFirst<End>, C:\var\lib\kubelet\volumeplugins\microsoft.com~iscsi.cmd\flexvolume.ps1: line 42 

at GetTargetForPortals, C:\var\lib\kubelet\volumeplugins\microsoft.com~iscsi.cmd\iscsi.ps1: line 126 

at ConnectIscsi, C:\var\lib\kubelet\volumeplugins\microsoft.com~iscsi.cmd\iscsi.ps1: line 179 

at mount_command_with_options, C:\var\lib\kubelet\volumeplugins\microsoft.com~iscsi.cmd\iscsi.ps1: line 466 

at mount_command, C:\var\lib\kubelet\volumeplugins\microsoft.com~iscsi.cmd\iscsi.ps1: line 427 

at RunFlexVolume, C:\var\lib\kubelet\volumeplugins\microsoft.com~iscsi.cmd\flexvolume.ps1: line 273 

at <ScriptBlock>, C:\var\lib\kubelet\volumeplugins\microsoft.com~iscsi.cmd\iscsi.ps1: line 648 

at <ScriptBlock>, <No file>: line 1

My gut says that it's related to the host not sending that request down the second interface. I could be wrong

I base64 encoded my node.session username and password, and tried to add the iscsi-secret.yaml file to my cluster, but I got this error:

The Secret "iscsi-secret" is invalid: type: Invalid value: "microsoft.com/iscsi.cmd": field is immutable

Is there some missing prereq I need to set in my cluster?

Try to mount share into windows pod. GKE cluster with windows node v1.16.9-gke.6 (windows server DC 2019ltsc 10.0.17763.1098)
kubelet.exe ... --volume-plugin-dir=C:\etc\kubernetes\flexvolume .....
I've unpacked archive with plugin, apply smb-secret.yml and smb.yml, but still getting mistake:
Warning FailedMount kubelet, gke-win-test-v2-windows-6be2eef6-jckk Unable to attach or mount volumes: unmounted volumes=[smb-volume], unattached volumes=[default-token-gpblr smb-volume]: failed to get Plugin from volumeSpec for volume "smb-volume" err=no volume plugin matched

What's wrong?

windows: 2019
k8s: v1.17.2

when i use
source: \sfs-nas1.cn-south-1c.xxx.com\share-2d58d4ca
is does not work, but when i use
source: \10.12.13.5\share-2d58d4ca
it works

thanks for your help!

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

When formatting an iSCSI volume. The root of the drive should get re-ACL'd such that it allows all users to create data. Normally the inherit ACE has this, it is just not there for the root ACE.

First seen in issue #9

See https://stackoverflow.com/questions/6622124/why-does-set-acl-on-the-drive-root-try-to-set-ownership-of-the-object for a discussion on setting drive acl's from powershell.

Notes about fix.

1. Should go before removal of format SCSI PR.
2. Should check if volume is empty incase volume came preformatted with user specified data.
3. Should ensure if volume exists to retry if no format was done, and no exclusive / readonly PR is on the disk

Hi, when we try to map to a directory with spaces in the name, e.g. \\servername.domain.com\folder1\some folder we got an error The specified network resource or device is no longer available.

The reason for that is that suddenly there are quotes around the directory name and the Trim functions in /flexvolume/windows/plugins/microsoft.com~smb.cmd/flexvolume.ps1 do not work correctly. The @JSON variable contains a beginning quote, but not ending one.

We fixed it with two edits. One in line 241, adding another Trim at the end:

$escapedJson = $all.Substring($command.Length).Trim().SubString($folder.Length).Trim('" ')

and secondly with adding quotes around the remotepath in line 128:

cmd /c "mklink /D $symlink `"$remotepath`"" 2>&1  | Out-Null

Hello,

I try to use the SMB plugin but I don't have authentication on the CIFS server.

I tried to modify the smd.ps1 script but I don't find the option of New-SmbGlobalMappingto not pass the credentials.

Does anyone have idea to have a workaround ?

I tried to use New-SmbMapping but I have to precise the letter...

Thank for your help

Hello,

I'm using Kubernetes v1.15.1 with windows server 2019 workers and was using v0.0.3 release of storage plugins on windows workers.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name:  smb-deployment
  labels:
    name:  smb-app
  namespace: windows
spec:
  replicas: 1
  selector:
    matchLabels:
      app: smb-app
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app:  smb-app
    spec:
      containers:
      - name: smb
        image: microsoft/nanoserver
        command: ["C:\\Windows\\System32\\cmd.exe"]
        args: ["/C", "echo", "0", ">", "c:\\d\\smb_test.txt"]
        volumeMounts:
        - name: smb-volume
          mountPath: c:/data
        resources:
          requests:
            cpu: 200m
            memory: 500Mi
          limits:
            cpu: 200m
            memory: 500Mi
      volumes:
      - name: smb-volume
        flexVolume:
          driver: "microsoft.com/smb.cmd"
          secretRef:
            name: "smb-secret"
          options:
            # source can be in any of the following formats 
            # \\servername\share\path  (\'s will need to be escaped)
            # smb://servername/share/path
            # //servername/share/path
            source: \\10.0.0.5\sharedfolder
      nodeSelector:
        beta.kubernetes.io/os: windows
        kubernetes.io/hostname: ip-0a230df7

Here is the error I'm getting in kubelet with debug enabled

""flexvolume-microsoft.com/smb.cmd/58b90ea4-e8e0-4549-a71c-6855f1f8b51a-smb-volume" ("58b90ea4-e8e0-4549-a71c-6855f1f8b51a")" failed. No retries permitted until 2019-09-19 04:08:01.8282223 +0000 GMT m=+997.903271801 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume "smb-volume" (UniqueName: "flexvolume-microsoft.com/smb.cmd/58b90ea4-e8e0-4549-a71c-6855f1f8b51a-smb-volume") pod "smb-deployment-8585cccf96-g77pv" (UID: "58b90ea4-e8e0-4549-a71c-6855f1f8b51a") : mount command failed, status: Failure, reason: Caught exception The parameter is incorrect. with stack "

Please let me know if I'm doing something wrong, thanks.