Azure CPI hangs talking to Azure for > 2 days about bosh-azure-cpi-release HOT 15 CLOSED

@sedouard I filed an issue #334 for azure-sdk-for-ruby.

from bosh-azure-cpi-release.

@jim80net Thanks for your reporting. I have several questions for you.

1. Which version are you using?
2. Could you send your manifest and the task log to me for advanced investigation? My email is [email protected].

from bosh-azure-cpi-release.

FYI -

Not sure if this is related but It's known that the azure ruby sdk doesn't have client side timeouts. We hit this issue when working with the storage APIs from less-than-favorable network connection (eg: uploading to an Azure container from an AWS data center).

The resulting symptoms will be what you see -- a deadlock of threads waiting on some some asynchronous web request.

from bosh-azure-cpi-release.

@AbelHu, email sent.

@sedouard Thanks for the +1, glad to see we're not the only ones impacted.

We'd like to see the client side timeout implemented.

from bosh-azure-cpi-release.

Additional info:

When we encountered this problem, we were using CLI Tool version 0.9.15 from our jumpbox on Azure.

from bosh-azure-cpi-release.

@landesherr Thanks for your information. How did you get the CPI version 0.9.15?

from bosh-azure-cpi-release.

@AbelHu
npm install -g azure-cli

from bosh-azure-cpi-release.

@landesherr I think you hit a different issue. Iandesherr hit the issue when using BOSH. But you hit the issue when using azure-cli. Could you describe your steps when you hit your issue?

from bosh-azure-cpi-release.

@AbelHu Sorry for the confusion - I was working with @jim80net when we ran into this problem. If I remember correctly, the BOSH job was triggered from the jumpbox.

from bosh-azure-cpi-release.

@landesherr the BOSH job is triggered from the jumpbox. But BOSH always calls Azure CPI to talk with Azure. azure-cli is only used when you prepare the service principal if you follow our guide. So I need to know how you hit the issue with azure-cli.

from bosh-azure-cpi-release.

@AbelHu For some reason, I thought you were asking for the CLI version we're using (which is totally unrelated to this problem), not CPI version. Serves me right for not paying enough attention. As @jim80net posted above, we were using what appears to be v2.0.0 of the CPI.

from bosh-azure-cpi-release.

@landesherr Please show me the CPI release link in your bosh.yml. It is similar to the link here.

from bosh-azure-cpi-release.

  - name: bosh-azure-cpi
    version: 6
    git: https://github.com/cloudfoundry-incubator/bosh-azure-cpi-release.git

from bosh-azure-cpi-release.

I'm seeing this too with CPI v7. I'll send a log.

from bosh-azure-cpi-release.

@jim80net @svrc-pivotal Before Ruby v2.3.0, Net::Http.read_timeout is set to 60 seconds by default but Net::Http.open_timeout is set to nil by default. I also find similar gdb backtrace in chef issue #2944 and puppet issue #1570.
I think it is fixed in the PR #109.

from bosh-azure-cpi-release.