Intermittent delegation failure about orca5 HOT 4 CLOSED

Also seen as stuff like this in pequod:
pequod:show>show reservations for current actor psc-broker
Reservations for slice 7cabd0c7-4d62-4046-91d8-ad1ac217c395:
36eaaab2-560c-4577-b4d3-903c67d0ab75 psc-broker
Slice: 7cabd0c7-4d62-4046-91d8-ad1ac217c395
500 pscvmsite.vlan [ ticketed, nascent]
Notices: Reservation 36eaaab2-560c-4577-b4d3-903c67d0ab75 (Slice psc-broker) is in state [Ticketed,None]
Start: Fri Jan 29 19:00:00 EST 2010 End:Wed Jan 29 19:00:00 EST 2031
c12dc53f-cccb-4925-bfbb-63d9a04786e2 psc-broker
Slice: 7cabd0c7-4d62-4046-91d8-ad1ac217c395
52 pscvmsite.vm [ ticketed, nascent]
Notices: Reservation c12dc53f-cccb-4925-bfbb-63d9a04786e2 (Slice psc-broker) is in state [Ticketed,None]
Start: Fri Jan 29 19:00:00 EST 2010 End:Wed Jan 29 19:00:00 EST 2031
dc793afd-3df7-42fe-ba1f-124240fed8fd psc-broker
Slice: 7cabd0c7-4d62-4046-91d8-ad1ac217c395
0 6e69b6ad-5cf1-43b7-b007-71fbd7812a08 [ failed, nascent]
Notices: Reservation dc793afd-3df7-42fe-ba1f-124240fed8fd (Slice psc-broker) is in state [Failed,None], err=Failing reservation due to non-recoverable RPC error (LocalError?), message=orca.shirako.proxies.soapaxis2.SoapAxis?2StubException: An error occurred while obtaining service stub, stack=Exception stack trace:
orca.shirako.proxies.soapaxis2.SoapAxis?2BrokerProxy.execute(SoapAxis?2BrokerProxy.java:91)
orca.shirako.proxies.soapaxis2.SoapAxis?2AuthorityProxy.execute(SoapAxis?2AuthorityProxy.java:109)
orca.shirako.kernel.RPCManager$RPCExecutor.run(RPCManager.java:808)
java.util.concurrent.ThreadPoolExecutor?.runWorker(Unknown Source)
java.util.concurrent.ThreadPoolExecutor?$Worker.run(Unknown Source)
java.lang.Thread.run(Unknown Source)

from orca5.

From Victor (for UMass rack, after getting multiple consecutive failures in claiming storage):
OK - between:

1. Clearing foreign keys out of the keystores
2. Starting AM/Broker container while SM container not running
   I was able to get everything online.
   Neither of those may have anything to do with anything
   but - that’s what I did, before it

from orca5.

@vjorlikowski have we seen this lately?

from orca5.

On Jan 15, 2016, at 3:02 PM, Ilya Baldin [email protected] wrote:

@vjorlikowski https://github.com/vjorlikowski have we seen this lately?

Not in a long while.

Perhaps related:
What I did see last evening was a failure to claim both BEN and PSC
We’ve seen this with BEN intermittently before; if you waited too long after starting the BEN AM without claiming at ndl-broker, you would have to restart BEN AM - and then, you’d be able to claim the delegated resources.

Above described happened last night, with both BEN and PSC AMs not being claimable at ndl-broker without a restart.
No exceptions logged on either AM.

from orca5.