Virtual Server listener address is not removed from Virtual Address list upon listener deletion about f5-openstack-agent HOT 8 CLOSED

How reproducible is this? Is there a log that I could look at? This has
been working all week.

Rich

On Fri, Apr 15, 2016 at 6:52 PM, Matt Greene [email protected]
wrote:

Assigned #80 #80
to @richbrowne https://github.com/richbrowne.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#80 (comment)

from f5-openstack-agent.

Not sure. I'll try it a few times- was simply following the RTD commands. But I do see some ERROR messages in the (attached) log file. Something to do with arp, so perhaps the listener is failing to delete due to some lingering arp entries pointing at the ifc.

f5-openstack-agent.log.txt

from f5-openstack-agent.

What I am seeing is an error in the BIG-IP Rest server. Maybe this is a
timing issue? If this is a transient issue, it should get cleared up by
the agent_manager sync_state thread. How did you clear the error condition?

I can try to reproduce it in another hour.

Rich

icontrol/session.py:238
2016-04-15 18:37:51.629 40058 DEBUG root
[req-731d3608-ad2b-4407-abeb-0f388beb38f6 b4f8478be7bf4437876f4df19ed4daa3
71d559383c5c46ccb4cd6ca6e95e976f - - -] RESPONSE::STATUS: 500 Content-Type:
application/json Content-Encoding: None
Text: u'{"code":500,"message":"arp entry not found
"/Test_71d559383c5c46ccb4cd6ca6e95e976f/10.2.2.3%2"","errorStack":[]}'
wrapper /usr/lib/python2.7/site-packages/icontrol/session.py:246
2016-04-15 18:37:51.629 40058 ERROR
f5_openstack_agent.lbaasv2.drivers.bigip.network_helper
[req-731d3608-ad2b-4407-abeb-0f388beb38f6 b4f8478be7bf4437876f4df19ed4daa3
71d559383c5c46ccb4cd6ca6e95e976f - - -] Error deleting arp 10.2.2.3%252.
Repsponse status code: 500. Response message: 500 Unexpected Error:
Internal Server Error for uri:
https://10.190.7.125/mgmt/tm/net/arp/~Test_71d559383c5c46ccb4cd6ca6e95e976f~10.2.2.3%252
Text: u'{"code":500,"message":"arp entry not found
"/Test_71d559383c5c46ccb4cd6ca6e95e976f/10.2.2.3%2"","errorStack":[]}'.
2016-04-15 18:37:51.630 40058 DEBUG
f5_openstack_agent.lbaasv2.drivers.bigip.network_helper
[req-731d3608-ad2b-4407-abeb-0f388beb38f6 b4f8478be7bf4437876f4df19ed4daa3
71d559383c5c46ccb4cd6ca6e95e976f - - -]
f5_openstack_agent.lbaasv2.drivers.bigip.network_helper.NetworkHelper
method get_fdb_entry called with arguments (<f5.bigip.BigIP object at
0x4290b50>, 'tunnel-vxlan-45') {'mac': None, 'partition':
u'Test_71d559383c5c46ccb4cd6ca6e95e976f'} wrapper
/usr/lib/python2.7/site-packages/oslo_log/helpers.py:45

On Sat, Apr 16, 2016 at 9:52 AM, Matt Greene [email protected]
wrote:

Not sure. I'll try it a few times- was simply following the RTD commands.
But I do see some ERROR messages in the (attached) log file. Something to
do with arp, so perhaps the listener is failing to delete due to some
lingering arp entries pointing at the ifc.

f5-openstack-agent.log.txt
https://github.com/F5Networks/f5-openstack-agent/files/222283/f5-openstack-agent.log.txt

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#80 (comment)

from f5-openstack-agent.

We didn't. Noticed it on manual proxy teardown but that was the end of the test. The BIGIP is still in the 'error' state. https://10.190.7.125/ browse to 'Local Traffic'/'Virtual Servers'/'Virtual Address List' and you'll see the listener ifc still hanging around.

from f5-openstack-agent.

Ok, I found one problem, not related to this. And I have a fix and a ticket opened. I also fixed the problem with the NoneType not having an attribute __getitem

from f5-openstack-agent.

Great work guys. I was just getting ready to tell @mattgreene that we should try to get this fixed because it will cause issues. But, as usual, you guys are way ahead of me.

from f5-openstack-agent.

@mattgreene @swormke The error logged is not related to virtual address. Matt's issue of proxy teardown is something else (which I can't re-create).

This error is logged when we delete a member and attempt to remove ARP entries. The error is appropriately caught and we continue with processing, so I do not believe it's the cause of any issue, other than the error in the log.

The error is caused by checking if an ARP entry exists, and BIG-IP doesn't like how we format the request (Apparently it doesn't like the URL encoding. Of course it doesn't like the request when we don't URL encode it either.) Here's the offending code:

    def arp_delete(self,
                   bigip,
                   ip_address,
                   partition=const.DEFAULT_PARTITION):
        if ip_address:
            address = urllib.quote(self._remove_route_domain_zero(ip_address))
            arp = bigip.net.arps.arp
            try:
                if arp.exists(name=address, partition=partition):

                    arp.load(name=address, partition=partition)
                    arp.delete()
            except HTTPError as err:
                LOG.error("Error deleting arp %s. "
                          "Repsponse status code: %s. Response "
                          "message: %s." % (address,
                                            err.response.status_code,
                                            err.message))

I have a fix in mind: get collection of ARP entries and iterate through to find a match. I've reassigned this to myself.

from f5-openstack-agent.

This error is no longer valid:

1. We have fixed many of the errors that were causing an interruption in the flow of cleanup.
2. I have added logic in the tenant cleanup to remove all folder contents if there is an error.

from f5-openstack-agent.