Azure Gateway and OpenVPN versions past 2.5.8 incompatible (being investigated by Azure Gateway production team) about openvpn HOT 58 CLOSED

The platform labels are more if the issue is platform specific, which this issue is not.

from openvpn.

Anyone heard anything new on this? Azure Support decided to close my ticket after adding me to list of impacted tenants, though I can technically open at any time. We've got about 32 endpoints sitting on 2.5.8 still.

from openvpn.

from openvpn.

The azure side does not even run OpenVPN. It runs Microsoft's own properitary implementation.

from openvpn.

Yes. Confirming again now that all endpoints are working after reissuing configuration without --disable-dco set, so it appears MS finally did their part, as well. We can actually close this thread now and consider it solved. :)

Just tested again, and now appears to be working WITHOUT disable-dco set. So maybe MS finally got around to correcting that?

from openvpn.

Thanks to everyone who helped me out in this thread. All issues have been resolved!

from openvpn.

Legacy service (OpenVPNServiceLegacy) has been deprectaed long time ago. Use its replacement called OpenVPNService backed by openvpnserv2.exe. Its installed by default.
Try sc start OpenVPNService. Also the auto-start config must be in the config-auto directory (C:\Program Files\OpenVPN\config-auto by default).

from openvpn.

@selvanair I appreciate this. I didn't realize this was the case! Cheers! I assume this works in the same way the legacy service does (auto-starts prior to login)?

from openvpn.

from openvpn.

from openvpn.

The config files are automatically moved by the installer if it finds auto-start configs in use -- i.e. OpenVPNService was running before upgrade. That service has been available since 2016 when the old one was renamed as legacy and deprecated.

but it doesn't look like the OpenVPNService is installed

What does sc query OpenVPNService show? Does C:\Program Files\OpenVPN\bin\openvpnserv2.exe exist? If the service is running you have to restart it for it to recognize changes to the config-auto folder.

from openvpn.

Thanks for this. I believe Webroot may be interacting with my install, which is causing the issue. Going to exempt the installation folder and see if that helps. I did get OpenVPNService showing up in list of services now, but it looks like the Wintun install was perhaps blocked by Webroot. Again, going to exempt and see if that helps.

from openvpn.

OK. SO now with that out of the way it is installed and running, but I'm noticing the connection is dropping over and over. Same config that worked with legacy service is not working with the OpenVPN service for some reason. Any ideas on what I should try next would be appreciated. I will check out the Azure guide for connecting via OpenVPN to the gateway in the meantime. Perhaps something has changed?

from openvpn.

Not sure why it isn't working, still. I do notice in the log that it looks like TCP_CLIENT link local is saying (not bound). Is this Wintun still not working right?

2023-02-02 10:55:27 Outgoing Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication
2023-02-02 10:55:27 Incoming Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication
2023-02-02 10:55:27 MANAGEMENT: >STATE:1675356927,RESOLVE,,,,,,
2023-02-02 10:55:28 TCP/UDP: Preserving recently used remote address: [AF_INET]XX.XXX.XXX.XXX:443
2023-02-02 10:55:28 ovpn-dco device [OpenVPN Data Channel Offload] opened
2023-02-02 10:55:28 TCP_CLIENT link local: (not bound)
2023-02-02 10:55:28 TCP_CLIENT link remote: [AF_INET]XX.XXX.XXX.XXX:443
2023-02-02 10:55:28 MANAGEMENT: >STATE:1675356928,WAIT,,,,,,
2023-02-02 10:55:28 MANAGEMENT: >STATE:1675356928,AUTH,,,,,,
2023-02-02 10:55:28 TLS: Initial packet from [AF_INET]XX.XXX.XXX.XXX:443, sid=XXXXXXXXXXXX
2023-02-02 10:55:28 Connection reset, restarting [-1]
2023-02-02 10:55:28 Closing DCO interface
2023-02-02 10:55:28 SIGUSR1[soft,connection-reset] received, process restarting
2023-02-02 10:55:28 MANAGEMENT: >STATE:1675356928,RECONNECTING,connection-reset,,,,,
2023-02-02 10:55:28 Restart pause, 300 second(s)
2023-02-02 10:58:13 SIGTERM[hard,init_instance] received, process exiting

from openvpn.

No that's DCO, the new driver. Somehow it is not able to establish TCP connection with remote. Does it work if you add disable-dco to the config?

from openvpn.

If that works with disable-dco, would be nice to get logs from the driver. Would you be able to provide logs following this instruction? TCP (and UDP) is surely supposed to work.

from openvpn.

Will check now and report back.

from openvpn.

No dice. Still doesn't work with disable-dco added. Same exact config works with legacy service on versions up until 2.6.0.

from openvpn.

Legacy service (OpenVPNServiceLegacy) has been deprectaed long time ago. Use its replacement called OpenVPNService backed by openvpnserv2.exe. Its installed by default. Try sc start OpenVPNService. Also the auto-start config must be in the config-auto directory (C:\Program Files\OpenVPN\config-auto by default).

Can confirm OpenVPNService is now there and config is in the config-auto directory, but no longer connecting. When trying to use the GUI to connect, I am running into issues from prior posts.

from openvpn.

Here is log with disable-dco enabled.

2023-02-02 12:12:44 OpenVPN 2.6.0 [git:v2.6.0/b999466418dddb89] Windows-MSVC [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [AEAD] [DCO] built on Jan 25 2023
2023-02-02 12:12:44 Windows version 10.0 (Windows 10 or greater), amd64 executable
2023-02-02 12:12:44 library versions: OpenSSL 3.0.7 1 Nov 2022, LZO 2.10
2023-02-02 12:12:44 MANAGEMENT: TCP Socket listening on [AF_INET]127.0.0.1:25340
2023-02-02 12:12:44 Need hold release from management interface, waiting...
2023-02-02 12:12:44 MANAGEMENT: Client connected from [AF_INET]127.0.0.1:50261
2023-02-02 12:12:44 MANAGEMENT: CMD 'state on'
2023-02-02 12:12:44 MANAGEMENT: CMD 'log on all'
2023-02-02 12:12:44 MANAGEMENT: CMD 'echo on all'
2023-02-02 12:12:44 MANAGEMENT: CMD 'bytecount 5'
2023-02-02 12:12:44 MANAGEMENT: CMD 'state'
2023-02-02 12:12:44 MANAGEMENT: CMD 'hold off'
2023-02-02 12:12:44 MANAGEMENT: CMD 'hold release'
2023-02-02 12:12:44 Outgoing Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication
2023-02-02 12:12:44 Incoming Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication
2023-02-02 12:12:44 MANAGEMENT: >STATE:1675361564,RESOLVE,,,,,,
2023-02-02 12:12:45 TCP/UDP: Preserving recently used remote address: [AF_INET]XX.XXX.XXX.XXX:443
2023-02-02 12:12:45 Socket Buffers: R=[65536->65536] S=[65536->65536]
2023-02-02 12:12:45 Attempting to establish TCP connection with [AF_INETXX.XXX.XXX.XXX:443
2023-02-02 12:12:45 MANAGEMENT: >STATE:1675361565,TCP_CONNECT,,,,,,
2023-02-02 12:12:45 TCP connection established with [AF_INET]XX.XXX.XXX.XXX:443
2023-02-02 12:12:45 TCPv4_CLIENT link local: (not bound)
2023-02-02 12:12:45 TCPv4_CLIENT link remote: [AF_INET]XX.XXX.XXX.XXX:443
2023-02-02 12:12:45 MANAGEMENT: >STATE:1675361565,WAIT,,,,,,
2023-02-02 12:12:45 MANAGEMENT: >STATE:1675361565,AUTH,,,,,,
2023-02-02 12:12:45 TLS: Initial packet from [AF_INET]XX.XXX.XXX.XXX:443, sid=b3e2cede 4de09285
2023-02-02 12:12:45 Connection reset, restarting [0]
2023-02-02 12:12:45 SIGUSR1[soft,connection-reset] received, process restarting
2023-02-02 12:12:45 MANAGEMENT: >STATE:1675361565,RECONNECTING,connection-reset,,,,,
2023-02-02 12:12:45 Restart pause, 1 second(s)

from openvpn.

I just rolled back to 2.5.8 and used the OpenVPNService instead of the legacy one and it connects without issue, so it is something with the newest version that my Azure VPN Gateway is not liking.

from openvpn.

The only thing the service does is to start openvpn executable and watch it so that it can restart if it exits. So this error is unrelated to the service.

Likely some incompatibility between 2.6 client and your server in azure. Could you post the server log at verb 4 to see whether that provides any clue?

from openvpn.

The only thing the service does is to start openvpn executable and watch it so that it can restart if it exits. So this error is unrelated to the service.

Likely some incompatibility between 2.6 client and your server in azure. Could you post the server log at verb 4 to see whether that provides any clue?

Correct. I moved back to the last version and used the OpenVPNService in lieu of sticking with the legacy service (now knowing that is deprecated) and it works fine on 2.5.8. I will pull those logs and get them up here as soon as I have a chance.

I do appreciate your time.

from openvpn.

At least we know that DCO seem to not be the cause :)

Microsoft has their own OpenVPN server implementation, so it would be really interesting to know if there is some compatibility issue.

from openvpn.

Hey guys. Sorry. I've been busy provisioning some new devices and working on some other things. I haven't forgotten about this, though. Still going to dig up those logs for you.

from openvpn.

So far, this is the only log I'm able to pull on the Azure Gateway:

[ERR] [default] [OVPN_9AFE60A3-B776-C1CC-1D0B-5114AB0FF4F0] Connection failed with error 0x3E3 => The I/O operation has been aborted because of either a thread exit or an application request.

I may need to open a ticket with Microsoft to get anything deeper than that. Would you like me to proceed?

from openvpn.

Yes. There is a known problem with Azure and getting into contact with Microsoft is probably the only way to get it resolved. Someone with an azure gateway account could do a git bisect to figure out which commit exactly broke the service.

from openvpn.

Schwabe: So for now, we should just stick with 2.5.8?

OpenVPN has made the administration of adding endpoints to our Azure network a breeze up until this point.

From a security standpoint, are we missing out on much versus 2.6.0?

from openvpn.

Just created the ticket with Azure Support, so hopefully, we'll get some movement on it soon.

from openvpn.

FYI. Azure Engineer replied to my request that they are aware of the issue and production team is working on it. They've added me to a list of customers affected. Thanks for the guidance on this issue. 🙏

from openvpn.

you may want to add also the Linux label as it happens to me on Arch with NetworkManager-openvpn (see the duplicate just above this)

from openvpn.

Looking for answers on this too if anyone has additional info

from openvpn.

@mbell85 Could you please try again? We were told by MSFT that 2.6 compatibility issue is now resolved.

from openvpn.

from openvpn.

Nope. I take it back. The latest versions are not working. When I go to connect on version 2.6.5 I get an error saying that the management option can not be parsed. Will attach log.

from openvpn.

Correction. 2.5.8 does indeed appear to work. But 2.6.5 (latest version) does not. I'll test around a bit more to see which version it stops working on again.

from openvpn.

This might be a different issue, so would be interesting to look at the log.

from openvpn.

from openvpn.

Still not working starting with 2.6.0.

Here is the log:

2023-07-27 20:12:57 OpenVPN 2.6.0 [git:v2.6.0/b999466418dddb89] Windows-MSVC [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [AEAD] [DCO] built on Feb 15 2023
2023-07-27 20:12:57 Windows version 10.0 (Windows 10 or greater), amd64 executable
2023-07-27 20:12:57 library versions: OpenSSL 3.0.8 7 Feb 2023, LZO 2.10
2023-07-27 20:12:57 MANAGEMENT: TCP Socket listening on [AF_INET]127.0.0.1:25340
2023-07-27 20:12:57 Need hold release from management interface, waiting...
2023-07-27 20:12:58 MANAGEMENT: Client connected from [AF_INET]127.0.0.1:57477
2023-07-27 20:12:58 MANAGEMENT: CMD 'state on'
2023-07-27 20:12:58 MANAGEMENT: CMD 'log on all'
2023-07-27 20:12:58 MANAGEMENT: CMD 'echo on all'
2023-07-27 20:12:58 MANAGEMENT: CMD 'bytecount 5'
2023-07-27 20:12:58 MANAGEMENT: CMD 'state'
2023-07-27 20:12:58 MANAGEMENT: CMD 'hold off'
2023-07-27 20:12:58 MANAGEMENT: CMD 'hold release'
2023-07-27 20:12:58 Outgoing Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication
2023-07-27 20:12:58 Incoming Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication
2023-07-27 20:12:58 MANAGEMENT: >STATE:1690506778,RESOLVE,,,,,,
2023-07-27 20:12:58 TCP/UDP: Preserving recently used remote address: [AF_INET]XX.XXX.XXX.XXX:443
2023-07-27 20:12:58 ovpn-dco device [OpenVPN Data Channel Offload] opened
2023-07-27 20:12:58 TCP_CLIENT link local: (not bound)
2023-07-27 20:12:58 TCP_CLIENT link remote: [AF_INET]XX.XXX.XXX.XXX:443
2023-07-27 20:12:58 MANAGEMENT: >STATE:1690506778,WAIT,,,,,,
2023-07-27 20:12:58 MANAGEMENT: >STATE:1690506778,AUTH,,,,,,
2023-07-27 20:12:58 TLS: Initial packet from [AF_INET]XX.XXX.XXX.XXX:443, sid=XXXXXXXXXXXXXX
2023-07-27 20:12:58 VERIFY OK: depth=2, C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root CA
2023-07-27 20:12:58 VERIFY OK: depth=1, C=US, O=DigiCert Inc, CN=DigiCert SHA2 Secure Server CA
2023-07-27 20:12:58 VERIFY KU OK
2023-07-27 20:12:58 Validating certificate extended key usage
2023-07-27 20:12:58 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
2023-07-27 20:12:58 VERIFY EKU OK
2023-07-27 20:12:58 VERIFY X509NAME OK: C=US, ST=Washington, L=Redmond, X.vpn.azure.com
2023-07-27 20:12:58 VERIFY OK: depth=0, C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=X.vpn.azure.com
2023-07-27 20:12:58 Control Channel: TLSv1.2, cipher TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, peer certificate: 2048 bit RSA, signature: RSA-SHA256
X.vpn.azure.com] Peer Connection Initiated with [AF_INET]XX.XXX.XXX.XXX:443
2023-07-27 20:12:58 TLS: move_session: dest=TM_ACTIVE src=TM_INITIAL reinit_src=1
2023-07-27 20:12:58 TLS: tls_multi_process: initial untrusted session promoted to trusted
2023-07-27 20:12:59 MANAGEMENT: >STATE:1690506779,GET_CONFIG,,,,,,
2023-07-27 20:12:59 SENT CONTROL [X.vpn.azure.com]: 'PUSH_REQUEST' (status=1)
2023-07-27 20:13:00 PUSH: Received control message: 'PUSH_REPLY,route 192.168.X.X 255.255.X.X,route 10.X.X.X 255.X.X.X,route 10.X.X.X 255.X.X.X,route 10.X.X.X 255.X.X.X,route-gateway 10.66.XX.XX,topology subnet,ifconfig 10.XX.XX.XX 255.XX.XX.XXX,dhcp-option DNS 10.XX.XX.XX.4,cipher AES-256-GCM'
2023-07-27 20:13:00 OPTIONS IMPORT: --ifconfig/up options modified
2023-07-27 20:13:00 OPTIONS IMPORT: route options modified
2023-07-27 20:13:00 OPTIONS IMPORT: route-related options modified
2023-07-27 20:13:00 OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified
2023-07-27 20:13:00 OPTIONS IMPORT: data channel crypto options modified
2023-07-27 20:13:00 OPTIONS IMPORT: Server did not request DATA_V2 packet format required for data channel offload
2023-07-27 20:13:00 OPTIONS ERROR: pushed options are incompatible with data channel offload. Use --disable-dco to connect to this server
2023-07-27 20:13:00 ERROR: Failed to apply push options
2023-07-27 20:13:00 Failed to open tun/tap interface
2023-07-27 20:13:00 Closing DCO interface
2023-07-27 20:13:00 SIGUSR1[soft,process-push-msg-failed] received, process restarting
2023-07-27 20:13:00 MANAGEMENT: >STATE:1690506780,RECONNECTING,process-push-msg-failed,,,,,
2023-07-27 20:13:00 Restart pause, 1 second(s)
2023-07-27 20:13:01 Outgoing Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication
2023-07-27 20:13:01 Incoming Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication
2023-07-27 20:13:01 MANAGEMENT: >STATE:1690506781,RESOLVE,,,,,,
2023-07-27 20:13:01 TCP/UDP: Preserving recently used remote address:

from openvpn.

Looks like the same error from before. Did MS specify whether disable-dco is needed (or anything else for that matter)? I plan on opening another ticket with them to ask myself.

from openvpn.

This is a different error - we are past TLS handshake now. Here server hasn't pushed peer-id, which is required for DATA_V2 format. This doesn't look right, since this has been a default behavior for many years - I think starting from 2.4 client pushes IV_PROTO with IV_PROTO_DATA_V2 bit set and server replies with peer-id in PUSH_REPLY and uses DATA_V2 format.

Let me ask MSFT again.

I would also recommend to use 2.6.5 since there were quite a few fixes to DCO implementation, although those are probably irrelevant to this specific issue.

from openvpn.

Indeed, progress!

Now we are at the point where a 2.6.5 client started with --disable-dco should work, while before that, we didn't even reach the "we could do something with options" stage.

Lack of support for DATA_V2 was no problem before DCO because both variants where "equally fine". With DCO, the new kernel data paths do no longer support the old framing. Reasoning behind this: DOC also requires AEAD ciphers. AEAD ciphers capability was introduced with 2.4.0, and P_DATA_V2 for clients was introduced somewhere in the 2.3 timeframe (commit 0e1fd33, November 2014). So all official OpenVPN sources that support AEAD also support DATA_V2, and thus, all combinations that would support DCO also need to support DATA_V2.

Which hints at MSFT using a totally different code basis, and not following upstream developments very closely...

from openvpn.

Guys. Thank you so much! Would you recommend then that I use 2.6.5 with --disable-dco or stick with 2.5.8 for now? I appreciate your expertise and assistance with this! I think most of us responsible for systems at any organization want to always stay current with updates, including third party ones, especially when they are serving a function as important as OpenVPN does.

from openvpn.

Just confirming that 2.6.5, with --disable-dco added to the config, does indeed work. Which is the more secure option at this point? To stick with 2.5.8 without --disable-dco or to move to 2.6.5 with --disable-dco enabled?

from openvpn.

Both 2.5.9 and 2.6.5 are fully maintained releases, and should be equally stable and secure. Without --disable-dco, 2.6 is much faster, and uses less CPU at the same time - but if DCO cannot be used, it does not really matter today.

At some point, OpenSSL 1.1.1 will become unsupported, and the OpenSSL 3.0 support in OpenVPN 2.5 is not as good as in 2.6 - at that time, moving to 2.6 might be a good idea, regardless of DCO.

from openvpn.

Thank you for that explanation! Makes perfect sense.

from openvpn.

Hey guys. I'm not sure if this is still related but some of my endpoints on 2.6.6 and above are sometimes getting APIPA addresses with OpenVPN. This was not occurring with 2.5.5. Strange thing is it only appears to be happening to a handful of endpoints.

from openvpn.

Logfile or it didn't happen... seriously, OpenVPN Clients will normally apply what the server gives them (PUSH_REPLY), so unless there's 169.254.* in the server reply, it's hard to see how it could end up on the interface.

from openvpn.

Logfile or it didn't happen... seriously, OpenVPN Clients will normally apply what the server gives them (PUSH_REPLY), so unless there's 169.254.* in the server reply, it's hard to see how it could end up on the interface.

Can't get the log file until I physically go to the location, but it's happened on several of our remote endpoints the past couple of weeks... Here's a screenshot of one with the APIPA address showing up where normally the 10.66.X.X network is showing up on other remote endpoints.

from openvpn.

I have gone ahead and run a reset command on the Azure VPN Gateway this morning and all other endpoints appear to be working OK. Needless to say, I've never had this behavior on any of our machines before the beginning of last week. The only change was that endpoints were updated to 2.6.7.

from openvpn.

Can't get the log file until I physically go to the location

You can verify what the server pushes to the client in the server log, it may help.

Not applicable to Azure.

from openvpn.

The only thing I'm seeing in the Azure Gateway logs that may be related are a bunch of these, which appear to have ceased since I reset the Gateway:

[ERR] [default] [OVPN_XXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX] Connection failed with error 0x3E3 => The I/O operation has been aborted because of either a thread exit or an application request.

from openvpn.

The vast majority of the logs end like this: Connection successful. Username=***N-ClientsChild IP=10.66.XX.XX, which are obviously successful connections. What could be happening on some of them I wonder where they end up with an APIPA address instead, even if the server is giving out a legit/correct IP?

from openvpn.

The azure side does not even run OpenVPN. It runs Microsoft's own properitary implementation.

Aware of that. I appreciate it. Just curious if there's anything in particular with the client I should be looking at, as I said it's only been happening last week into this week. I've reset the networking stack on the endpoints and that clears it for a while (sometimes it takes multiple resets or even uninstallation of the NIC) and then sometimes it appears to come back. I'm not blaming or attacking anyone here. Just trying to find an answer to the issue I am having. Thanks again!

from openvpn.

Believe I may have found the issue after all. Webroot DNS Protection Agent being funky. Appears to have messed with the DNS resolution on affected endpoints (it sets the DNS to always be localhost and resolution happens through the agent). The OpenVPN Client log shows its failing to find the gateway a number of times and then I see the APIPA address show up. Reinstalling Webroot DNS appears to resolve it. Now to go to Webroot support to find out why it is failing to resolve at times. Thanks guys.

from openvpn.

Thanks for coming back with the underlying reason. APIPA could be "it's set to use the tap adapter, the tap adapter is opened (so windows tries DHCP) and then OpenVPN gets stuck, so no proper address is provided".

If you are indeed using TAP and not DCO - any special reason for this? (DCO performs better, and does not play tricks with magic DHCP, but might have compatibility issues).

from openvpn.

Last I tested, DCO had to be disabled for Microsoft's implementation of the OpenVPN Server. Which they are going to have to fix eventually, I would hope. When the time comes, I may just decommission the MS Azure GW Server and run an OpenVPN Server proper of my own in an Azure VM.

from openvpn.

@mbell85 There could be something else in your config, apart from disable-dco, which disables DCO. Last time I've heard from MS, they haven't implemented DATA_V2 at all. I wonder if it is still the case. May I ask you to show us connection log?

from openvpn.