GCM works intermittently with Azure DevOps Server about git-credential-manager HOT 13 CLOSED

Ok - so this is weird. I tried messing with it a bit yesterday to no avail. Because of meetings I decided not to use git, so I did NOT even bother to get my one-day password. Today, I signed on and just for funsies decided to try a git pull and lo and behold it works! I literally changed nothing. And my 1 day password (from Monday) for sure has expired (today is Wed), and it clearly didn't work yesterday (Tues) either.

Logs are attached. About the only thing that jumps out at me is that when it works, it appears to issue a store command whereas when it's in fail mode, it issues an erase command. I don't know anything about these commands or how they get triggered. I had seen the "Invalid OAUTH configuration" log message and thought maybe that was it, but as you can see it's in the "success" log too, so it must be a red herring.

logsAfterDoingNothingDidNotWorkMonTueNowWorksWed.txt

Another observation is that in my Windows Credential Manager there are literally NO git credentials. I made sure to refresh it and I had previously seen an entry in there before. One SO post had recommended deleting it, so I've tried that a couple times too, as well as backing it up and trying to restore. I've attached a screenshot for reference:

Finally, here are my settings as I left them yesterday:

[credential "http://abc-teamfound:8080"]
	helper = manager
	provider = generic
	usehttppath = true
	allowWindowsAuth = true

I had taken out the preserve = true and I had been trying the allowWindowsAuth setting as well, to see if that affected anything. I've kind of been shooting in the dark, trying to get GCM to work.

Any help or guidance would be greatly appreciated.

Thanks!
Michael

Note; My plan is now to not touch these settings and continue to use git. When it breaks, which it most likely will, in a couple days, I'll then snag the logs for "yet another log diff compare" and update my findings here.

from git-credential-manager.

Hi! Thanks for all the information and detailed logs. It looks like your environment is very locked down; our best guess so far is that there is a problem with Windows authentication between Git (libcurl) and TFS. GCM shows in the logs that Windows authentication is advertised by the server and instructs Git to attempt Kerberos/NTLM. This is failing and the fallback is the basic username and password prompt that you see.

GCM does not actually perform Windows authentication - it delegates this to Git/CURL. We may be able to help further, however, if we see the CURL logs from Git. You can obtain this by setting the following environment variables:

GIT_TRACE=1
GIT_TRACE_CURL=1
GCM_TRACE=1

Note that the output from GIT_TRACE_CURL can be quite verbose. It should not contain any secrets; however, it's recommended to double check before uploading out of an abundance of caution.

from git-credential-manager.

Thank you so much for the response!! Yes, indeed, our environment is relatively locked down (which is a good thing). We do basically have single sign on for most things. For example, once logged into to Windows, I can access Sql Server Management Studio w/o entering a user/pass. It just works.

I will enable those env variables and start capturing logs and update here with any findings.

What I don't understand is why it works some of the time for me. Case in point, I signed on before getting to this message and did a quick git pull and it still works today. Haven't changed anything. However, as previously stated, I had it working last week and then it suddenly stopped.

When all is done and we have this figured out, I should probably write a nice blog post or SO question on it!

Thanks,
Michael

from git-credential-manager.

Yes, it's definitely odd that you're seeing the intermittent behavior - I can only assume it's an intermittent issue with NTLM/Kerberos, but I'm also unsure of why that would happen. Hopefully the updated logs will shed some light.

from git-credential-manager.

Hi - we are collecting more logs, but are also off tomorrow, so I probably won't have any new information until next week. There are some image encoding in the logs - can I delete those? I'll also sanitize project/url/etc names before uploading.

Thanks,
Michael

from git-credential-manager.

Next week is fine. And it also should be fine to remove the image encoding.

from git-credential-manager.

Hi,

I added the following to my bash_profile so that it wouldn't print out to the console every time:

export GIT_TRACE=~/git.log
export GIT_TRACE_CURL=~/git-curl-trace.log
export GCM_TRACE=~/gcm-trace.log

In any case, I am attaching the two curl trace logs. The first is from Thursday, when it was still working. The second log is from today and sure enough is a non-working example. I signed in the exact same way I do every day and proceeded to do a git pull, at which point I got the "Auth failed" message. I've compared the two curl logs and don't see much of a difference. Do you?

Thursday log when it worked
git-curl-trace.log

Monday log when it did not work
git-curl-trace-not-working.log

Note: I did change a few urls, names, and some text within the image data just to be safe. Otherwise, it's pretty much left alone.

I also have the other two logs, but didn't see anything notable in them, besides which, you already have the two GCM trace logs above. The only thing that jumps out at me there is the 'store' vs 'erase' commands. I can attach them if you like, however.

(btw - you weren't kidding about the verbosity of the curl trace log!)

Thanks,
Michael

from git-credential-manager.

And go figure, I signed in today, no changes, and now magically it works! I can attach logs, but I'm guessing there's little point in doing so.

One thing I considered: After logging in to laptop + vpn, I still have to log into internal websites through Chrome one time (first thing) in the morning. For example, if I go to our Confluence wiki, I won't be logged in. Then I'll click log in and have to enter my pin (same pin for my laptop and vpn). But that's one time only. Now, I will get timed out during the day, but all I have to do for the rest of the day is click log in and it automatically logs me in (no pin entry required). So:

• I don't know how SSO is set up here. Maybe there's something wonky there?
• I don't know what Windows is doing behind the scenes with those credentials (caching??)
• Perhaps I hadn't logged in once through chrome on the days git didn't work, but
  • I would expect a subsequent login to, say confluence, would fix that (and it doesn't).

Thanks,
Michael

from git-credential-manager.

Looking at the differences between the working and not-working curl logs, I see this in the latter:

..
07:43:35.359272 http.c:845              == Info: NTLM handshake rejected
07:43:35.359272 http.c:845              == Info: Authentication problem. Ignoring this.
..

Looks like there is indeed something weird going on with NTLM between the client and server. GCM does not actually perform NTLM or Kerberos authentication itself, and just lets Git/curl handle it.

One thing I considered: After logging in to laptop + vpn, I still have to log into internal websites through Chrome one time (first thing) in the morning. For example, if I go to our Confluence wiki, I won't be logged in.

This sounds like there's definitely an issue with your SSO. Is this affecting others in your environment or just you?

It may be possible that although you've managed to log-in to Windows, your machine didn't 'see' the domain controller so is using cached creds until the VPN connects, and then you next perform a NTLM handshake? I'm not an NTLM expert, so this is just a guess.

Does this change if you are physically on the corporate network rather than via
the VPN?

from git-credential-manager.

Good catch - I missed that when I was pouring through the logs. I also see the following before the handshake rejection:

working: Recv header: HTTP/1.1 200 OK
not-working: Recv header: HTTP/1.1 401 Unauthorized

To answer your first question, this affects everyone. The whole dev team has to get a temp one-day password every day for git, which is why I'm pushing hard to solve this.

I suppose the next step is to figure out who even is in charge of SSO. I'm new, so that's been difficult. We're big (over 10,000 employees), but I've had a hard time getting straight answers on that front. 🤣 And for what it's worth, people on the team said it "used to work" like a year ago. And because we can all log on to the ADO server with no problem, I pursued a solution from our end.

To answer your second question - no idea. I'm fully remote two states away. Most of the team is fully remote too. However, if I can get someone to do that testing I will. Even so, it still needs to be solved for all the fully remote folks.

Thanks,
Michael

from git-credential-manager.

And for what it's worth, people on the team said it "used to work" like a year ago.

Something that may be worth trying is installing an older version of Git/GCM to see if it changes anything. If you still run into the same problems, it indicates that your network environment has likely changed in some way that caused this issue to manifest. Here's a link to a version of Git from about a year and a half ago you can try:

https://github.com/git-for-windows/git/releases/tag/v2.37.0.windows.1

Please let us know what you find.

Also, would it be possible to ask the maintainer of your Azure DevOps Server instance to see if they can see anything in the logs (specifically the web server logs, which handle NTLM)? The message from CURL that the handshake was rejected is all we can see on the client side - the server is rejecting your connection for some reason.

from git-credential-manager.

Just wanted to post an update here. I was finally able to meet with our Engineering team. One takeaway from that is apparently there are cached local credentials. This makes sense, because how else would we be able to log on to our laptops with a token card?

However, our VPN client does not seem to refresh them upon logging in. So, while they are valid for a local log in, they might expire for other resources requiring authentication.

Solution: Lock and unlock your screen after logging in. This apparently forces a refresh of said credentials. I have even confirmed this is the case by seeing failure, then having a user do the lock/unlock, and then being able to use git. A rather "disappointing" solution. ;)

All that said, I still have some anomalies. One user still can't get theirs to work, despite trying the above. Will be getting them to enable trace logging soon, I hope. Another user, I found out, has never had to get a temp one-day password. It's always just worked for them. So that one is weird.

Thanks again for all your help. If I can get things sorted w/ those couple users soon, then I'll probably close this.

from git-credential-manager.

Hi,

I think it's time to close this issue. Logged in to work today and a git push failed. So, I did the lock/unlock trick and it then succeeded.

The user who couldn't get theirs to work is now working, so /shrug? And the user who always worked, well, I guess a similar shrug. We do have several different ways of logging in (PIV cards, tokens, and crypto cards), but so far User2's set up is identical to mine. I did find one other user who remotes in to an "on-site" computer as opposed to a laptop+vpn, so that's huge flag there as far as environmental differences go.

I do want to say that I apparently still need to have that useHttpPath option set to true, but otherwise am working fine. If I set it to false (the default), then the authentication fails and GCM falls back to user/pass dialog. Set back to true, and we are good to go!

Thank you both for all your help. I learned a lot along the way and it was a beneficial troubleshooting session. I hope that if someone else has issues, they stumble upon this and it helps them troubleshoot.

Thanks,
Michael Kolakowski

from git-credential-manager.