ProtonUp-Qt Flatpak ver. 2.9.2 - UnicodeDecodeError about protonup-qt HOT 13 CLOSED

No problem at all, thank you for the follow-up here! I didn't realise this myself either (I guess I had this a long time ago? Unsure, it has been many years since i did this installation so I'm not sure what past-sonic2kk did 😅) but those steps sound reasonable and that seems like a very little-known step. Particularly if you're not using an en_US locale it would not be obvious to me to include that.

Hopefully if anyone hits this in future this can bring a bit of visibility to the issue, so thank you as well 😄

from protonup-qt.

I must apologize that I didn't realize that I missed typing the Flatpak version of Steam which I am using.

I think I've fixed it thanks to you. I removed # from [email protected] (can be done with any ISO) from /etc/locale.gen and then in /etc/locale.conf I added LC_CTYPE=es_ES@euro (or whichever ISO you choose after LC_CTYPE=). Then I did sudo locale-gen and rebooted. After rebooting I launched Steam (Flatpak) and closed it. Then I launched ProtonUp-Qt and it worked like before.

I do not understand why Steam chose to use ISO instead of UTF-8. That's strange... I hope they change to UTF-8.

Thank you!

from protonup-qt.

I'm not sure why Steam uses ISO either, but I did check and I have both the ISO and UTF-8 locale enabled on my system. My install is from a long time ago so I'm not sure why I have it set up this way. I'm glad it's resolved though!

Maybe you should report this upstream, even just to ask and maybe get a clearer answer on what is going on? That should clear up whether it's expected that both locales are enabled, and/or if it is expected that Steam should not use UTF-8. To me it isn't clear from reading around that both locales should be enabled, but I also had both enabled, so I'm not sure if I read something a long time ago when I set up this installation 😅 I'm not sure what my other devices have, but I would wager since ProtonUp-Qt works fine on those devices that they have both locales set up as well (if that is indeed the root cause). I would be interesting to know what the Steam Deck uses, since at launch it had complaints about misconfigured locales that appear to be resolved now.

If you want to report it (although you don't have to!) you're probably best reporting it on the Steam Flatpak repo for now and reporting further upstream to Valve if this is a general thing with Steam that you'd like to see changed.

It is worth noting that the Steam Flatpak is entirely unofficial, and it is known to have a few oddities. Valve do label issues on the Steam for Linux issue tracker when they are experienced with Steam Flatpak, but to my understanding they only directly develop and maintain the Steam .deb which other distros' repos build off of (including Valve themselves on SteamOS).

from protonup-qt.

After reading the Steam Wiki from Arch Linux carefully https://wiki.archlinux.org/title/Steam I didn't realize that in the Installation section under "In order to run Steam on Arch Linux:" it states:

• If not already done during installation, you must generate the en_US.UTF-8 locale to prevent invalid pointer errors.

So I proceeded to uninstall Steam Flatpak and eliminate the directory (games are on another SSD) and redid the locale and locale-gen installation without the ISO and only the UTF-8 of both en_US (as fallback) and es_ES to see if it works. After rebooting and install Steam Flatpak, I closed it and quickly launched ProtonUp-Qt to see if it will work or not. It worked. I even tried out the command you told me to run chardetect config.vdf in ~/.var/app/com.valvesoftware.Steam/data/Steam/config/ now it comes out as config.vdf: ascii with confidence 1.0. Now ProtonUp-Qt works without any issues!
The locale of en_US.UTF-8 was the missing link all along which I didn't have before. I didn't think it would be necessary for the Flatpak version of Steam either, but it's needed.
Thank you @sonic2kk for your time, patience, and help with this. 👍👍👍

from protonup-qt.

Interesting, learned something new again.

Hopefully if anyone hits this in future this can bring a bit of visibility to the issue, so thank you as well 😄

Good to hear it is fixed, thanks for helping! I will close this issue.

from protonup-qt.

(Note: I believe this is distinct from #415)

Can't reproduce on Arch Linux with Flatpak (v2.9.2) or 887702a.

I wonder if this is locale-related? It seems like this could be a bug in the upstream VDF library with handling unicode characters, which may come about from parsing a config.vdf.

I'll try inserting some unicode characters into a VDF file and see if ProtonUp-Qt (or more specifically, the upstream VDF library) can parse it.

from protonup-qt.

From doing some research it appears that the error may actually be pretty much as stated; the format of the file may not be UTF-8. Can you check the format of your config.vdf file? The location may vary but the most common location on Arch Linux is either:

• ~/.steam/root/config/config.vdf for regular Steam
• ~/.var/app/com.valvesoftware.Steam/data/Steam/config/config.vdf for Flatpak Steam (I believe the path is right)

I couldn't convert to a new charset to test but it might be a good idea to check what the encoding of the file is. Any text editor should tell you, similarly file config.vdf from the folder with the file should reference this as well. chardetect on your system should also be able to tell you.

But I suspect this issue comes from the file simply not being UTF-8. This is odd because Python's open should be able to determine the encoding, although you can specify encoding=. But we shouldn't need to do this. Maybe Python is defaulting to the locale of the Flatpak environment (probably UTF-8?) but that does not match the config.vdf encoding?

As per the Python open docs

In text mode, if encoding is not specified the encoding used is platform-dependent: locale.getencoding() is called to get the current locale encoding.

We would be using text mode as this is the default (I believe you have to specify something like rb for bytes mode). On my system, locale.getencoding() returns UTF-8. So perhaps the result of locale.getencoding() does not match the config.vdf encoding. The question would become, why is this file not UTF-8 and in what circumstances would it not be UTF-8? It seems unusual.

from protonup-qt.

From doing some research it appears that the error may actually be pretty much as stated; the format of the file may not be UTF-8. Can you check the format of your config.vdf file? The location may vary but the most common location on Arch Linux is either:

* `~/.steam/root/config/config.vdf` for regular Steam

* `~/.var/app/com.valvesoftware.Steam/data/Steam/config/config.vdf` for Flatpak Steam (I believe the path is right)

I couldn't convert to a new charset to test but it might be a good idea to check what the encoding of the file is. Any text editor should tell you, similarly file config.vdf from the folder with the file should reference this as well. chardetect on your system should also be able to tell you.

But I suspect this issue comes from the file simply not being UTF-8. This is odd because Python's open should be able to determine the encoding, although you can specify encoding=. But we shouldn't need to do this. Maybe Python is defaulting to the locale of the Flatpak environment (probably UTF-8?) but that does not match the config.vdf encoding?

As per the Python open docs

In text mode, if encoding is not specified the encoding used is platform-dependent: locale.getencoding() is called to get the current locale encoding.

We would be using text mode as this is the default (I believe you have to specify something like rb for bytes mode). On my system, locale.getencoding() returns UTF-8. So perhaps the result of locale.getencoding() does not match the config.vdf encoding. The question would become, why is this file not UTF-8 and in what circumstances would it not be UTF-8? It seems unusual.

Thank you for your response.
First, I tested it with vim and, strangely, the only text that appears is "InstallConfigStore". Then I tried it with GNOME's Text Editor and was able to see the text file normally. Finally, I did the chardetect and this is what came up:
config.vdf: ISO-8859-1 with confidence 0.73

from protonup-qt.

Hmm, it sounds like something is wrong with your config.vdf. What size is the file? I think it should be in the region of about 200kb give-or-take depending on various factors (my file is around 350kb but I have a lot of entries in my CompatToolMapping and a lot of games in my library).

The text "InstallConfigStore" would be expected to show up in the file. In fact it should be the first entry in the file. The text-based VDF format is very similar to JSON, with the structure being like this (just a snippet of the first few lines, these sections get closed later on).

"InstallConfigStore"
{
	"Software"
	{
		"Valve"
		{
			"Steam"
			{

If your file size is more than a few bytes (the text "InstallConfigStore" probably only takes up about 40 bytes or less) then I would say when viewing the file it can only display this as the first part of the file. But if that is the entire size of the file, then I suspect there's a problem somewhere!

For what it's worth, ProtonUp-Qt might've crashed if it tried to read this file anyway. I can't remember how strict we are about the contents of config.vdf, we did get stricter recently but I'm not sure how much we verify the format.

Some other things that might noteworthy are:

• Does the file modified date line up with the last time you opened / closed Steam? This file gets modified each time the Steam Client is opened and closed.
• If you open Steam, does this file get re-created and look correct afterwards?

from protonup-qt.

Hmm, it sounds like something is wrong with your config.vdf. What size is the file? I think it should be in the region of about 200kb give-or-take depending on various factors (my file is around 350kb but I have a lot of entries in my CompatToolMapping and a lot of games in my library).

The text "InstallConfigStore" would be expected to show up in the file. In fact it should be the first entry in the file. The text-based VDF format is very similar to JSON, with the structure being like this (just a snippet of the first few lines, these sections get closed later on).

"InstallConfigStore"
{
	"Software"
	{
		"Valve"
		{
			"Steam"
			{

If your file size is more than a few bytes (the text "InstallConfigStore" probably only takes up about 40 bytes or less) then I would say when viewing the file it can only display this as the first part of the file. But if that is the entire size of the file, then I suspect there's a problem somewhere!

For what it's worth, ProtonUp-Qt might've crashed if it tried to read this file anyway. I can't remember how strict we are about the contents of config.vdf, we did get stricter recently but I'm not sure how much we verify the format.

The file size is 34,1 kb.

Some other things that might noteworthy are:

* Does the file modified date line up with the last time you opened / closed Steam? This file gets modified each time the Steam Client is opened and closed.

* If you open Steam, does this file get re-created and look correct afterwards?

The file does get modified when I open and close Steam. The file size changes to 34,0 kb.

from protonup-qt.

The file size is probably fine, maybe the average size is smaller than I anticipated 😅 But approx 34kb surely suggests the file stores more than just "InstallConfigStore", and I imagine it just isn't getting displayed.

I did a bit of digging and it seems that, particularly on Arch where the locale is often expected to be configured by the user, you can configure your locale to be non-UTF-8. So this is something to check; can you type locale in your terminal and see if your locales, in particular the LANG variable (should be the first one) ends with UTF-8 or ISO-8859?

You can see more about the locales on Arch on the Arch Wiki Locale page.

I imagine that this is a locale-related issue, and that Python is trying to use UTF-8 when opening the file, but Steam is using an alternative locale and trying to use ISO-8859.

From a development perspective, I'm not sure if this is something ProtonUp-Qt should handle. On one hand, Steam appears to respect the system locale. On the other hand, Python should be doing that but it appears that it still tries to use UTF-8 to open the file. I wonder if there's a way we can get a file locale and then pass that to open()... This won't be 100% foolproof (there is no way to be 100% confident in a charset to my knowledge) and there may not even be a good way to do this from Python across distributions so if this is locale-related, perhaps chalking this up to system configuration is fine? I'm not sure if there's a way we should handle this on our end.

from protonup-qt.

The file size is probably fine, maybe the average size is smaller than I anticipated 😅 But approx 34kb surely suggests the file stores more than just "InstallConfigStore", and I imagine it just isn't getting displayed.

I did a bit of digging and it seems that, particularly on Arch where the locale is often expected to be configured by the user, you can configure your locale to be non-UTF-8. So this is something to check; can you type locale in your terminal and see if your locales, in particular the LANG variable (should be the first one) ends with UTF-8 or ISO-8859?

You can see more about the locales on Arch on the Arch Wiki Locale page.

I imagine that this is a locale-related issue, and that Python is trying to use UTF-8 when opening the file, but Steam is using an alternative locale and trying to use ISO-8859.

From a development perspective, I'm not sure if this is something ProtonUp-Qt should handle. On one hand, Steam appears to respect the system locale. On the other hand, Python should be doing that but it appears that it still tries to use UTF-8 to open the file. I wonder if there's a way we can get a file locale and then pass that to open()... This won't be 100% foolproof (there is no way to be 100% confident in a charset to my knowledge) and there may not even be a good way to do this from Python across distributions so if this is locale-related, perhaps chalking this up to system configuration is fine? I'm not sure if there's a way we should handle this on our end.

My locale is: LANG=es_ES.UTF-8
I'm wondering if removing # from en_US.UTF-8 UTF-8 (to use as fallback as stated on the Arch Wiki Locale) and regenerate locale will help... 🤔

from protonup-qt.

It might be worth trying to enable a UTF-8 fallback. I'm not too sure of all the specifics around this, but it does seem odd to me that Steam is writing the file as what is guessed to be ISO-8859, and that this also happens to be a common non-UTF-8 locale.

It seems Python is assuming UTF-8 when that is clearly not the case, and Python should be making this determination based on your system locale settings. We already know that the Flatpak is returning UTF-8 (based on the error causing this issue) but it might be useful to check that your system Python is using, assuming it is installed (iirc Steam at least relies on Python, so if you're not using Flatpak Steam you probably have Python). You can run python3 -c "import locale;print(locale.getencoding())" to execute the Python command to get the encoding it will default to when opening files (-c is a flag you can give to Python to pass a string as a command instead of running an interactive session and exiting).

If this also returns UTF-8, and we know the Flatpak returns UTF-8, and your system is configured to use UTF-8 as you've shown, then I guess for some reason Steam is not using this.

from protonup-qt.