(mag ook in nederlands, hoor) Fryslân wrong in PDF and TEX: probably

FWIW: files with wrong wording: <div class="snippet-clipboard-content notranslate

Fryslân wrong as FryslÃ¢n in PDF and TEX: double unicode about covid-19 HOT 3 OPEN

mzelst commented on September 18, 2024

Fryslân wrong as FryslÃ¢n in PDF and TEX: double unicode

from covid-19.

Comments (3)

sanderjo commented on September 18, 2024

I don't know what the flow is that leads to the file in reports.

So I used this hack to correct to "Fryslân"

sed -i 's/FryslÃ¢n/Fryslân/g' daily_report.tex
sed -i 's/FryslÃ¢n/Fryslân/g' daily_report.ttt

After pdflatex daily_report, I had a corrected PDF:

from covid-19.

sanderjo commented on September 18, 2024

FWIW: files with wrong wording:

~/git/covid-19$ grep -irn FryslÃ¢n *
misc/maps/Gemeentegrenzen2021_simpelRD.geojson:331:{ "type": "Feature", "properties": { "id": 326, "statcode": "GM1900", "jrstatcode": "2021GM1900", "statnaam": "SÃºdwest-FryslÃ¢n", "rubriek": "gemeente" }, "geometry": { "type": "MultiPolygon", "coordinates": [ [ [ [ 160686.19700000063, 573323.60399999842 ], [ 161499.23200000077, 572253.34699999914 ], [ 164397.41000000015, 573255.42000000179 ], [ 165952.11600000039, 571938.61899999902 ], [ 167589.32499999925, 571965.32699999958 ], [ 168998.86100000143, 571042.4 ], [ 170504.08500000089, 572177.06500000134 ], [ 171745.21299999952, 570849.22700000182 ], [ 173032.8898999989, 569542.50239999965 ], [ 174600.94610000029, 571352.58810000122 ], [ 175680.798599999, 570396.28940000013 ], [ 177986.9990000017, 570473.19099999964 ], [ 179386.0540000014, 571145.20899999887 ], [ 179791.65300000086, 569611.45300000161 ], [ 181862.69399999827, 567828.42000000179 ], [ 180738.7490000017, 566988.95499999821 ], [ 180998.07699999958, 564490.64400000125 ], [ 180399.875, 563902.5390000008 ], [ 180738.79, 562299.53700000048 ], [ 179694.06500000134, 562279.65509999916 ], [ 178447.1640000008, 560416.52299999818 ], [ 178613.0, 557195.0 ], [ 175164.57600000128, 555052.1400000006 ], [ 175232.74500000104, 549954.80299999937 ], [ 172161.17599999905, 549931.27100000158 ], [ 172000.0, 548406.39900000021 ], [ 168927.49980000034, 547611.19929999858 ], [ 166915.19949999824, 551161.69009999931 ], [ 163380.71770000085, 548885.19660000131 ], [ 161049.86560000107, 546167.7533 ], [ 162715.70690000057, 543661.60619999841 ], [ 161052.69640000165, 542404.33080000058 ], [ 158051.69000000134, 543056.24700000137 ], [ 158624.11580000073, 540415.59380000085 ], [ 157622.00930000097, 540105.46950000152 ], [ 155732.47190000117, 541102.47080000117 ], [ 153599.34629999846, 542932.90080000088 ], [ 152761.19990000129, 544469.59939999878 ], [ 153934.55579999834, 545809.57710000128 ], [ 156317.47060000151, 547001.864 ], [ 155929.02230000123, 550236.49329999834 ], [ 156718.61389999837, 552945.68690000102 ], [ 155643.78330000117, 557318.09299999848 ], [ 156401.61320000142, 559263.21020000055 ], [ 154252.96339999884, 563097.04 ], [ 155363.50400000066, 564033.68800000101 ], [ 154155.35949999839, 567133.97410000116 ], [ 152030.39829999954, 565211.69139999896 ], [ 148961.70699999854, 564491.02100000158 ], [ 140200.27299999818, 556944.39779999852 ], [ 140141.44350000098, 557012.25039999932 ], [ 148700.90300000086, 564414.54800000042 ], [ 152408.52199999988, 565652.24100000039 ], [ 154374.45100000128, 567569.36499999836 ], [ 154451.6790000014, 569029.82400000095 ], [ 156150.985, 570528.35099999979 ], [ 156631.77030000091, 573913.07030000165 ], [ 160686.19700000063, 573323.60399999842 ] ] ] ] } },
misc/maps/Gemeentegrenzen2021_simpelRD.geojson:356:{ "type": "Feature", "properties": { "id": 351, "statcode": "GM1970", "jrstatcode": "2021GM1970", "statnaam": "Noardeast-FryslÃ¢n", "rubriek": "gemeente" }, "geometry": { "type": "MultiPolygon", "coordinates": [ [ [ [ 208482.5165, 603053.10649999976 ], [ 208452.5724, 603008.77490000054 ], [ 206539.75800000131, 602724.07600000128 ], [ 206075.7, 600937.04899999872 ], [ 207838.34699999914, 597058.56509999931 ], [ 210292.92599999905, 596688.31500000134 ], [ 210562.8740000017, 595414.42700000107 ], [ 212700.82200000063, 596077.67300000042 ], [ 214925.79659999907, 595331.182 ], [ 214483.45010000095, 591129.26720000058 ], [ 212822.38720000163, 589474.85940000042 ], [ 212355.76000000164, 587204.59 ], [ 211390.75800000131, 585907.80799999833 ], [ 208904.079, 587310.693 ], [ 207762.37000000104, 586236.92000000179 ], [ 205301.27299999818, 586690.03000000119 ], [ 198844.02, 584302.81899999827 ], [ 199489.83229999989, 589568.37099999934 ], [ 200644.25200000033, 590722.14799999818 ], [ 200282.46200000122, 593234.83300000057 ], [ 197327.80200000107, 593251.88500000164 ], [ 197255.01139999926, 591578.41220000014 ], [ 194239.27899999917, 591322.00800000131 ], [ 193425.546, 593087.92599999905 ], [ 190661.68, 591926.21299999952 ], [ 188980.85700000077, 590680.6950000003 ], [ 189641.50299999863, 589162.1790000014 ], [ 188022.47700000182, 587920.18800000101 ], [ 186581.31679999828, 589213.90929999948 ], [ 185115.66000000015, 587697.05000000075 ], [ 183664.3900000006, 587848.41000000015 ], [ 180076.71000000089, 589842.98000000045 ], [ 178216.21499999985, 589859.20899999887 ], [ 179133.67199999839, 591959.89600000158 ], [ 177335.58700000122, 592925.961 ], [ 176992.43910000101, 594772.00600000098 ], [ 181927.14499999955, 597412.30799999833 ], [ 184527.29050000012, 598376.6838000007 ], [ 185640.78400000185, 599377.86199999973 ], [ 188309.60229999945, 600619.4925 ], [ 191322.7434, 600356.64380000159 ], [ 196798.75600000098, 602028.41699999943 ], [ 202059.06399999931, 602649.735 ], [ 205006.98699999973, 602173.66099999845 ], [ 207507.305, 603439.364 ], [ 208482.5165, 603053.10649999976 ] ] ] ] } },
misc/scripts/coronaboetes.R:7:                                "Noardeast-FryslÃ¢n" = "Noardeast-Fryslân",
misc/scripts/coronaboetes.R:8:                                "SÃºdwest-FryslÃ¢n" = "Súdwest-Fryslân")
plot_scripts/daily_maps_plots.R:72:ggd.data$statnaam <- recode(ggd.data$statnaam, "GGD FryslÃ¢n" = "GGD Fryslân",
reports/daily_report.Rmd:200:dat.weekago$Province <- recode(dat.weekago$Province,"Friesland" = "FryslÃ¢n")
workflow/dashboards/cases_ggd_agegroups.R:5:dat$Municipal_health_service <- recode(dat$Municipal_health_service, "GGD FryslÃ¢n" = "GGD Fryslân",

So ... first step might be: brute-forcing the above sed command on these files to see if the problem goes away on next run ... ?

from covid-19.

sanderjo commented on September 18, 2024

Root cause:

â in utf8 is C3 A2

But if those two bytes are not interpreted as UTF8 but incorrectly as two seperate ASCII chars, then you get

>>> chr(0xC3)
'Ã'
>>> chr(0xA2)
'¢'

So: Ã¢

And then ... if you then encode them into UTF8 (because they are non-US-ASCII), you get ... 4 bytes:

>>> 'Ã¢'.encode()
b'\xc3\x83\xc2\xa2'

... which we saw earlier on: c3 83 c2 a2

So, somewhere, probably, a correct UTF8 file is incorrectly interpreted as ASCII, and then put into UTF8 again ...

from covid-19.

Fryslân wrong as FryslÃ¢n in PDF and TEX: double unicode about covid-19 HOT 3 OPEN

Comments (3)

Related Issues (17)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent