Why does Windows have issues with the encoding, but Linux doesn't? - windows

Question

For my toy project mpu I have two CI solutions running:
Travis / Linux: Works
Azure / Windows: Fails
It fails with this message:
_______________________________ test_read_json ________________________________
def test_read_json():
path = "files/example.json"
source = pkg_resources.resource_filename(__name__, path)
data_real = read(source)
data_exp = {
"a list": [1, 42, 3.141, 1337, "help", "�"],
"a string": "bla",
"another dict": {"foo": "bar", "key": "value", "the answer": 42},
}
> assert data_real == data_exp
E AssertionError: assert {'a list': [1... answer': 42}} == {'a list': [1... answer': 42}}
E Omitting 2 identical items, use -vv to show
E Differing items:
E {'a list': [1, 42, 3.141, 1337, 'help', '€']} != {'a list': [1, 42, 3.141, 1337, 'help', '�']}
E Use -v to get the full diff
tests\test_io.py:175: AssertionError
Why can it read the € sign from the JSON, but within the test it fails? (Python 3.6)

Answer 1

I assume that the read function which is used in the test wraps open in some way or another.
TL;DR Try adding encoding='utf8' to the call to open.
From my experience, Windows does not always play nice with non-ascii characters when reading files unless the encoding is set explicitly.
Also, it does not help that the default value for encoding is platform-dependent:
encoding is the name of the encoding used to decode or encode the
file. This should only be used in text mode. The default encoding is
platform dependent (whatever locale.getpreferredencoding() returns),
but any text encoding supported by Python can be used. See the codecs
module for the list of supported encodings.
some tests (ran on Win 10, Python 3.7, locale.getpreferredencoding() returns cp1262):
test.csv
€
with open('test.csv') as f:
print(f.read())
# €
with open('test.csv', encoding='utf8') as f:
print(f.read())
# '€'