Python2 crashes with non-ascii caracters.

[thanosgn]

Hi,

When using PyInquirer with python2 I face a problem when the options include non-ascii characters.
Python3 works fine though.

Here is a minimal working example that demonstrates this behaviour.

# -*- coding: utf-8 -*-
from __future__ import print_function, unicode_literals
from PyInquirer import prompt

def main():
    choice = 'John’s Pizza'
    questions = [
        {
            'type': 'list',
            'name': 'pizza',
            'message': 'Pizzeria:',
            'choices' : [choice]
        }
    ]
    answer = prompt(questions)
    pizza = answer['pizza']
    print('Answer is |' + pizza + '|' )

if __name__ == '__main__':
    main()

Python3 execution:

❯ python3 encoding-test.py  
? Pizzeria:  John’s Pizza
Answer is |John’s Pizza|

Python2 execution:

❯ python2 encoding-test.py
? Pizzeria:  (Use arrow keys)                                                                                                                                       
Traceback (most recent call last):
  File "encoding-test.py", line 20, in <module>
    main()
  File "encoding-test.py", line 15, in main
    answer = prompt(questions)
  File "/usr/local/lib/python2.7/dist-packages/PyInquirer/prompt.py", line 71, in prompt
    eventloop=eventloop)
  File "/usr/local/lib/python2.7/dist-packages/prompt_toolkit/shortcuts.py", line 625, in run_application
    result = cli.run()
  File "/usr/local/lib/python2.7/dist-packages/prompt_toolkit/interface.py", line 413, in run
    self._redraw()
  File "/usr/local/lib/python2.7/dist-packages/prompt_toolkit/interface.py", line 358, in _redraw
    self.renderer.render(self, self.layout, is_done=self.is_done)
  File "/usr/local/lib/python2.7/dist-packages/prompt_toolkit/renderer.py", line 429, in render
    extended_height=size.rows,
  File "/usr/local/lib/python2.7/dist-packages/prompt_toolkit/layout/containers.py", line 142, in write_to_screen
    sizes = self._divide_heigths(cli, write_position)
  File "/usr/local/lib/python2.7/dist-packages/prompt_toolkit/layout/containers.py", line 177, in _divide_heigths
    dimensions = [get_dimension_for_child(c, index) for index, c in enumerate(self.children)]
  File "/usr/local/lib/python2.7/dist-packages/prompt_toolkit/layout/containers.py", line 175, in get_dimension_for_child
    return c.preferred_height(cli, write_position.width, write_position.extended_height)
  File "/usr/local/lib/python2.7/dist-packages/prompt_toolkit/layout/containers.py", line 1652, in preferred_height
    return self.content.preferred_height(cli, width, max_available_height)
  File "/usr/local/lib/python2.7/dist-packages/prompt_toolkit/layout/containers.py", line 1000, in preferred_height
    cli, width - total_margin_width, max_available_height, wrap_lines),
  File "/usr/local/lib/python2.7/dist-packages/prompt_toolkit/layout/controls.py", line 254, in preferred_height
    content = self.create_content(cli, width, None)
  File "/usr/local/lib/python2.7/dist-packages/prompt_toolkit/layout/controls.py", line 259, in create_content
    tokens_with_mouse_handlers = self._get_tokens_cached(cli)
  File "/usr/local/lib/python2.7/dist-packages/prompt_toolkit/layout/controls.py", line 239, in _get_tokens_cached
    cli.render_counter, lambda: self.get_tokens(cli))
  File "/usr/local/lib/python2.7/dist-packages/prompt_toolkit/cache.py", line 37, in get
    value = getter_func()
  File "/usr/local/lib/python2.7/dist-packages/prompt_toolkit/layout/controls.py", line 239, in <lambda>
    cli.render_counter, lambda: self.get_tokens(cli))
  File "/usr/local/lib/python2.7/dist-packages/PyInquirer/prompts/list.py", line 98, in _get_choice_tokens
    append(i, choice)
  File "/usr/local/lib/python2.7/dist-packages/PyInquirer/prompts/list.py", line 92, in append
    tokens.append((T.Selected if selected else T, str(choice[0]),
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 4: ordinal not in range(128)

The apostrophe unicode character is what is breaking the python2 execution.

[CITGuru]

Currently, though I haven't figured out how to work around that without escaping the ascii codec. Though you can use unicode as below to get through it without error.

unicode('John’s Pizza').encode("unicode_escape")

which will give out what we dont really want

>>> John\u2019s Pizza

So its better to avoid it. In case you were able to fix it, do share how you are able to. I'll be working on it in my free time to get it fixed.

[thanosgn]

I have not tested it thoroughly, but it seems that removing the str() call from PyInquirer/prompts/list.py in line 92 solves the issue for now. See diff below:

@@ -89,7 +89,7 @@ class InquirerControl(TokenListControl):
                 tokens.append((T.Selected if selected else T,
                                '- %s (%s)' % (choice[0], choice[2])))
             else:
-                tokens.append((T.Selected if selected else T, str(choice[0]),
+                tokens.append((T.Selected if selected else T, choice[0],
                                select_item))
             tokens.append((T, '\n'))
 

Maybe this will guide you in the right direction even if it won't solve the issue yet.

Thanos

[CITGuru]

Oh lemme check that 👍 . Thanks

[tmbo]

Just ran into the same issue - would be great if we could merge that fix.

[CITGuru]

I have added a fix to the dev branch. I'll push it up to the master with some other fix.

[CITGuru]

So I am gonna close this issue. :+1

[tmbo]

Sounds great, we want to use it as part of https://github.com/rasahq/rasa_core and since we still need to support py2 & py3 that would make this a lot easier 🚀

[CITGuru]

Nice one, 👍 . Thanks for your time

[tmbo]

Do you have any plans when the next release will happen?

[CITGuru]

No, not anytime soon. Cos I am planning to add editor and some other fix. I need to carry out test against some environments. Just too busy these days in my company. Also I wanted to add some of the TODO feature. But, I can create a pre release version, so people can use most recent fix with no new feature. 👍

[tmbo]

Yes that would be great, if you tag it with 1.1.0a1 (or whatever your next release is) "normal" people wont get it if they install from pip only if you specifically use the version number, that would be best 👍