tqdm not printing

[behrica]

Doing this

(py/run-simple-string "from tqdm import tqdm")
(py/run-simple-string "from time import sleep")
(def s "with tqdm(total=100) as pbar:\n    for i in range(10):\n        sleep(1)\n        pbar.update(10)")
(py/run-simple-string s)

does not print anything in the repl.

[jjtolton]

What is tqdm?

[behrica]

https://github.com/tqdm/tqdm

Nearly every python library is using it for progress report.

[behrica]

I am not sure, if something can be done on te level of libpython-clj
I think tqdm tries to detect of it has a "tty" or not.

[behrica]

I think the "root cause" of the non printing of tqdm in libpython-clj is that stderr is not a tty:

(py/run-simple-string "import sys;print(sys.stderr.isatty())")
-> false

while in a python shell:

>>> sys.stderr.isatty()
True

[behrica]

There is something strange ongoing.
the code here:

libpython-clj/src/libpython_clj2/python/io_redirect.clj

Line 37 in 11140d6

(constantly (py-ffi/py-false)))})))

Seems to set the isatty() for both (out and err) to false.

But verifying it via "run-simple-string",
shows that err -> false, and stdin -> true

[jjtolton]

Interesting. Well, that is technically correct, isn't it? Repl work with libpython-clj is not tty. Or is this for scripting?

[behrica]

A python interpreter is a tty. So I would expect the same for the python process started via libpython-clj running in a REPL.

[behrica]

I get printing by patching:

libpython-clj/src/libpython_clj2/python/io_redirect.clj

Line 34 in 11140d6

"flush" (py-class/make-tuple-instance-fn

to

 "flush" (py-class/make-tuple-instance-fn
              (fn [self] (.flush *err*)))

[behrica]

This is my complete "test code", so we know what we talk about:
(run in a "fresh" repl)

  (require '[libpython-clj2.python :as py])
  (py/initialize!)
  (def _ (py/run-simple-string "from tqdm import tqdm"))
  (def _ (py/run-simple-string "from time import sleep"))
  (def _ (py/run-simple-string "with tqdm(total=100) as pbar:\n    for i in range(10):\n        sleep(1)\n        pbar.update(10)"))

[behrica]

This code should print something (in realtime(, so every second for 10 seconds,
this is behavoiur in teh python interpreter.
But it does not in current `libpython-clj.

[behrica]

I can "trigger" the printing manually, by "doing something" after th above lines, which make Clojure print someting to "err" or "flush it".
(it will not print every second, but all togerther at the end)

 (require '[libpython-clj2.python :as py])
  (py/initialize!)
  (def _ (py/run-simple-string "from tqdm import tqdm"))
  (def _ (py/run-simple-string "from time import sleep"))
  (def _ (py/run-simple-string "with tqdm(total=100) as pbar:\n    for i in range(10):\n        sleep(1)\n        pbar.update(10)"))
  (.flush *err*)

Doing this a pressing "return" in the repl !!! afterwards
will the print the full progress in one go.

-> there is flushing problem.

[behrica]

Changing here:

libpython-clj/src/libpython_clj2/python/io_redirect.clj

Line 34 in 11140d6

"flush" (py-class/make-tuple-instance-fn

which does "nothing" on flush into:

 "flush" (py-class/make-tuple-instance-fn
              (fn [self] (.flush *err*)))

so force teh flushing of err results in a fully working progress bar, (so it prints every second for 10 seconds)
(the isatty being false or true in our code seem not make a dfference)

[behrica]

We have teh same issue with stdout.
This

 (def _ (py/run-simple-string "for i in range(10):\n        sleep(1)\n        print( 1)"))

does neither print something every second. It does so at the end (all is printed after 10 seconds)

[behrica]

Doing the same fix for out (as above), makes this working:

(def _ (py/run-simple-string "for i in range(10):\n        sleep(1)\n        print( 1, flush=True)"))

but not

(def _ (py/run-simple-string "for i in range(10):\n        sleep(1)\n        print( 1,)"))

Better said: In teh python interpreter stdoud seem to be se to "auto-flush" or similar.

(but without the above patch, neither "stdout" nor "stderr" can be flushed from python code. (only from acting on the Java streams)

[behrica]

So for me it seems clearly wrong to implement "flush" empty in here:

libpython-clj/src/libpython_clj2/python/io_redirect.clj

Line 34 in 11140d6

"flush" (py-class/make-tuple-instance-fn