Server crashed when client tty was closed

[raphCode]

• I had a zellij session running on my server
• I connected via mosh and zelllij a
• I noticed another mosh session + zellij client was already running, restricting the zellij window size
• killall mosh-server
• Upon re-login I discovered that the zellij session died

This workflow worked in the past, so I am not sure what broke here.
zellij 0.33.0 on Arch

ERROR  |zellij_utils::errors::not| 2022-12-05 21:22:07.662 [stdin_handler] [zellij-utils/src/errors.rs:452]: Panic occured:
             thread: stdin_handler
             location: At zellij-client/src/os_input_output.rs:141:39
             message: called `Result::unwrap()` on an `Err` value: Os { code: 5, kind: Uncategorized, message: "Input/output error" }
ERROR  |zellij_utils::errors::not| 2022-12-05 21:22:07.662 [stdin_handler] [zellij-utils/src/errors.rs:452]: Panic occured:
             thread: stdin_handler
             location: At zellij-client/src/os_input_output.rs:141:39
             message: called `Result::unwrap()` on an `Err` value: Os { code: 5, kind: Uncategorized, message: "Input/output error" }
WARN   |zellij_utils::ipc        | 2022-12-05 21:22:07.664 [server_router] [zellij-utils/src/ipc.rs:204]: Error in IpcReceiver.recv(): InvalidMarkerRead(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })
ERROR  |zellij_utils::errors::not| 2022-12-05 21:22:07.664 [server_router] [zellij-utils/src/errors.rs:452]: Panic occured:
             thread: server_router
             location: At zellij-server/src/lib.rs:279:42
             message: Program terminates: a fatal error occured

Caused by:
    0: failed to handle instruction for client 2
    1: received empty message from client
INFO   |zellij_server::wasm_vm   | 2022-12-05 21:22:07.668 [wasm      ] [zellij-server/src/wasm_vm.rs:328]: wasm main thread exits
ERROR  |zellij_utils::errors::not| 2022-12-05 21:22:07.674 [async-std/runti] [zellij-utils/src/errors.rs:452]: Panic occured:
             thread: async-std/runtime
             location: At zellij-server/src/pty.rs:776:30
             message: Program terminates: a fatal error occured

Caused by:
    0: failed to spawn terminals for layout
    1: failed to listen for bytes from PTY
    2: failed to async-send to screen
    3: failed to send message to screen
    4: Originating Thread(s)

    5: failed to send message to channel
ERROR  |zellij_utils::errors::not| 2022-12-05 21:22:07.674 [async-std/runti] [zellij-utils/src/errors.rs:452]: Panic occured:
             thread: async-std/runtime
             location: At zellij-server/src/pty.rs:776:30
             message: Program terminates: a fatal error occured

Caused by:
    0: failed to spawn terminals for layout
    1: failed to listen for bytes from PTY
    2: failed to async-send to screen
    3: failed to send message to screen
    4: Originating Thread(s)

    5: failed to send message to channel
ERROR  |zellij_utils::errors::not| 2022-12-05 21:22:07.675 [async-std/runti] [zellij-utils/src/errors.rs:452]: Panic occured:
             thread: async-std/runtime
             location: At zellij-server/src/pty.rs:538:22
             message: Program terminates: a fatal error occured

Caused by:
    0: failed to run async task for terminal 13
    1: failed to listen for bytes from PTY
    2: failed to async-send to screen
    3: failed to send message to screen
    4: Originating Thread(s)

    5: failed to send message to channel
ERROR  |zellij_utils::errors::not| 2022-12-05 21:22:07.683 [async-std/runti] [zellij-utils/src/errors.rs:452]: Panic occured:
             thread: async-std/runtime
             location: At zellij-server/src/pty.rs:776:30
             message: Program terminates: a fatal error occured

Caused by:
    0: failed to spawn terminals for layout
    1: failed to listen for bytes from PTY
    2: failed to async-send to screen
    3: failed to send message to screen
    4: Originating Thread(s)

    5: failed to send message to channel
WARN   |zellij_utils::ipc        | 2022-12-05 21:22:07.705 [server_router] [zellij-utils/src/ipc.rs:204]: Error in IpcReceiver.recv(): InvalidMarkerRead(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })
ERROR  |zellij_utils::errors::not| 2022-12-05 21:22:07.705 [server_router] [zellij-utils/src/errors.rs:452]: Panic occured:
             thread: server_router
             location: At zellij-server/src/lib.rs:279:42
             message: Program terminates: a fatal error occured

Caused by:
    0: failed to handle instruction for client 1
    1: received empty message from client

[raphCode]

@har7an
I like the new error reporting!
Is it possible to add the file and line number in the "Caused by:" traces? In my case the context strings are unique so a simple grep on the codebase brings up the correct locations, but I am not sure if that is the case globally.

@imsnif
Probably relevant for you since crashes are severe.

[imsnif]

I'm pretty sure this is: #1955

[raphCode]

You are right, this is a dup, sorry.
I already reported this in #1949 and fixed by #1965

Looking through the git blames, seems that this was introduced by: #1881
where one instance of logging was replaced with an Err which was not marked as non-fatal.
I looked at the PR to spot any additional crash locations, but found nothing too obvious.

[har7an]

I like the new error reporting!

Glad to hear it!

Is it possible to add the file and line number in the "Caused by:" traces?

Well... Due to the way LoggableError is currently implemented, this depends on rust-lang/rust#87417, which hasn't come very far yet, I'm afraid. I can try and explore an alternative implementation of the LoggableError, of course, and see whether that will maintain error locations.

In my case the context strings are unique so a simple grep on the codebase brings up the correct locations, but I am not sure if that is the case globally.

I assume they mostly are. But even if we had duplicates, the "trace" of the error cause will allow us to nail it down to one location in the end.

Looking through the git blames, seems that this was introduced by: #1881

Oh well, in that case:

• Sorry for having crashed your zellij session!
• Thanks for double-checking. :)

[har7an]

Well... Due to the way LoggableError is currently implemented, this depends on rust-lang/rust#87417,

Forget that, I misunderstood the question.

I think it should be possible to add more information to the error locations mentioned in Caused by:. This would probably require a partial reimplementation of with_context, or our own anyhow::Context trait for that matter. I'm just not sure what impact this would have on application performance/size, because I don't know how std::panic::Location::caller() works under the hood. I'll open a new issue for it so we can discuss it there.

[raphCode]

I assume they mostly are. But even if we had duplicates, the "trace" of the error cause will allow us to nail it down to one location in the end.

You are right. So probably file locations aren't really important, just work on this if you feel like it :)