Redrawing room list buffer can be slow when in many rooms

[Rutherther]

OS/platform

Guix System or NixOS

Emacs version and provenance

Tried from both Nix and Guix, both times with jit native compilation enabled.
GNU Emacs 29.4 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.41, cairo version 1.18.0)

Emacs command

./with-emacs.sh --dir /tmp/test

Emacs frame type

GUI

Ement package version and provenance

I've used method in README to obtain from latest commit. That is 3f87a95 when trying this.

Actions taken

1. execute ement-connect
2. log in
3. wait for a bit after the ement room list opens

Observed results

After few seconds Emacs lags, for a few seconds without being able to do anything. This repeats every few seconds between the lags.

Expected results

I should be able to use emacs normally when having ement room list open.

Backtrace

No response

Etc.

       51922  54% + timer-event-handler
       40607  42% + command-execute
        3010   3% + ...
         274   0% + redisplay_internal (C function)
           3   0%   mouse--click-1-maybe-follows-link

I have not used profiler at all till now and I am not sure how to get relevant information. I can run for specific functions, but I don't know what to target.

I observe this behavior both in my regular Emacs installation and on a fresh new emacs version obtained with with-emacs.sh script.

I am on Conduit server. I also have sliding sync running on the server. I am in more than 200 rooms. Few of the rooms are bigger groups. But most of them are bridged DM rooms. I use Mautrix Discord (no guild bridged, only DMs), Mautrix Telegram, Mautrix Whatsapp, Mautrix Meta, Heisenbridge (bridging a few irc groups).

I have tried registering a separate account and join a few rooms (I am in those on my main account) that have more members. I did not observe this behavior on that account.

I have already reached alphapapa through Ement.el room, where it was suggested to open an issue. This was like two weeks ago (sorry for taking so long)

[Rutherther]

Additionally since I named all the bridges I use, I've tried stopping all of those so the bot accounts don't get a chance to update anything temselves. This didn't solve anything

[phil-s]

You can fully-expand a node in the profile report with C-uRET

You want to read C-hig (elisp)Profiling

[Rutherther]

Right, that makes sense, should've read the manual thoroughly...

       17169  94% - timer-event-handler
       17169  94%  - apply
       17169  94%   - plz--respond
       17169  94%    - #<compiled 0x8d5096906d58e94>
       17165  94%     - #<compiled -0xade022a6f6d61b0>
       17165  94%      - apply
       17165  94%       - ement--sync-callback
       17165  94%        - run-hook-with-args
       17157  94%         - ement-room-list-auto-update
       17157  94%          - revert-buffer
       17157  94%           - ement-room-list-revert
       17157  94%            - ement-room-list
       17049  93%             - taxy-magit-section-format-items
       17049  93%              - #<compiled -0xa7c2710a7245255>
       17049  93%               - #<compiled -0xa7c2710a7245255>
       17029  93%                - #<compiled -0xa7c2710a7245255>
       15961  87%                 - #<compiled 0x1f264c6d0a102517>
       15961  87%                  - #<compiled 0x1ccd7f36e7b5ed28>
       15439  84%                   - ement-room-list-column-format-name
       15362  84%                    - ement--room-display-name
       15354  84%                     - #<compiled -0xa2f04426fcf8645>
       15326  84%                      - mapcar
       15318  84%                       - #<compiled -0xc0db1ca136f5bc5>
        6335  34%                        - ement--user-displayname-in
        1387   7%                           #<compiled 0x53b2892b14fda62>
          12   0%                      - cl-remove-if-not
           4   0%                       - cl-remove
           4   0%                        - cl-delete
           4   0%                         - #<compiled -0xd07981da80d5c6f>
           4   0%                            apply
           4   0%                       #<compiled -0x66e54a1270e5219>
          72   0%                      truncate-string-to-width
           5   0%                    - ement--room-direct-p
           5   0%                     - #<compiled 0x1c375647c4cf9cae>
           5   0%                        seq-contains-p
         503   2%                     #<compiled 0x2689ac9dafa9eef>
           8   0%                   - ement-room-list-column-format-latest
           4   0%                      ement--human-format-duration
           4   0%                     map-put!
           3   0%                   - ement-room-list-column-format-notifications
           3   0%                    - ement-room-notification-state
           3   0%                     - map-nested-elt
           3   0%                      - seq-reduce
           3   0%                       - seq-do
           3   0%                        - mapc
           3   0%                         - #<compiled 0x1a0c8572a1b1c6fe>
           3   0%                          - #<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_14>
           3   0%                             map-elt
        1068   5%                 - #<compiled -0xa7c2710a7245255>
        1064   5%                  - #<compiled 0x1f264c6d0a102517>
        1064   5%                   - #<compiled 0x1ccd7f36e7b5ed28>
         972   5%                      #<compiled 0x2689ac9dafa9eef>
          48   0%                    - ement-room-list-column-format-name
          28   0%                     - ement--room-direct-p
          28   0%                      - #<compiled 0x1c375647c4cf9cae>
          24   0%                       - seq-contains-p
          12   0%                        - seq-do
           8   0%                         - mapc
           4   0%                            #<compiled 0xa1508dc680577d0>
           8   0%                       ement--room-display-name
           4   0%                       truncate-string-to-width
           4   0%                     - ement--room-unread-p
           4   0%                      - map-nested-elt
           4   0%                       - seq-reduce
           4   0%                        - seq-do
           4   0%                         - mapc
           4   0%                          - #<compiled 0x1a0cbb08a1b1c6fe>
           4   0%                           - #<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_14>
           4   0%                            - map-elt
           4   0%                               apply
          16   0%                    - map-put!
          16   0%                     - apply
          12   0%                      - #<compiled 0x1fe906d3872180>
           4   0%                         assoc
          12   0%                    - ement-room-list-column-format-notifications
          12   0%                     - ement-room-notification-state
           8   0%                      - #<compiled -0x171045b66f07ec2e>
           8   0%                       - map-nested-elt
           8   0%                        - seq-reduce
           8   0%                         - seq-do
           8   0%                          - mapc
           8   0%                           - #<compiled 0x1a0c833aa1b1c6fe>
           8   0%                            - #<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_14>
           8   0%                             - map-elt
           4   0%                                apply
           4   0%                      - cl-find-if
           4   0%                       - cl-find
           4   0%                          cl-position
           4   0%                    - ement-room-list-column-format-topic
           4   0%                       replace-regexp-in-string
           4   0%                      map-elt
          20   0%                - #<compiled 0x1f264c6d0a102517>
          20   0%                 - #<compiled 0x1ccd7f36e7b5ed28>
          12   0%                    #<compiled 0x2689ac9dafa9eef>
           4   0%                  - ement-room-list-column-format-name
           4   0%                     ement--room-direct-p
           4   0%                  - ement-room-list-column-format-🐱
           4   0%                     map-elt
          60   0%             - taxy-fill
          60   0%              - #<compiled 0xd9ce7609f704410>
          24   0%               - #<lambda 0x5b7290e5d7>
          24   0%                - taxy-take-keyed
          24   0%                 - taxy-take-keyed
          24   0%                  - taxy-take-keyed
          24   0%                   - taxy-take-keyed
          24   0%                    - taxy-take-keyed
          24   0%                     - taxy-take-keyed
          24   0%                      - taxy-take-keyed
          20   0%                       - taxy-take-keyed
          20   0%                        - taxy-take-keyed
          16   0%                         - #<lambda 0x9884c6d4680d254>
          16   0%                          - when
          16   0%                           - if
          12   0%                            - cl-loop
           8   0%                             - cl-block
           4   0%                              - cl--block-wrapper
           4   0%                               - catch
           4   0%                                  let*
           4   0%                                #<compiled -0x118465efb9c3cb7a>
           4   0%                         - taxy-take-keyed
           4   0%                          - cl-find-if
           4   0%                           - cl-find
           4   0%                            - cl-position
           4   0%                               cl--position
           4   0%                       - #<lambda -0x1667dbc0235e70fe>
           4   0%                        - when
           4   0%                         - if
           4   0%                          - cl-loop
           4   0%                             #<compiled -0x18ca13412de6202b>
          20   0%               - #<lambda 0x5b7290e5d7>
          20   0%                - taxy-take-keyed
          20   0%                 - taxy-take-keyed
          20   0%                  - taxy-take-keyed
          20   0%                   - taxy-take-keyed
          20   0%                    - taxy-take-keyed
          20   0%                     - taxy-take-keyed
          20   0%                      - taxy-take-keyed
          20   0%                       - taxy-take-keyed
          16   0%                        - taxy-take-keyed
          16   0%                         - taxy-take-keyed
          12   0%                          - taxy-take-keyed
           8   0%                           - taxy-take-keyed
           8   0%                            - ement-room-list-key-space
           8   0%                             - #<compiled -0xb12d938b9c3b04a>
           8   0%                              - cl-find
           8   0%                               - cl-position
           4   0%                                - cl--position
           4   0%                                   ement-room-id
           4   0%                           - ement-room-list-key-people
           4   0%                            - ement--room-direct-p
           4   0%                             - #<compiled 0x1c375647c4cf9cae>
           4   0%                                seq-contains-p
           4   0%                          - cl-find-if
           4   0%                           - cl-find
           4   0%                              cl-position
           4   0%                        - #<lambda -0x1667dbc0235e6d66>
           4   0%                         - when
           4   0%                          - if
           4   0%                             cl-loop
          16   0%               - #<lambda 0x5b7290e5d7>
          16   0%                - taxy-take-keyed
          16   0%                 - taxy-take-keyed
          16   0%                  - taxy-take-keyed
          16   0%                   - taxy-take-keyed
          16   0%                    - taxy-take-keyed
          16   0%                     - taxy-take-keyed
          16   0%                      - taxy-take-keyed
          16   0%                       - taxy-take-keyed
          16   0%                        - taxy-take-keyed
          12   0%                         - taxy-take-keyed
           8   0%                          - ement-room-list-key-people
           8   0%                           - ement--room-direct-p
           8   0%                            - #<compiled 0x1c375647c4cf9cae>
           8   0%                               seq-contains-p
           4   0%                          - taxy-take-keyed
           4   0%                           - ement-room-list-key-people
           4   0%                            - ement--room-direct-p
           4   0%                             - #<compiled 0x1c375647c4cf9cae>
           4   0%                                seq-contains-p
           4   0%                         - #<lambda -0x1667dbc0235e6d66>
           4   0%                          - when
           4   0%                             if
          36   0%             - taxy-magit-section-insert
          36   0%              - #<compiled -0x1e8784fb555b0cfa>
          36   0%               - #<compiled -0x1aa0cdce6d225f87>
          36   0%                - #<compiled -0x1e8784fb555b0cfa>
          36   0%                 - #<compiled -0x1fa4b3834370d217>
          32   0%                  - #<compiled -0x1e8784fb555b0cfa>
          32   0%                   - #<compiled 0x7c103c4459c9ae>
          28   0%                    - #<compiled -0x1e8784fb555b0cfa>
          24   0%                     - #<compiled -0x3bd58746dd0a204>
          24   0%                      - #<compiled 0x10d6bdcfa45d7357>
          12   0%                       - magit-insert-section--create
           8   0%                        - run-hook-with-args-until-success
           4   0%                         - magit-section-cached-visibility
           4   0%                          - magit-section-ident
           4   0%                             eieio-oref
           4   0%                           taxy-magit-section-visibility
           4   0%                        - apply
           4   0%                         - ement-room-list-section
           4   0%                            apply
           4   0%                     - magit-insert-section--create
           4   0%                        eieio-oset
           4   0%                    - #<compiled 0x10d6bdcfa45d7357>
           4   0%                       #<compiled 0x351f1af2b08050d>
           4   0%                    #<compiled 0x10d6bdcfa45d7357>
           8   0%             - taxy-sort
           8   0%              - taxy-mapc*
           8   0%               - taxy-mapc-taxys
           4   0%                - #<compiled 0x485719bda9c4691>
           4   0%                 - cl-sort
           4   0%                  - #<compiled -0xd2b365d0d882dc3>
           4   0%                   - #<compiled 0x1587f0a2f36a7ac6>
           4   0%                    - ement--room-unread-p
           4   0%                       map-nested-elt
           4   0%                - taxy-mapc-taxys
           4   0%                 - #<compiled 0x1f478977502fc02>
           4   0%                    cl-sort
           4   0%             - taxy-sort*
           4   0%              - taxy-mapc*
           4   0%               - taxy-mapc-taxys
           4   0%                - taxy-mapc-taxys
           4   0%                   #<compiled 0xde8c33a96b565f4>
           8   0%         - ement--auto-sync
           8   0%          - ement--sync
           8   0%           - ement-api
           8   0%            - apply
           8   0%               plz
           4   0%     - plz--coding-system
           4   0%        coding-system-from-name
         624   3% - command-execute
         535   2%  - funcall-interactively
         334   1%   - next-line
         242   1%    - line-move
          12   0%     - line-move-partial
           9   0%      - window-screen-lines
           5   0%       - window-inside-pixel-edges
           2   0%          window-edges
           4   0%         default-line-height
           3   0%      - default-line-height
           1   0%         default-font-height
           8   0%       default-line-height
         201   1%   - previous-line
         178   0%    - line-move
           4   0%     - line-move-partial
           4   0%        default-line-height
          89   0%  - byte-code
          89   0%   - read-extended-command
          89   0%    - read-extended-command-1
          35   0%     - completing-read-default
           7   0%        command-execute
         377   2% + ...
          26   0% + magit-section-post-command-hook
          17   0% + redisplay_internal (C function)
          12   0%   internal-timer-start-idle
           2   0% + magit-section-pre-command-hook

[phil-s]

I think you'll want to test 2212b38

[Rutherther]

Maybe I should've been more specific. Yes, I know about it, I've tested it. alphapapa made it when I told them about the lags. It makes it a bit better, since it doesn't lag when doing operations with emacs quickly, but after I am idle for only a second, it usually lags again, if not right after the second, then after few more seconds.

[phil-s]

You can, of course, (setq ement-room-list-update-idle-delay N) to some N which is more than the 1 second value currently specified.

If you have ement-room-list-auto-update enabled at all then it's going to Do Things periodically, but the above lets you decide how often that might happen.

[alphapapa]

There are a few fundamental issues here:

1. You're in many rooms. If they are active, that means the server is constantly sending new events as they arrive in the rooms.
2. Every time new events are received, the room list buffer is updated, if it exists.
3. Rendering a large number of rooms can be somewhat slow, depending on various factors. AFAIK the chief issue is that all of the values for all of the columns for all of the rooms must be realized so that their width can be calculated, so that the columns' widths can be calculated.
4. It's not clear whether the relevant Elisp code is being natively compiled. You said, "Tried from both Nix and Guix, both times with jit native compilation enabled," but you may not be aware that, e.g. Guix's Emacs packaging does not native-compile Elisp packages which are installed outside of Guix (an unfortunate decision IMO, which they have declined to change). So the relevant Elisp code here may be only byte-compiled, which may be reducing its performance compared to what I experience on my system.

So there are various possible ways to address these issues:

1. Improve the performance of taxy-magit-section. There might be some caching that could be implemented. That probably won't happen anytime soon, as I have other priorities for the near future. Another possibility would be to see if individual rooms could be updated, but AFAIK magit-section doesn't really support that, so it's unlikely to work; and it might take a few hours of work just to find out.
2. Increase the idle delay so the room buffer won't be updated unless you're really not using Emacs for a little while. Of course, that means it won't get updated as often, so it will be outdated sometimes.
3. Stop updating the room list automatically altogether. Either kill the buffer, or disable automatic updates.
4. Reduce the number of rooms you're in. Obviously that has side effects, so it would probably be a last resort.
5. The ement-tabulated-room-list library is still present. It may still work. It may work more quickly than the taxy-magit-section-based one, but it does no grouping. Maybe it could be an alternative for you.
6. Maybe someone could implement a vtable-based room list. It would also have no grouping features, but it might render more quickly.
7. You could potentially re-evaluate all of the relevant code so as to make all of the functions interpreted, so that the profiler would give a bit more information (i.e. not show functions as compiled, which obscures their purpose). Maybe there are some hot spots that could be optimized.

None of these are great options, but they're all I know of now.