Chart Generation Taking a Long Time (or not happening at all)

[ryanleesipes]

Describe the bug
Generating charts for my inbox seem to take a very long time (I have yet to see the charts actually display). Have waited for charts to appear, assuming it is because of my large inbox. But have waited for likely an hour with the tab open at this point and still no charts. The spinner is spinning so I assume something is happening, but I'm not sure if I should continue to wait or if it's a bug,

To Reproduce

1. I click on the button to display the stats dashboard.
2. The top-line stats display, such as: Mails total, Mails unread, Mails received, etc
3. None of the charts display information. Just black where the charts should show up, the titles however are there.

Expected behavior
Expected the charts to appear and update as more information was processed, or the charts to show up faster.

Screenshots

System (please complete the following information):

• OS: elementaryOS 5.1.7
• Thunderbird Version 78.3.1 (64-bit)

[devmount]

This definitely is a bug. I have an inbox with ~30k emails which takes 13s to generate the charts. This is still quite long, so performance should definitely be improved, but never finishing chart generation is a bug.

Do you have multiple email accounts? Does the popup loading spinner finish (before clicking your account and opening the stats page)?

[ryanleesipes]

Just one account.

The spinner does not seem to stop if I let it sit before opening the page (in the pop over menu).

[devmount]

Ok, thank you for your feedback! I'll try to reproduce this and come back as soon as I have a solution.

[ryanleesipes]

I can look at the console or anywhere you think there might be useful information.

[devmount]

I set up a virtual environment with elementaryOS 5.1.7 and Thunderbird Version 78.3.1 (64-bit) and wasn't able to reproduce your issue. So I assume it must be something with your emails or folder structure.

Can you please do the following:

• open the console (add-ons > gear > debug add-ons > scroll to ThirdStats > Inspect)
• clear console (small trash icon on the upper left)
• open (or right click > reload if its already open) the Stats tab
• make a screenshot or copy any errors/warnings and paste it here

Thank you!

[devmount]

Plus:
I just released v0.4.0 with improved loading indication (you shouldn't see empty charts anymore now). Maybe this changes anything for you, @ryanleesipes?

[ryanleesipes]

Here is the screenshot you requested.

I do like the new indicator:

[devmount]

Thank you for the screenshot! I created a possible fix. To test it, please do the following:

1. Clone this repo git clone https://github.com/devmount/third-stats
2. Change into the created directory cd third-stats
3. Switch branch git checkout fix-infinite-loading
4. Open Thunderbird and uninstall ThirdStats, if you already installed it
5. Open Thunderbird > main menu > add-ons > gear menu > debug add-ons > load temporary add-on > choose the manifest file in the cloned repo's directory
6. Check, if charts are loaded now. If not, please create a screenshot of the console again.

Thank you very much @ryanleesipes for taking the time to test and debug this issue.

[ryanleesipes]

So two things to note (one might be the issue).

I installed Thunderbird on this machine via Flatpak so I wonder if that is causing issues. When attempting to load the temporary add-on I get the following message:

Extension is invalid
Loading locale file _locales/en/messages.json: Error: Error while loading 'file:///run/user/1000/doc/c28881a6/_locales/en/messages.json' (NS_ERROR_FILE_NOT_FOUND)

This could be because Flatpaks have their own filesystem, I'm troubleshooting.

[devmount]

Interesting, is that file actually existing on that location? Looks like the root directory of ThirdStats was not properly recognized... I don't have any experience with Flatpak, so thank you for troubleshooting!

[koobs]

I'm seeing the same issue on:

• Windows: 10 Pro x64 (2004, 19041.572)
• Thunderbird: 82.0b3 (64-bit) (Installed via AMO)
• Third-Stats: 0.5.0

Additional symptoms to report:

• The processing hangs after the same number of messages are parsed. In my case: 82,562 (now 82,564 after 2 new messages are received)
• I see the same errors in debug addons console output as @ryanleesipes

However, when I view the standard console output (not via debug addons), I see some additional errors that dont appear to show up during an 'addon debug' context. This is with the full-page third-stats loading, not just stats via the menu icon.

Full Console Log: console-export-2020-10-20_21-33-0.txt

Looking at the process stack (process explorer, see screenshot), the only potentially interesting thing I can see is en elevated number of calls to RegExprParser::* and configthreadlocale activity after the mail parsing process hangs, which continues until closing the third-stats tab. The number of calls to TargetNTUnmapViewofSection is at least two orders of magnitude higher too (baseline is <= 0.03 CPU when the issue is visible)

Additional information. Unsure if relevent or related:

• The processing (count) seems to restart at zero each time the toolbar icon is opened -> closed -> opened (may be intended)

[devmount]

Thank you @koobs for this detailed description and error analysis! 👏🏻 This definitely helps. Have you checked the possible fix I created? Does it change anything?

Thanks to your error log I can do some additional tests and see if I can find what's causing this issue.

The processing (count) seems to restart at zero each time the toolbar icon is opened -> closed -> opened (may be intended)

A click of the toolbar icon always reloads the popup content, so the counts are processed again - I can see if it's possible to implement a storage for already processed email/folder counts. If this is a feature you would like, you can create a feature request for that.