"Plot log file" page crashes after too many data are collected. #619

sebi5361 · 2024-02-24T18:31:31Z

sebi5361
Feb 24, 2024

I am logging data on a SD card.
I am plotting those data using the "Plot log file" webpage (/DG option):

It works perfectly till a certain moment in time when data must be too numerous as after clicking on the "Plot log file" I reach this page:

... that remains the same for a long time till it crashes:

In my situation this occurs after 6 parameters being logged every 30 seconds for about 2 days.
And changing the time interval in that webpage before it crashes doesn't prevent it from crashing anyway (I'll test with a URL command that fixes the time interval to see if it works though).

It would be nice to estimate a maximum time interval that can be plotted without crashing that webpage and to set it as a limit.
Additionally, it would be nice to have the possibility to select a different sampling period for the data being sent by MQTT and the one being logged on the SD card.

fredlcore · 2024-02-24T18:46:09Z

fredlcore
Feb 24, 2024
Maintainer

We have discussed this at length here: #539 and here: #542
In all our testings the crashes you mention do not occur (at this rather "short" interval).
Several people have requested a different log interval, but I don't see the rationale behind that: SD card logging was introduced before we had MQTT, so that you could have some kind of graphical display at least. Once you log externally, there is simply no use-case for SD card logging anymore, since storage capacity, access speed and analytical features will always be (much) less effective compared to any home automation system you will have to be using in order to work with MQTT. If you want to have your own kind of MQTT log interval, this could always be achieved with custom code snippets, and maybe I'll write one to show how this would be done at some point, but for a more general solution that just adds clutter in the configuration for most users, I see no justification...

0 replies

sebi5361 · 2024-02-24T20:15:00Z

sebi5361
Feb 24, 2024
Author

A look at the time interval in the picture above shows it's only two days.
48 h * 60 min * 2 samples/min * 6 simultaneous samples = 34560 samples.
For me it is not clear why you would spend time on interfacing an SD card adapter if you believe the SD card logging is an outdated feature. But reading the posts you mentioned I understand improving the code is somehow complicated.
Having that custom code snippet would be of great help.

0 replies

fredlcore · 2024-02-25T02:07:50Z

fredlcore
Feb 25, 2024
Maintainer

If you mean our previous discussion regarding the SD card adapter for the NodeMCU, that doesn't really cost me a lot of time compared to refactoring the code the way you suggested it (plus the knowledge of adding SD card functionality to a NodeMCU device can come in handy for other projects). I didn't say that SD card logging is outdated, I said that there is no use-case once you do external logging via MQTT. There are, however, a number of cases where that is not easily possible (for example when running at at site with no (permanent) internet connection, such as a holiday home) and where SD card logging is a viable option. But in that case you don't need to differentiate between SD and MQTT logging intervals.

As for your crashes, it's hard to diagnose from afar because as shown in the mentioned threads, we have done testing with far greater datasets without any problems: #539 (comment)
At the least I would need is a SerMo log before, during and after the request for the graph display. The only thing I could act upon is if there is actually a crash due to a bug in the code. If BSB-LAN, however, keeps on running, then it's a network/browser issue on your side.

0 replies

sebi5361 · 2024-02-25T03:17:19Z

sebi5361
Feb 25, 2024
Author

Yes I was referring to our previous discussion; I get your point.
My issue seems to be related to the browser type as with Chrome it crashes but with Firefox it works:

Thanks for BSB-LAN, a great piece of software!

0 replies

fredlcore · 2024-02-25T03:42:11Z

fredlcore
Feb 25, 2024
Maintainer

You're welcome :) - as for the browser crash in Chrome, I remember that sometimes users have reported issues with different browsers (have to check if it was always Chrome), but I wonder if it's just a timeout problem or something else. Could you measure after which time Chrome crashes? If it's a "round" number of seconds/minutes, it might be a setting that can simply be adjusted. I'm not sure if BSB-LAN could change something in the code to prevent that because there is a continuous stream of data coming from BSB-LAN to the browser...
EDIT: @DE-cr did tests with Chromium here without any problems: #539 (comment) - did you also use the filtering that limits to 10k lines or did you download the full-size log?

0 replies

DE-cr · 2024-02-25T08:44:57Z

DE-cr
Feb 25, 2024

If it works in Firefox, but crashes in Chrome, it's likely to be a browser issue, not a problem with the BSB-LAN code, and as you seem to have Firefox available, I'd just use that then.

What you could do to narrow the problem down is:

see if /D (instead of /DG) works alright
see if loading the /D output in the datalog viewer from the scripts directory crashes

How much RAM does Chrome have available on your system?

As for the separate log intervals for logging and MQTT: I do something similar by using custom code to send data via UDP to interested devices in my network, at a different interval from what I use for logging in BSB-LAN (different parameters, too) -> https://github.com/DE-cr/BSBmonCR, using "Variant 2". The logging in BSB-LAN I mostly use to analyze the boiler's behavior (and I re-configure the logging every once in a while), my custom one for "home automation" (well, mostly display) purposes. I very much appreciate that I can have the two separate.

P.s. Your two days log should be about 2.5 MB large. Mine cannot grow this big, as I have no SD card in my system. However, loading about 90,000 data samples (distributed over 7 parameters) in the external datalog viewer does not crash my current version of Chromium.

P.p.s. I'm not aware of a 10 KB filter in the current /DG implementation, only limiting it to a date range via the start/stop date controls.

2 replies

fredlcore Feb 25, 2024
Maintainer

It's not a 10kB, but 10k lines filter, as you mentioned in the linked comment. I have never checked upon that, just quoted the statement.

DE-cr Feb 25, 2024

In the current implementation, that only exists in the external viewer, which upon loading a log file limits the start date to about 10,000 data samples before the end date, for a faster initial plot display.

There's no such thing left in BSB-LAN's /DG. For that there's only a days limit for the initial display: #define DEFAULT_DAYS_TO_PLOT "3" in BSB_LAN_config.h. Iirc we didn't expect that much data with most log settings and three days to initially be displayed per default. Lowering that setting to "1" could also help here. To then plot the previous day, one should first set the end date to that value, then the start date to the same value. This could be used as a workaround to the problem with Chrome on the thread starter's system.

DE-cr · 2024-02-25T09:41:53Z

DE-cr
Feb 25, 2024

Does the "Learn more" link offered by Chrome upon "crashing" provide helpful information?

0 replies

sebi5361 · 2024-03-02T11:43:45Z

sebi5361
Mar 2, 2024
Author

Yes the "learn more" link offered by Chrome upon "crashing" provides helpful information!
https://support.google.com/chrome/answer/95669
... as I was able to successfully plot the log file in an Incognito window!
I might erase my Chrome history to see if the solution offered by Chrome works, but currently I am using firefox to plot the log file as I need to keep my Chrome history currently.

Yes the /D (instead of /DG) works alright and prints the log data directly in a Chrome webpage.

see if loading the /D output in the datalog viewer from the scripts directory crashes

I downloaded locally the log file using wget and then launched your BSBmonCR_log_viewer.html file on a local webserver in Chrome. Is it what you mean?
I got this output (same with Firefox):

1 reply

DE-cr Mar 2, 2024

The plot is not quite what I expected (obviously).
Could you post a shortened version of your D.csv file, e.g. the first 100 lines or so?

sebi5361 · 2024-03-02T14:14:54Z

sebi5361
Mar 2, 2024
Author

Here it is:
D100.csv

4 replies

DE-cr Mar 3, 2024

Hmm, your D100.csv loads okay with me, both in Firefox 123.0 and Chrome 120.0.6099.216:

DE-cr Mar 3, 2024

Could you please try again with a current version of https://github.com/fredlcore/BSB-LAN/blob/master/BSB_LAN/scripts/BSB-LAN_datalog-viewer.html? Your version must be at least a year old, as there's no date controls to set the range to plot.

sebi5361 Mar 3, 2024
Author

It works with the current version on my local server!!
(The version that doesn't work on my local server is the one in your GitHub repository)

DE-cr Mar 3, 2024

Okay, that explains it: BSBmonCR is a completely different program (to run on a second ESP32, collecting data from a BSB-LAN unit via UDP) from BSB-LAN, and uses a completely different log format.

sebi5361 · 2024-03-03T08:26:04Z

sebi5361
Mar 3, 2024
Author

Oh OK.
This later issue might be related to my local server then, and is a different topic. I will try to investigate it however.

I found the reason why I couldn't plot the log data in Chrome!
As clearing my history wasn't enough, I deactivated extensions one by one to find that Click and Read CNRS was responsible for the issue:
Click and Read CNRS, 3.13.0, Extension that offers links to resources from documentary identifiers

So strange to me that this innocent extension is responsible for such an issue...
However, without it I can successfully plot my log data in Chrome!

0 replies

fredlcore · 2024-06-05T16:04:16Z

fredlcore
Jun 5, 2024
Maintainer

Is there still something open for discussion here or can I close this topic?

2 replies

DE-cr Jun 5, 2024

Okay to close imo.

sebi5361 Jun 5, 2024
Author

ok to close

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"Plot log file" page crashes after too many data are collected. #619

{{title}}

Replies: 11 comments 9 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

"Plot log file" page crashes after too many data are collected. #619

Replies: 11 comments · 9 replies

fredlcore Feb 24, 2024 Maintainer

sebi5361 Feb 24, 2024 Author

fredlcore Feb 25, 2024 Maintainer

sebi5361 Feb 25, 2024 Author

fredlcore Feb 25, 2024 Maintainer

fredlcore Feb 25, 2024 Maintainer

sebi5361 Mar 2, 2024 Author

sebi5361 Mar 2, 2024 Author

sebi5361 Mar 3, 2024 Author

sebi5361 Mar 3, 2024 Author

fredlcore Jun 5, 2024 Maintainer

sebi5361 Jun 5, 2024 Author

Replies: 11 comments 9 replies

fredlcore
Feb 24, 2024
Maintainer

sebi5361
Feb 24, 2024
Author

fredlcore
Feb 25, 2024
Maintainer

sebi5361
Feb 25, 2024
Author

fredlcore
Feb 25, 2024
Maintainer

fredlcore Feb 25, 2024
Maintainer

sebi5361
Mar 2, 2024
Author

sebi5361
Mar 2, 2024
Author

sebi5361 Mar 3, 2024
Author

sebi5361
Mar 3, 2024
Author

fredlcore
Jun 5, 2024
Maintainer

sebi5361 Jun 5, 2024
Author