Delay in data availability from real-time to historical download

Hi Matthias,

Good question, well presented, deserves an answer! 

The downloadable CSVs are generated when the graphs are drawn, and the graphs are drawn when the server feels like it, after 30 minutes. Really, it is like that. Dan wrote some clever code to see how busy the server is at any given time. If the server is too busy it won't attempt to generate the graphs, and when it attempts graph generation it checks if the last time was 30 minutes or less. If less it skips it, if not it will generate the CSVs then build the graphs from the CSVs. Such as: submission comes in --> data stored --> is server busy? yes = stop / no --> is it less than 30 mins since last generation? yes = stop / no --> generate CSVs and graphs. I'll add a flow chart to the bottom of this post, that should make it easier to understand.

A couple of Three additional things regarding this - you can see your latest 30 submissions live in your control panel. This is handy for checking to see if your counter is submitting OK. The other thing is you can force graph refreshes via the API - here . The third thing is viewing the graphs or downloading the CSVs from the station page has no bearing on graph generation. That page simply displays what the server has already generated. The only ways are data submission logic and forcing a refresh.

You can force just one graph, or all of them. However the server busy logic still exists. So even with a force refresh, if the server is too busy it will gracefully stop.
This is the function to redraw all:

Function: forcegraphrefresh
Parameters: user
Description: Forces a refresh of the station's graphs.
Example: https://radmon.org/radmon.php?function=forcegraphrefresh&user=Simomax
Returns: 'Graphs successfully generated.' if successful, or 'Server is too busy. Try again another time.' if server load is high.

Forcing a refresh ignores the 30 minute delay between generations, and it is not recommended to use in an automated way. The function is more for diagnostics than actual use. When graphs are generated it really impacts server performance. It has to pull a load of data from the database, average, compress and smush it into several CSVs, then draw several PNGs, one from each CSV. This really uses some CPU and IO so isn't recommended to spam the command or automate it. I just timed a full refresh and it took 1 minute 27 seconds. And finally, that flowchart:

 

This thread prompted me to have a dig through the radmon logs. Due to what I found in the logs I have restricted some API function use. More on that here:  https://radmon.org/index.php/forum/radmon-org-news/1543-restriction-for-some-api-functions