× RadLog

Edit Long Term Trend?

More
1 month 1 week ago - 1 week 3 days ago #7024 by GetSirius
Wondering if there is a way to edit the data for Long Term Trend graphs. I have one running for several years that has 1 very high spike that I would like to remove so that the remainder of the graph is useable. Any suggestions? Thank you in advance. 
Here is a sample graph:



 

Mineral Point radiation/lightning monitor: getsirius.com/radiation
Last edit: 1 week 3 days ago by GetSirius.

Please Log in or Create an account to join the conversation.

More
3 weeks 6 days ago #7027 by Simomax
Replied by Simomax on topic Edit Long Term Trend?
Have a look at your my documents\radlog folder. In there you should find longtermtrenddata.txt and longtermtrenddata.xml. I think the text file is the actual data and the xml are the coords for plotting the graph png. I have never edited either of these files, so I can't be sure, but if I were to have a go at changing the long term graph I would simply edit the data in the text file and wait for the next time radlog.exe generates the graphs and see if that worked, or the changes were reverted somehow. It would be prudent to back up your files too. Maybe the whole radlog folder.

I could be wrong, but I think the the process works by averaging the entire graph period, or day - it may be what the graph refresh period is, or daily. I say this as in my longtermtrenddata.txt the records cover 24 hours, which is what I have my graphing period set to. So it gets the average for the day, also the minimum and maximum for the period, calculates µSv/hr and mR/hr, and their minimum and maximum, and then writes that to longtermtrenddata.txt. Then radlog.exe uses the text file data to create (or maybe radlog.exe does) and creates the xml file, from which the graph is plotted. Dan would have a much better idea, if he can remember.

If you edit the text file and your changes are overwritten back to the originals then you could try by closing radlog.exe down before making the changes in case the data is already in memory and generated from that. If that fails then there is a good chance the data is generated from all of the individual day's (or logging period) data, in which case tracking down the actual day and editing the files for that day (graphdata.xml I think).
The following user(s) said Thank You: GetSirius

Please Log in or Create an account to join the conversation.

More
3 weeks 4 days ago - 3 weeks 4 days ago #7028 by Simomax
Replied by Simomax on topic Edit Long Term Trend?
I tried editing my own longtermtrenddata.txt and whilst the changes did stick, the graph appears to be unchanged. I have a spike that I have tracked down to 2023-10-16 and changed that line in longtermtrenddata.txt, but didn't affect my graph. I'm going to try and remove (move to another location for testing) all the data for that whole day and see if that updates the graph. I'm starting to think that longtermtrenddata.txt may simply be appended to with the latest day's data, or period's data. I'll post back tomorrow with the results of removing that entire day.

Edit: Can't do that as I have already archived off everything prior to 2024, so the long term graph can't be generated by the day files. It may be that longtermtrenddata.xml contains the data for the graph. I'll try editing that and see what happens!

Edit 2: Not sure I can do that either.... 
It seems the xml file doesn't contain actual dates, only some number (which I think relates to graph plotting coords), so not sure how to proceed further with this. It's something I'd like to know.
In the xml the data is like the sample below. The XValue is apparently the date, so I guess it is a matter of working out the date, then change that line. Maybe...
<Chart BackColor="Black">
  <Series>
    <Series Name="CPM mean" Legend="Legend" XValueType="DateTime" ChartType="Line" ChartArea="Main graph" Color="Blue">
      <Points>
        <DataPoint XValue="42464.783584247685" YValues="22.3565768148437" />
        <DataPoint XValue="42466.706572662035" YValues="22.3866796016693" />
        <DataPoint XValue="42467.706530567128" YValues="22.0776149352392" />
        <DataPoint XValue="42468.706487083335" YValues="21.9956476238039" />
        <DataPoint XValue="42470.414000775461" YValues="22.8366131308677" />
        <DataPoint XValue="42471.4139475" YValues="22.4055923614237" />
        <DataPoint XValue="42472.413909629628" YValues="23.2995710770289" />
        <DataPoint XValue="42473.413866365743" YValues="22.5791956146558" />
        <DataPoint XValue="42475.088463888889" YValues="22.696176243955" />
        <DataPoint XValue="42476.088421006942" YValues="23.0805163171556" />
        <DataPoint XValue="42478.445493634259" YValues="22.8865909423987" />
Last edit: 3 weeks 4 days ago by Simomax.

Please Log in or Create an account to join the conversation.

More
3 weeks 4 days ago - 3 weeks 4 days ago #7029 by Simomax
Replied by Simomax on topic Edit Long Term Trend?
I found the line with the spike in my longtermtrenddata.xml and I can edit it there, but I don't really want to yet without understanding how the date numbering works. This is the line in my longtermtrenddata.xml:
<DataPoint XValue="45216.238533958334" YValues="313" />
I could just edit it and see what happens, but that wouldn't help in the future if I wanted to do it again. I may have to ask Dan and see if he remembers how he coded this in radlog.exe.
One thing I noticed is the XML seems to have nine sections to it, so the maximums for CPM, µSv/hr and mR/hr would have to be edited for completeness. The sections are CPM mean, µSv/hr mean, mR/hr mean, CPM max, µSv/hr max, mR/hr max, CPM min, µSv/hr min, mR/hr min in that order.
Last edit: 3 weeks 4 days ago by Simomax.

Please Log in or Create an account to join the conversation.

More
3 weeks 4 days ago #7030 by Bert490
Replied by Bert490 on topic Edit Long Term Trend?
Good sleuthing on the data spike!  I tried this and had success.  I'm using WIndows, so I opened longtermtrenddata.xml in a spreadsheet to locate the offending max value, and got the corresponding XValue.  Searching on that, found all 9 YValues, 6 of which needed editing (max and mean, not min).  I tried unsuccessfully to edit/save the .xml file: "Cannot save or export XML data.  The XML maps in this workbook are not exportable".  And trying to save in an older (2003) format .xml worked but was 30x bigger and unreadable by Radlog.  What worked was just editing longtermtrenddata.txt.  Radlog apparently re-builds the .xml file from the .txt file.

Note to anyone wanting to 'clean up' spikes that don't belong, if you notice it in your recent data, Radlog Professional makes it very easy to delete data points *as long as you catch it within 1 week*.  On the Radmon.org tab enter a number of hours up to 168, and enter the value above which to clip, then click Delete.  Longer than 1 week calls for the manual editing of 6 x the number of unwanted values.

Please Log in or Create an account to join the conversation.

More
3 weeks 3 days ago #7031 by Simomax
Replied by Simomax on topic Edit Long Term Trend?
@Bert490, I tried editing the longtermtrenddata.txt and whilst the changes stuck in the text file, it had no effect on the graph itself, nor longtermtrenddata.xml, which the graph is actually plotted from. I had a chat with Dan last night about this and he said this about the longtermtrenddata.txt: "I think its a breakdown of the long term measurements". He also mentioned that this was done a long, long time ago and without him digging into the code he can't remember much about it.

@All, Anyway, I had a poke and a prod about and have found that editing the longtermtrenddata.xml does push the changes to the graph. I edited my longtermtrenddata.xml last night and come 07:15 this morning it did make the changes to the actual graph png.

Changing the xml file is fairly easy, however, finding the spike that you want to change isn't so easy. The only way I could find it was by simply looking at the xml line by line until I found the correct 'CPM max'. I then edited that line to remove the spike (YValues). Then I copied the DateTime (XValue) and searched for it in my text editor to find the 'µSv/hr max' and 'mR/hr max' for that DateTime and edited them also. Upon saving the longtermtrenddata.xml my text editor (Notepad++) hung for a couple of minutes, but eventually freed up and saved the changes. Then I waited until the graphs were next generated the morning after and the changes could be seen in the graph. Job done!

There are some caveats though.
  • Finding the data to edit in the longtermtrenddata.xml isn't easy and I could only find the line I wanted by actually looking at the 'CPM max' on every line until I found it.
  • I'm guessing the 'µSv/hr max' and 'mR/hr max' are irrelevant if the graphs are set to use CPM, but are probably important if the graph uses µSv/hr or mR/hr.
  • When saving the xml file my text editor locked up for a couple of minutes. I suspect this is because radlog.exe is keeping the file open, but closing every so often. Closing down radlog.exe may sort this and allow the file to be saved instantly.
  • You may have to clear your browser cache, or perform a CTRL + Refresh on the page to force the new graph png to display on the web.
From chatting to Dan about this, the process of plotting the long term trend graph is as follows:
  1. Every day (or graph logging period - still not confirmed, I forgot to ask) 9 lines are written to longtermtrenddata.xml with the day's (or the period's) CPM mean, CPM max, CPM min, µSv/hr mean, µSv/hr max, µSv/hr min, mR/hr mean, mR/hr max, mR/hr min (YValues)
  2. The DateTime (XValue) is serialized using .Net to store the date as some stupid number.
  3. The xml is then read back in to radlog.exe, the data is then deserialized by .Net and then plotted to the graph
So in a nutshell to edit the long term trend graph, simply edit the longtermtrenddata.xml, make the changes, save the file, then wait for radlog.exe to plot a new graph on it's next graph plotting period.

Please Log in or Create an account to join the conversation.

Moderators: Gamma-Man
Time to create page: 0.186 seconds
Powered by Kunena Forum
Everything's free. Please support us by considering a donation. Log in first!
Solar powered Raspberry Pi 4 server stats: CPU 56% Memory 14% Swap 41% CPU temp=61.8'C Uptime 44 Days