Whoops outage yesterday
5 years 8 months ago - 5 years 8 months ago #4390
by mw0uzo
Whoops outage yesterday was created by mw0uzo
Power tripped out due to faulty toaster and server did not automatically boot due to a configuration change. Noticed a few hours later.
Last edit: 5 years 8 months ago by mw0uzo.
Please Log in or Create an account to join the conversation.
5 years 8 months ago #4391
by Alpha01
Replied by Alpha01 on topic Whoops outage yesterday
Since this long server outage, I encounter unusual and sporadic timeouts while submitting the data from RadLogger Pro. Of course, I've restarted all my systems, including my fiber optic router which manages my high speed Internet connection, but without improvement to the sporadic timeouts... Maybe something had not recovered well on the server side?
Please Log in or Create an account to join the conversation.
5 years 8 months ago #4392
by mw0uzo
Replied by mw0uzo on topic Whoops outage yesterday
Thanks for the information, there have been several reports of this and I have yet to track down the cause. I have noticed some increase in disk activity. Do the outages occur during busy times or randomly? It could be a spike in graph processing... I designed the data processing to spread itself out, but if its not working and a large number of stations are processed at the same time then that could cause sporadic outages while server experiences high mem usage and disk load.
Please Log in or Create an account to join the conversation.
5 years 8 months ago #4393
by Juzzie
Owner and operator of "southofhobart" monitoring stations.
Replied by Juzzie on topic Whoops outage yesterday
Maybe, if there is a spike in automated graph processing... back to "refresh" button?
Also, could there be a hacking attempt in progress? I have noticed at times there are 10s of thousands of Guests online (now forum home page - 5660 guests.) I noticed the same thing before the big hack.
Also, could there be a hacking attempt in progress? I have noticed at times there are 10s of thousands of Guests online (now forum home page - 5660 guests.) I noticed the same thing before the big hack.
Owner and operator of "southofhobart" monitoring stations.
The following user(s) said Thank You: mw0uzo
Please Log in or Create an account to join the conversation.
5 years 8 months ago - 5 years 8 months ago #4394
by Alpha01
Replied by Alpha01 on topic Whoops outage yesterday
Yes, it's happening completely randomly but quite often. I'm not sure if I got a very very delayed "200, OK" reply or if I didn't get any reply: Radlog reports only a "timeout". If I've enough time, I will try to setup a wireshark to figure out the reply time to each Radlog tcp submission.
Last edit: 5 years 8 months ago by Alpha01.
The following user(s) said Thank You: mw0uzo
Please Log in or Create an account to join the conversation.
5 years 7 months ago - 5 years 7 months ago #4395
by Alpha01
Replied by Alpha01 on topic Whoops outage yesterday
Hello,
Finally I could record one event using Wireshark.
It seems that the server completely miss the opening of a connection (port 52224 in the attached picture), the first attempt failed, but also the two following retries 3 and 9 seconds later. Of course as the connection failed to establish at the tcp level, no http request could be sent, reason whyradmon Radlogger Pro reported a timeout.
The previous one (port 52222) and the next one, 60s later (port 52226) were successful.
I let you investigate on your side if you could confirm that...
Kind regards,
PS: I noticed also several retransmission attempts at different level (often a missing ACK) and systematic RST to close a previous connection just before opening a new one (don't know the root cause).
Finally I could record one event using Wireshark.
It seems that the server completely miss the opening of a connection (port 52224 in the attached picture), the first attempt failed, but also the two following retries 3 and 9 seconds later. Of course as the connection failed to establish at the tcp level, no http request could be sent, reason why
The previous one (port 52222) and the next one, 60s later (port 52226) were successful.
I let you investigate on your side if you could confirm that...
Kind regards,
PS: I noticed also several retransmission attempts at different level (often a missing ACK) and systematic RST to close a previous connection just before opening a new one (don't know the root cause).
Last edit: 5 years 7 months ago by Alpha01.
The following user(s) said Thank You: mw0uzo
Please Log in or Create an account to join the conversation.
Moderators: Gamma-Man
Time to create page: 0.176 seconds