Whoops outage yesterday

More
5 years 1 week ago - 5 years 1 week ago #4390 by mw0uzo
Power tripped out due to faulty toaster and server did not automatically boot due to a configuration change. Noticed a few hours later.
Last edit: 5 years 1 week ago by mw0uzo.

Please Log in or Create an account to join the conversation.

More
5 years 5 days ago #4391 by Alpha01
Replied by Alpha01 on topic Whoops outage yesterday
Since this long server outage, I encounter unusual and sporadic timeouts while submitting the data from RadLogger Pro. Of course, I've restarted all my systems, including my fiber optic router which manages my high speed Internet connection, but without improvement to the sporadic timeouts... Maybe something had not recovered well on the server side?

Please Log in or Create an account to join the conversation.

More
5 years 2 days ago #4392 by mw0uzo
Replied by mw0uzo on topic Whoops outage yesterday
Thanks for the information, there have been several reports of this and I have yet to track down the cause. I have noticed some increase in disk activity. Do the outages occur during busy times or randomly? It could be a spike in graph processing... I designed the data processing to spread itself out, but if its not working and a large number of stations are processed at the same time then that could cause sporadic outages while server experiences high mem usage and disk load.

Please Log in or Create an account to join the conversation.

More
5 years 2 days ago #4393 by Juzzie
Replied by Juzzie on topic Whoops outage yesterday
Maybe, if there is a spike in automated graph processing... back to "refresh" button?
Also, could there be a hacking attempt in progress? I have noticed at times there are 10s of thousands of Guests online (now forum home page - 5660 guests.) I noticed the same thing before the big hack.

Owner and operator of "southofhobart" monitoring stations.
The following user(s) said Thank You: mw0uzo

Please Log in or Create an account to join the conversation.

More
4 years 11 months ago - 4 years 11 months ago #4394 by Alpha01
Replied by Alpha01 on topic Whoops outage yesterday
Yes, it's happening completely randomly but quite often. I'm not sure if I got a very very delayed "200, OK" reply or if I didn't get any reply: Radlog reports only a "timeout". If I've enough time, I will try to setup a wireshark to figure out the reply time to each Radlog tcp submission.
Last edit: 4 years 11 months ago by Alpha01.
The following user(s) said Thank You: mw0uzo

Please Log in or Create an account to join the conversation.

More
4 years 11 months ago - 4 years 11 months ago #4395 by Alpha01
Replied by Alpha01 on topic Whoops outage yesterday
Hello,
Finally I could record one event using Wireshark.
It seems that the server completely miss the opening of a connection (port 52224 in the attached picture), the first attempt failed, but also the two following retries 3 and 9 seconds later. Of course as the connection failed to establish at the tcp level, no http request could be sent, reason why radmon Radlogger Pro reported a timeout.
The previous one (port 52222) and the next one, 60s later (port 52226) were successful.
I let you investigate on your side if you could confirm that...
Kind regards,

PS: I noticed also several retransmission attempts at different level (often a missing ACK) and systematic RST to close a previous connection just before opening a new one (don't know the root cause).
Attachments:
Last edit: 4 years 11 months ago by Alpha01.
The following user(s) said Thank You: mw0uzo

Please Log in or Create an account to join the conversation.

Moderators: Gamma-Man
Time to create page: 0.193 seconds
Powered by Kunena Forum
Everything's free. Please support us by considering a donation. Log in first!
Solar powered Raspberry Pi 4 server stats: CPU 33% Memory 14% Swap 17% CPU temp=59.4'C Uptime 40 Days