Buffer filling to 100% after about 36h

[staze]

Hi All,

Not sure what's going on, but since 3.9 (17/3/2019) after about 36 hours my buffer will fill to 100% (1000 items) and it will stop uploading to services. This is a Nano SD.

Not sure what's going on. I do have Weathercat connecting to it over port 22222, so maybe it's something with the new LOOP2 code (I can't see anyway to enable/disable that), or it might be a service I have running connecting to tempate.cgi every 30 minutes (that doesn't seem likely).

Any ideas? It's getting supper annoying to restart. =/

[staze]

What is super odd is everything seems fine for a while. Buffer sits at like 1 item for hours and hours. It's seemingly filling up overnight and I don't catch it until it's full and Weathercloud sends me an email saying my station stopped reporting. =(

[staze]

Wonder if it had to do with DB backup issue reported in other forum.

Now on 24/3/2019, guess we'll see. =)

[staze]

Still seeing issue, I think. Started after about 10 hours this time. May have been wifi network congestion or something, but things seem to have reset (system log showed the Nano SD reconnecting to the console), and several of my weather network services had failed and given up retrying.

Not sure what the heck is going on. =( But now I'm on a build that isn't listed on wiki... 25/3/2019 (Build 2018). So maybe it's got something fixed. =)

[ConligWX]

can your Nano speak with the outside world?

try pinging a few sites from an SSH session

[staze]

Yes. My WeatherCat connection stays active (seemingly). And I can login to web interface. :/

[admin]

Are you using Meteoplug cloud services?

[staze]

Are you using Meteoplug cloud services?

Not that I'm aware of.

[staze]

So, seems like this might be tied to Meteotemplate. For some reason, it is causing a "timeout in headers". This then happens repeatedly (I can find no reason why it's timing out. Everything SEEMS to work just fine), and the buffer fills up.

Would really like some assistance figuring out what's going on. This has only started since the last couple 3.9 builds. =/

[staze]

Occasionally, it'll succeed, and the buffer will drain.

Also of interest, latest log entry (right when it successfully connected to Meteotemplate):

system (25.03.2019 20:20:49): Access point operation disabled 86580 secs after boot. Not sure why the access point was enabled for 24 hours... there was no gap in my internet coverage. =/

Wonder if my Nano is getting confused when it's access point is active and can't figure out multihoming.

[ConligWX]

best to SSH to the Nano and start testing a few connections to outside websites and services. are you double NATted or through any high end firewall?

Also what DNS server are you using? I use cloudfare since it is by far the quickest responding DNS server for me.

DNS: 1.1.1.1 and 1.0.0.1

[ConligWX]

system (25.03.2019 20:20:49): Access point operation disabled 86580 secs after boot. Not sure why the access point was enabled for 24 hours... there was no gap in my internet coverage. =/

Likewise, though the script that checks the internet connection is test5m.sh. when I check this it runs a check against www.google.de and www.meteobridge.com afaik.

I found every 2-3 pings it would fail. I ended up using my ISP's DNS server IPs as a test rather than the failing DNS server pings.

lately it has behaved itself so I dont need to edit this file anymore, though I now have new network hardware and split my network into 2 VLANs. IoT devices (NEST, WebCams, phones, Security stuff on one VLAN, then rest on my default VLAN with Weather Station and a few PCs. The IoT Vlan cannot initiate a connection to the default VLAN.

I posted my initial findings on the connection issue here:

https://forum.qnap.com/viewtopic.php?f=5&t=147652

Pesonally I would SSH onto your Nano, the run the command below. keep running the command very quickly and see if you get packet loss.

ping -q -c 1 "www.google.de"

[staze]

Caught it in the middle of failing. It's up to 330 buffer items. Interestingly, it thinks all the services are working (except none of them have updated in nearly 40 minutes). So something is failing, not showing it's failed, and then causing stuff to back up.

I am ssh'd in. I don't see anything in logs that looks suspicious. It can contact everything. Buffer seems to fill up pretty quickly... up to 411 items now.

Is there any way to either go back to a previous 3.9 firmware, or disable the LOOP2 experimental code? This started with that release...

Rebooting the unit since it's stuck at this point. =( Otherwise I might as well set a cronjob to reboot it nightly. =(

[staze]

Hmm, interestingly, meteotemplate seems to indicate it has kept updating even though this screenshot seems to disagree.

Screen Shot 2019-03-26 at 7.39.24 AM.png (103.63 KiB) Viewed 3933 times

[staze]

lately it has behaved itself so I dont need to edit this file anymore, though I now have new network hardware and split my network into 2 VLANs. IoT devices (NEST, WebCams, phones, Security stuff on one VLAN, then rest on my default VLAN with Weather Station and a few PCs. The IoT Vlan cannot initiate a connection to the default VLAN.

My network is already configured for IoT stuff to be segregated. I'm using Unifi gear. Router is a USG, AP is a UAP-AC-HD, and Unifi switches and cloud controller. Meteobridge is within 10ft of AP. Always has nice healthy signal. It's on the IoT network.

And likewise, IoT stuff can't initiate contact to normal LAN.

DNS is just the USG for the IoT network (which forwards DNS inquiries to Cloudflare DNS). That hasn't changed since my initial install of the Meteobridge. My normal computer vlan uses a Pi-Hole.

I'm honestly at a loss. I can't figure out what's dying on the Meteobridge that's causing stuff to backup. I dislike just rebooting the Nano since it erases any evidence of the fault.

Would really like some help providing whatever is needed to Boris to help troubleshoot this. But right now, I'm running dry.

Another point of interest, and maybe completely unrelated. The RAM usage does slowly tick up over the MB running. Booting up it starts at mid-50% and by the time it fails it's up in the high 70%'s. Could just be DB caching or something, but it does make me wonder.