Can templates collide/overwrite other templates?

[SANdood]

I am now running numerous HTTP/FTP services (7 total), plus uploading to 16 weather/webcam services, plus running 4-6 apps that request data using template.cgi. ALl these on my new TL-MR3020V3 MeteoBridge.

Load-wise, these seem to work reasonably well, with load averages between 0.4-0.6 most of the time. Memory utilization leaves 19360kB free.

Code: Select all

System
Platform:	TL-MR3020V3   (no USB hub)
RAM:	60592 kB total, 19360 kB free (68% used)
SW Version:	Meteobridge 3.9 (Mar 26 2019, build 2030), FW 1.3
Uptime:	4 days, 16 hours, 55 minutes    Buffer: 5 items (1%)

However, from time to time, I will see failed transmits in the Live Data tab that display HTML/JSON that doesn't look appropriate - often 50-100 characters. Also, *SOMETIMES* one or more of my apps will (apparently) get garbage/unexpected data back from their template.cgi calls - most often at the top of the hour, and often for as long as 5 minutes after the hour (doing 60-second GETs of the template).

Also, the system log frequently reports `no template file "/var/run/meteohub/template" for doing write on socket 5551`, as you can see at the bottom of this snapshot:

Code: Select all

wnetd (28.03.2019 19:42:28): process started.
system (28.03.2019 19:42:28): start of logging process initiated.
logger (28.03.2019 19:42:29): historical data loaded (36 records).
logger (28.03.2019 19:42:29): data logger (version 5.1a, build 2030) started.
logger (28.03.2019 19:42:30): connect station 0 (Vantage via TCP/IP).
logger (28.03.2019 19:42:31): wakeup sent.
logger (28.03.2019 19:42:33): no need to update console date/time.
logger (28.03.2019 19:42:34): waiting for sensors (wind0, rain0, th0, thb0) to provide data.
logger (28.03.2019 19:42:34): services paused.
logger (28.03.2019 19:42:34): setup script executed.
logger (28.03.2019 19:42:34): internal logger activated.
logger (28.03.2019 19:42:34): syncing.
logger (28.03.2019 19:42:44): gettime request sent.
logger (28.03.2019 19:42:44): no Vantage time offset.
logger (28.03.2019 19:42:44): eeprom request sent.
logger (28.03.2019 19:42:45): rain collector size 0.01 inch per tick.
logger (28.03.2019 19:42:45): transmitter 1 active as station type #0 "ISS".
logger (28.03.2019 19:42:45): firmware version request sent.
logger (28.03.2019 19:42:45): firmware version 3.80 (supports LOOP2)
logger (28.03.2019 19:42:45): no reading of Vantage's internal logger.
logger (28.03.2019 19:42:50): sensors are now providing data.
logger (28.03.2019 19:42:50): services started.
logger (28.03.2019 23:12:21): no template file "/var/run/meteohub/template" for doing write on socket 5551
logger (02.04.2019 11:43:48): message above repeats 34 times.

From this, it would appear perhaps that `template.cgi` requests are persisted in a file, and perhaps even that they are all using the same exact file name.

Is it possible that this is the case, and that my templates are being unintentionally overwritten?

FWIW, some of the templates I am using takes 6-10 seconds to complete (timing from when I execute the GET to when I get back a response). I have shortened the templates as much as possible, but the problem appears to persist.

[Mattk]

Does all this activity have individual time slots or are they (or many of them) trying to be pushed at the same instant?

[SANdood]

Everything happens at a "random" number of seconds past the top of a minute. I can't control the services configured on the Meteobridge itself, and the each app selects a pseudo-random offset for its pull requests - I have no way to coordinate them other than to hard code them out of the way of each other (which will leave them still randomly colliding with the Meteobridge-based push services).

But if every call to template.cgi creates a file with the same name, there will inevitably be collisions. I would think it simple enough to have the template file name extended with the time (as in now() milliseconds-since-epoch) to avoid the issue altogether.

[admin]

Polling data in parallel via the template.cgi script result into the bugs you see. Reason is that template.cgi copies the payload into a file on the tempfs, starts a request on port 5551 to have the filled in data returned and then deletes the file again. Running two jobs in parallel will of course make trouble.

I will think about making the temp files more unique, but you guys are really pushing hard here. It is just a tiny computing device, have some mercy

[SANdood]

I/we *REALLY* appreciate everything you do for us with this amazing creation. I recommend Meteobridge to all of my weather-geek buddies primarily because of you and your team's commitment to the stability and functionality of this device.



I know that I am pushing the limits of the device (and of template.cgi), but there is no way to reliably assign time slots to different applications running on different platforms so as to avoid collisions.

I have broken my cgi requests into 3 separate sessions, requesting only the most volatile data every minute, things that change less frequently every 3-15 minutes, and stuff that only changes when the day changes into once or maybe twice per day. I've also shortened my previous approach of returning JSON-structured data to a simple comma-deliminated list of values (reduces from over 2500 characters of request/1500 characters of response to about 900 request/140 response - which has significantly reduced the time it takes template.cgi to return the data I need. I've also reduced the number of template.cgi clients by having all of my IoT applets get their data from one weather-server app, instead of each going directly to the Meteobridge for their data.

If you need a beta tester, I can reliably create collisions in my environment (by reverting back to the long-data request form). I will be travelling for the next few weeks without reliable internet, but I'll find a way to test it if you build it...

Thanks again!!!