Test of permanent per minute data storing on NANO SD and RPI

This section covers the Meteobridge PRO, NANO SD and Raspberry Pi units exclusively

Moderator: Mattk

Post Reply
User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 6459
Joined: Mon Oct 01, 2007 10:51 pm

Test of permanent per minute data storing on NANO SD and RPI

Post by admin »

I my attempt to give the Meteohub user base a migration path to switch over to Meteobridge without loosing important features, I am experimenting with a Meteobridge add-on that allows to permamently store per minute data. Meteobridge versions NANO SD, PRO and RPI are capable to store hourly data persistently, but not per minute data. Currently per minute data is held in a RAM DB only and just for the last 72 hours. This allows for detailed daily charts and exports but not for long-term storage.

Reason behind this limitation was price of SLC flash storage (about $20 for a 2 GB microSD card) and higher flash wear because of more often writes. With the introduction of cheaper SLC microSD cards from Western Digital (former SanDisk) the price burden is no longer there. You can now buy 16 GB of SLC flash (SDSDQED-016G-XI) for about $30 at mouser (https://www.mouser.com/ProductDetail/Sa ... ecqg%3D%3D).

To reduce wear, the aggregation of data changes within an hour is still handled by the RAM DB only. This avoids to have min/max of hour/day/month/year/all data be recomputed and stored in flash every time a new data point comes in. However, per minute data is now also loggerd in the DB on flash storage to make it long-term available.

As you might assume this needs quite some changes, like
* major rework of data handling
* menu to edit per minute data on the web interface
* making data imports also on per minute base (when data granularity allows and the user is willing to tolerate the longer import times this takes)

Before going public, I would like to get feedback from beta testers. So if you are running a NANO SD or RPI and are interested to give it a try, please drop me a note.


"Storage" row adds an estimate how many days/years of additional data the SD card might be able to store. This is computed from the daily increase of the DB size compared to available storage. Therefore numbers will go crazy when you store large files on the SD card or import data. When Meteobridge is in standard operation for a day or two, you can expect realistic estimates.
Bildschirmfoto 2020-11-15 um 13.05.15.png
Bildschirmfoto 2020-11-15 um 13.05.15.png (62.64 KiB) Viewed 222 times


Feature needs to be switched on here (please don't forget to press "Save and Apply".
Bildschirmfoto 2020-11-15 um 13.13.18.png
Bildschirmfoto 2020-11-15 um 13.13.18.png (48.44 KiB) Viewed 218 times


Editing per minute data is working exactly as with hourly data before. Changes done on the per minute data are not propagated to hourly/daily/monthly/yearly data. If you want to achive this you also have to do changes on the "Edit Hour Data" tab.
Bildschirmfoto 2020-11-15 um 13.02.12.png
Bildschirmfoto 2020-11-15 um 13.02.12.png (114.2 KiB) Viewed 222 times

User avatar
galfert
Gold Boarder
Gold Boarder
Posts: 221
Joined: Sun Jun 24, 2018 10:31 pm
Location: Orlando, FL

Re: Test of permanent per minute data storing on NANO SD and RPI

Post by galfert »

What happens after 17.5 years when the storage fills up? Is there a way to manage keeping say the record high/low but throw out the per minute data that is older than say 16 years?

Can you change the recording interval to every 5 minutes instead of every minute? That would right there extend the life to 87.5 years.
MR3020 v1
GW1000

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 6459
Joined: Mon Oct 01, 2007 10:51 pm

Re: Test of permanent per minute data storing on NANO SD and RPI

Post by admin »

Per minute data can be deleted separately and you can define which sensor to delete and the time frames being considered. By that you have plenty of control. All these operations on per minute data do not interfere with the higher aggregation levels of data.

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 6459
Joined: Mon Oct 01, 2007 10:51 pm

Re: Test of permanent per minute data storing on NANO SD and RPI

Post by admin »

Logging data on per minute schedule allows some really exciting charts. Attached you find an interactive chart in per-minute resolution for a complete month. The chart is HTML with embedded javascript and about 6 MB of size. NANO SD needs about 60 seconds to generate this chart, RPI 4 about 5 seconds.

Link for self inspection: https://admin.meteobridge.com/files/dem ... 01903.html

When you call it you get the complete view of the month. Selection of a time period is rather simple by just stating the "start" and "stop" time marker on the calling URL. Example below charts the complete March 2019 in per minute resolution:

Code: Select all

http://192.168.1.xxx/public/chart.cgi?chart=allinone-iso.chart&res=min&lang=en&start=20190301&stop=20190401



Bildschirmfoto 2020-11-21 um 15.38.48.png
Bildschirmfoto 2020-11-21 um 15.38.48.png (107.01 KiB) Viewed 144 times
By moving the ruler at the bottom you can zoom into any time frame down to inspection on minute level.


Bildschirmfoto 2020-11-21 um 15.37.14.png
Bildschirmfoto 2020-11-21 um 15.37.14.png (112.63 KiB) Viewed 144 times

User avatar
galfert
Gold Boarder
Gold Boarder
Posts: 221
Joined: Sun Jun 24, 2018 10:31 pm
Location: Orlando, FL

Re: Test of permanent per minute data storing on NANO SD and RPI

Post by galfert »

Now that is really neat. I really like the scrubber on the bottom of the graph. Works just like when you are editing video.
MR3020 v1
GW1000

Post Reply