Test of permanent per minute data storing on NANO SD and RPI

[admin]

I my attempt to give the Meteohub user base a migration path to switch over to Meteobridge without loosing important features, I am experimenting with a Meteobridge add-on that allows to permamently store per minute data. Meteobridge versions NANO SD, PRO and RPI are capable to store hourly data persistently, but not per minute data. Currently per minute data is held in a RAM DB only and just for the last 72 hours. This allows for detailed daily charts and exports but not for long-term storage.

Reason behind this limitation was price of SLC flash storage (about $20 for a 2 GB microSD card) and higher flash wear because of more often writes. With the introduction of cheaper SLC microSD cards from Western Digital (former SanDisk) the price burden is no longer there. You can now buy 16 GB of SLC flash (SDSDQED-016G-XI) for about $30 at mouser (https://www.mouser.com/ProductDetail/Sa ... ecqg%3D%3D).

To reduce wear, the aggregation of data changes within an hour is still handled by the RAM DB only. This avoids to have min/max of hour/day/month/year/all data be recomputed and stored in flash every time a new data point comes in. However, per minute data is now also loggerd in the DB on flash storage to make it long-term available.

As you might assume this needs quite some changes, like
* major rework of data handling
* menu to edit per minute data on the web interface
* making data imports also on per minute base (when data granularity allows and the user is willing to tolerate the longer import times this takes)

Before going public, I would like to get feedback from beta testers. So if you are running a NANO SD or RPI and are interested to give it a try, please drop me a note.


"Storage" row adds an estimate how many days/years of additional data the SD card might be able to store. This is computed from the daily increase of the DB size compared to available storage. Therefore numbers will go crazy when you store large files on the SD card or import data. When Meteobridge is in standard operation for a day or two, you can expect realistic estimates.

Bildschirmfoto 2020-11-15 um 13.05.15.png (62.64 KiB) Viewed 977 times

Feature needs to be switched on here (please don't forget to press "Save and Apply".

Bildschirmfoto 2020-11-15 um 13.13.18.png (48.44 KiB) Viewed 973 times

Editing per minute data is working exactly as with hourly data before. Changes done on the per minute data are not propagated to hourly/daily/monthly/yearly data. If you want to achive this you also have to do changes on the "Edit Hour Data" tab.

Bildschirmfoto 2020-11-15 um 13.02.12.png (114.2 KiB) Viewed 977 times

[galfert]

What happens after 17.5 years when the storage fills up? Is there a way to manage keeping say the record high/low but throw out the per minute data that is older than say 16 years?

Can you change the recording interval to every 5 minutes instead of every minute? That would right there extend the life to 87.5 years.

[admin]

Per minute data can be deleted separately and you can define which sensor to delete and the time frames being considered. By that you have plenty of control. All these operations on per minute data do not interfere with the higher aggregation levels of data.

[admin]

Logging data on per minute schedule allows some really exciting charts. Attached you find an interactive chart in per-minute resolution for a complete month. The chart is HTML with embedded javascript and about 6 MB of size. NANO SD needs about 60 seconds to generate this chart, RPI 4 about 5 seconds.

Link for self inspection: https://admin.meteobridge.com/files/dem ... 01903.html

When you call it you get the complete view of the month. Selection of a time period is rather simple by just stating the "start" and "stop" time marker on the calling URL. Example below charts the complete March 2019 in per minute resolution:

Code: Select all

http://192.168.1.xxx/public/chart.cgi?chart=allinone-iso.chart&res=min&lang=en&start=20190301&stop=20190401

Bildschirmfoto 2020-11-21 um 15.38.48.png (107.01 KiB) Viewed 899 times

By moving the ruler at the bottom you can zoom into any time frame down to inspection on minute level.

Bildschirmfoto 2020-11-21 um 15.37.14.png (112.63 KiB) Viewed 899 times

[galfert]

Now that is really neat. I really like the scrubber on the bottom of the graph. Works just like when you are editing video.

[Sean]

Hi there,

My name is Sean I have read this interesting post.
I have a new Raspberry PI 4 B using a Meteostick
the results are here:
http://skynetweather.com/html/meteo/index.php
I was using a Meteobridge Pro Red up until now

I have a Davis Vantage Pro 2 plus

I also use WeeWX
results are here:
http://skynetweather.com/html/weather34/index.php

I would be very interested to help you beta test your new idea !

Cheers Sean

[admin]

I would be very interested to help you beta test your new idea !

Please send MAC to "info(at)meteobridge.com".

[galfert]

What happens when you log at the minute level and you need to edit historical data? Before if you had to you edited on the hourly level. Seems that to edit with this new feature in place that then you'd be needing to edit on the per minute level. Unless there was perhaps a smart editing feature that let you change multiple entries at a time.

I log data to an mySQL database for safe keeping. I log at the 5 minute level. That to me offers a reasonable granularity and if I need to edit something it isn't tedious.

I think I'd like to test this new feature. MAC address coming up via email later today.

[admin]

For aggregation purposes editing is still done on the per hour base and changes are propagated to the higher aggregation levels to keep it all consistent.

Editing the per-minute data does not change hourly/daily/monthly/yearly data. View it as a separate data bucket that stands on its own. Imho editing per-minute data will not be much used apart from getting rid of spikes or artifacts, but the also avaibale delete functions might be very usefull when the microSD card gets filled up and you decide to get rid of per-minute data for certain sensors or certain time periods. Good thing is, that while deleting per-minute data - which is very storage consuming - all other time aggregations will not be impacted and still provide their data. You just drop the high-resolution version but keep the rest.

[36260]

This is quite interesting. Does that mean that per minute data aggregation is natively available is using meteobridge pro black or red version?

[admin]

I prefer to give the per-minute persistent storing some experience on the stronger NANO SD and RPI platforms (faster microSD, higher capacity, more processing power) before thinking about also enabling this on the MB PRO.

[36260]

Fair enough, thank you.

[Gyvate]

Hi Boris,
The with 14151 introduced per minute data storing and editing seems to work on the MB Pro.
I only noticed some "beauty error" - in the per minute section it says "edit hour data" - that
should be "edit minute data", shouldn't it ?
See attachment

[admin]

Thanks, will be fixed with next update.

[garybridge]

Hi,

Am I right in assuming that this could be a DIY upgrade for existing users of a Nano SD with 2GB cards?
Will the 16GB Card listed above from Mouser work in a Nano work now?
Could someone write a guide for performing this upgrade and minimising the risk of data loss?

Thanks