Test of permanent per minute data storing on NANO SD and RPI

This section covers the Meteobridge PRO, NANO SD and Raspberry Pi units exclusively

Moderator: Mattk

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 6524
Joined: Mon Oct 01, 2007 10:51 pm

Test of permanent per minute data storing on NANO SD and RPI

Post by admin »

I my attempt to give the Meteohub user base a migration path to switch over to Meteobridge without loosing important features, I am experimenting with a Meteobridge add-on that allows to permamently store per minute data. Meteobridge versions NANO SD, PRO and RPI are capable to store hourly data persistently, but not per minute data. Currently per minute data is held in a RAM DB only and just for the last 72 hours. This allows for detailed daily charts and exports but not for long-term storage.

Reason behind this limitation was price of SLC flash storage (about $20 for a 2 GB microSD card) and higher flash wear because of more often writes. With the introduction of cheaper SLC microSD cards from Western Digital (former SanDisk) the price burden is no longer there. You can now buy 16 GB of SLC flash (SDSDQED-016G-XI) for about $30 at mouser (https://www.mouser.com/ProductDetail/Sa ... ecqg%3D%3D).

To reduce wear, the aggregation of data changes within an hour is still handled by the RAM DB only. This avoids to have min/max of hour/day/month/year/all data be recomputed and stored in flash every time a new data point comes in. However, per minute data is now also loggerd in the DB on flash storage to make it long-term available.

As you might assume this needs quite some changes, like
* major rework of data handling
* menu to edit per minute data on the web interface
* making data imports also on per minute base (when data granularity allows and the user is willing to tolerate the longer import times this takes)

Before going public, I would like to get feedback from beta testers. So if you are running a NANO SD or RPI and are interested to give it a try, please drop me a note.


"Storage" row adds an estimate how many days/years of additional data the SD card might be able to store. This is computed from the daily increase of the DB size compared to available storage. Therefore numbers will go crazy when you store large files on the SD card or import data. When Meteobridge is in standard operation for a day or two, you can expect realistic estimates.
Bildschirmfoto 2020-11-15 um 13.05.15.png
Bildschirmfoto 2020-11-15 um 13.05.15.png (62.64 KiB) Viewed 978 times


Feature needs to be switched on here (please don't forget to press "Save and Apply".
Bildschirmfoto 2020-11-15 um 13.13.18.png
Bildschirmfoto 2020-11-15 um 13.13.18.png (48.44 KiB) Viewed 974 times


Editing per minute data is working exactly as with hourly data before. Changes done on the per minute data are not propagated to hourly/daily/monthly/yearly data. If you want to achive this you also have to do changes on the "Edit Hour Data" tab.
Bildschirmfoto 2020-11-15 um 13.02.12.png
Bildschirmfoto 2020-11-15 um 13.02.12.png (114.2 KiB) Viewed 978 times

User avatar
galfert
Gold Boarder
Gold Boarder
Posts: 271
Joined: Sun Jun 24, 2018 10:31 pm
Location: Orlando, FL

Re: Test of permanent per minute data storing on NANO SD and RPI

Post by galfert »

What happens after 17.5 years when the storage fills up? Is there a way to manage keeping say the record high/low but throw out the per minute data that is older than say 16 years?

Can you change the recording interval to every 5 minutes instead of every minute? That would right there extend the life to 87.5 years.
Meteobridge RPI | GW1000

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 6524
Joined: Mon Oct 01, 2007 10:51 pm

Re: Test of permanent per minute data storing on NANO SD and RPI

Post by admin »

Per minute data can be deleted separately and you can define which sensor to delete and the time frames being considered. By that you have plenty of control. All these operations on per minute data do not interfere with the higher aggregation levels of data.

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 6524
Joined: Mon Oct 01, 2007 10:51 pm

Re: Test of permanent per minute data storing on NANO SD and RPI

Post by admin »

Logging data on per minute schedule allows some really exciting charts. Attached you find an interactive chart in per-minute resolution for a complete month. The chart is HTML with embedded javascript and about 6 MB of size. NANO SD needs about 60 seconds to generate this chart, RPI 4 about 5 seconds.

Link for self inspection: https://admin.meteobridge.com/files/dem ... 01903.html

When you call it you get the complete view of the month. Selection of a time period is rather simple by just stating the "start" and "stop" time marker on the calling URL. Example below charts the complete March 2019 in per minute resolution:

Code: Select all

http://192.168.1.xxx/public/chart.cgi?chart=allinone-iso.chart&res=min&lang=en&start=20190301&stop=20190401



Bildschirmfoto 2020-11-21 um 15.38.48.png
Bildschirmfoto 2020-11-21 um 15.38.48.png (107.01 KiB) Viewed 900 times
By moving the ruler at the bottom you can zoom into any time frame down to inspection on minute level.


Bildschirmfoto 2020-11-21 um 15.37.14.png
Bildschirmfoto 2020-11-21 um 15.37.14.png (112.63 KiB) Viewed 900 times

User avatar
galfert
Gold Boarder
Gold Boarder
Posts: 271
Joined: Sun Jun 24, 2018 10:31 pm
Location: Orlando, FL

Re: Test of permanent per minute data storing on NANO SD and RPI

Post by galfert »

Now that is really neat. I really like the scrubber on the bottom of the graph. Works just like when you are editing video.
Meteobridge RPI | GW1000

Sean
Fresh Boarder
Fresh Boarder
Posts: 1
Joined: Sun Nov 29, 2020 3:32 pm

Re: Test of permanent per minute data storing on NANO SD and RPI

Post by Sean »

Hi there,

My name is Sean I have read this interesting post.
I have a new Raspberry PI 4 B using a Meteostick
the results are here:
http://skynetweather.com/html/meteo/index.php
I was using a Meteobridge Pro Red up until now

I have a Davis Vantage Pro 2 plus

I also use WeeWX
results are here:
http://skynetweather.com/html/weather34/index.php

I would be very interested to help you beta test your new idea !

Cheers Sean :-)

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 6524
Joined: Mon Oct 01, 2007 10:51 pm

Re: Test of permanent per minute data storing on NANO SD and RPI

Post by admin »

Sean wrote:
Sun Nov 29, 2020 3:40 pm
I would be very interested to help you beta test your new idea !
Please send MAC to "info(at)meteobridge.com".

User avatar
galfert
Gold Boarder
Gold Boarder
Posts: 271
Joined: Sun Jun 24, 2018 10:31 pm
Location: Orlando, FL

Re: Test of permanent per minute data storing on NANO SD and RPI

Post by galfert »

What happens when you log at the minute level and you need to edit historical data? Before if you had to you edited on the hourly level. Seems that to edit with this new feature in place that then you'd be needing to edit on the per minute level. Unless there was perhaps a smart editing feature that let you change multiple entries at a time.

I log data to an mySQL database for safe keeping. I log at the 5 minute level. That to me offers a reasonable granularity and if I need to edit something it isn't tedious.

I think I'd like to test this new feature. MAC address coming up via email later today. :D
Meteobridge RPI | GW1000

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 6524
Joined: Mon Oct 01, 2007 10:51 pm

Re: Test of permanent per minute data storing on NANO SD and RPI

Post by admin »

For aggregation purposes editing is still done on the per hour base and changes are propagated to the higher aggregation levels to keep it all consistent.

Editing the per-minute data does not change hourly/daily/monthly/yearly data. View it as a separate data bucket that stands on its own. Imho editing per-minute data will not be much used apart from getting rid of spikes or artifacts, but the also avaibale delete functions might be very usefull when the microSD card gets filled up and you decide to get rid of per-minute data for certain sensors or certain time periods. Good thing is, that while deleting per-minute data - which is very storage consuming - all other time aggregations will not be impacted and still provide their data. You just drop the high-resolution version but keep the rest.

36260
Expert Boarder
Expert Boarder
Posts: 80
Joined: Sun Feb 17, 2008 11:17 am
Location: Saint Pierre de Jards - France

Re: Test of permanent per minute data storing on NANO SD and RPI

Post by 36260 »

This is quite interesting. Does that mean that per minute data aggregation is natively available is using meteobridge pro black or red version?
Davis VP2 + Solar - Nano SD

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 6524
Joined: Mon Oct 01, 2007 10:51 pm

Re: Test of permanent per minute data storing on NANO SD and RPI

Post by admin »

I prefer to give the per-minute persistent storing some experience on the stronger NANO SD and RPI platforms (faster microSD, higher capacity, more processing power) before thinking about also enabling this on the MB PRO.

36260
Expert Boarder
Expert Boarder
Posts: 80
Joined: Sun Feb 17, 2008 11:17 am
Location: Saint Pierre de Jards - France

Re: Test of permanent per minute data storing on NANO SD and RPI

Post by 36260 »

Fair enough, thank you.
Davis VP2 + Solar - Nano SD

Gyvate
Senior Boarder
Senior Boarder
Posts: 49
Joined: Thu May 14, 2020 4:36 pm

Re: Test of permanent per minute data storing on NANO SD and RPI

Post by Gyvate »

Hi Boris,
The with 14151 introduced per minute data storing and editing seems to work on the MB Pro. :)
I only noticed some "beauty error" - in the per minute section it says "edit hour data" - that
should be "edit minute data", shouldn't it ? 8)
See attachment
Attachments
MB-Pro_14151_minute-data.JPG
MB-Pro_14151_minute-data.JPG (213.61 KiB) Viewed 399 times
WH4000SE 1.5.4/GW1000 1.6.3/HP1000SE Pro 1.6.9
2 x Meteobridge Pro [B+R] 5.1 (Jan 14 2021, build 14236/271),RPi4-5.1/2069
Ecowitt 5763, 5764; WU ISAARB3; ISAARB22; Weathercloud 3011399141; http://meshka.eu/meteo/template - http://meshka.eu/Weather34

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 6524
Joined: Mon Oct 01, 2007 10:51 pm

Re: Test of permanent per minute data storing on NANO SD and RPI

Post by admin »

Thanks, will be fixed with next update.

garybridge
Senior Boarder
Senior Boarder
Posts: 47
Joined: Tue Nov 05, 2019 7:04 pm
Contact:

Re: Test of permanent per minute data storing on NANO SD and RPI

Post by garybridge »

Hi,

Am I right in assuming that this could be a DIY upgrade for existing users of a Nano SD with 2GB cards?
Will the 16GB Card listed above from Mouser work in a Nano work now?
Could someone write a guide for performing this upgrade and minimising the risk of data loss?

Thanks
Gary Bridge MIET, MIPS.

https://whaleybridge-weather.com

Post Reply