Changes on MB PRO data storage with release 3.0

This section covers the Meteobridge PRO units exclusively

Moderator: Mattk

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 4965
Joined: Mon Oct 01, 2007 10:51 pm

Changes on MB PRO data storage with release 3.0

Post by admin » Tue Jun 28, 2016 11:46 pm

As you can see from many postings regarding MB PRO problems in this forum,
there is a constant pattern that internal flash storage (realized by an internal
USB stick) tends to get faulty over time. It happens not to all systems, but
more are affected than I feel is acceptable.

That made us replacing the USB storage by a more robust one. We conducted
wear out tests etc to pick a suitable brand and model. While expectation was that
this should solve the issue completely, we again see again a few systems failing on
flash storage.

To have this reliability issue finally solved, I don't see an alternative to dramatically
reduce wear of the flash by reducing write intervals from "per minute" to
"per hour". Downside is that for passed days the database will only provide
data down to a hourly level (min/avg/max) for all sensors. Min/max times will stay
precise to the minute. For the last 24 hours data will also be kept in RAM and
will stay available on a per minute level.

This design change will reduce wear on flash by factor 60 which means when USB
stick works reliable today for a lets say 3 month minimum, this will be 15 years now.
I can imagine that you won't be happy that an initially designed data resolution is
reduced, but I don't know a better solution to bring back reliability.

Current version 2.9 is still archiving data on "per minute" mode. Next release 3.0
will switch to "per hour", so you can stay on "per minute" if some reasons do force you.

Thank you for your understanding and sorry for all the hassle.

xcom
Senior Boarder
Senior Boarder
Posts: 46
Joined: Mon Jun 16, 2014 5:11 am

Re: Changes on MB PRO data storage with release 3.0

Post by xcom » Tue Jul 05, 2016 10:27 pm

Boris,

Are you open to suggestions?

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 4965
Joined: Mon Oct 01, 2007 10:51 pm

Re: Changes on MB PRO data storage with release 3.0

Post by admin » Wed Jul 06, 2016 12:45 am

yes, of course.

frogfoot
Junior Boarder
Junior Boarder
Posts: 34
Joined: Thu Feb 25, 2016 11:06 am

Re: Changes on MB PRO data storage with release 3.0

Post by frogfoot » Fri Jul 08, 2016 8:29 pm

how about an option to save the database to a remote folder share? or the possibility of hosting the database on a remote system such as a NAS, and just using SQL to update/retrieve data as required, therefore offloading much of the processing as well.

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 4965
Joined: Mon Oct 01, 2007 10:51 pm

Re: Changes on MB PRO data storage with release 3.0

Post by admin » Sat Jul 09, 2016 1:16 am

looks like a rather fragile setup where assumptions about a NAS in the LAN needs to be made.
I am currently working a solution that might be give per minute data on last 36 hours and
will move to hourly data beyond that. I don't see that this will kill typical use cases and it
will not depend from additional infrastructure being there or not.

xcom
Senior Boarder
Senior Boarder
Posts: 46
Joined: Mon Jun 16, 2014 5:11 am

Re: Changes on MB PRO data storage with release 3.0

Post by xcom » Sun Jul 10, 2016 4:22 pm

Boris,

I am going to email you with a more detail idea and other options.
But my question is... Why couldn't you create a tmpfs out of memory for temporary storage? This file could than be copied to persistent storage in which ever intervals you would like.
This is very common practice with embedded devices.

Mattk
Gold Boarder
Gold Boarder
Posts: 275
Joined: Mon Sep 22, 2014 3:24 am

Re: Changes on MB PRO data storage with release 3.0

Post by Mattk » Sun Jul 10, 2016 11:26 pm

What ever is put in place the over riding end result is that the unit must remain totally self contained and not reliant on any form of external devices, NAS units etc

bubulino
Fresh Boarder
Fresh Boarder
Posts: 13
Joined: Tue Mar 30, 2010 3:24 pm
Location: Schwellbrunn Schweiz

Re: Changes on MB PRO data storage with release 3.0

Post by bubulino » Mon Jul 11, 2016 2:33 pm

Mattk wrote:
What ever is put in place the over riding end result is that the unit must
remain totally self contained and not reliant on any form of external devices,
NAS units etc
Absolutely, because some systems work in 3G/4G environments.

Best regards, Markus

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 4965
Joined: Mon Oct 01, 2007 10:51 pm

Re: Changes on MB PRO data storage with release 3.0

Post by admin » Tue Jul 12, 2016 11:33 pm

I just released new update of 3.0 that holds data up to 36 hours in a DB on RAM disk. This DB holds data in per minute resolution .
Data older than 36 hours is kept on internal USB storage, but only goes down in resolution onto an per hour detail.
Doing so average/min/max values are kept per hour, so you don't loose detail about min/max values and time stamp on this.
While this sounds easy, interlocking two databases with overlapping time frames is rather complicated, but this is
something I have to master. :oops:

In parallel I do research on SLC USB storage, which will bring wear margins up by at least factor 10. Unfortunately, these
are only provided by industrial non-consumer companies like ATP and they cost more than 5 times the consumer USB sticks
do cost. Going the new way of not keeping minute data forever, a 4 GB internal storage might do, which could bring true SLC
flash into an affordable reach. I have ordered some for testing... we ll see.

I hope not to tax your patience too much and to get this finally solved soon.

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 4965
Joined: Mon Oct 01, 2007 10:51 pm

Re: Changes on MB PRO data storage with release 3.0

Post by admin » Tue Aug 02, 2016 10:17 am

As using SLC USB storage from ATP is a real expensive option I tried to get some more insight how much GB are needed based on the new hourly storage schema.

My experiments show that the database has about 1MB of size, when storing 20 days of data with 46 "atomic" sensors, which is is 50 KB a day. Atomic sensors are "th0temp", "th0hum", "th0dew", "th0lowbat", "th0signal", etc. By that each sensor variable you can use in templates is such an atomic sensor. in my standard setup of an ISS being readout by MB PROs internal RF receiver including the virtual data sensors, I have 46 atomic sensors:

Code: Select all

rain0rate, rain0total, rain0lowbat, rain0signal_hour, wind0dir, wind0wind, wind0avgwind, wind0chill, wind0lowbat, wind0signal, th0temp, th0hum, th0dew, th0heatindex, th0lowbat, th0signal, sol0rad, sol0lowbat, sol0signal, uv0index, uv0lowbat, uv0signal, data10num, data11num, data12num, data13num, data15num, data16num, data14num, data17num, data18num, data19num, thb9temp, thb9hum, thb9dew, thb9press, thb9seapress, thb9lowbat, thb0temp, thb0hum, thb0dew, thb0press, thb0seapress, thb0lowbat, thb0signal, data20num
Assuming that you want to store 10 years of data with 150 atomic sensor (to cover even very large sensor arrays) we have 365 * 10 = 3650 days as time period and 3 times the sensors in the example above. When my math is not fooling me total demand of storage for this database is 3650 * 3 * 50KB = 550 MB.

By that a 1GB USB stick will cover nearly 20 years of data for even extremely large weather station setups.

sebi
Fresh Boarder
Fresh Boarder
Posts: 8
Joined: Wed Aug 26, 2015 4:41 pm
Contact:

Re: Changes on MB PRO data storage with release 3.0

Post by sebi » Tue Sep 19, 2017 10:22 pm

Hello,

have there been any changes to the flash problem? Is there still only the hourly level available for history (min/avg/max) values?

Thanks in advance!

Sebi

Mattk
Gold Boarder
Gold Boarder
Posts: 275
Joined: Mon Sep 22, 2014 3:24 am

Re: Changes on MB PRO data storage with release 3.0

Post by Mattk » Wed Sep 20, 2017 2:49 am

Flash problem? Are you referring to the internal USB storage of the MBP?

sebi
Fresh Boarder
Fresh Boarder
Posts: 8
Joined: Wed Aug 26, 2015 4:41 pm
Contact:

Re: Changes on MB PRO data storage with release 3.0

Post by sebi » Wed Sep 20, 2017 8:03 pm

Hello,

yes i meant the USB storage problem discribed above, sorry for the confusion. Is there a solution so far?

Thanks

Mattk
Gold Boarder
Gold Boarder
Posts: 275
Joined: Mon Sep 22, 2014 3:24 am

Re: Changes on MB PRO data storage with release 3.0

Post by Mattk » Wed Sep 20, 2017 11:45 pm

Basically what Boris has outlined above re decreasing the capacity of the USB so as to move to an industrial version at a reasonable cost, modified the archive/history period so as to not to continually hammer the daylights out of the USB, which is what kills consumer grade USB.

I've simply replaced all mine with SLC ATP 4GB & 8GB but based on Boris's calc a 1Gb could be feasible at around US$25

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 4965
Joined: Mon Oct 01, 2007 10:51 pm

Re: Changes on MB PRO data storage with release 3.0

Post by admin » Thu Sep 28, 2017 11:05 pm

Hi Sebi, where are you located?

Post Reply