Two systems lisning to the same weather station get different results **solved**

This section covers the Meteobridge PRO, NANO SD and Raspberry Pi units exclusively

Moderator: Mattk

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 6631
Joined: Mon Oct 01, 2007 10:51 pm

Re: Two systems lisning to the same weather station get different results **solved**

Post by admin »

Please insert a stick and give me a remote login. Are you sure the pin headers connect fine?

DAP
Senior Boarder
Senior Boarder
Posts: 70
Joined: Fri Jan 15, 2016 8:37 am

Re: Two systems lisning to the same weather station get different results

Post by DAP »

I currently have a known good memory stick installed (SanDisk Cruzer 8GB FAT32 format volume name Pictures, no files on the disk). The login I gave you a few days ago should still be good.

The memory stick was removed from its case, but is still too long to put inside the Meteobridge PROs case, so the Meteobridge PRO is running outside its case at the moment. I have measured the voltage at the front USB port, and it measures correctly at slightly above 5V.

I examined the circuit boards under a stereo microscope and did not see any faulty solder joints, or ICs that looked like they had let the smoke out.
I did not look under the RF shielding.

A bit of info that may or may not be relevant. A few years ago I was having trouble getting the WIFI to connect. The system has been connected by a cat5 cable ever since.

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 6631
Joined: Mon Oct 01, 2007 10:51 pm

Re: Two systems lisning to the same weather station get different results

Post by admin »

When a new stick is inserted it also needs to call

Code: Select all

http://ip-of-your-meteobridge/cgi-bin/system.cgi?rm /root/mbstorage.fmt
and reboot with the new stick. doing so the stick gets partitioned and formatted the way it needs to be.

I did this on your system and the sandisk you have inserted is now operational. But let me say that you will not have much joy with it as it will not provide the needed write endurance. I strongly recommend to use SLC USB sticks only, which are unforunately expensive and not to buy at your local radio shack. I recommend ATP NANODURA, which is also in the current generation of MB PROs.

Rain totals should be fine again.

DAP
Senior Boarder
Senior Boarder
Posts: 70
Joined: Fri Jan 15, 2016 8:37 am

Re: Two systems lisning to the same weather station get different results **solved**

Post by DAP »

[s]What is the procedure to execute that call? That does not look like a command line command.[/s]
I think I figured out that I enter that line in my browser with the "ip-of-your-meteobridge" replaced with an actual IP address.

The memory stick I have installed is a temporary proof of concept hack. I wanted to be sure I could get this working again before I spent any money.
I spent some time on Amazon trying to find a SLC USB memory stick, but their search function sucks.

I will run an experiment with a uSD card reader as soon as I figure out how to make the call you suggested. If that works, I will buy a high endurance uSD card, and run with that.
This looks to be substantially cheaper than the USB SLC memory you suggest. If it does not last, I'll replace it with the memory stick you suggest.

Thanks.

Mattk
Platinum Boarder
Platinum Boarder
Posts: 885
Joined: Mon Sep 22, 2014 3:24 am

Re: Two systems lisning to the same weather station get different results **solved**

Post by Mattk »

Really not sure why you are continuing to go against the suggestions and use sub standard USB devices and especially ones that don't fit anyway?

Forget Amazon and go to places like Digi-Key or Mouser etc. Some Nanodura (ATP) part #'s that will do the job without any argument are

AF4GUFNDNC
AF8GUFNDNC
AF1GUFNDNC

User avatar
galfert
Platinum Boarder
Platinum Boarder
Posts: 326
Joined: Sun Jun 24, 2018 10:31 pm
Location: Orlando, FL

Re: Two systems lisning to the same weather station get different results **solved**

Post by galfert »

DAP wrote:
Sat Feb 06, 2021 9:37 pm
... This looks to be substantially cheaper than the USB SLC memory you suggest. If it does not last, I'll replace it with the memory stick you suggest.
An SLC SD card from the approved list is not a suggestion!... It is enforced! If you want to run Meteobridge on a Raspberry Pi it is the only solution. It is enforced in the Meteobridge software by way of the software identifying the SD card model. Why? Because Smartbedded wants it that way as they have learned (from previous experience on other projects) that they don't want to have to deal with upset users a few years from now when their data is gone. If it was left as a choice the majority of users would not understand the difference and many would probably not read the manual nor other documentation to even learn this. It amounts to negligence to not proactively enforce the use of SLC SD cards.

This is not to say that regular SD cards do not have a use case. Plenty of reasons where data storage is not concerned where a regular SD card would be fine for a project. If it blows up, no problem, you get a new card and reload the program. If it's a Raspberry Pi robot, or it controls a 3D printer, it is used to learn Linux or programming or to browse the web...etc...then who cares...there is no data.

It is not unusual for companies to want to control and guarantee the level of performance for their products. I just read an article about Synology enforcement of specific hard drives in their enterprise NAS devices. Countless other examples exist like Apple have also done this. Look at the mess Tesla is in because of their first generation MCU and the use of poor quality eMMC flash storage. Slightly different situation involving a costly recall, but it drives the point of what the outcome is when you ignore what is inevitable. Because it isn't a question of if it fails, but rather it is known that it will fail 100% of the time.

If you don't like it, then it is always your choice to run some other software.
Meteobridge RPI | GW1000

DAP
Senior Boarder
Senior Boarder
Posts: 70
Joined: Fri Jan 15, 2016 8:37 am

Re: Two systems lisning to the same weather station get different results **solved**

Post by DAP »

I have just tested with a uSD card reader, and a 64GB uSD card I had lying around. I was able to get this to work.
I plan to replace the random uSD card with this: https://documents.westerndigital.com/co ... icrosd.pdf
Marketing people being marketing people, I have to assume that their "13 years continuous recording of HD video" means 720p 60, the lowest data rate that is still called "HD".
This also assumes the behaviour of dash cams which is to always overwrite the oldest video file when the disk is full.
It is not clear if the uSD card implements any kind of wear leveling, it may not, in which case it will not last 13 years since in this application it is overwriting the same file every hour. But I am willing to take that risk. If it fails in less than 13 years, I'll replace it again, and base my decision on what I replace it with on how long this memory lasts.
My Meteobridge pro has now worn out two memory sticks. The first one died in less than a year. My biggest complaint with the Meteobridge pro is not that the memory failed. All FLASH storage will eventually fail. My biggest complaint is that it failed by silently, sending incorrect data to the weather websites. Once I noticed that the data was wrong, it was clear from the log files that something had gone wrong, but not what. I would have found and fixed the problem much sooner if the Meteobride pro had stopped sending data to the websites altogether rather than poisoning their databases with bogus data that looked sort of right. Also a clear message in the log like "can't write to internal USB memory stick" would have simplified the diagnosis of the problem.

I do not use the Davis weather system for the rainfall totals I trust, I have a manual rain gauge and a piece of paper for that. I have not found any electronic system reliable enough to trust. (I have not yet found a way to clean the rain gauge each year without it recording bogus rainfall. No, disconnecting the rain sensor does NOT work. and the weather websites I upload data to do not allow me to delete bogus data.)

As far as the Tesla, I own a Tesla, so I am well aware of what they did wrong. The stupid thing that they did is to solder the eMMC memory IC directly to the circuit board so that to replace that memory IC, they must replace the whole computer in the car. Then they continuously wrote log files with lots of unnecessary information to it. Newer cars use a uSD card in a socket.

User avatar
galfert
Platinum Boarder
Platinum Boarder
Posts: 326
Joined: Sun Jun 24, 2018 10:31 pm
Location: Orlando, FL

Re: Two systems lisning to the same weather station get different results **solved**

Post by galfert »

DAP wrote:
Mon Feb 08, 2021 11:30 pm
I have just tested with a uSD card reader, and a 64GB uSD card I had lying around. I was able to get this to work.
I plan to replace the random uSD card with this: https://documents.westerndigital.com/co ... icrosd.pdf
That Sandisk Max Endurance SD card is just still just a TLC type card. They probably have done some improvements to improve on earlier TLC designs but it is still not at the level of SLC in terms of erase cycles possible.
Marketing people being marketing people, I have to assume that their "13 years continuous recording of HD video" means 720p 60, the lowest data rate that is still called "HD".
This also assumes the behaviour of dash cams which is to always overwrite the oldest video file when the disk is full.
Aren't you the least bit worried about the fact that this Max Endurance card spec sheet makes no mention of erase cycles possible nor of TBW? They are so cryptic with the "hours" of video possible. Why do they not list the TBW and erase cycles? Because it is marketing crap. I read some articles that delved into this card and they determined it was a TLC type card. You get what you pay for.

Which brings me to my next point. Do you not get that you don't have the choice? If you insert anything but a pre-approved SLC SD card the Meteobridge will not do any data logging. I'm not quite sure why you keep pressing about using anything but an SLC SD card. It isn't going to work. It isn't about when it fails...it is that the software will not let you use what you want. I don't know how else to communicate this to you which I thought was already clear from other documentation and from everything I and others have said. :roll:

Maybe you need to see it to believe it:
rpi.jpg
rpi.jpg (21.36 KiB) Viewed 556 times
Meteobridge RPI | GW1000

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 6631
Joined: Mon Oct 01, 2007 10:51 pm

Re: Two systems lisning to the same weather station get different results **solved**

Post by admin »

Meteobridge PRO does not enforce SLC storage, but NANO and RPI do.

I have learnt it the hard way and decided to make it mandatory on NANO and RPI when I was watching the Meteobridge PROs failing caused by USB stick wear outs. The hassle is not worth the cheaper buy, imho.

Using a USB microSD card adapter and fitting a SLC SD card into it might also do the job in a PRO, but it looks really a bit complicated to me and it is a setup with more contacts and components and by that more options to fail long term.

User avatar
galfert
Platinum Boarder
Platinum Boarder
Posts: 326
Joined: Sun Jun 24, 2018 10:31 pm
Location: Orlando, FL

Re: Two systems lisning to the same weather station get different results **solved**

Post by galfert »

Ah okay. So the Meteobridge PRO does not enforce SLC USB but it does ship with one installed. Correct?
Meteobridge RPI | GW1000

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 6631
Joined: Mon Oct 01, 2007 10:51 pm

Re: Two systems lisning to the same weather station get different results **solved**

Post by admin »

yes. First generation PROs had non-SLC USB sticks and this is where my negative experience regarding consumer USB sticks comes from. For some years now we only ship with SLC USB sticks and all issues related to USB storage were gone from that point in time. Up to now we did not had a single failed or worn-out USB stick from the SLC batches (>1000 pcs so far).

DAP
Senior Boarder
Senior Boarder
Posts: 70
Joined: Fri Jan 15, 2016 8:37 am

Re: Two systems lisning to the same weather station get different results **solved**

Post by DAP »

My meteobridg pro is definitely from the first batch. The first memory card failed within a year. It was replaced under warrantee.
The dead memory stick I just pulled out of my replacement unit was a Toshiba 8GB memory card. I don't know if this was a SLC card, it may have been from before the switch to SLC.

While SLC cells last longer than MLCs, they limit the size of the memory to 8GB. MLCs can be as high as 1TB. If the memory is wear leveled, a larger MLC memory can last as long as the smaller SLC memory since each cell is not written as often.

The problem is that one can never tell how good the wear leveling is since manufactures tend to keep that information secret.

Part of the reason I went with uSD is that I have a choice of about 10 uSD card readers that will fit inside the Meteobridge PRO and any uSD card will fit in a uSD card reader. with memory sticks, not only do I have to find one that has a chance of lasting, it also must come in a form factor that will fit inside the meteobridge pro case. Most of them don't.

As I mentioned I have a Tesla, and I had a memory stick fail being used with TeslaCAM. Other tesla users have reported having good luck with a uSD card reader and the SanDisk Max Endurance cards.

I decided it was worth it to me to find out if it will work in this application. If I outlive this memory card, you can be sure I will come back to this forum to report on the failure. I know exactly what risk I am taking and no blame will fall on the Meteobridge if my experiment fails.

(note that the Max Endurance card has not yet arrived, the card currently in the system is a cheap 64GB uSD card)
Meteobridge.png
Meteobridge.png (56.02 KiB) Viewed 503 times

Mattk
Platinum Boarder
Platinum Boarder
Posts: 885
Joined: Mon Sep 22, 2014 3:24 am

Re: Two systems lisning to the same weather station get different results **solved**

Post by Mattk »

It appears you are getting carried away with size over quality and server grade SLC, Can not even imagine why there is any discussion on anything even remotely bigger than 8GB and even 8GB is not warranted. Maybe you need to review your data management policy?

DAP
Senior Boarder
Senior Boarder
Posts: 70
Joined: Fri Jan 15, 2016 8:37 am

Re: Two systems lisning to the same weather station get different results **solved**

Post by DAP »

If there is proper wear leveling, a memory that is twice as large will last twice as long in a particular application, if the memory technology is the same, as each cell is written half as often on the larger memory.

The larger memory has a useful function, even if I don't ever intend to use more than 1% of the memory.

MLC memories are mass produced, so even a 1TB uSD is cheaper than an 8GB SLC memory.

With a 256GB MLC drive, the SLCs must survive 32 times as many writes as the cells of this MLC for the 8GB SLC drive to last longer.

At some point, just by making the memories larger, a consumer grade MLC memory will last as long, or longer than the smaller SLC memory for the same application.

It is not clear if we have yet reached that point. Manufactures of consumer grade memories don't tend to release useful data on this.

It is my belief that even if they have not surpassed SLC memories in this application, a good quality MLC should be good enough (last 5+ years). So I put my money where my mouth is.

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 6631
Joined: Mon Oct 01, 2007 10:51 pm

Re: Two systems lisning to the same weather station get different results **solved**

Post by admin »

@DAP: Your opinion about MLC capabilities in general is not something anyone argues against. The problem is that consumer SD cards and USB sticks are not made with elaborated wear leveling algorithms because these media is 99% used for very low wear demanding applications like storing pictures or movies or a backup of data here and there. Therefore, there is no incentive for the manufacturer to embed sophisticated wear-leveling algortihms. Story is completely different with SSD, which will be ongoing hammered with data and where this issue got attention from the manufacturers from the beginning.

In a nutshell: The fact that your SSD does fine with MLC/TLC/QLC (by wear-leveling and overprovisioning) does not tell that the same is true for SD cards and USB sticks.

My experience from dealing with this problems for >10 years (since I started 2007 with Meteohub) from a manufacturer perspective tells me that you cannot rely on any non-SLC SD card or USB stick. Some might be reasonable fine, others will fail within a couple of months. There is no way to know in advance and internals can also change within a product series without being noticed. I don't think the spared money is worth the hassle.

If you don't agree and think to know better, fine with me. At least I can say I told you so and you know my recommendation.

Post Reply