Data export problem with release 3.0

[chrysnic]

Hi all,

My MB Pro is configured to export at 00:15 each day the previous day's data at minute resolution. Based on the admin's description of the new release 3.0: "For the last 24 hours data will also be kept in RAM and will stay available on a per minute level". So, I assumed that the Data Export service for the previous day would still export minute-resolution data (directly from RAM). However, below is a snippet from the exported file. As you can see, there is data for 5 consecutive minutes, then a 10-minute gap, then data for 5 minutes, then another 10-minute gap, and so on. This pattern repeats for the entire day.

Is this how it is supposed to work?


2016-07-05,04:15,26.3,64,18.8,1001.5,1.1,3.6,267,0.0
2016-07-05,04:16,26.3,64,18.8,1001.5,1.1,3.6,267,0.0
2016-07-05,04:17,26.3,64,18.8,1001.5,1.1,3.6,267,0.0
2016-07-05,04:18,26.3,64,18.8,1001.5,1.1,3.6,267,0.0
2016-07-05,04:19,26.3,64,18.8,1001.5,1.1,3.6,267,0.0
2016-07-05,04:20,26.3,64,18.8,1001.5,1.1,3.6,267,0.0
2016-07-05,04:21,,,,,,,,
2016-07-05,04:22,,,,,,,,
2016-07-05,04:23,,,,,,,,
2016-07-05,04:24,,,,,,,,
2016-07-05,04:25,,,,,,,,
2016-07-05,04:26,,,,,,,,
2016-07-05,04:27,,,,,,,,
2016-07-05,04:28,,,,,,,,
2016-07-05,04:29,,,,,,,,
2016-07-05,04:30,26.4,60,18.1,1001.5,0.0,0.0,273,0.0
2016-07-05,04:31,26.4,60,18.1,1001.5,0.0,0.0,273,0.0
2016-07-05,04:32,26.4,60,18.1,1001.5,0.0,0.0,273,0.0
2016-07-05,04:33,26.4,60,18.1,1001.5,0.0,0.0,273,0.0
2016-07-05,04:34,26.4,60,18.1,1001.5,0.0,0.0,273,0.0
2016-07-05,04:35,26.4,60,18.1,1001.5,0.0,0.0,273,0.0
2016-07-05,04:36,,,,,,,,
2016-07-05,04:37,,,,,,,,
2016-07-05,04:38,,,,,,,,
2016-07-05,04:39,,,,,,,,
2016-07-05,04:40,,,,,,,,
2016-07-05,04:41,,,,,,,,
2016-07-05,04:42,,,,,,,,
2016-07-05,04:43,,,,,,,,
2016-07-05,04:44,,,,,,,,
2016-07-05,04:45,26.4,59,17.8,1001.5,0.0,0.0,295,0.0
2016-07-05,04:46,26.4,59,17.8,1001.5,0.0,0.0,295,0.0
2016-07-05,04:47,26.4,59,17.8,1001.5,0.0,0.0,295,0.0
2016-07-05,04:48,26.4,59,17.8,1001.5,0.0,0.0,295,0.0
2016-07-05,04:49,26.4,59,17.8,1001.5,0.0,0.0,295,0.0
2016-07-05,04:50,26.4,59,17.8,1001.5,0.0,0.0,295,0.0
2016-07-05,04:51,,,,,,,,
2016-07-05,04:52,,,,,,,,
2016-07-05,04:53,,,,,,,,
2016-07-05,04:54,,,,,,,,
2016-07-05,04:55,,,,,,,,
2016-07-05,04:56,,,,,,,,
2016-07-05,04:57,,,,,,,,
2016-07-05,04:58,,,,,,,,
2016-07-05,04:59,,,,,,,,
2016-07-05,05:00,26.3,62,18.4,1001.6,0.0,0.0,294,0.0
2016-07-05,05:01,26.3,62,18.4,1001.6,0.0,0.0,294,0.0
2016-07-05,05:02,26.3,62,18.4,1001.6,0.0,0.0,294,0.0
2016-07-05,05:03,26.3,62,18.4,1001.6,0.0,0.0,294,0.0
2016-07-05,05:04,26.3,62,18.4,1001.6,0.0,0.0,294,0.0
2016-07-05,05:05,26.3,62,18.4,1001.6,0.0,0.0,294,0.0
2016-07-05,05:06,,,,,,,,
2016-07-05,05:07,,,,,,,,
2016-07-05,05:08,,,,,,,,
2016-07-05,05:09,,,,,,,,
2016-07-05,05:10,,,,,,,,
2016-07-05,05:11,,,,,,,,
2016-07-05,05:12,,,,,,,,
2016-07-05,05:13,,,,,,,,
2016-07-05,05:14,,,,,,,,
2016-07-05,05:15,26.1,65,19.0,1001.8,0.0,0.0,294,0.0
2016-07-05,05:16,26.1,65,19.0,1001.8,0.0,0.0,294,0.0
2016-07-05,05:17,26.1,65,19.0,1001.8,0.0,0.0,294,0.0
2016-07-05,05:18,26.1,65,19.0,1001.8,0.0,0.0,294,0.0
2016-07-05,05:19,26.1,65,19.0,1001.8,0.0,0.0,294,0.0
2016-07-05,05:20,26.1,65,19.0,1001.8,0.0,0.0,294,0.0
2016-07-05,05:21,,,,,,,,
2016-07-05,05:22,,,,,,,,
2016-07-05,05:23,,,,,,,,
2016-07-05,05:24,,,,,,,,
2016-07-05,05:25,,,,,,,,


Thanks.

[admin]

With changes in release 3.0 database requests are supposed not to be on per minute resolution. When you modify your request to pick hourly data it will work fine. You could also set "#pragma fillgap=900" in your export template to fill up gaps inbetween the the current DB update of every 15 mins. I know, that this might not be an acceptable approach as it just repeats data.

So there is no great solution I can offer right now. What I can think of is directing the per minute export requests directly to RAM and to report data from there. Will be rather complicated but has the potential to fulfill your needs. However, this will take a while to implement.

[chrysnic]

I fully understand the difficulty and complication. Nevertheless, the ability to (somehow) have minute-resolution data logging is a very important and desirable attribute (at least in my case). A solution to this issue (i.e., the forced transition to hourly intervals due to storage reliability) would be greatly appreciated.

[admin]

I am working a a solution that provides per minute data for the last 36 hours. I guess this will fit your demand.

[Mattk]

I am working a a solution that provides per minute data for the last 36 hours. I guess this will fit your demand.

Is it possible then to have a sum36h or even a sum30h?

[admin]

I am still developing/testing, but from the conceptual side this should do.

[bseeger]

With changes in release 3.0 database requests are supposed not to be on per minute resolution. When you modify your request to pick hourly data it will work fine. You could also set "#pragma fillgap=900" in your export template to fill up gaps inbetween the the current DB update of every 15 mins. I know, that this might not be an acceptable approach as it just repeats data.

So there is no great solution I can offer right now. What I can think of is directing the per minute export requests directly to RAM and to report data from there. Will be rather complicated but has the potential to fulfill your needs. However, this will take a while to implement.

Da mein Englisch nicht gut ist, wäre ev eine kurze Deutsche Uebersetztung sehr gut. Mein stündlicher Datenexport hat jetzt den gleichen fehlerhaften Aufbau wie von Chrysnic beschrieben. 5 Minuten Daten--> 10 Minuten Pause usw. Was muss ich machen dass ich trotzdem die Minutenwerte erhalte. Notfalls auch 5 Minutenwerte aber sicher kontinuierlich und nicht mit 10 Minuten Unterbruch.

Herzlichen Dank

[admin]

Ich arbeite an einer Lösung dafür. Das ist bannig komplex, da die minütlichen Daten nun in einer RAM Datenbank gehalten werden, sowie die minütlich aktualisierten Werte für die höheren Aggregierungsstufen (Stunde/Tag/Monat/Jahr). Ich hoffe ein erstes Release dafür dieses Wochenende fertig zu bekommen, dann gehen auch die minütlichen Exporte für die letzten 36 Stunden.

Der Lohn all der Mühen ist, dass die Last auf den USB-Speicher wirklich dramatisch (ca Faktor 50) zurückgeht und diese nervigen Speicherausfälle Historie sein werden. Zudem können die 8GB dann wirklich über ein Jahrzehnt an Daten halten. Alles wird gut!

[bseeger]

Als provisorische "Lösung" bin ich wieder zurück zu Version 2.9, und habe den automatischen Versionsupdate im Moment deaktiviert, jetzt ist alles wieder wie vorher.

[admin]

There is a new 3.0 online, please give it a try. I tested on my rig best I can and it works fine.