meteohub.de

I've been running MB (various versions) for well over a year now on a TP-LINK TL-MR3020 taking data from an ObserverIP connected to a Ambient WS-1400-IP. At the moment of this report all versions are current.

Over the last several months I occasionally had scenarios where the WU page for my station (KCAMOUNT75) would not load completely and the map and graph/tables would be missing. Today I finally had enough time to dig into this. I had already noticed, in the past, that when this happens, my browser's console has error messages (seen both on FireFox and Safari), along the lines of: "SyntaxError: Left hand side of operator '=' must be a reference." This error aborts the running JavaScript, which then in turn "kills" the rest of the page.

Today I looked into this a little deeper and found (using the debugger in the webkit inspector) that the problem happens when an Ajax call on the page retrieves the station historical data (so the table and graphs can be produced) and receives syntactically incorrect JSON/Javascript expression back. The line in question that produces the error looks like this: Looking at the rest of the JSON around it, it appears that entries before and after are (often) free of this problem. Here is a small excerpt (WU apparently produces datapoints 5 minutes apart): The entry in the middle has two such problem cases, the one before and after are fine. Furthermore it is interesting that the problem entries contains MeteoBridge template variable expressions. It should be noted here that I do not use MB's standard WU upload function, but rather use a custom crafted http request. The relevant portion of the custom URL contains: So it appears that somehow in the process of forming the actual URL MB failed to substitute a value for the expression and leaves the expression itself instead (this is documented to mean missing or malformed data). To further explain this it must also mean that WU receives all URL parameters and stores their values as text (if not, it would have failed to parse the template expressions left in there, as float values are apparently expected). Furthermore, WU's JSON generator is apparently then also written to render the value stored (the text of the template expression) without quotes because it "believes" it is rendering a float value (which should not have quotes).

One would come to the conclusion that WU is rather brain dead in accepting such obviously non-sense values to begin with, let alone echo them without any check of correctness, causing a browser Javascript crash. Apparently, for reasons I have not yet determined, my sensors, or the ObserverIP, occasionally produce missing/invalid data. I am not going to "fix" that because right now it is helpful in debugging this, and possible solutions, but it is probably as simple as a new battery in the indoor unit.

So the following questions come up:

1. What is going wrong in expression eval?
   • Is it not receiving any data for these sensor values from the ObserverIP?
   • Or incorrect data from the ObserverIP (something that does not parse)?
   • Or some rare case with valid data, but a bug in MB code (seems unlikely, but just mentioning it)
2. What makes sense to do in these cases? All the other sensors do have good values. I can think of several options:
   • Use the "replacement" string option. Without specifying a replacement, we get exactly the behavior described here (and as documented: "When a variable is not defined or there is no data for a specified sensor, information specified as "replacement" string will be represented instead."). Therefore this would require the template creator (user) to specify a value for every sensor where this could possibly happen (which generally will mean all of them). This might work (although may be not for the WU case presented here), but will make templates a lot more tedious.
   • Skip the whole HTTP request. This is in contradiction to the current documentation, so would not be backwards compatible. However, for a generic http request facility, this might be desirable behavior. If we would want to enable this, we would need a backwards compatible changes to template processing.
   • Leave out the whole URL parameter: instead of producing "indoorhumidity=[thb0hum-act]&" produce just "&". This too contradicts current documentation, might be the desired behavior for a specific case, and would need a template processing change.

First lets have a look at the first option and the WU case. Assuming the analysis is correct, there is not really a suitable "replacement" value. We'd like to communicate "null", but that is not possible in URLs with the possible exception of using the empty string (I'll test this with WU later). However, unless WU would take an empty value and truly not store it, as opposed to what I think will happen and store an empty string. Storing an empty string would still produce invalid JSON output because nothing would follow the colon after the property name. So YES, WU is not making this easy and one could take the stance that WU is wrong and therefore no fix/change needs to be offered by MB. Practically, however, lots of people use MB with WU so an easy to use solution would probably please many folks. Since similar problems may exists with other services for which the generic HTTP request template is needed, the problem to be addressed is actually larger than just WU.

The third option could be implemented with backwards compatibility. It would require first parsing/deconstructing the URL. Next process all parameter values and if the value is precisely a single template variable expression (starts and ends with brackets), then evaluate and substitute. Then reconstruct, and process the whole URL as a string to catch cases where template expressions where used in other ways. For backwards compatibility we would only have to address the case where no replacement value is specified, and, somehow, the user indicated a new behavior was desired. For example, if we allow a converter to be prefixed by the exclamation mark, it would indicate that a parameter would have to be eliminated if data is missing or malformed. It should be noted that, in many cases, the actual converter is empty: the exclamation by itself would mean "standard converter but leave out if missing/malformed). So my example would then look: Alternatively would could introduce "#" to replace "=!": A different approach altogether is to require "{}" instead of "[]" for parameter substitutions that we want to be treated this way: All of these would be a lot more concise than putting replacement values on everything (except if they are all empty strings, but that would not solve all cases).

The second option could be handled in a similar manner by introducing a "pseudo"variable that, when seen even once in the whole template, if any conversion fails, the whole request would be abandoned. Since we would not want to repeat that for every parameter perhaps we could introduce [nogo] as a special pseudo variable that requests this behavior. Or, if we would like a specific expression to trigger this (but not some others), it could be done with another special prefix before the converter. May be something like "!!". Any single one of such failed "conversion" would drop the request.

Since I believe that even using the replacement string option will not allow me to completely "fix" this problem with this particular service (WU), even if it did it would be extremely tedious and I would prefer if one of the proposed alternatives were implemented (or something like it or equivalent to it).

When a sensor does not receive fresh data after a while or directly after starting the logger a variable might be undefined. Therefore, you need to define ALWAYS replacement strings to handle situations like that.

Yes, understood that is necessary in the current template model (and I thought I already indicated that). As pointed out in the post, this is not going to work for at least the WU case (and possibly other services). I have already substituted "--", but WU still does not produce valid output (as predicted in the original post). What is really needed here is to leave the whole parameter out. This cannot be done with any replacement string I can think of (but I'd be happy to stand corrected).

Of course we could send "0" as a temperature, but that too does not convey an actual missing or invalid measurement and would produce weird temp graphs, as opposed to graphs that are simply missing a datapoint.

From the lack of further response, it seems there is no intention (or time) to do anything about this, either using my suggestions, or otherwise.

Nonetheless, I want to convey the situation is real and poses a problem. I have checked my sensor's battery and location and all is fine. It just appears that once every so often (few times a week), the value from the indoor Ambient unit is not received, or not received correctly. In such cases, it would be best to either not upload to WU at all (since it is very transient, this would hardly make a difference in the overall value of the dataset), or to upload, leaving just those two sensor values out.

Neither is possible at the moment. Yes, I have used the replacement value option, but there simply is not a text one could put there to tell WU that the sensor value is invalid, without causing the station's display page to crash on the returned Javascript. Yes, I'll try to get WU to apply better validation and output, but no guarantees I can get them to move on this. If the same problem would appear with any other HTTP request a user wishes to use, there would be, again no solution.

If you disagree this is a real problem faced by a real user, let me know. If it is a matter of my not explaining it adequately for you to understand, we should discuss (perhaps off the forum?). If this is a matter of lack of resources, please let us know and perhaps I can offer help to get it done. If there is no desire to fix it that too would be helpful to know.

You cannot send "--" to WU when a value is missing, you have to drop the whole name-value pair. You can do that with ''#if# #then# #else# #fi#" closures explained in template replacement section of the wiki. But this is rather advanced stuff.

Did you try this? From all your examples I can't see that Meteobridge is not doing as specified in the wiki. If template replacement options (which are already rather powerful imho) don't match all aspects of your very specialized use case, then this is not a product fault one must immediately take care of. Please adjust your expectation on individual solution support to a reasonable level.

It seems you think I am saying MB does not work as advertised. That is not what I was trying to convey. What I was trying to convey is that there is a scenario that, with existing functionality, is troublesome because it requires a very verbose template. There are some 10 parameters and having to apply the conditional approach to all of them makes for an enormous template.

I am well aware of both the replace functionality, as well as the conditional options (and so I should not have said it is not possible; my bad) although, as you see, one can consider that advanced. I was making a suggestion on how to make handling that very easy for the user (where most users will never get to understand the advanced stuff perhaps), and in a backwards compatible manner.

Meanwhile, in the particular case of wunderground, they have let me know that sending -999 as replacement value will, effectively, achieve the same thing as not sending the complete pair. Conceptually I don't like their solution, but I suppose it will work for this case.