Automatic export to mysql

Requests for future functionality

Moderator: Mattk

Post Reply
manu13
Fresh Boarder
Fresh Boarder
Posts: 7
Joined: Thu Jun 06, 2013 2:07 pm

Automatic export to mysql

Post by manu13 » Mon Feb 17, 2014 11:43 am

Hello,
I tried there a few months to develop a php script, which retrieves data from clientraw.txt file to store the data in a sql database, the script is able to archive and display data a php page in tabular form.

But I am having a big worry, syntax clientraw file that data recovery is not obvious.
In fact everything is on one line, and the script counts the number of characters for data recovery, but according to the number of characters that can create a gap and recording erroneous data.
Depending on the structure of the source file I could code a script simply, and offer all freely.
But for that I need that is implanted in Meteohub function generates a text or data file each is on a different line.
  • example:
    Station Name
    Date and time
    Time of last data reception
    outside temperature
    windchill
    relative humidity
    Dewpoint
    pressure
    Wind direction
    Instantaneous wind speed
    Mean wind speed
    gust
    Mini outdoor temperature daily
    Maximum daily outdoor temperature
    Monthly mini outdoor temperature
    Maximum Outdoor Temperature Monthly

    Windchill mini daily
    Windchill maximum daily
    Windchill monthly mini
    Windchill monthly maximum
    RH mini daily
    Daily maximum relative humidity
    Mini monthly relative humidity
    Monthly maximum relative humidity
    Dewpoint mini daily
    Maximum daily dew point
    Dewpoint mini monthly
    Dew point of maximum monthly
    Daily minimum pressure
    Daily maximum pressure
    Monthly minimum pressure
    Monthly maximum pressure
    Average daily maximum wind
    Maximum monthly mean wind
    Maximum daily gust
    Monthly maximum burst
    Rainfall
    Maximum daily rainfall
    Maximum monthly rainfall
Without the creation of this function and of this file, I do not feel able to make this functional script.
I ask people interested, to support my application
Thank you to you

User avatar
YJB
Platinum Boarder
Platinum Boarder
Posts: 387
Joined: Thu Feb 19, 2009 5:53 pm
Location: Venhuizen, Netherlands
Contact:

Re: Automatic export to mysql

Post by YJB » Mon Feb 17, 2014 8:33 pm

I guess that you can easily accomplish this using the existing meteolog.cgi interface:

This will give you a maximum of 500 records starting at 20140201000000

Code: Select all

/usr/bin/wget -q  -t 10 -T 20 -O - "http://meteohub/meteolog.cgi?mode=data&start=20140201000000&sensor=thb0"
And this one gives you the most recent value:

Code: Select all

/usr/bin/wget -q  -t 10 -T 20 -O - "http://meteohub/meteolog.cgi?mode=data&sensor=thb0"
I can give you more examples, but everything is nicely documented in the "HTTP Data Logging Protocol" manual. current version 1.5 (http://www.meteohub.de/files/HTTP-Data- ... l-v1.5.pdf)

Ysbrand
Last edited by YJB on Mon Feb 17, 2014 11:29 pm, edited 1 time in total.

manu13
Fresh Boarder
Fresh Boarder
Posts: 7
Joined: Thu Jun 06, 2013 2:07 pm

Re: Automatic export to mysql

Post by manu13 » Mon Feb 17, 2014 11:22 pm

Fabulous, I wget control quite well.
But I can not find documentation on HTTP Data Logging Protocol.
Sorry if my English is bad.

Ps: my script once operational will be under the GPL, allowing everyone to improve it and make it evolve.

Php side but I am a beginner, but I make it my business.

thank you again for your help.

manu13
Fresh Boarder
Fresh Boarder
Posts: 7
Joined: Thu Jun 06, 2013 2:07 pm

Re: Automatic export to mysql

Post by manu13 » Sat Mar 14, 2015 12:56 pm

Hello everyone,
I go back about it, having started this project, I had the problem with my NSLU2 and various bugs on my script.

I propose to test this code in a php page, you just change the IP address to one of your Meteohub server.

The script only displays real-time data (single Meteohub not your weather station), I tested with my Oregon WMR200 is ok, but if you could test with other station that would be great.

Currently the script displays atmospheric pressure, temperature and outdoor humidity and dew point.

This does have a draft, I will improve it quickly and set up the data recording.

By cons, just a question, I can not get via wget to retrieve the minimum and maximum values of the day, can you help me?

Second question, I added the rain, so I have full cumulation and intensity but may have recovered accumulated over the past hour and 24 hours?

good day

Manu




Code: Select all

<?php

$ipmeteohub ="192.168.0.38";

// Pression atmosphérique, Atmospheric pressure

$pression = shell_exec("wget -q  -t 10 -T 20 -O - \"http://$ipmeteohub/meteolog.cgi?mode=data&sensor=thb0\"");

$pieces = explode(" ", $pression);
$pieces = $pieces[6];
$pression = rtrim($pieces);

// Suppression décimale, remove decimal

$pieces = explode(".", $pression);
$pieces = $pieces[0];
$pression = rtrim($pieces);

echo $pression.' hpa';

?>

<br><br>

<?php

// Température extérieure, outside temperature

$temp = shell_exec("wget -q  -t 10 -T 20 -O - \"http://$ipmeteohub/meteolog.cgi?mode=data&sensor=th0\"");

$pieces = explode(" ", $temp);
$pieces = $pieces[2];
$temp = rtrim($pieces);

echo $temp.' °c';

?>

<br><br>

<?php

// Humidité extérieure, outside moisture

$hum = shell_exec("wget -q  -t 10 -T 20 -O - \"http://$ipmeteohub/meteolog.cgi?mode=data&sensor=th0\"");

$pieces = explode(" ", $hum);
$pieces = $pieces[3];
$hum = rtrim($pieces);

echo $hum.' %';

?>

<br><br>

<?php

//  Point de rosée, dew Point   

$pdr = shell_exec("wget -q  -t 10 -T 20 -O - \"http://$ipmeteohub/meteolog.cgi?mode=data&sensor=th0\"");

$pieces = explode(" ", $pdr);
$pieces = $pieces[4];
$pdr = rtrim($pieces);

echo $pdr.' °c';

?>

<br><br>

<?php

//  precipitation  

$rain = shell_exec("wget -q  -t 10 -T 20 -O - \"http://$ipmeteohub/meteolog.cgi?mode=data&sensor=rain0\"");

$pieces = explode(" ", $rain);
$pieces = $pieces[3];
$rain = rtrim($pieces);

echo $rain.' mm';

?>

<br><br>

<?php

//  Intensiter precipitation  

$rain = shell_exec("wget -q  -t 10 -T 20 -O - \"http://$ipmeteohub/meteolog.cgi?mode=data&sensor=rain0\"");

$pieces = explode(" ", $rain);
$pieces = $pieces[2];
$rain = rtrim($pieces);

echo $rain.' mm/h';

?>


Bonjour à tous,
je remonte ce sujet, après avoir lancé ce projet, j'ai eu des problème avec mon NSLU2, et divers bugs sur mon script.

Je vous propose de tester ce code dans une page php, il faut juste modifier l'adresse ip par celle de votre serveur météohub.

Le script affiche seulement les données en temps réel (vue par météohub et non par votre station météo), j'ai testé avec ma Orégon WMR200, c'est ok, mais si vous pouviez tester avec d'autres station cela serait super.

Actuellement le script affiche la pression atmosphérique, température et humidité exterieure, ainsi que le point de rosée.

Cela n'ai qu'une ébauche, je vais améliorer cela rapidement et mettre en place l'enregistrement des données.

Par contre, juste une question, je n'arrive pas via wget à récupérer les valeurs minimale et maximale de la journée, pouvez vous m'aider ?

Deuxième question, j'ai ajouté la pluie, j'ai donc le cumul total et l'intensité, mais peut ont récupérer le cumul sur la dernière heure et sur 24 heures ?

bonne journée

Manu

manu13
Fresh Boarder
Fresh Boarder
Posts: 7
Joined: Thu Jun 06, 2013 2:07 pm

Re: Automatic export to mysql

Post by manu13 » Mon Mar 16, 2015 7:50 pm

Hello,
I continued to work on the script, this advance.
I have not found an answer to get the max min data of the day, so I pulled over clientraw.txt, you will find below a test script to display the values of the mini maxi days.

I must still my disappointment no response.

Regarding the recording script data in a SQL database, I set up a script configuration page for the selected language (French, because I am French, English because it is the most spoken language world and German, as the developer of Meteohub and German) choice of measurement unit (° C, ° F, hPa, mb, in, mmHg, etc ...) I would suggest you to quickly test everything.

good evening

Code: Select all

<?php  

$monfichier = fopen("http://192.168.0.37/myweb/uploads/clientraw.txt", "r");

 $meteohub = fgets($monfichier);

$pieces = explode(" ", $meteohub);
$pieces = $pieces[47];
echo $tempmini = rtrim($pieces);

?>

<br><br>

<?php  

$pieces = explode(" ", $meteohub);
$pieces = $pieces[46];
echo $tempmaxi = rtrim($pieces);

?>

<br><br>

<?php  

$pieces = explode(" ", $meteohub);
$pieces = $pieces[132];
$pressmini = rtrim($pieces);

// Suppression décimale, remove decimal, Entfernen dezimal

$pieces = explode(".", $pressmini);
$pieces = $pieces[0];
echo $pressmini = rtrim($pieces);

?>

<br><br>

<?php  

$pieces = explode(" ", $meteohub);
$pieces = $pieces[131];
$pressmax = rtrim($pieces);

// Suppression décimale, remove decimal, Entfernen dezimal

$pieces = explode(".", $pressmax);
$pieces = $pieces[0];
echo $pressmax = rtrim($pieces);

?>

<br><br>

<?php  

$pieces = explode(" ", $meteohub);
$pieces = $pieces[164];
$hummini = rtrim($pieces);

// Suppression décimale, remove decimal, Entfernen dezimal

$pieces = explode(".", $hummini);
$pieces = $pieces[0];
echo $hummini = rtrim($pieces);

?>

<br><br>

<?php  

$pieces = explode(" ", $meteohub);
$pieces = $pieces[163];
$hummax = rtrim($pieces);

// Suppression décimale, remove decimal, Entfernen dezimal

$pieces = explode(".", $hummax);
$pieces = $pieces[0];
echo $hummax = rtrim($pieces);

?>

<br><br>
 

<?php  

$pieces = explode(" ", $meteohub);
$pieces = $pieces[138];
echo $tempmaxi = rtrim($pieces);

?>

<br><br>

<?php  

$pieces = explode(" ", $meteohub);
$pieces = $pieces[139];
echo $tempmaxi = rtrim($pieces);

?>


Bonjour,
j'ai continué à travailler sur le script, cela avance.
Je n'ai pas trouvé de réponse pour récupérer les données mini maxi du jour, alors je me suis rabattu sur clientraw.txt, vous trouverez ci dessous un script de test pour afficher les valeurs mini maxi du jours.

J'avoue tout de même ma déception de voir aucune réponse.

Concernant le script d'enregistrement des données dans une base sql, j'ai mis en place une page de configuration du script, pour sélectionné la langue ( Français, car je suis Français, l'anglais car c'est la langue la plus parlé au monde et l'allemand, car le développeur de météohub et allemand) choix des unité de mesure, (°c, °f, hPa, mb, in, mmHg, etc ...) je vous proposerais rapidement de tester le tout.

bonne soirée

uakm2201
Fresh Boarder
Fresh Boarder
Posts: 1
Joined: Tue Jul 21, 2015 12:12 pm

Re: Automatic export to mysql

Post by uakm2201 » Tue Jul 21, 2015 12:28 pm

Hi all

Also try to work on this way, but using XML file. Main idea is that we could retrieve history using this file.

Actually, I'm a little bit lost on php using arrays ... Need help on this if someone could take a look :)

Ps : Also french ...

Here is an extract of some var_dump :

array(3) { 'date' => string(14) "20150714000058" 'id' => string(3) "uv0" 'index' => string(3) "0.0" }
array(4) { 'date' => string(14) "20150714000058" 'id' => string(4) "sol0" 'rad' => string(3) "0.0" 'evapo' => string(3) "2.0" }
array(4) { 'date' => string(14) "20150714000058" 'id' => string(5) "rain0" 'rate' => string(3) "0.0" 'total' => string(5) "112.8" }
array(6) { 'date' => string(14) "20150714000101" 'id' => string(5) "wind0" 'dir' => string(3) "274" 'gust' => string(3) "0.0" 'wind' => string(3) "0.0" 'chill' => string(4) "17.5" }
array(8) { 'date' => string(14) "20150714000105" 'id' => string(4) "thb0" 'temp' => string(4) "28.7" 'hum' => string(2) "34" 'dew' => string(4) "11.3" 'press' => string(5) "968.2" 'seapress' => string(6) "1017.1" 'fc' => string(1) "2" }
array(5) { 'date' => string(14) "20150714000110" 'id' => string(3) "th0" 'temp' => string(4) "17.5" 'hum' => string(2) "74" 'dew' => string(4) "12.8" }

function traverse( DomNode $node, $level=0 ){
handle_node( $node, $level );
if ( $node->hasChildNodes() ) {
$children = $node->childNodes;
foreach( $children as $kid ) {
if ( $kid->nodeType == XML_ELEMENT_NODE ) {
traverse( $kid, $level+1 );
}
}
}
}

function handle_node( DomNode $node, $level ) {
for ( $x=0; $x<$level; $x++ ) {
$first=true;
print " ";
}
if ( $node->nodeType == XML_ELEMENT_NODE ) {
//print $node->tagName;
if ($node->hasAttributes()) {
$attributes = array();
foreach ($node->attributes as $attrName => $attrValue) {
//if(!$first)print "\n";
//print "@".$attrName."[".$attrValue->value."]";
//$first=false;
$attributes[$attrName] = $attrValue->value;
}
//$type=$attributes['id'];
var_dump($attributes);
}
print "<br />\n";
}
}

// Main

$dom = new DomDocument();
$dom->load('./test.xml');
$root = $dom->firstChild;

traverse($root);

Post Reply