Hi,
As I can tell, when Meteohub uploads files via FTP, it uploads them as <filename>.<ext>.<temp>, after the upload it renames them to <filename>.<ext>.
This not only causes a (slight) increase in data transfer, but more seriously (for me), it confuses other applications quite a lot.
The data which is upload my Meteohub is analyzed/used by e.g. http://www.beyondmeasure.se, but also a couple of batchfiles running off a w7 machine.
I would really like Meteohub just to upload the file directly as <filename>.<ext> which, to me, is the most logical way of doing it. Also it would solve a lot of "external" issues that I'm having.
FTP uload confuses other applications
Moderator: Mattk
-
YJB
- Platinum Boarder
- Posts: 387
- Joined: Thu Feb 19, 2009 5:53 pm
- Location: Venhuizen, Netherlands
- Contact:
Re: FTP uload confuses other applications
HI,
This is configured in cron:
ncftpput
-S XX Upload into temporary files suffixed by XX.
Not sure why your applications are having issues, just ignore files with the tmp extension when processing data.
To be honest, I think that uploading a tmp file is more reliable, as another application will never read a file that is incomplete (and can cause weird problems).
Ysbrand
This is configured in cron:
ncftpput
-S XX Upload into temporary files suffixed by XX.
Not sure why your applications are having issues, just ignore files with the tmp extension when processing data.
To be honest, I think that uploading a tmp file is more reliable, as another application will never read a file that is incomplete (and can cause weird problems).
Ysbrand
Re: FTP uload confuses other applications
don´t get your last argument
From time to time the tmp file is disrupting my clientraw upload
That is caused by an incomplete upload of the tmp file which is then blocking the whole clientraw.txt upload process, because the current meteohub commands do not allow to delete these hidden files via ncftp.
I had then manually deleting the .tmp file to get the upload started again ...
Not a big deal, only happens once or twice in three months.
I let cron run this command at night to get rid of possible damaged clientraw .tmp files >>>
curl -Q 'DELE /path/.in.clientraw.txt.tmp.' -u username:password ftp://ftpserver.com
From time to time the tmp file is disrupting my clientraw upload
That is caused by an incomplete upload of the tmp file which is then blocking the whole clientraw.txt upload process, because the current meteohub commands do not allow to delete these hidden files via ncftp.
I had then manually deleting the .tmp file to get the upload started again ...
Not a big deal, only happens once or twice in three months.
I let cron run this command at night to get rid of possible damaged clientraw .tmp files >>>
curl -Q 'DELE /path/.in.clientraw.txt.tmp.' -u username:password ftp://ftpserver.com
Re: FTP uload confuses other applications
Problem is that the applications look for created files, not renamed files. If I look for .temp file, they disappear very quickly as they get renamed.
For those that are paying for their broadband usage, it also makes a small (big over time) change compared to just uploading the file.
For those that are paying for their broadband usage, it also makes a small (big over time) change compared to just uploading the file.
Re: FTP uload confuses other applications
Uploading to a tempfile first and renaming it when upload is complete is something needed by 99% of the users to avoid that their web pages try to show files still being in the upload process. One could define an option not to do it, but I hardly understand why your application needs that. Please check for ways to change your application as it will also have trouble when it grabs a not yet completely uploaded data file.
One way to solve it might be to let your application watch another folder and by regularily copying non-temp files to the folder your application is working on.
The argument about additional data tansfer volumen caused by the renaming doesnt really convince me, as we just talk about a few bytes of a ftp command, that will be less than 1% of the data uploaded in total.
One way to solve it might be to let your application watch another folder and by regularily copying non-temp files to the folder your application is working on.
The argument about additional data tansfer volumen caused by the renaming doesnt really convince me, as we just talk about a few bytes of a ftp command, that will be less than 1% of the data uploaded in total.