The scripting documentation states that SYNC will transfer a file only if it is newer. Does SYNC only check the file creation time stamp to determine if a file is newer?
In our situation, we're seeing a file size difference between a file on the FTP server and the same file on the client after a SYNC. On the next SYNC pass, it looks like the file is ignored because of the time stamp, even though the file contents have changed. We're downloading from the server to the client. Both client and server are running Windows 2003.
The file size difference is likely being caused by a situation where the file on the server is still being written when ScriptFTP initiates the download. This seems to be confirmed by the fact that the process that consumes the files on the client sees the file as "corrupted" and can't process the file.
If anyone else has experienced this problem and devised a way around it, please share.
In our situation, we're seeing a file size difference between a file on the FTP server and the same file on the client after a SYNC. On the next SYNC pass, it looks like the file is ignored because of the time stamp, even though the file contents have changed. We're downloading from the server to the client. Both client and server are running Windows 2003.
The file size difference is likely being caused by a situation where the file on the server is still being written when ScriptFTP initiates the download. This seems to be confirmed by the fact that the process that consumes the files on the client sees the file as "corrupted" and can't process the file.
If anyone else has experienced this problem and devised a way around it, please share.
Hi UnMonkey,
The best way to do that, in my own opinion, is to configure the process that runs in the server to modify the files outside the directory where ScriptFTP is downloading files. I don't know if this is possible or not. Other way is to use the command GETFILESIZE to compare local and remote file sizes and call GETFILE if it's different. Something like:
Does SYNC only check the file creation time stamp to determine if a file is newer?Yes, SYNC is based only on the file modification time stamp.
The file size difference is likely being caused by a situation where the file on the server is still being written when ScriptFTP initiates the download.Yes, I think this the origin of the problem. The file time stamp is usually updated once a write operation has finished. If ScriptFTP downloads a corrupted file confirms this fact.
The best way to do that, in my own opinion, is to configure the process that runs in the server to modify the files outside the directory where ScriptFTP is downloading files. I don't know if this is possible or not. Other way is to use the command GETFILESIZE to compare local and remote file sizes and call GETFILE if it's different. Something like:
Code: Select all
You can use FOREACH to do the same thing to more than one file.# Get local and remote file sizes
$local_file_size=GETFILESIZE(LOCAL,"data.log")
$remote_file_size=GETFILESIZE(REMOTE,"data.log")
# if sizes are different download the file
IF($local_file_size!=$remote_file_size)
GETFILE("data.log")
END IF
liked Script FTP? Try our other product Email Parser
Yes, SYNC is based only on the file modification time stamp.Thank you for the confirmation.
How efficient is a call to GETFILESIZE() on the remote server? Does GETLIST also gets file size and time and cache that information for subsequent calls to GETFILESIZE()? I can understand why that would not be ideal in all cases.
Your solution to compare file sizes and initiate a download if different works fine for a single file, but in our case, we need to do this on about 1600 files to ensure the SYNC across both file time and file size. We can loop through them each time, but how much latency should we expect that to create?
How efficient is a call to GETFILESIZE() on the remote server?Each time GETFILESIZE is called ScriptFTP will send the FTP protocol SIZE command to the FTP server.
Does GETLIST also gets file size and time and cache that information for subsequent calls to GETFILESIZE()?No, the information is not cached.
(...) about 1600 files (...) We can loop through them each time, but how much latency should we expect that to create?Well, SYNC is much faster because the file listing with file atributes is retrieved once per directory. The solution with GETFILESIZE takes longer. I don't know how much time, it depends on the connection to the FTP server.
Note that if you don't want to use GETFILESIZE I could modify SYNC to compare also file sizes (as well as file time). This is called a customized version of ScriptFTP and the fees are usually around USD 85.
Your solution to compare file sizes and initiate a download if different works fine for a single file, but in our case, we need to do this on about 1600If the 1600 files are in the same directory you can achieve this using FOREACH. If the directory tree does not have more than two levels you can also use FOREACH but the script gets slightly complicated. Let me know if you need further help on this.
liked Script FTP? Try our other product Email Parser
Thank you for the offer of custom development of the SYNC command. We'll see if we can work around that limitation for now and get back to you if we must have the custom development. Maybe there are other licensed users out there wanting the same thing.
For anyone following along and for future reference for the community...
Yesterday we implemented the SYNC as a loop:
One thing we have going for us is that our files are created in a chronological fashion and follow a strict naming convention that reflects the year, month, day, hour, and minute the file is created. Our next step will be to implement a mechanism by which the loop above will ignore files that were previously SYNC'd. To do this, we'll write the "number" (YYYYMMDDhhmm) of the last file processed out to a file when the script completes. The next time the script starts, we'll read in that "number" and while looping, only check files which have a higher time stamp than the last file SYNC'd. This will cut down on the number of GETFILESIZE() calls that need to be made to the remote server and should decrease the overall script execution time.
Of course, right now we're retrieving the file list from the remote server directly after connecting and looping forward through the files. As this is taking an hour, it could be that hour time is giving the files time to complete writing to disk before the SYNC call (See original problem). If this happens with the numbering logic, we could always implement a SLEEP() call in a loop until we see the downloaded file size match.
Anyone see anything there that could backfire on us?
For anyone following along and for future reference for the community...
Yesterday we implemented the SYNC as a loop:
Code: Select all
We caught half a dozen files of mismatched file size, but never got to the point in execution where after the GETFILE() the files were still of different sizes. Keep in mind that the mismatched file sizes may have been the result of the last run of our previous script in which the SYNC was not done as part of a loop as above. The good news is that this is working well for us and related processes relying on this one are happy again. Subsequent runs of the same script above saw no mismatched file sizes. The bad news is that one pass takes a little over an hour.FOREACH $fileName IN $fileList
PRINT("Work file: ". $fileName)
$result=SYNC($LocalStorePath, $RemotePath, DOWNLOAD, $fileName)
IF($result=="OK")
PRINT("File SYNC ok, checking sizes: " . $fileName)
$remoteFileSize = GETFILESIZE(REMOTE, $fileName)
$localFileSize = GETFILESIZE(LOCAL, $fileName)
IF($remoteFileSize != $localFileSize)
PRINT("File size did not match after sync: " . $fileName)
PRINT("Remote: " . $remoteFileSize." Local: " . $localFileSize)
PRINT("Forcing re-download of file")
GETFILE($fileName)
$remoteFileSize = GETFILESIZE(REMOTE, $fileName)
$localFileSize = GETFILESIZE(LOCAL, $fileName)
IF($remoteFileSize != $localFileSize)
PRINT("File sizes still don't match after RAW GET")
PRINT("Remote: " . $remoteFileSize . " Local: " . $localFileSize)
END IF
END IF
ELSE
PRINT("SYNC call failed on file: " . $fileName)
PRINT("Attempting GET")
GETFILE($fileName)
END IF
END FOREACH
One thing we have going for us is that our files are created in a chronological fashion and follow a strict naming convention that reflects the year, month, day, hour, and minute the file is created. Our next step will be to implement a mechanism by which the loop above will ignore files that were previously SYNC'd. To do this, we'll write the "number" (YYYYMMDDhhmm) of the last file processed out to a file when the script completes. The next time the script starts, we'll read in that "number" and while looping, only check files which have a higher time stamp than the last file SYNC'd. This will cut down on the number of GETFILESIZE() calls that need to be made to the remote server and should decrease the overall script execution time.
Of course, right now we're retrieving the file list from the remote server directly after connecting and looping forward through the files. As this is taking an hour, it could be that hour time is giving the files time to complete writing to disk before the SYNC call (See original problem). If this happens with the numbering logic, we could always implement a SLEEP() call in a loop until we see the downloaded file size match.
Anyone see anything there that could backfire on us?
I am trying to do the same sort of thing here based on your nifty script but I am getting errors.
Getting same error for each file in the loop
Work file: 1012.TXT
Attempting GET
I have enclosed a copy of my script.
Getting same error for each file in the loop
Work file: 1012.TXT
SYNC("","",DOWNLOAD,"1012.TXT")SYNC call failed on file: 1012.TXT
***** SYNC Error #123 : Cannot change current local directory to .
***** The system said: The filename, directory name, or volume label syntax is incorrect.
Attempting GET
I have enclosed a copy of my script.
Code: Select all
Any help would be appreciated
# Connect to FTP server
OPENHOST("ftp.myftpsite.com","username","pass")
# All files will now be downloaded here
LOCALCHDIR("D:\inetpub\64.118.67.198\inbound")
# Change current remote directory
CHDIR("/inbound)
# Retrieve remote file listing
GETLIST($filelist, REMOTE_FILES)
FOREACH $fileName IN $fileList
PRINT("Work file: ". $fileName)
$result=SYNC($LocalStorePath, $RemotePath, DOWNLOAD, $fileName)
IF($result=="OK")
PRINT("File SYNC ok, checking sizes: " . $fileName)
$remoteFileSize = GETFILESIZE(REMOTE, $fileName)
$localFileSize = GETFILESIZE(LOCAL, $fileName)
IF($remoteFileSize != $localFileSize)
PRINT("File size did not match after sync: " . $fileName)
PRINT("Remote: " . $remoteFileSize." Local: " . $localFileSize)
PRINT("Forcing re-download of file")
GETFILE($fileName)
$remoteFileSize = GETFILESIZE(REMOTE, $fileName)
$localFileSize = GETFILESIZE(LOCAL, $fileName)
IF($remoteFileSize != $localFileSize)
PRINT("File sizes still don't match after RAW GET")
PRINT("Remote: " . $remoteFileSize . " Local: " . $localFileSize)
END IF
END IF
ELSE
PRINT("SYNC call failed on file: " . $fileName)
PRINT("Attempting GET")
GETFILE($fileName)
END IF
END FOREACH
# Close connection
CLOSEHOST
Hello jmayer,
You are getting that error because you have to define the content of the variables $LocalStorePath and $RemotePath, for example:
http://www.ScriptFTP.com/ScriptFTP_3_3_ ... mesinc.exe
Of course this customized version is available for anyone. If it gets popular I will add a switch to the mainstream build of ScriptFTP to change the synchronization criteria. If someone founds that the link is broken (I may remove this file in the future) please contact support or post here to get it.
You are getting that error because you have to define the content of the variables $LocalStorePath and $RemotePath, for example:
Code: Select all
But you don't need to use this script as I have prepared a customized version of ScriptFTP that synchronizes files based on the file size instead of the file modification time. You can download it at:$LocalStorePath="C:\mylocalstorepath\"
$RemotePath="/remotedir/remotesubdir"
http://www.ScriptFTP.com/ScriptFTP_3_3_ ... mesinc.exe
Of course this customized version is available for anyone. If it gets popular I will add a switch to the mainstream build of ScriptFTP to change the synchronization criteria. If someone founds that the link is broken (I may remove this file in the future) please contact support or post here to get it.
liked Script FTP? Try our other product Email Parser
- All times are UTC -