ftp automation

ScriptFTP

The professional tool to automate FTP, SFTP, FTPS and schedule FTP batch jobs

The forum is now read only. Please, go to the the main ScriptFTP website if you need help.
Post here if you experience file transfer problems or unexpected errors.
The scripting documentation states that SYNC will transfer a file only if it is newer. Does SYNC only check the file creation time stamp to determine if a file is newer?

In our situation, we're seeing a file size difference between a file on the FTP server and the same file on the client after a SYNC. On the next SYNC pass, it looks like the file is ignored because of the time stamp, even though the file contents have changed. We're downloading from the server to the client. Both client and server are running Windows 2003.

The file size difference is likely being caused by a situation where the file on the server is still being written when ScriptFTP initiates the download. This seems to be confirmed by the fact that the process that consumes the files on the client sees the file as "corrupted" and can't process the file.

If anyone else has experienced this problem and devised a way around it, please share.
Hi UnMonkey,
Does SYNC only check the file creation time stamp to determine if a file is newer?
Yes, SYNC is based only on the file modification time stamp.
The file size difference is likely being caused by a situation where the file on the server is still being written when ScriptFTP initiates the download.
Yes, I think this the origin of the problem. The file time stamp is usually updated once a write operation has finished. If ScriptFTP downloads a corrupted file confirms this fact.

The best way to do that, in my own opinion, is to configure the process that runs in the server to modify the files outside the directory where ScriptFTP is downloading files. I don't know if this is possible or not. Other way is to use the command GETFILESIZE to compare local and remote file sizes and call GETFILE if it's different. Something like:
Code: Select all# Get local and remote file sizes $local_file_size=GETFILESIZE(LOCAL,"data.log") $remote_file_size=GETFILESIZE(REMOTE,"data.log") # if sizes are different download the file IF($local_file_size!=$remote_file_size) GETFILE("data.log") END IF  
You can use FOREACH to do the same thing to more than one file.
Yes, SYNC is based only on the file modification time stamp.
Thank you for the confirmation.

How efficient is a call to GETFILESIZE() on the remote server? Does GETLIST also gets file size and time and cache that information for subsequent calls to GETFILESIZE()? I can understand why that would not be ideal in all cases.

Your solution to compare file sizes and initiate a download if different works fine for a single file, but in our case, we need to do this on about 1600 files to ensure the SYNC across both file time and file size. We can loop through them each time, but how much latency should we expect that to create?
How efficient is a call to GETFILESIZE() on the remote server?
Each time GETFILESIZE is called ScriptFTP will send the FTP protocol SIZE command to the FTP server.
Does GETLIST also gets file size and time and cache that information for subsequent calls to GETFILESIZE()?
No, the information is not cached.
(...) about 1600 files (...) We can loop through them each time, but how much latency should we expect that to create?
Well, SYNC is much faster because the file listing with file atributes is retrieved once per directory. The solution with GETFILESIZE takes longer. I don't know how much time, it depends on the connection to the FTP server.

Note that if you don't want to use GETFILESIZE I could modify SYNC to compare also file sizes (as well as file time). This is called a customized version of ScriptFTP and the fees are usually around USD 85.
Your solution to compare file sizes and initiate a download if different works fine for a single file, but in our case, we need to do this on about 1600
If the 1600 files are in the same directory you can achieve this using FOREACH. If the directory tree does not have more than two levels you can also use FOREACH but the script gets slightly complicated. Let me know if you need further help on this.
Thank you for the offer of custom development of the SYNC command. We'll see if we can work around that limitation for now and get back to you if we must have the custom development. Maybe there are other licensed users out there wanting the same thing.

For anyone following along and for future reference for the community...

Yesterday we implemented the SYNC as a loop:
Code: Select allFOREACH $fileName IN $fileList     PRINT("Work file: ". $fileName)     $result=SYNC($LocalStorePath, $RemotePath, DOWNLOAD, $fileName)     IF($result=="OK")         PRINT("File SYNC ok, checking sizes: " . $fileName)         $remoteFileSize = GETFILESIZE(REMOTE, $fileName)         $localFileSize = GETFILESIZE(LOCAL, $fileName)         IF($remoteFileSize != $localFileSize)             PRINT("File size did not match after sync: " . $fileName)             PRINT("Remote: " . $remoteFileSize." Local: " . $localFileSize)             PRINT("Forcing re-download of file")             GETFILE($fileName)             $remoteFileSize = GETFILESIZE(REMOTE, $fileName)             $localFileSize = GETFILESIZE(LOCAL, $fileName)             IF($remoteFileSize != $localFileSize)                 PRINT("File sizes still don't match after RAW GET")                 PRINT("Remote: " . $remoteFileSize . " Local: " . $localFileSize)             END IF         END IF     ELSE            PRINT("SYNC call failed on file: " . $fileName)         PRINT("Attempting GET")         GETFILE($fileName)     END IF END FOREACH  
We caught half a dozen files of mismatched file size, but never got to the point in execution where after the GETFILE() the files were still of different sizes. Keep in mind that the mismatched file sizes may have been the result of the last run of our previous script in which the SYNC was not done as part of a loop as above. The good news is that this is working well for us and related processes relying on this one are happy again. Subsequent runs of the same script above saw no mismatched file sizes. The bad news is that one pass takes a little over an hour.

One thing we have going for us is that our files are created in a chronological fashion and follow a strict naming convention that reflects the year, month, day, hour, and minute the file is created. Our next step will be to implement a mechanism by which the loop above will ignore files that were previously SYNC'd. To do this, we'll write the "number" (YYYYMMDDhhmm) of the last file processed out to a file when the script completes. The next time the script starts, we'll read in that "number" and while looping, only check files which have a higher time stamp than the last file SYNC'd. This will cut down on the number of GETFILESIZE() calls that need to be made to the remote server and should decrease the overall script execution time.

Of course, right now we're retrieving the file list from the remote server directly after connecting and looping forward through the files. As this is taking an hour, it could be that hour time is giving the files time to complete writing to disk before the SYNC call (See original problem). If this happens with the numbering logic, we could always implement a SLEEP() call in a loop until we see the downloaded file size match.

Anyone see anything there that could backfire on us?
I am trying to do the same sort of thing here based on your nifty script but I am getting errors.

Getting same error for each file in the loop

Work file: 1012.TXT
SYNC("","",DOWNLOAD,"1012.TXT")
***** SYNC Error #123 : Cannot change current local directory to .
***** The system said: The filename, directory name, or volume label syntax is incorrect.
SYNC call failed on file: 1012.TXT
Attempting GET


I have enclosed a copy of my script.

Code: Select all # Connect to FTP server OPENHOST("ftp.myftpsite.com","username","pass") # All files will now be downloaded here LOCALCHDIR("D:\inetpub\64.118.67.198\inbound") # Change current remote directory CHDIR("/inbound) # Retrieve remote file listing GETLIST($filelist, REMOTE_FILES) FOREACH $fileName IN $fileList PRINT("Work file: ". $fileName) $result=SYNC($LocalStorePath, $RemotePath, DOWNLOAD, $fileName) IF($result=="OK") PRINT("File SYNC ok, checking sizes: " . $fileName) $remoteFileSize = GETFILESIZE(REMOTE, $fileName) $localFileSize = GETFILESIZE(LOCAL, $fileName) IF($remoteFileSize != $localFileSize) PRINT("File size did not match after sync: " . $fileName) PRINT("Remote: " . $remoteFileSize." Local: " . $localFileSize) PRINT("Forcing re-download of file") GETFILE($fileName) $remoteFileSize = GETFILESIZE(REMOTE, $fileName) $localFileSize = GETFILESIZE(LOCAL, $fileName) IF($remoteFileSize != $localFileSize) PRINT("File sizes still don't match after RAW GET") PRINT("Remote: " . $remoteFileSize . " Local: " . $localFileSize) END IF END IF ELSE PRINT("SYNC call failed on file: " . $fileName) PRINT("Attempting GET") GETFILE($fileName) END IF END FOREACH # Close connection CLOSEHOST  
Any help would be appreciated
Hello jmayer,

You are getting that error because you have to define the content of the variables $LocalStorePath and $RemotePath, for example:
Code: Select all$LocalStorePath="C:\mylocalstorepath\" $RemotePath="/remotedir/remotesubdir"
But you don't need to use this script as I have prepared a customized version of ScriptFTP that synchronizes files based on the file size instead of the file modification time. You can download it at:

http://www.ScriptFTP.com/ScriptFTP_3_3_ ... mesinc.exe

Of course this customized version is available for anyone. If it gets popular I will add a switch to the mainstream build of ScriptFTP to change the synchronization criteria. If someone founds that the link is broken (I may remove this file in the future) please contact support or post here to get it.