Script

FTP

The professional tool to automate FTP, SFTP, FTPS

and schedule FTP batch jobs

MENU MENU
Post here if you experience file transfer problems or unexpected errors.
I have recently ran into an issue with a script. It appears that some files that have not been fully downloaded from the remote host are being deleted from the remote host. Here is how the issue appears to work.

-Remote file is attempted for download
-Download begins but is not finished and a 0 byte file is place on the local disk
-Connection times out or is interrupted
-Connection is reestablished
-ScriptFTP continues script were it left off, which is at the delete remote file if successful download has occured
-Remote file is deleted without ever being fully downloaded

Full verbose output of said scenario can be found below.

Is there anyway that the connection reestablishment can be prevented? If the connection is broken, regardless of whether or not the file has been downloaded successfully I would rather have the file downloaded a second time rather than be deleted on the remote host.

I cannot seem to pinpoint what I have missed in my script (which can be found below). Any thought on what I need to change?

Thank you for your assistance.
FTP Script
  1. #Declare variables
  2.  
  3. #FTP connection information.
  4.  
  5. #Enable Verbose Logging
  6. Verbose (ON)
  7.  
  8. $ftpserver="xxx.xxxxxx.com"
  9. $user="xxx-xxx"
  10. $password="*******"
  11.  
  12. #Local download directory
  13. $local_download_dir="C:\directory\directory\directory\directory\directory"
  14.  
  15. #Remote download directory
  16. $remote_download_dir="/remote_dir"
  17.  
  18. # Blat emailing parameters
  19. # Documentation can be found at http://www.blat.net/
  20.  
  21. $blat_path="C:\bin\blat262\full\blat.exe"
  22.  
  23. $smtp_server="222.222.222.222"
  24. $email_from="email@email.com"
  25. $email_to_list="C:\bin\blat262\opt\list.txt"
  26. $email_subject="FTP Download Errors"
  27. $email_body1="Could not connect to FTP server.  Check for connectivity related issues.  Additional details can be found below."
  28. $email_body2="Could not download files.   Check hosted FTP folder contents.  Additional details can be found below."
  29.  
  30. # Build the log file path and append with current date and time.
  31. $log_file_path=("C:\directory\ScriptFTP\Logs")."\logfile-ftp-".GETTIME(FORMAT0).".txt"
  32.  
  33. # The blat command line to send an email should be
  34. # like this (without the line breaks):
  35. #
  36. # C:\blat\blat.exe
  37. # -server smtp.myserver.com
  38. # -u myuser_at_myserver.com
  39. # -pw mypassword
  40. # -f scriptftp@myserver.com
  41. # -to me@myserver
  42. # -subject "ScriptFTP error message"
  43. # -body "Error messsage here"
  44. # -ps "C:\directory\ScriptFTP\Logs\logfile-ftp2011_08_28-16_48_49.txt"
  45. #
  46. # As there are two different kind of emails depending
  47. # on the two errors that may happen we need to build
  48. # two different blat command lines.
  49. #
  50. # Both have a common part, that is the variables
  51. # $common_part_1, $common_part_2 and $common_part_3.
  52. # The fourth part contains the different body and
  53. # they are called $cmd_line_part_4_1 and $cmd_line_part_4_2
  54. #
  55. $common_part_1=$blat_path." -server "
  56. $common_part_2=$smtp_server." -f ".$email_from." -tf ".$email_to_list." -subject ".'"'
  57. $common_part_3=$email_subject.'"'
  58.  
  59. $cmd_line_part_4_1=" -body ".'"'.$email_body1.'"'." -priority 1 -ps ".'"'.$log_file_path.'"'
  60. $cmd_line_part_4_2=" -body ".'"'.$email_body2.'"'." -priority 1 -ps ".'"'.$log_file_path.'"'
  61.  
  62. # Concatenate the text string to build the complete command lines
  63. $blat_cmd_line_1=$common_part_1.$common_part_2.$common_part_3.$cmd_line_part_4_1
  64. $blat_cmd_line_2=$common_part_1.$common_part_2.$common_part_3.$cmd_line_part_4_2
  65.  
  66. # Start logging the script output to previously created log file path.
  67. LOGTO($log_file_path)
  68.  
  69. # Set the connection attempts counter to 0
  70. $attempts=0
  71.  
  72. # Return to this point if a connection attempt fails.
  73. :connect
  74.  
  75. # Display connecting message.
  76. PRINT("****Connecting To Hosted FTP****")
  77.  
  78. # Connect to ftp.myhost.com as myuser
  79. $result=OPENHOST($ftpserver,$user,$password)
  80.  
  81. # Increment the connection attempts counter
  82. $attempts=$attempts+1
  83.  
  84. # Check if $result is different from "OK"
  85. IF($result!="OK")
  86.         # If this is the third attempt stop execution
  87.         IF($attempts==3)
  88.                 PRINT("Cannot connect to FTP server. Sending an email and aborting in 5 seconds.")
  89.                 EXEC($blat_cmd_line_1)
  90.                 SLEEP(5)
  91.                 STOP
  92.         ELSE
  93.                 PRINT("Cannot connect! Trying again.")
  94.                 # Jump to the label :connect to retry
  95.                 # the connection
  96.                 GOTO :connect
  97.         END IF
  98.  
  99. # Change the current local directory.
  100. $result=LOCALCHDIR($local_download_dir)
  101.  
  102. # If LOCALCHDIR failed stop the script
  103. IF($result!="OK")
  104.     STOP
  105.  
  106. # Change remote directory to proper directory
  107. $result=CHDIR($remote_download_dir)
  108.  
  109. # If CHDIR failed stop the script
  110. IF($result!="OK")
  111.     STOP
  112.  
  113. # Get the remote file listing, store it in $list
  114. $result=GETLIST($list,REMOTE_FILES,"*.*")
  115.  
  116. # If GETLIST failed stop the script
  117. IF($result!="OK")
  118.     STOP
  119.  
  120. # Display a message on screen
  121. PRINT("****Performing File Download With Delete On Succesful Download****")
  122.  
  123. # For each file in $list...
  124. FOREACH $item IN $list
  125.     # Download the file
  126.     $result=GETFILE($item)  
  127.     # If the file has been succesfully downloaded
  128.     # delete the remote copy. If not stop the script.
  129.     IF($result=="OK")
  130.         DELETEFILE($item)
  131.     ELSE
  132.       PRINT("Cannot download files. Sending an email and aborting in 5 seconds.")
  133.       EXEC($blat_cmd_line_2)
  134.       SLEEP(5)
  135.       STOP
  136.     END IF
  137.  
  138. # Close the connection
Attachments
Output from ScriptFTP
(2.98 KiB) Downloaded 359 times
Hello danorth,

I have put the script output you posted in your message as an attached file.

According to the script output ScriptFTP should return an error. And GETFILE should not return OK. This is a ScriptFTP issue we will investigate...

In the meantime I suggest you to use the command GETFILESIZE to check that the downloaded file has more than 0 bytes. Of course this is only a workaround but it will help:
FTP Script
  1. FOREACH $item IN $list
  2.     # Download the file
  3.     $result=GETFILE($item)  
  4.     # If the file has been succesfully downloaded
  5.     # delete the remote copy. If not stop the script.
  6.     IF($result=="OK")
  7.     #--------------- Edited by ScriptFTP support
  8.     IF(GETFILESIZE(LOCAL,$item)>0)
  9.             DELETEFILE($item)
  10.     END IF
  11.     #-------------------------------------------
  12.     ELSE
  13.       PRINT("Cannot download files. Sending an email and aborting in 5 seconds.")
  14.       EXEC($blat_cmd_line_2)
  15.       SLEEP(5)
  16.           STOP
  17.     END IF
Thank you for the response.

Let me know what you find out during your review.
I don't know if this is the same thing that is happening to me. If the file is being uploaded it shouldn't try and grab it correct? I am getting partial files being brought in, then the file was deleting

We are seeing instances where it looks like the ftp script is downloading partial files from the ftp. It kicks the script off every 3 minutes
It is getting a result of ok and deleting the file.

Have you seen this before or is there anything I can do to make sure it doesn\'t download partial files?
FTP Script
  1. # For each file in $list...
  2. FOREACH $item IN $list
  3.     # Download the file
  4.     $result=GETFILE($item)
  5.     #Get the filesize of folder
  6.     $size=GETFILESIZE(REMOTE,$item)
  7.     PRINT("File Size for ".$item. ": ".$size)
  8.     # If the file has been succesfully downloaded
  9.     # delete the remote copy. If not stop the script.
  10.     IF($result=="OK")
  11.        DELETEFILE($item)
  12.        EXEC('xcopy "\\producer\Hires Graphics\AutoFTPDownload\ROP_Color\"'.$item.' "\\producer\Hires Graphics\CTP Broad Color\" /Y')
  13.     END IF
It kicks the script off every 3 minutes
It may happen that the file located in the FTP site (uploaded by other FTP client) is not complete. If you run the script so often this may be something that it should happen sooner or later.

Consider using lock files. Take a look at this forum topic:

http://www.scriptftp.com/forum/viewtopic.php?f=6&t=566