Errant Deleting of Remote Files After Incomplete Download

[danorth]

I have recently ran into an issue with a script. It appears that some files that have not been fully downloaded from the remote host are being deleted from the remote host. Here is how the issue appears to work.

-Remote file is attempted for download
-Download begins but is not finished and a 0 byte file is place on the local disk
-Connection times out or is interrupted
-Connection is reestablished
-ScriptFTP continues script were it left off, which is at the delete remote file if successful download has occured
-Remote file is deleted without ever being fully downloaded

Full verbose output of said scenario can be found below.

Is there anyway that the connection reestablishment can be prevented? If the connection is broken, regardless of whether or not the file has been downloaded successfully I would rather have the file downloaded a second time rather than be deleted on the remote host.

I cannot seem to pinpoint what I have missed in my script (which can be found below). Any thought on what I need to change?

Thank you for your assistance.

FTP Script: [ Download ] [ Hide ]

FTP Script: [ Download ] [ Show ]

#Declare variables

#FTP connection information.

#Enable Verbose Logging
Verbose (ON)

$ftpserver="xxx.xxxxxx.com"
$user="xxx-xxx"
$password="*******"

#Local download directory
$local_download_dir="C:\directory\directory\directory\directory\directory"

#Remote download directory
$remote_download_dir="/remote_dir"

# Blat emailing parameters
# Documentation can be found at http://www.blat.net/

$blat_path="C:\bin\blat262\full\blat.exe"
 
$smtp_server="222.222.222.222"
$email_from="email@email.com"
$email_to_list="C:\bin\blat262\opt\list.txt"
$email_subject="FTP Download Errors"
$email_body1="Could not connect to FTP server.  Check for connectivity related issues.  Additional details can be found below."
$email_body2="Could not download files.   Check hosted FTP folder contents.  Additional details can be found below."
 
# Build the log file path and append with current date and time.
$log_file_path=("C:\directory\ScriptFTP\Logs")."\logfile-ftp-".GETTIME(FORMAT0).".txt"
 
# The blat command line to send an email should be
# like this (without the line breaks):
#
# C:\blat\blat.exe
# -server smtp.myserver.com
# -u myuser_at_myserver.com
# -pw mypassword
# -f scriptftp@myserver.com
# -to me@myserver
# -subject "ScriptFTP error message"
# -body "Error messsage here"
# -ps "C:\directory\ScriptFTP\Logs\logfile-ftp2011_08_28-16_48_49.txt"
#
# As there are two different kind of emails depending
# on the two errors that may happen we need to build
# two different blat command lines.
#
# Both have a common part, that is the variables
# $common_part_1, $common_part_2 and $common_part_3.
# The fourth part contains the different body and
# they are called $cmd_line_part_4_1 and $cmd_line_part_4_2
#
$common_part_1=$blat_path." -server "
$common_part_2=$smtp_server." -f ".$email_from." -tf ".$email_to_list." -subject ".'"'
$common_part_3=$email_subject.'"'
 
$cmd_line_part_4_1=" -body ".'"'.$email_body1.'"'." -priority 1 -ps ".'"'.$log_file_path.'"'
$cmd_line_part_4_2=" -body ".'"'.$email_body2.'"'." -priority 1 -ps ".'"'.$log_file_path.'"'
 
# Concatenate the text string to build the complete command lines
$blat_cmd_line_1=$common_part_1.$common_part_2.$common_part_3.$cmd_line_part_4_1
$blat_cmd_line_2=$common_part_1.$common_part_2.$common_part_3.$cmd_line_part_4_2

# Start logging the script output to previously created log file path.
LOGTO($log_file_path)

# Set the connection attempts counter to 0
$attempts=0
 
# Return to this point if a connection attempt fails.
:connect

# Display connecting message.
PRINT("****Connecting To Hosted FTP****")

# Connect to ftp.myhost.com as myuser
$result=OPENHOST($ftpserver,$user,$password)

# Increment the connection attempts counter
$attempts=$attempts+1

# Check if $result is different from "OK"
IF($result!="OK")
        # If this is the third attempt stop execution
        IF($attempts==3)
                PRINT("Cannot connect to FTP server. Sending an email and aborting in 5 seconds.")
                                EXEC($blat_cmd_line_1)
                                SLEEP(5)
                                STOP
        ELSE
                PRINT("Cannot connect! Trying again.")
                # Jump to the label :connect to retry
                # the connection
                GOTO :connect
        END IF
END IF
 
# Change the current local directory.
$result=LOCALCHDIR($local_download_dir)
 
# If LOCALCHDIR failed stop the script
IF($result!="OK")
    STOP
END IF
 
# Change remote directory to proper directory
$result=CHDIR($remote_download_dir)
 
# If CHDIR failed stop the script
IF($result!="OK")
    STOP
END IF

# Get the remote file listing, store it in $list
$result=GETLIST($list,REMOTE_FILES,"*.*")
 
# If GETLIST failed stop the script
IF($result!="OK")
    STOP
END IF

# Display a message on screen
PRINT("****Performing File Download With Delete On Succesful Download****")
 
# For each file in $list...
FOREACH $item IN $list
    # Download the file
    $result=GETFILE($item)  
    # If the file has been succesfully downloaded
    # delete the remote copy. If not stop the script.
    IF($result=="OK")
        DELETEFILE($item)
    ELSE
      PRINT("Cannot download files. Sending an email and aborting in 5 seconds.")
      EXEC($blat_cmd_line_2)
      SLEEP(5)
          STOP
    END IF
END FOREACH
 
# Close the connection
CLOSEHOST

[ScriptFTP support]

Hello danorth,

I have put the script output you posted in your message as an attached file.

According to the script output ScriptFTP should return an error. And GETFILE should not return OK. This is a ScriptFTP issue we will investigate...

In the meantime I suggest you to use the command GETFILESIZE to check that the downloaded file has more than 0 bytes. Of course this is only a workaround but it will help:

FTP Script: (check_file_size.ftp) [ Download ] [ Hide ]

FTP Script: (check_file_size.ftp) [ Download ] [ Show ]

FOREACH $item IN $list
    # Download the file
    $result=GETFILE($item)  
    # If the file has been succesfully downloaded
    # delete the remote copy. If not stop the script.
    IF($result=="OK")
        #--------------- Edited by ScriptFTP support
        IF(GETFILESIZE(LOCAL,$item)>0)
                DELETEFILE($item)
        END IF
        #-------------------------------------------
    ELSE
      PRINT("Cannot download files. Sending an email and aborting in 5 seconds.")
      EXEC($blat_cmd_line_2)
      SLEEP(5)
          STOP
    END IF
END FOREACH

[danorth]

Thank you for the response.

Let me know what you find out during your review.

[wglenn]

I don't know if this is the same thing that is happening to me. If the file is being uploaded it shouldn't try and grab it correct? I am getting partial files being brought in, then the file was deleting

We are seeing instances where it looks like the ftp script is downloading partial files from the ftp. It kicks the script off every 3 minutes
It is getting a result of ok and deleting the file.

Have you seen this before or is there anything I can do to make sure it doesn\'t download partial files?

FTP Script: [ Download ] [ Hide ]

FTP Script: [ Download ] [ Show ]

# For each file in $list...
FOREACH $item IN $list
    # Download the file
    $result=GETFILE($item)
    #Get the filesize of folder
    $size=GETFILESIZE(REMOTE,$item)
    PRINT("File Size for ".$item. ": ".$size)
    # If the file has been succesfully downloaded
    # delete the remote copy. If not stop the script.
    IF($result=="OK")
       DELETEFILE($item)
       EXEC('xcopy "\\producer\Hires Graphics\AutoFTPDownload\ROP_Color\"'.$item.' "\\producer\Hires Graphics\CTP Broad Color\" /Y')
    END IF
END FOREACH

[ScriptFTP support]

It kicks the script off every 3 minutes

It may happen that the file located in the FTP site (uploaded by other FTP client) is not complete. If you run the script so often this may be something that it should happen sooner or later.

Consider using lock files. Take a look at this forum topic:

viewtopic.php?f=6&t=566