ftp automation

ScriptFTP

The professional tool to automate FTP, SFTP, FTPS and schedule FTP batch jobs

The forum is now read only. Please, go to the the main ScriptFTP website if you need help.
Need help writing a script? Have any question about ScriptFTP?
I am having trouble adding exclusions to my SYNC command. My script is trying to access and download somewhere around 3000 individual files from the US Census FTP site. As you might expect, I am running into connection problems. Given the amount of time it takes to download all of these files, I can accept that fact. Since I didn't want to re-download all of my successful files I figured using SYNC would be my best option. But because of my connection problems, I would like to skip all of the subdirs I have successfully accessed and downloaded from. My attempt to skip these subdirs took the form of ADDEXCLUSIONs. Yet every time I run the script, SYNC will still try and look for an update. I have two questions:

1. Is it even possible to ADDEXCLUSIONS to SYNC?
2. And if so, where have I gone wrong?

Thanks in advance!
Code: Select all# Author: Logan Suhr # Company: First American Spatial Solutions # Date: 11.5.09 # Email: lsuhr@firstam.com # web: faspatial.com # This script accesses the US Census Bureau FTP site.  Specifically it locates # and downloads the chosen Street Line shapefiles for every county in the US # to a local server. SILENT(OFF) VERBOSE(OFF) # Go to Q:\Tiger Data\2009 Street Centerlines LOCALCHDIR("Q:\Tiger Data\2009 Street Centerlines") :reconnect $attempts=0 :connect # Connect to FTP server $result=OPENHOST("ftp2.census.gov","anonymous","") # Increment the connection attempts counter $attempts=$attempts+1 # Check if $result is different from "OK" IF($result!="OK") # If this is the third attempt stop execution IF($attempts==3) STOP ELSE PRINT("Cannot connect! Trying again.")         # Sleep 10 seconds          SLEEP(10) # Jump to the label :connect to retry                 # the connection        GOTO :connect END IF END IF # This is a list of state paths to ignore during download. ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/01_ALABAMA/") ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/02_ALASKA/") ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/04_ARIZONA/") ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/05_ARKANSAS/") ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/06_CALIFORNIA/") ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/08_COLORADO/") ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/09_CONNECTICUT/") ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/10_DELAWARE/") ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/11_DISTRICT_OF_COLUMBIA/") ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/12_FLORIDA/") ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/13_GEORGIA/") ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/15_HAWAII/") ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/16_IDAHO/") # Download all the modified and new edges.zip files from the directory # /geo/tiger/TIGER2009 located in the FTP site to the local # directory Q:\Tiger Data\2009 Street Centerlines. Subdirectories are included # in the synchronization.  Manual clock differnce setting is required for this # connection. SETCLOCKDIFF(3600) IF(SYNC("Q:\Tiger Data\2009 Street Centerlines","/geo/tiger/TIGER2009",DOWNLOAD,SUBDIRS,"*edges.zip")!="OK")  PRINT("Synchronization error. Exiting")  GOTO :reconnect END IF # Transfer finished, close the connection CLOSEHOST  
Hello,

The syntax of ADDEXCLUSION is a bit tricky. I copy/paste from the scripting guide:
Syntax: ADDEXCLUSION(list,filename,path)
  • list: ScriptFTP maintains two exclusion lists: one for uploads and another one for downloads. Use this parameter to indicate which list you want to add the file to.
    UPLOAD Select the upload exclusion list.
    DOWNLOAD Select the download exclusion list.
  • filename: local or remote filename (depending on the list you have selected). Wildcards are also supported. Do not include any path in this parameter, only file names.
  • path (optional): local or remote path to the file you want to exclude. The path must be a full path, for example C:\Docs\Whatever\ or /remote_dir/whatever/. If you do not supply this parameter ScriptFTP will ignore all files matching the given file name regardless of their path.
Remarks:
This command is capable of ignoring directories as well. Use "/" or "\" at the end of the file name to indicate that it is a directory. See the examples.
If you want to exclude from the download, for example, the remote directory geo/tiger/TIGER2009/01_ALABAMA/, the correct way is:
Code: Select allADDEXCLUSION(DOWNLOAD,"01_ALABAMA/","geo/tiger/TIGER2009/")
if it still does not work try also:
Code: Select allADDEXCLUSION(DOWNLOAD,"01_ALABAMA/","/geo/tiger/TIGER2009/")
or:
Code: Select all# This one means "exclude from the download any remote directory with the name 01_ALABAMA regardless of its path" ADDEXCLUSION(DOWNLOAD,"01_ALABAMA/")

If it still does not work do not hesitate to post here again.
Excellent suggestion. I guess I was trying to be all inclusive when I should have been more selective. I followed your third script example. Works perfectly.

I tried another route for my time out problem that didn't respond how I thought it should. I'll keep the post on this thread because it involves ADDEXCLUSION with SNYC. I'm working around the issue due to a time budget but figured I would get your take on the script.

More often that not, my time out error happens while 'skipping' the previously downloaded files. There are hundreds of them. My solution, which didn't work, was to create running exclusions based on a remote directory list. As one state's files were downloaded completely or skipped over, that state was then added to the exclusion list. This attempt was performed within a FOREACH loop. What I found out is that the FOREACH loop overrides any exclusions I had set and still check every state. While I'm not the most accurate script writer, I thought I had it right.

Performing the IF statement without the FOREACH statement resulted in proper exclusions if I established them before the SYNC. This is why I am looking at the FOREACH statement as the problem.

As an aside, I really enjoying working with ScriptFTP. It's my new best friend and makes me the most productive guy in the office. Cheers.

After I establish my FTP connection, I wrote the following code:
Code: Select allCHDIR("/geo/tiger/TIGER2009") GETLIST($remote_subdir_list,REMOTE_DIRECTORIES) FOREACH $item IN $remote_subdir_list     PRINT ($item) END FOREACH # Download all the modified and new edges.zip files from the directory # /geo/tiger/TIGER2009 located in the FTP site to the local # directory Q:\Tiger Data\2009 Street Centerlines. Subdirectories are included # in the synchronization.  Manual clock differnce setting is required for this # connection. SETCLOCKDIFF(3600) FOREACH $item IN $remote_subdir_list     IF(SYNC("Q:\Tiger Data\2009 Street Centerlines\".$item,"/geo/tiger/TIGER2009/".$item,DOWNLOAD,SUBDIRS,"*edges.zip")!="OK")      PRINT("Synchronization error. Exiting")      GOTO :reconnect     END IF     ADDEXCLUSION(DOWNLOAD,$item."/","/geo/tiger/TIGER2009/") END FOREACH # Transfer finished, close the connection CLOSEHOST
Hi,
What I found out is that the FOREACH loop overrides any exclusions I had set and still check every state
Yes, the exclussions are only applied to the commands that transfer files. These are GETFILE, PUTFILE and SYNC.