I am having trouble adding exclusions to my SYNC command. My script is trying to access and download somewhere around 3000 individual files from the US Census FTP site. As you might expect, I am running into connection problems. Given the amount of time it takes to download all of these files, I can accept that fact. Since I didn't want to re-download all of my successful files I figured using SYNC would be my best option. But because of my connection problems, I would like to skip all of the subdirs I have successfully accessed and downloaded from. My attempt to skip these subdirs took the form of ADDEXCLUSIONs. Yet every time I run the script, SYNC will still try and look for an update. I have two questions:
1. Is it even possible to ADDEXCLUSIONS to SYNC?
2. And if so, where have I gone wrong?
Thanks in advance!
1. Is it even possible to ADDEXCLUSIONS to SYNC?
2. And if so, where have I gone wrong?
Thanks in advance!
Code: Select all
# Author: Logan Suhr
# Company: First American Spatial Solutions
# Date: 11.5.09
# Email: lsuhr@firstam.com
# web: faspatial.com
# This script accesses the US Census Bureau FTP site. Specifically it locates
# and downloads the chosen Street Line shapefiles for every county in the US
# to a local server.
SILENT(OFF)
VERBOSE(OFF)
# Go to Q:\Tiger Data\2009 Street Centerlines
LOCALCHDIR("Q:\Tiger Data\2009 Street Centerlines")
:reconnect
$attempts=0
:connect
# Connect to FTP server
$result=OPENHOST("ftp2.census.gov","anonymous","")
# Increment the connection attempts counter
$attempts=$attempts+1
# Check if $result is different from "OK"
IF($result!="OK")
# If this is the third attempt stop execution
IF($attempts==3)
STOP
ELSE
PRINT("Cannot connect! Trying again.")
# Sleep 10 seconds
SLEEP(10)
# Jump to the label :connect to retry
# the connection
GOTO :connect
END IF
END IF
# This is a list of state paths to ignore during download.
ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/01_ALABAMA/")
ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/02_ALASKA/")
ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/04_ARIZONA/")
ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/05_ARKANSAS/")
ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/06_CALIFORNIA/")
ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/08_COLORADO/")
ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/09_CONNECTICUT/")
ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/10_DELAWARE/")
ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/11_DISTRICT_OF_COLUMBIA/")
ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/12_FLORIDA/")
ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/13_GEORGIA/")
ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/15_HAWAII/")
ADDEXCLUSION(DOWNLOAD,"geo/tiger/TIGER2009/16_IDAHO/")
# Download all the modified and new edges.zip files from the directory
# /geo/tiger/TIGER2009 located in the FTP site to the local
# directory Q:\Tiger Data\2009 Street Centerlines. Subdirectories are included
# in the synchronization. Manual clock differnce setting is required for this
# connection.
SETCLOCKDIFF(3600)
IF(SYNC("Q:\Tiger Data\2009 Street Centerlines","/geo/tiger/TIGER2009",DOWNLOAD,SUBDIRS,"*edges.zip")!="OK")
PRINT("Synchronization error. Exiting")
GOTO :reconnect
END IF
# Transfer finished, close the connection
CLOSEHOST
Hello,
The syntax of ADDEXCLUSION is a bit tricky. I copy/paste from the scripting guide:
If it still does not work do not hesitate to post here again.
The syntax of ADDEXCLUSION is a bit tricky. I copy/paste from the scripting guide:
Syntax: ADDEXCLUSION(list,filename,path)If you want to exclude from the download, for example, the remote directory geo/tiger/TIGER2009/01_ALABAMA/, the correct way is:Remarks:
- list: ScriptFTP maintains two exclusion lists: one for uploads and another one for downloads. Use this parameter to indicate which list you want to add the file to.
UPLOAD Select the upload exclusion list.
DOWNLOAD Select the download exclusion list.- filename: local or remote filename (depending on the list you have selected). Wildcards are also supported. Do not include any path in this parameter, only file names.
- path (optional): local or remote path to the file you want to exclude. The path must be a full path, for example C:\Docs\Whatever\ or /remote_dir/whatever/. If you do not supply this parameter ScriptFTP will ignore all files matching the given file name regardless of their path.
This command is capable of ignoring directories as well. Use "/" or "\" at the end of the file name to indicate that it is a directory. See the examples.
Code: Select all
if it still does not work try also:ADDEXCLUSION(DOWNLOAD,"01_ALABAMA/","geo/tiger/TIGER2009/")
Code: Select all
or:ADDEXCLUSION(DOWNLOAD,"01_ALABAMA/","/geo/tiger/TIGER2009/")
Code: Select all
# This one means "exclude from the download any remote directory with the name 01_ALABAMA regardless of its path"
ADDEXCLUSION(DOWNLOAD,"01_ALABAMA/")
If it still does not work do not hesitate to post here again.
liked Script FTP? Try our other product Email Parser
Excellent suggestion. I guess I was trying to be all inclusive when I should have been more selective. I followed your third script example. Works perfectly.
I tried another route for my time out problem that didn't respond how I thought it should. I'll keep the post on this thread because it involves ADDEXCLUSION with SNYC. I'm working around the issue due to a time budget but figured I would get your take on the script.
More often that not, my time out error happens while 'skipping' the previously downloaded files. There are hundreds of them. My solution, which didn't work, was to create running exclusions based on a remote directory list. As one state's files were downloaded completely or skipped over, that state was then added to the exclusion list. This attempt was performed within a FOREACH loop. What I found out is that the FOREACH loop overrides any exclusions I had set and still check every state. While I'm not the most accurate script writer, I thought I had it right.
Performing the IF statement without the FOREACH statement resulted in proper exclusions if I established them before the SYNC. This is why I am looking at the FOREACH statement as the problem.
As an aside, I really enjoying working with ScriptFTP. It's my new best friend and makes me the most productive guy in the office. Cheers.
After I establish my FTP connection, I wrote the following code:
I tried another route for my time out problem that didn't respond how I thought it should. I'll keep the post on this thread because it involves ADDEXCLUSION with SNYC. I'm working around the issue due to a time budget but figured I would get your take on the script.
More often that not, my time out error happens while 'skipping' the previously downloaded files. There are hundreds of them. My solution, which didn't work, was to create running exclusions based on a remote directory list. As one state's files were downloaded completely or skipped over, that state was then added to the exclusion list. This attempt was performed within a FOREACH loop. What I found out is that the FOREACH loop overrides any exclusions I had set and still check every state. While I'm not the most accurate script writer, I thought I had it right.
Performing the IF statement without the FOREACH statement resulted in proper exclusions if I established them before the SYNC. This is why I am looking at the FOREACH statement as the problem.
As an aside, I really enjoying working with ScriptFTP. It's my new best friend and makes me the most productive guy in the office. Cheers.
After I establish my FTP connection, I wrote the following code:
Code: Select all
CHDIR("/geo/tiger/TIGER2009")
GETLIST($remote_subdir_list,REMOTE_DIRECTORIES)
FOREACH $item IN $remote_subdir_list
PRINT ($item)
END FOREACH
# Download all the modified and new edges.zip files from the directory
# /geo/tiger/TIGER2009 located in the FTP site to the local
# directory Q:\Tiger Data\2009 Street Centerlines. Subdirectories are included
# in the synchronization. Manual clock differnce setting is required for this
# connection.
SETCLOCKDIFF(3600)
FOREACH $item IN $remote_subdir_list
IF(SYNC("Q:\Tiger Data\2009 Street Centerlines\".$item,"/geo/tiger/TIGER2009/".$item,DOWNLOAD,SUBDIRS,"*edges.zip")!="OK")
PRINT("Synchronization error. Exiting")
GOTO :reconnect
END IF
ADDEXCLUSION(DOWNLOAD,$item."/","/geo/tiger/TIGER2009/")
END FOREACH
# Transfer finished, close the connection
CLOSEHOST
Hi,
What I found out is that the FOREACH loop overrides any exclusions I had set and still check every stateYes, the exclussions are only applied to the commands that transfer files. These are GETFILE, PUTFILE and SYNC.
liked Script FTP? Try our other product Email Parser