Jump to content
PDS Geosciences Node Community

Dan Scholes

Geo Staff
  • Content count

    25
  • Joined

  • Last visited

Community Reputation

4 Neutral

About Dan Scholes

  • Rank
    Geo Staff

Profile Information

  • Gender
    Male
  • Location
    Washington University in St. Louis
  • Interests
    databases, application development

Recent Profile Visitors

The recent visitors block is disabled and is not being shown to other users.

  1. Version 1.0.0

    97 downloads

    This zip file includes example Wget commands for downloading files from the PDS Geosciences Node. The first example demonstrates downloading a PDS data set from the PDS Geosciences Node archive. The second example demonstrates using Wget to download an Orbital Data Explorer (ODE) cart request.
  2. Version 1.0.0

    96 downloads

    This zip file contains a sample Python 3.6 script for downloading files from the PDS Geosciences Node. The sample includes a configuration for downloading data files from both a PDS Geosciences Node archive and files from an Orbital Data Explorer (ODE) cart request. The script supports multiple levels of sub directories, as well. The script includes variables that should be set by the user for his or her environment. The example PDS data set and ODE cart request both exist, and they are available for test executions of the script. Python 3.6 is required for the script to function.
  3. Below I have included a Python 3.6 sample script for downloading files from the PDS Geosciences Node. The sample includes a configuration for downloading data files from both a PDS Geosciences Node archive and files from an Orbital Data Explorer (ODE) cart request. The script supports multiple levels of sub directories, as well. The script includes variables that should be set by the user for his or her environment. The example PDS data set and ODE cart request both exist, and they are available for test executions of the script. Python 3.6 is required for the script to function. This script is also available for download in the downloads section of the forum. # PDSGeosciencesNode_FileDownload.py # Dan Scholes 2/19/18 # Pypthon 3.6 compatible version # Example of downloading data files using # links from HTTP PDS Geosciences Node Data Archive # or Orbital Data Explorer (ODE) Cart location # Note: One drawback of this script is that it downloads one file at a time, rather than multiple streams. # Additional Note: In the future, changes to the PDS Geosciences Node website and Orbital Data Explorer website may cause this example to no longer function. # Disclaimer: This sample code is provided "as is", without warranty of any kind, express or implied. In no event shall the author be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the sample code or the use or other dealings with the sample code. # Phython download website: https://www.python.org/downloads/ import urllib.request import re import time from pathlib import Path # Variables for user to populate---------- saveFilesToThisDirectory = 'c:/temp/data/' # local destination path to save files #next two lines are for downloading from the PDS Geosciences Node archive url = "http://pds-geosciences.wustl.edu/mro/mro-m-rss-5-sdp-v1/mrors_1xxx/" #enter the directory you would like to download relativeLinkPathBase = "http://pds-geosciences.wustl.edu" #this is the default location for the relative paths on the website (just leave this value) #next two lines are for downloading an ODE cart request #url = "http://ode.rsl.wustl.edu/cartdownload/data/sample/" #enter the directory you would like to download #relativeLinkPathBase = "http://ode.rsl.wustl.edu/" #this is the default location for the relative paths on the ode cart website (just leave this value) recursiveVal = True # True/False whether to download files in subdirectories of the specified location in the url variable verboseMessages = False # True/False whether to display verbose messages during the script processing # End of variables for user to populate---------- relativeLinkPathBase = relativeLinkPathBase.rstrip('/') maxDownloadAttempts = 3 filesToDownloadList = [] def get_pageLinks(inUrl,inRecursive): if verboseMessages: print("Cataloging Directory: ",inUrl) #directory to process myURLReader = urllib.request.urlopen(inUrl.rstrip('/')) myResults = myURLReader.read().decode('utf-8').replace("<a href=","<A HREF=").replace("</a>","</A>") myURLReader.close() data=myResults.split("</A>") tag="<A HREF=\"" endtag="\">" for item in data: if "<A HREF" in item: try: ind = item.index(tag) item=item[ind+len(tag):] end=item.index(endtag) except: pass else: #The link is found itemToDownload = item[:end] if "." in itemToDownload: #the link is to a file if relativeLinkPathBase not in itemToDownload: #is the path relative, so we add the base url itemToDownload = relativeLinkPathBase + itemToDownload filesToDownloadList.append(itemToDownload) else: # it's a directory, so let's go into it if recursive is chosen if inRecursive: if itemToDownload not in inUrl: #we make sure it isn't a link to parent directory if relativeLinkPathBase not in itemToDownload: itemToDownload = relativeLinkPathBase + itemToDownload # the directory is a subdirectory, so we will follow it if verboseMessages: print("subdirectory to process ", itemToDownload) get_pageLinks(itemToDownload,inRecursive) def download_files(): # download the files that were identified # this is refering to the global list of files to download localSuccessfulDownloads = 0 print("==Downloads starting ==============") for link in filesToDownloadList: downloadAttempts = 0 fileDownloaded = False if verboseMessages: print("downloading file: ",link) local_link = link; local_link = saveFilesToThisDirectory + local_link.replace(relativeLinkPathBase,"") local_filename = link.split('/')[-1] #make sure the local directory stucture has been created path = Path(local_link.replace(local_filename,"")) path.mkdir(parents=True, exist_ok=True) while not fileDownloaded and downloadAttempts < maxDownloadAttempts: try: urllib.request.urlretrieve(link,local_link) localSuccessfulDownloads += 1 fileDownloaded = True except urllib.error.URLError as e: downloadAttempts += 1 #we will retry the download the number of times allowed by maxDownloadAttempts variable if verboseMessages: print("downloadError: ",e.reason) if verboseMessages: print("downloadErrorFile: ",link," attempt:",downloadAttempts) if downloadAttempts < maxDownloadAttempts: time.sleep(15) #wait 15 seconds before the next attempt else: print("Could not successfully download: ",link," after ",downloadAttempts," download attempts") print("==Downloads complete ==============") print("SuccessfulDownloads: ",localSuccessfulDownloads," out of ",len(filesToDownloadList)) print('==Process is starting ===================') #get the file links get_pageLinks(url, recursiveVal) print("==Collected ", len(filesToDownloadList), " file links ======") #now download the files download_files()
  4. Below I have included example Wget commands for downloading files from the PDS Geosciences Node. The first example demonstrates downloading a PDS data set from the PDS Geosciences Node archive. The second example demonstrates using Wget to download an Orbital Data Explorer (ODE) cart request. Dan Scholes 2/20/18 Example of downloading data files using links from HTTP PDS Geosciences Node Data Archive or Orbital Data Explorer (ODE) Cart location Note: In the future, changes to the PDS Geosciences Node website and Orbital Data Explorer website may cause this example to no longer function. Disclaimer: This sample code is provided "as is", without warranty of any kind, express or implied. In no event shall the author be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the sample code or the use or other dealings with the sample code. Wget website: https://www.gnu.org/software/wget/ Example of downloading a PDS Geosciences Node archive subdirectory wget -rkpN -P c:\temp\data -nH --cut-dirs 2 --level=15 --no-parent --reject "index.html*" -e robots=off http://pds-geosciences.wustl.edu/mro/mro-m-crism-4-typespec-v1/mrocr_8001/ Example of downloading ODE Cart Request wget -rkpN -P c:\temp\data -nH --cut-dirs 2 --level=15 --no-parent --reject "index.html*" -e robots=off http://ode.rsl.wustl.edu/cartdownload/data/sample -r means recursively download files -k means convert links. Links on the webpage will be localhost instead of example.com/path. -p means get all webpage resources, so wget will obtain images and javascript files to make website work properly. -N is to retrieve timestamps, so if local files are newer than files on remote website, the remote files will be skipped. -P sets the local destination directory for the downloaded files. -e is a flag option that must be set for the robots=off to work. robots=off means ignore robots file. -c allows the command to pick up where it left off if the connection is dropped and the command I re-run. --no-parent keeps the command from downloading all the files in the directories above the requested level. --reject "index.html*" keeps wget from downloading every directory's default index.html. -nH will disable the generation of the host-prefixed directories. In the example above, a directory ode.rsl.wustl.edu will not be created locally. --cut-dirs 2 Ignore the count of directory components. Basically, this example will omit the first 2 directory levels from the path it creates locally for the files that are downloaded. Example: http://ode.rsl.wustl.edu/cartdownload/data/sample The first directory in the destination directory will be "sample". --level=depth --level=15 Levels to recursively search. The default is 5, but we will need to go farther with ODE cart and PDS Geosciences Archive. --------------------------------------------------------------------------------------------------------------------------------------------------------- -nd or --no-directories it is used to put all the requested files in one directory. We are not using this feature, but a user may prefer this option.
  5. ODE has recently released a new ODE cart request download page. This is a new option for accessing a requested ODE cart request. A link to the page is provided in the order fulfillment email that a user receives after their cart request is ready for download. The new ODE cart request download page supports multiple download methods and offers faster options for users to retrieve requested ODS files. The ODE cart request download page contains a single click link to download the entire user cart request using the Aspera Connect web browser plug-in. This option provides a convenient and fast download method for most users. The Aspera Connect browser plug-in employs the FASP data transfer technology to better utilize the user’s available bandwidth. HTTP and FTP links are provided on the page for users who prefer to download the files through a client application, a different web browser plug-in, a local script, or directly through the web browser. Additionally, the page supports browsing and downloading specific directories and files of the cart request through the Aspera Connect web browser plug-in or using direct HTTP links. Multiple files can be downloaded simultaneously through this page. Try the new page the next time you request PDS products through the ODE cart!
  6. ODE has recently added a new option to download all of a PDS product's files through a single click. The new option is available from the product detail page of PDS products hosted by the PDS Geosciences Node. This option uses the Aspera Connect browser plug-in. Aspera technology more fully utilizes one's available bandwidth than traditional TCP connections, such as HTTP and FTP, which will improve the speed the file download speed. Give the new feature a try and let us know how it works for you. Example page with the option: http://ode.rsl.wustl.edu/mars/indexproductpage.aspx?product_id=FRT0000B278_07_IF164S_TRR3&product_idGeo=17880484 Help page further describing the feature and installation of the Aspera Connect browser plug-in: http://ode.rsl.wustl.edu/mars/pagehelp/quickstartguide/index.html?product_detail_page.htm#aspera
  7. The PDS Geosciences Node released the MakeLabels tool this July. The program can be used to generate PDS4 labels using a label template and one or two Excel spreadsheets. It replaces placeholders in the template with values from the spreadsheets. In fact it can be used to output any type of ASCII file. More information about the program and it's downloadable file can be found on the PDS Geosciences Node website. http://pds-geosciences.wustl.edu/tools/makelabels.html
  8. Dan Scholes

    ODE REST Python examples

    The ODE team has posted three simple Python examples of querying the ODE REST interface. These scripts are found in the forum's download section. https://geoweb.rsl.wustl.edu/community/index.php?/files/file/26-ode-rest-python-examples/
  9. Dan Scholes

    ODE REST Python Examples

    Version 1.0.0

    316 downloads

    Attached are three simple Python examples of querying the ODE REST interface. odeProductCount_Python27.py - A Python 2.7 example of querying the ODE REST interface for the count of products of particular product type. The results are written to the command prompt interface. odeProductCount_Python36.py - A Python 3.6 example of querying the ODE REST interface for the count of products of particular product type. The results are written to the command prompt interface. odeREST_FileDownload.py - A Python 3.6 example of querying ODE for products of a particular mission/instrument/product type. The product files of the first 10 query results will be downloaded to the local machine.
  10. Dan Scholes

    Map Projection for ENVI Users

    Note for ENVI users of map-projected data: Some Geosciences Node hosted map-projected data sets (.IMG files) are not read by ENVI as projected data sets. Using the Geospatial Data Abstraction Library (GDAL) gdal_translate command, users can easily convert most of these data products into GeoTIFF or ENVI file formats which are read by ENVI as projected data. GDAL packages for various computer systems can be found here: https://trac.osgeo.org/gdal/wiki/DownloadingGdalBinaries. The Windows package used in this example is the OSGEO4W 64-bit version. A MacOS X build is available, as well. Download and install a GDAL package that has gdal_translate.exe. gdal_translate.exe runs from the command line. Make the folder that contains gdal_translate.exe your current directoryTo do this, type "cd path" without the quotes into the command line, where "path" is the full path to the folder that contains gdal_translate.exe. gdal_translate.exe is executed with the following syntax: gdal_translate –of outputtype inputfilepath outputfilepath "outputtype" is GTiff for a GeoTiff (.tif) and ENVI for an ENVI file (.dat) Both file paths are the full path to the file. In the case of a file with a detached PDS label, the "inputfilepath" points to the label, not the data file. Open the file as normal in ENVI. Example using OSGeo4W 64-bit (Windows): C:\>cd OSGeo4W64\bin C:\OSGeo4W64\bin>gdal_translate -of GTiff C:\hrsc\h2064_0000_dt4.img C:\hrsc\h2064_0000_dt4.tif Example using GDAL 2.1 Complete (MacOS X): :/ cd Library/Frameworks/GDAL.framework/Programs :Programs ./gdal_translate -of ENVI /Desktop/hrsc/h2064_0000_dt4.img /Desktop/hrsc/h2064_0000_dt4.dat Examples of data sets that benefit from the GDAL translate command: MRO - HiRISE DTM Mars Express - all HRSC, OMEGA DDRGM Odyssey - THEMIS IRGEO2, VGEO1, VGEO2 MGS - MOLA MEGDR, TES TIMAP Additional instructions for specific data sets: Mars Odyssey THEMIS Visible map-projected .CUB files (VGEO1 and VGEO2) have detached PDS labels, but the map-projection data is in the .CUB file, so the "inputfilepath" should be the data file itself. Mars Odyssey THEMIS Infrared map-projected .CUB.gz files (IRGEO2) need to be un-zipped before using gdal_translate. These data are similar to the visible data files in "inputfilepath" syntax (.CUB, not .LBL). More details about the gdal_translate command can be found at http://www.gdal.org/gdal_translate.html.
  11. Dan Scholes

    Lola altitude data problem

    No problem. Just let me know if you experience any further problems. Thanks, Dan
  12. Dan Scholes

    Lola altitude data problem

    Hi Phil, I believe I have fixed the problem. Please give it a try with Chrome. Thank you very much for bringing the problem to our attention! Dan
  13. Dan Scholes

    Lola altitude data problem

    Hi Phil, Thank you for contacting us about the problem you experienced. I ran some tests. I can recreate the problem with Chrome and Edge, but files download properly with Firefox. I will need to research this further. Which browser and OS are you using? Thanks, Dan
  14. The following ODE website enhancements were deployed May 31, 2016: Functionality to search by a list of PDS product ids, including wildcards Mars ODE - ability to filter search results by Mars Year Mars ODE - ability to download non-PDS processed versions of MRO CTX images from ASU's Mars Space Flight Facility's Image Explorer from product detail pages or at cart checkout Mars ODE - ability to download MRO CRISM TRDR's corresponding DDR files at cart checkout, for improved interface usability Deployed help documentation updates for website features and existing data sets available through ODE
  15. May 5th, 2016 – MRO HiRISE Updates - MRO HiRISE EDR, RDR, DTM and Anaglyph data products released through April 2nd, 2016 (Orbit 45,399) See https://wufs.wustl.edu/ode/odeholdings/Mars_holdings.html for a full list of holdings in Mars ODE.
×