Jump to content
PDS Geosciences Node Community

Dan Scholes

Geo Staff
  • Posts

    56
  • Joined

  • Last visited

Everything posted by Dan Scholes

  1. Hi Jairo, Thank you for emailing us with your question. You are receiving the “corrupted file” response because your computer does not have a native application for opening the PDS IMG files. Also, sometimes a cd/dvd image file is called an IMG, so an application used for mounting disk images tries to open the file in a different expected format. A typical PDS image is stored as a simple binary array. It is described by a PDS label (a set of ASCII-text keyword=value statements), which can be either attached or detached. An attached label is embedded at the beginning of the image file; a detached label is in a separate file with the same name, extension LBL. The label tells you everything you need to know to read the image -- the number of lines, number of pixels (samples) per line, the size and data type of a sample, and map projection information if the image is a map. (PDS deliberately does not rely on any commercial or proprietary formats to archive image data, in order to ensure the long-term viability of the data.) Typically IMG files are opened with programs including ENVI, NASA View, or GDAL. ArcMap might support IMG files, but I don’t remember. The IMG files can also be parsed and manipulated with code (Python, IDL, Matlab, C++, C#). In the future, I hope to add functionality into ODE to allow a user to request the files in various formats, but it is not on the near list. In the meantime, GDAL is a standard application for changing formats. http://www.gdal.org/ (general information and the link to download) http://www.gdal.org/gdal_translate.html (basic GDAL commands) Here is an example commands that can be used to convert an img to tif: Gdal_translate -of GTiff D:\test\data\MGN\fmap\fl13s309.img D:\test\data\MGN\fmap\fl13s309.tif NASA View is better for just viewing images: https://pds.nasa.gov/tools/about/pds3-tools/nasa-view.shtml Let us know if you have any further questions. Thanks, Dan
  2. October 10th, 2018 – MRO HiRISE Updates HiRISE EDR, RDR, DTM and Anaglyph data products released through August 31, 2018 (Orbit 56,699) See https://wufs.wustl.edu/ode/odeholdings/Mars_holdings.html
  3. October 8th, 2018 - Mar Odyssey Release 65 Data Updates Updated THEMIS standard data products (IREDR, IRRDR, IRBTR, VISEDR, VISRDR and VISABR data) from February 19, 2002 to March 8, 2018 Updated THEMIS geometry data products (IRGEO2, IRPBT2, VGEO2, and VISALB data) from September 28, 2013 to March 8, 2018 Updated GRS data products (CGS, SGS, DND, AHD, ELEMTS) from February 19, 2002 to January 9, 2018 See https://wufs.wustl.edu/ode/odeholdings/Mars_holdings.html
  4. September 18, 2018 - LRO Release 35 has been loaded into ODE. Updated LOLA EDR products from June 18th, 2009 to July 7th, 2018 Updated LOLA RDR products from July 13th, 2009 to August 8th, 2018 Updated LAMP EDR and RDR products from July 6th, 2009 to June 14th, 2018 Updated LEND EDR_SCI, EDR_HK, RDR_RSCI, and RDR_CHK products from June 18th, 2009 to June 14th, 2018 Updated LEND RDR_DLD and RDR_DLX products from June 27th, 2009 to March 14th, 2018 Updated Diviner EDR and RDR products from July 5th, 2009 to June 15th, 2018 Updated Mini-RF Bistatic data products (EDR, RDR, and DDR) from April 6th, 2011 to December 16th, 2017. Updated LROC EDR, CDR and RDR products from June 30th, 2009 to June 15th, 2018 Updated Diviner RDR query tool database from July 5th, 2009 to June 15th, 2018. Updated LOLA RDR query tool database from July 13th, 2009 to August 8th, 2018 Please see ODE Lunar Holdings - https://wufs.wustl.edu/ode/odeholdings/Moon_holdings.html
  5. Version 1.0.0

    995 downloads

    This zip file includes example Wget commands for downloading files from the PDS Geosciences Node. The first example demonstrates downloading a PDS data set from the PDS Geosciences Node archive. The second example demonstrates using Wget to download an Orbital Data Explorer (ODE) cart request.
  6. Version 1.0.0

    974 downloads

    This zip file contains a sample Python 3.6 script for downloading files from the PDS Geosciences Node. The sample includes a configuration for downloading data files from both a PDS Geosciences Node archive and files from an Orbital Data Explorer (ODE) cart request. The script supports multiple levels of sub directories, as well. The script includes variables that should be set by the user for his or her environment. The example PDS data set and ODE cart request both exist, and they are available for test executions of the script. Python 3.6 is required for the script to function.
  7. Below I have included a Python 3.6 sample script for downloading files from the PDS Geosciences Node. The sample includes a configuration for downloading data files from both a PDS Geosciences Node archive and files from an Orbital Data Explorer (ODE) cart request. The script supports multiple levels of sub directories, as well. The script includes variables that should be set by the user for his or her environment. The example PDS data set and ODE cart request both exist, and they are available for test executions of the script. Python 3.6 is required for the script to function. This script is also available for download in the downloads section of the forum. # PDSGeosciencesNode_FileDownload.py # Dan Scholes 2/19/18 # Pypthon 3.6 compatible version # Example of downloading data files using # links from HTTP PDS Geosciences Node Data Archive # or Orbital Data Explorer (ODE) Cart location # Note: One drawback of this script is that it downloads one file at a time, rather than multiple streams. # Additional Note: In the future, changes to the PDS Geosciences Node website and Orbital Data Explorer website may cause this example to no longer function. # Disclaimer: This sample code is provided "as is", without warranty of any kind, express or implied. In no event shall the author be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the sample code or the use or other dealings with the sample code. # Phython download website: https://www.python.org/downloads/ import urllib.request import re import time from pathlib import Path # Variables for user to populate---------- saveFilesToThisDirectory = 'c:/temp/data/' # local destination path to save files #next two lines are for downloading from the PDS Geosciences Node archive url = "http://pds-geosciences.wustl.edu/mro/mro-m-rss-5-sdp-v1/mrors_1xxx/" #enter the directory you would like to download relativeLinkPathBase = "http://pds-geosciences.wustl.edu" #this is the default location for the relative paths on the website (just leave this value) #next two lines are for downloading an ODE cart request #url = "http://ode.rsl.wustl.edu/cartdownload/data/sample/" #enter the directory you would like to download #relativeLinkPathBase = "http://ode.rsl.wustl.edu/" #this is the default location for the relative paths on the ode cart website (just leave this value) recursiveVal = True # True/False whether to download files in subdirectories of the specified location in the url variable verboseMessages = False # True/False whether to display verbose messages during the script processing # End of variables for user to populate---------- relativeLinkPathBase = relativeLinkPathBase.rstrip('/') maxDownloadAttempts = 3 filesToDownloadList = [] def get_pageLinks(inUrl,inRecursive): if verboseMessages: print("Cataloging Directory: ",inUrl) #directory to process myURLReader = urllib.request.urlopen(inUrl.rstrip('/')) myResults = myURLReader.read().decode('utf-8').replace("<a href=","<A HREF=").replace("</a>","</A>") myURLReader.close() data=myResults.split("</A>") tag="<A HREF=\"" endtag="\">" for item in data: if "<A HREF" in item: try: ind = item.index(tag) item=item[ind+len(tag):] end=item.index(endtag) except: pass else: #The link is found itemToDownload = item[:end] if "." in itemToDownload: #the link is to a file if relativeLinkPathBase not in itemToDownload: #is the path relative, so we add the base url itemToDownload = relativeLinkPathBase + itemToDownload filesToDownloadList.append(itemToDownload) else: # it's a directory, so let's go into it if recursive is chosen if inRecursive: if itemToDownload not in inUrl: #we make sure it isn't a link to parent directory if relativeLinkPathBase not in itemToDownload: itemToDownload = relativeLinkPathBase + itemToDownload # the directory is a subdirectory, so we will follow it if verboseMessages: print("subdirectory to process ", itemToDownload) get_pageLinks(itemToDownload,inRecursive) def download_files(): # download the files that were identified # this is refering to the global list of files to download localSuccessfulDownloads = 0 print("==Downloads starting ==============") for link in filesToDownloadList: downloadAttempts = 0 fileDownloaded = False if verboseMessages: print("downloading file: ",link) local_link = link; local_link = saveFilesToThisDirectory + local_link.replace(relativeLinkPathBase,"") local_filename = link.split('/')[-1] #make sure the local directory stucture has been created path = Path(local_link.replace(local_filename,"")) path.mkdir(parents=True, exist_ok=True) while not fileDownloaded and downloadAttempts < maxDownloadAttempts: try: urllib.request.urlretrieve(link,local_link) localSuccessfulDownloads += 1 fileDownloaded = True except urllib.error.URLError as e: downloadAttempts += 1 #we will retry the download the number of times allowed by maxDownloadAttempts variable if verboseMessages: print("downloadError: ",e.reason) if verboseMessages: print("downloadErrorFile: ",link," attempt:",downloadAttempts) if downloadAttempts < maxDownloadAttempts: time.sleep(15) #wait 15 seconds before the next attempt else: print("Could not successfully download: ",link," after ",downloadAttempts," download attempts") print("==Downloads complete ==============") print("SuccessfulDownloads: ",localSuccessfulDownloads," out of ",len(filesToDownloadList)) print('==Process is starting ===================') #get the file links get_pageLinks(url, recursiveVal) print("==Collected ", len(filesToDownloadList), " file links ======") #now download the files download_files()
  8. Below I have included example Wget commands for downloading files from the PDS Geosciences Node. The first example demonstrates downloading a PDS data set from the PDS Geosciences Node archive. The second example demonstrates using Wget to download an Orbital Data Explorer (ODE) cart request. Dan Scholes 2/20/18 Example of downloading data files using links from HTTP PDS Geosciences Node Data Archive or Orbital Data Explorer (ODE) Cart location Note: In the future, changes to the PDS Geosciences Node website and Orbital Data Explorer website may cause this example to no longer function. Disclaimer: This sample code is provided "as is", without warranty of any kind, express or implied. In no event shall the author be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the sample code or the use or other dealings with the sample code. Wget website: https://www.gnu.org/software/wget/ Example of downloading a PDS Geosciences Node archive subdirectory wget -rkpN -P c:\temp\data -nH --cut-dirs 2 --level=15 --no-parent --reject "index.html*" -e robots=off http://pds-geosciences.wustl.edu/mro/mro-m-crism-4-typespec-v1/mrocr_8001/ Example of downloading ODE Cart Request wget -rkpN -P c:\temp\data -nH --cut-dirs 2 --level=15 --no-parent --reject "index.html*" -e robots=off http://ode.rsl.wustl.edu/cartdownload/data/sample -r means recursively download files -k means convert links. Links on the webpage will be localhost instead of example.com/path. -p means get all webpage resources, so wget will obtain images and javascript files to make website work properly. -N is to retrieve timestamps, so if local files are newer than files on remote website, the remote files will be skipped. -P sets the local destination directory for the downloaded files. -e is a flag option that must be set for the robots=off to work. robots=off means ignore robots file. -c allows the command to pick up where it left off if the connection is dropped and the command I re-run. --no-parent keeps the command from downloading all the files in the directories above the requested level. --reject "index.html*" keeps wget from downloading every directory's default index.html. -nH will disable the generation of the host-prefixed directories. In the example above, a directory ode.rsl.wustl.edu will not be created locally. --cut-dirs 2 Ignore the count of directory components. Basically, this example will omit the first 2 directory levels from the path it creates locally for the files that are downloaded. Example: http://ode.rsl.wustl.edu/cartdownload/data/sample The first directory in the destination directory will be "sample". --level=depth --level=15 Levels to recursively search. The default is 5, but we will need to go farther with ODE cart and PDS Geosciences Archive. --------------------------------------------------------------------------------------------------------------------------------------------------------- -nd or --no-directories it is used to put all the requested files in one directory. We are not using this feature, but a user may prefer this option.
  9. ODE has recently released a new ODE cart request download page. This is a new option for accessing a requested ODE cart request. A link to the page is provided in the order fulfillment email that a user receives after their cart request is ready for download. The new ODE cart request download page supports multiple download methods and offers faster options for users to retrieve requested ODS files. The ODE cart request download page contains a single click link to download the entire user cart request using the Aspera Connect web browser plug-in. This option provides a convenient and fast download method for most users. The Aspera Connect browser plug-in employs the FASP data transfer technology to better utilize the user’s available bandwidth. HTTP and FTP links are provided on the page for users who prefer to download the files through a client application, a different web browser plug-in, a local script, or directly through the web browser. Additionally, the page supports browsing and downloading specific directories and files of the cart request through the Aspera Connect web browser plug-in or using direct HTTP links. Multiple files can be downloaded simultaneously through this page. Try the new page the next time you request PDS products through the ODE cart!
  10. ODE has recently added a new option to download all of a PDS product's files through a single click. The new option is available from the product detail page of PDS products hosted by the PDS Geosciences Node. This option uses the Aspera Connect browser plug-in. Aspera technology more fully utilizes one's available bandwidth than traditional TCP connections, such as HTTP and FTP, which will improve the speed the file download speed. Give the new feature a try and let us know how it works for you. Example page with the option: http://ode.rsl.wustl.edu/mars/indexproductpage.aspx?product_id=FRT0000B278_07_IF164S_TRR3&product_idGeo=17880484 Help page further describing the feature and installation of the Aspera Connect browser plug-in: http://ode.rsl.wustl.edu/mars/pagehelp/quickstartguide/index.html?product_detail_page.htm#aspera
  11. The PDS Geosciences Node released the MakeLabels tool this July. The program can be used to generate PDS4 labels using a label template and one or two Excel spreadsheets. It replaces placeholders in the template with values from the spreadsheets. In fact it can be used to output any type of ASCII file. More information about the program and it's downloadable file can be found on the PDS Geosciences Node website. http://pds-geosciences.wustl.edu/tools/makelabels.html
  12. The ODE team has posted three simple Python examples of querying the ODE REST interface. These scripts are found in the forum's download section. https://geoweb.rsl.wustl.edu/community/index.php?/files/file/26-ode-rest-python-examples/
  13. Version 1.0.0

    3522 downloads

    Attached are three simple Python examples of querying the ODE REST interface. odeProductCount_Python27.py - A Python 2.7 example of querying the ODE REST interface for the count of products of particular product type. The results are written to the command prompt interface. odeProductCount_Python36.py - A Python 3.6 example of querying the ODE REST interface for the count of products of particular product type. The results are written to the command prompt interface. odeREST_FileDownload.py - A Python 3.6 example of querying ODE for products of a particular mission/instrument/product type. The product files of the first 10 query results will be downloaded to the local machine.
  14. Note for ENVI users of map-projected data: Some Geosciences Node hosted map-projected data sets (.IMG files) are not read by ENVI as projected data sets. Using the Geospatial Data Abstraction Library (GDAL) gdal_translate command, users can easily convert most of these data products into GeoTIFF or ENVI file formats which are read by ENVI as projected data. GDAL packages for various computer systems can be found here: https://trac.osgeo.org/gdal/wiki/DownloadingGdalBinaries. The Windows package used in this example is the OSGEO4W 64-bit version. A MacOS X build is available, as well. Download and install a GDAL package that has gdal_translate.exe. gdal_translate.exe runs from the command line. Make the folder that contains gdal_translate.exe your current directoryTo do this, type "cd path" without the quotes into the command line, where "path" is the full path to the folder that contains gdal_translate.exe. gdal_translate.exe is executed with the following syntax: gdal_translate –of outputtype inputfilepath outputfilepath "outputtype" is GTiff for a GeoTiff (.tif) and ENVI for an ENVI file (.dat) Both file paths are the full path to the file. In the case of a file with a detached PDS label, the "inputfilepath" points to the label, not the data file. Open the file as normal in ENVI. Example using OSGeo4W 64-bit (Windows): C:\>cd OSGeo4W64\bin C:\OSGeo4W64\bin>gdal_translate -of GTiff C:\hrsc\h2064_0000_dt4.img C:\hrsc\h2064_0000_dt4.tif Example using GDAL 2.1 Complete (MacOS X): :/ cd Library/Frameworks/GDAL.framework/Programs :Programs ./gdal_translate -of ENVI /Desktop/hrsc/h2064_0000_dt4.img /Desktop/hrsc/h2064_0000_dt4.dat Examples of data sets that benefit from the GDAL translate command: MRO - HiRISE DTM Mars Express - all HRSC, OMEGA DDRGM Odyssey - THEMIS IRGEO2, VGEO1, VGEO2 MGS - MOLA MEGDR, TES TIMAP Additional instructions for specific data sets: Mars Odyssey THEMIS Visible map-projected .CUB files (VGEO1 and VGEO2) have detached PDS labels, but the map-projection data is in the .CUB file, so the "inputfilepath" should be the data file itself. Mars Odyssey THEMIS Infrared map-projected .CUB.gz files (IRGEO2) need to be un-zipped before using gdal_translate. These data are similar to the visible data files in "inputfilepath" syntax (.CUB, not .LBL). More details about the gdal_translate command can be found at http://www.gdal.org/gdal_translate.html.
  15. No problem. Just let me know if you experience any further problems. Thanks, Dan
  16. Hi Phil, I believe I have fixed the problem. Please give it a try with Chrome. Thank you very much for bringing the problem to our attention! Dan
  17. Hi Phil, Thank you for contacting us about the problem you experienced. I ran some tests. I can recreate the problem with Chrome and Edge, but files download properly with Firefox. I will need to research this further. Which browser and OS are you using? Thanks, Dan
  18. The following ODE website enhancements were deployed May 31, 2016: Functionality to search by a list of PDS product ids, including wildcards Mars ODE - ability to filter search results by Mars Year Mars ODE - ability to download non-PDS processed versions of MRO CTX images from ASU's Mars Space Flight Facility's Image Explorer from product detail pages or at cart checkout Mars ODE - ability to download MRO CRISM TRDR's corresponding DDR files at cart checkout, for improved interface usability Deployed help documentation updates for website features and existing data sets available through ODE
  19. May 5th, 2016 – MRO HiRISE Updates - MRO HiRISE EDR, RDR, DTM and Anaglyph data products released through April 2nd, 2016 (Orbit 45,399) See https://wufs.wustl.edu/ode/odeholdings/Mars_holdings.html for a full list of holdings in Mars ODE.
  20. April 27th, 2016 - MEX HRSC New Version 3 map projected data (REFDR3) loaded into ODE The third release of HRSC new Version 3 of map projected RDR data (REFDR3) have been loaded into ODE, with coverage through November 25, 2005. Additional version 3 data will be released incrementally until it completely replaces the version 2 data set. Six months after that the version 2 data will be taken offline. See https://wufs.wustl.edu/ode/odeholdings/Mars_holdings.html for a full list of data holdings in ODE.
  21. April 21, 2016 - Revised Diviner GDR Level-3 data has been loaded into ODE Review the data set's errata file for details of the changes. http://pds-geosciences.wustl.edu/lro/lro-l-dlre-4-rdr-v1/lrodlr_1001/errata.txt See https://wufs.wustl.edu/ode/odeholdings/Moon_holdings.html for a full list of data holdings in ODE.
  22. April 13th, 2016 - MEX HRSC New Version 3 map projected data (REFDR3) loaded into ODE The second release of HRSC new Version 3 of map projected RDR data (REFDR3) have been loaded into ODE, with coverage through March 18, 2005 (through Orbit 1500). Additional version 3 data will be released incrementally until it completely replaces the version 2 data set. Six months after that the version 2 data will be taken offline. See https://wufs.wustl.edu/ode/odeholdings/Mars_holdings.html for a full list of data holdings in ODE.
  23. December 29th, 2014 - LRO Release 20 has been loaded into ODE. Updated LAMP EDR and RDR products from July 6th, 2009 to September 14th, 2014 Updated LEND EDR and RDR products from July 3rd, 2009 to September 14th, 2014 Updated LOLA EDR products from June 18th, 2009 to September 10th, 2014. Updated LOLA RDR products from July 7th, 2009 to September 10th, 2014. Updated Diviner EDR and RDR products from July 5th, 2009 to September 15th, 2014 Updated Diviner RDR Query Tool Database from July 5th, 2009 to September 15th, 2014 Updated LOLA RDR Query Tool Database from June 18th, 2009 to September 10th, 2014 Updated LROC EDR, CDR, and RDR products from June 30th, 2009 to September 15th, 2014 Please see ODE Lunar Holdings - https://wufs.wustl.edu/ode/odeholdings/Moon_holdings.html
  24. December 16th, 2014 - LRO Release 20 has been loaded into ODE. Updated LAMP EDR and RDR products from July 6th, 2009 to September 14th, 2014 Updated LEND EDR and RDR products from July 3rd, 2009 to September 14th, 2014 Updated LOLA EDR products from June 18th, 2009 to September 10th, 2014. Updated LOLA RDR products from July 7th, 2009 to September 10th, 2014. Updated Diviner EDR and RDR products from July 5th, 2009 to September 15th, 2014 Updated LOLA RDR Query Tool Database from June 18th, 2009 to September 10th, 2014 In Progress: LROC EDR, CDR, and RDR products from June 30th, 2009 to September 15th, 2014 In Progress: Diviner RDR Query Tool Database from July 5th, 2009 to September 15th, 2014 Please see ODE Lunar Holdings - https://wufs.wustl.edu/ode/odeholdings/Moon_holdings.html
  25. Version 7.3.1

    3006 downloads

    A collection of ENVI and IDL procedures for reading, displaying, and analyzing CRISM data, produced by the CRISM Science Team. Related downloads CAT V7.3.1 Release Notes CAT History String Description CAT Installation Guide Introduction to CAT CAT User's Guide Additional information is available from the Geosciences Node CRISM page.
×
×
  • Create New...