Dan Scholes

For data users · April 22, 2020

Hi There,

I am one of the developers who maintains the ODE website. I ran the same query in the ODE LOLA RDR Query tool, and I received the same response. I believe it is reasonable. Remember, you are just limiting the output to one day. The tool is not setup very well for global coverage searches, but it can be done for narrow time windows. If you wish to query 2014-2019, I would suggest doing a region of interest. That is more of the objective of the query tool.

The limited areas of coverage for 2014 and later is expected. The orbiter was in a different orbit and the instrument issues you mentioned. To see the different in coverage for later years, you can use the ODE map search page to filter LRO LOLA RDR coverage to specific time ranges. Link to map search interface: https://ode.rsl.wustl.edu/moon/indexMapSearch.aspx

Map interface limited to LOLA RDR (image mapSearch1.jpg)

LOLA RDR filtered to 2014 observations. (image mapSearch2.jpg)

Thanks,

Dan

December 17, 2019

The ODE map search has been updated to have product coverage map layers grouped by mission/instrument/processing level. This map layer organization matches the product search page.

Mars Orbital Data Explorer Map Search

Lunar Orbital Data Explorer Map Search

Mercury Orbital Data Explorer Map Search

Venus Orbital Data Explorer Map Search

Mars ODE Map Search Example:

July 11, 2019

The PDS Geosciences Node and the LROC Data Node have established a faster data transfer method. As a result, ODE cart downloads of LROC data will be fulfilled noticeably faster than in the past. We encourage users to give it a try through the Lunar ODE website.

June 25, 2019

New functionality has been added to the ODE product search page to allow users to select a freehand polygon location for coverage searches. Similar freehand polygon search filtering has been added to the map search page.

Mars ODE Product Search - https://ode.rsl.wustl.edu/mars/indexProductSearch.aspx

Mercury ODE Product Search - https://ode.rsl.wustl.edu/mercury/indexProductSearch.aspx

Lunar ODE Product Search - https://ode.rsl.wustl.edu/moon/indexProductSearch.aspx

Venus ODE Product Search - https://ode.rsl.wustl.edu/venus/indexProductSearch.aspx

Map Projection for ENVI Users · April 9, 2019

Hi Raj Patel,

Thank you for contacting us about your question.

First, I confirmed there are no .QUB files in the THEMIS IRBTR (Infrared Brightness Temperature Record) data set.

Newer versions of GDAL support the conversion of a .QUB file type into GeoTiff (.tif).

Here is an example of the command:

Gdal_translate -of GTiff D:\test\data\I00818001RDR.QUB D:\test\data\I00818001RDR.tif

The first link on the following FAQ describes the tools ASU's Mars Space Flight Facility recommends for opening THEMIS images.
http://viewer.mars.asu.edu/faq#t6n18

Let me know if you have further questions.

Best wishes,
Dan

February 26, 2019

Hi Deepak,

Thank you for the valuable suggestions and feedback. Our staff will be discussing these suggestions over the next couple of weeks to determine our current capabilities and prioritizing future ODE updates. I will be able to provide you with more details after these discussions occur. Also, we will have a booth at LPSC where your presentation ideas can be discussed. Please stop by the PDS booth during LPSC.

Best wishes,

Dan

October 30, 2018

Hi Jairo,

Thank you for emailing us with your question.

You are receiving the “corrupted file” response because your computer does not have a native application for opening the PDS IMG files. Also, sometimes a cd/dvd image file is called an IMG, so an application used for mounting disk images tries to open the file in a different expected format.

A typical PDS image is stored as a simple binary array. It is described by a PDS label (a set of ASCII-text keyword=value statements), which can be either attached or detached. An attached label is embedded at the beginning of the image file; a detached label is in a separate file with the same name, extension LBL. The label tells you everything you need to know to read the image -- the number of lines, number of pixels (samples) per line, the size and data type of a sample, and map projection information if the image is a map. (PDS deliberately does not rely on any commercial or proprietary formats to archive image data, in order to ensure the long-term viability of the data.)

Typically IMG files are opened with programs including ENVI, NASA View, or GDAL. ArcMap might support IMG files, but I don’t remember. The IMG files can also be parsed and manipulated with code (Python, IDL, Matlab, C++, C#).

In the future, I hope to add functionality into ODE to allow a user to request the files in various formats, but it is not on the near list. In the meantime, GDAL is a standard application for changing formats.
http://www.gdal.org/ (general information and the link to download)
http://www.gdal.org/gdal_translate.html (basic GDAL commands)

Here is an example commands that can be used to convert an img to tif:
Gdal_translate -of GTiff D:\test\data\MGN\fmap\fl13s309.img D:\test\data\MGN\fmap\fl13s309.tif

NASA View is better for just viewing images:
https://pds.nasa.gov/tools/about/pds3-tools/nasa-view.shtml

Let us know if you have any further questions.
Thanks,
Dan

October 10, 2018

October 10th, 2018 – MRO HiRISE Updates

HiRISE EDR, RDR, DTM and Anaglyph data products released through August 31, 2018 (Orbit 56,699)

See https://wufs.wustl.edu/ode/odeholdings/Mars_holdings.html

October 10, 2018

October 8th, 2018 - Mar Odyssey Release 65 Data Updates

Updated THEMIS standard data products (IREDR, IRRDR, IRBTR, VISEDR, VISRDR and VISABR data) from February 19, 2002 to March 8, 2018

Updated THEMIS geometry data products (IRGEO2, IRPBT2, VGEO2, and VISALB data) from September 28, 2013 to March 8, 2018

Updated GRS data products (CGS, SGS, DND, AHD, ELEMTS) from February 19, 2002 to January 9, 2018

See https://wufs.wustl.edu/ode/odeholdings/Mars_holdings.html

September 19, 2018

September 18, 2018 - LRO Release 35 has been loaded into ODE.

Updated LOLA EDR products from June 18th, 2009 to July 7th, 2018
Updated LOLA RDR products from July 13th, 2009 to August 8th, 2018
Updated LAMP EDR and RDR products from July 6th, 2009 to June 14th, 2018
Updated LEND EDR_SCI, EDR_HK, RDR_RSCI, and RDR_CHK products from June 18th, 2009 to June 14th, 2018
Updated LEND RDR_DLD and RDR_DLX products from June 27th, 2009 to March 14th, 2018
Updated Diviner EDR and RDR products from July 5th, 2009 to June 15th, 2018
Updated Mini-RF Bistatic data products (EDR, RDR, and DDR) from April 6th, 2011 to December 16th, 2017.
Updated LROC EDR, CDR and RDR products from June 30th, 2009 to June 15th, 2018

Updated Diviner RDR query tool database from July 5th, 2009 to June 15th, 2018.
Updated LOLA RDR query tool database from July 13th, 2009 to August 8th, 2018

Please see ODE Lunar Holdings - https://wufs.wustl.edu/ode/odeholdings/Moon_holdings.html

For data users · February 22, 2018

Below I have included a Python 3.6 sample script for downloading files from the PDS Geosciences Node. The sample includes a configuration for downloading data files from both a PDS Geosciences Node archive and files from an Orbital Data Explorer (ODE) cart request. The script supports multiple levels of sub directories, as well. The script includes variables that should be set by the user for his or her environment. The example PDS data set and ODE cart request both exist, and they are available for test executions of the script. Python 3.6 is required for the script to function. This script is also available for download in the downloads section of the forum.

# PDSGeosciencesNode_FileDownload.py
# Dan Scholes 2/19/18
# Pypthon 3.6 compatible version
# Example of downloading data files using 
# links from HTTP PDS Geosciences Node Data Archive
# or Orbital Data Explorer (ODE) Cart location
# Note: One drawback of this script is that it downloads one file at a time, rather than multiple streams.
# Additional Note: In the future, changes to the PDS Geosciences Node website and Orbital Data Explorer website may cause this example to no longer function.
# Disclaimer: This sample code is provided "as is", without warranty of any kind, express or implied. In no event shall the author be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the sample code or the use or other dealings with the sample code.

# Phython download website: https://www.python.org/downloads/

import urllib.request
import re
import time
from pathlib import Path

# Variables for user to populate----------
saveFilesToThisDirectory = 'c:/temp/data/'  # local destination path to save files

#next two lines are for downloading from the PDS Geosciences Node archive
url = "http://pds-geosciences.wustl.edu/mro/mro-m-rss-5-sdp-v1/mrors_1xxx/"  #enter the directory you would like to download
relativeLinkPathBase = "http://pds-geosciences.wustl.edu"										#this is the default location for the relative paths on the website (just leave this value) 		

#next two lines are for downloading an ODE cart request
#url = "http://ode.rsl.wustl.edu/cartdownload/data/sample/" 			#enter the directory you would like to download
#relativeLinkPathBase = "http://ode.rsl.wustl.edu/"							#this is the default location for the relative paths on the ode cart website (just leave this value) 		

recursiveVal = True					# True/False whether to download files in subdirectories of the specified location in the url variable
verboseMessages = False			# True/False whether to display verbose messages during the script processing

# End of variables for user to populate----------

relativeLinkPathBase = relativeLinkPathBase.rstrip('/')
maxDownloadAttempts = 3
filesToDownloadList = []

def get_pageLinks(inUrl,inRecursive):
	if verboseMessages: print("Cataloging Directory: ",inUrl)  #directory to process
	
	myURLReader = urllib.request.urlopen(inUrl.rstrip('/'))
	myResults = myURLReader.read().decode('utf-8').replace("<a href=","<A HREF=").replace("</a>","</A>")
	myURLReader.close()
		
	data=myResults.split("</A>")
	tag="<A HREF=\""
	endtag="\">"
	for item in data:
	
	    if "<A HREF" in item:
	        try:
	            ind = item.index(tag)
	            item=item[ind+len(tag):]
	            end=item.index(endtag)
	        except: pass
	        else:
	            #The link is found
	            itemToDownload = item[:end]
	            if "." in itemToDownload:
	            	#the link is to a file
	            	if relativeLinkPathBase not in itemToDownload:
	            		#is the path relative, so we add the base url		            	
	            		itemToDownload = relativeLinkPathBase + itemToDownload
	            	filesToDownloadList.append(itemToDownload)
	            else:
	            	# it's a directory, so let's go into it if recursive is chosen
	            	if inRecursive:
		            	if itemToDownload not in inUrl:  #we make sure it isn't a link to parent directory
		            		if relativeLinkPathBase not in itemToDownload:
		            			itemToDownload = relativeLinkPathBase + itemToDownload
		            		# the directory is a subdirectory, so we will follow it
		            		if verboseMessages: print("subdirectory to process ", itemToDownload)
		            		get_pageLinks(itemToDownload,inRecursive)

def download_files():
	# download the files that were identified
	# this is refering to the global list of files to download
	localSuccessfulDownloads = 0

	print("==Downloads starting ==============")
	for link in filesToDownloadList:
		downloadAttempts = 0		
		fileDownloaded = False
		if verboseMessages: print("downloading file: ",link)
		local_link = link;
		local_link = saveFilesToThisDirectory + local_link.replace(relativeLinkPathBase,"")
		local_filename = link.split('/')[-1]	

		#make sure the local directory stucture has been created
		path = Path(local_link.replace(local_filename,"")) 
		path.mkdir(parents=True, exist_ok=True)
		
		while not fileDownloaded and downloadAttempts < maxDownloadAttempts:			
			try: 
				urllib.request.urlretrieve(link,local_link)	
				localSuccessfulDownloads += 1	
				fileDownloaded = True
			except urllib.error.URLError as e:
				downloadAttempts += 1 #we will retry the download the number of times allowed by maxDownloadAttempts variable
				if verboseMessages: print("downloadError: ",e.reason)
				if verboseMessages: print("downloadErrorFile: ",link," attempt:",downloadAttempts)
				if downloadAttempts < maxDownloadAttempts:
					time.sleep(15)  #wait 15 seconds before the next attempt
				else:
					print("Could not successfully download: ",link," after ",downloadAttempts," download attempts")
				
	print("==Downloads complete ==============")
	print("SuccessfulDownloads: ",localSuccessfulDownloads," out of ",len(filesToDownloadList))
	
	
print('==Process is starting ===================')
#get the file links
get_pageLinks(url, recursiveVal)
print("==Collected ", len(filesToDownloadList), " file links ======")
#now download the files
download_files()

For data users · February 22, 2018

Below I have included example Wget commands for downloading files from the PDS Geosciences Node. The first example demonstrates downloading a PDS data set from the PDS Geosciences Node archive. The second example demonstrates using Wget to download an Orbital Data Explorer (ODE) cart request.

Dan Scholes 2/20/18
Example of downloading data files using
links from HTTP PDS Geosciences Node Data Archive
or Orbital Data Explorer (ODE) Cart location

Note: In the future, changes to the PDS Geosciences Node website and Orbital Data Explorer website may cause this example to no longer function.
Disclaimer: This sample code is provided "as is", without warranty of any kind, express or implied. In no event shall the author be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the sample code or the use or other dealings with the sample code.

Wget website: https://www.gnu.org/software/wget/

Example of downloading a PDS Geosciences Node archive subdirectory
wget -rkpN -P c:\temp\data -nH --cut-dirs 2 --level=15 --no-parent --reject "index.html*" -e robots=off http://pds-geosciences.wustl.edu/mro/mro-m-crism-4-typespec-v1/mrocr_8001/

Example of downloading ODE Cart Request
wget -rkpN -P c:\temp\data -nH --cut-dirs 2 --level=15 --no-parent --reject "index.html*" -e robots=off http://ode.rsl.wustl.edu/cartdownload/data/sample

-r means recursively download files

-k means convert links. Links on the webpage will be localhost instead of example.com/path.

-p means get all webpage resources, so wget will obtain images and javascript files to make website work properly.

-N is to retrieve timestamps, so if local files are newer than files on remote website, the remote files will be skipped.

-P sets the local destination directory for the downloaded files.

-e is a flag option that must be set for the robots=off to work.

robots=off means ignore robots file.

-c allows the command to pick up where it left off if the connection is dropped and the command I re-run.

--no-parent keeps the command from downloading all the files in the directories above the requested level.

--reject "index.html*" keeps wget from downloading every directory's default index.html.

-nH will disable the generation of the host-prefixed directories.
In the example above, a directory ode.rsl.wustl.edu will not be created locally.

--cut-dirs 2 Ignore the count of directory components. Basically, this example will omit the first 2 directory levels from the path it creates locally for the files that are downloaded.
Example: http://ode.rsl.wustl.edu/cartdownload/data/sample The first directory in the destination directory will be "sample".

--level=depth
--level=15 Levels to recursively search. The default is 5, but we will need to go farther with ODE cart and PDS Geosciences Archive.

---------------------------------------------------------------------------------------------------------------------------------------------------------
-nd or --no-directories it is used to put all the requested files in one directory. We are not using this feature, but a user may prefer this option.

January 8, 2018

ODE has recently released a new ODE cart request download page. This is a new option for accessing a requested ODE cart request. A link to the page is provided in the order fulfillment email that a user receives after their cart request is ready for download. The new ODE cart request download page supports multiple download methods and offers faster options for users to retrieve requested ODS files.

The ODE cart request download page contains a single click link to download the entire user cart request using the Aspera Connect web browser plug-in. This option provides a convenient and fast download method for most users. The Aspera Connect browser plug-in employs the FASP data transfer technology to better utilize the user’s available bandwidth. HTTP and FTP links are provided on the page for users who prefer to download the files through a client application, a different web browser plug-in, a local script, or directly through the web browser.

Additionally, the page supports browsing and downloading specific directories and files of the cart request through the Aspera Connect web browser plug-in or using direct HTTP links. Multiple files can be downloaded simultaneously through this page.

Try the new page the next time you request PDS products through the ODE cart!

January 8, 2018

ODE has recently added a new option to download all of a PDS product's files through a single click. The new option is available from the product detail page of PDS products hosted by the PDS Geosciences Node. This option uses the Aspera Connect browser plug-in. Aspera technology more fully utilizes one's available bandwidth than traditional TCP connections, such as HTTP and FTP, which will improve the speed the file download speed.

Give the new feature a try and let us know how it works for you.

Example page with the option:

http://ode.rsl.wustl.edu/mars/indexproductpage.aspx?product_id=FRT0000B278_07_IF164S_TRR3&product_idGeo=17880484

Help page further describing the feature and installation of the Aspera Connect browser plug-in:

http://ode.rsl.wustl.edu/mars/pagehelp/quickstartguide/index.html?product_detail_page.htm#aspera

October 13, 2017

The PDS Geosciences Node released the MakeLabels tool this July. The program can be used to generate PDS4 labels using a label template and one or two Excel spreadsheets. It replaces placeholders in the template with values from the spreadsheets. In fact it can be used to output any type of ASCII file.

More information about the program and it's downloadable file can be found on the PDS Geosciences Node website.
http://pds-geosciences.wustl.edu/tools/makelabels.html

September 5, 2017

The ODE team has posted three simple Python examples of querying the ODE REST interface. These scripts are found in the forum's download section.

https://geoweb.rsl.wustl.edu/community/index.php?/files/file/26-ode-rest-python-examples/

Map Projection for ENVI Users · July 13, 2017

Note for ENVI users of map-projected data: Some Geosciences Node hosted map-projected data sets (.IMG files) are not read by ENVI as projected data sets. Using the Geospatial Data Abstraction Library (GDAL) gdal_translate command, users can easily convert most of these data products into GeoTIFF or ENVI file formats which are read by ENVI as projected data.

GDAL packages for various computer systems can be found here: https://trac.osgeo.org/gdal/wiki/DownloadingGdalBinaries. The Windows package used in this example is the OSGEO4W 64-bit version. A MacOS X build is available, as well.

Download and install a GDAL package that has gdal_translate.exe.
gdal_translate.exe runs from the command line. Make the folder that contains gdal_translate.exe your current directory
- To do this, type "cd path" without the quotes into the command line, where "path" is the full path to the folder that contains gdal_translate.exe.
gdal_translate.exe is executed with the following syntax:
- gdal_translate –of outputtype inputfilepath outputfilepath
- "outputtype" is GTiff for a GeoTiff (.tif) and ENVI for an ENVI file (.dat)
- Both file paths are the full path to the file.
- In the case of a file with a detached PDS label, the "inputfilepath" points to the label, not the data file.
Open the file as normal in ENVI.

Example using OSGeo4W 64-bit (Windows):

C:\>cd OSGeo4W64\bin

C:\OSGeo4W64\bin>gdal_translate -of GTiff C:\hrsc\h2064_0000_dt4.img C:\hrsc\h2064_0000_dt4.tif

Example using GDAL 2.1 Complete (MacOS X):

:/ cd Library/Frameworks/GDAL.framework/Programs
:Programs ./gdal_translate -of ENVI /Desktop/hrsc/h2064_0000_dt4.img /Desktop/hrsc/h2064_0000_dt4.dat

Examples of data sets that benefit from the GDAL translate command:

MRO - HiRISE DTM
Mars Express - all HRSC, OMEGA DDRGM
Odyssey - THEMIS IRGEO2, VGEO1, VGEO2
MGS - MOLA MEGDR, TES TIMAP

Additional instructions for specific data sets:

Mars Odyssey THEMIS Visible map-projected .CUB files (VGEO1 and VGEO2) have detached PDS labels, but the map-projection data is in the .CUB file, so the "inputfilepath" should be the data file itself.
Mars Odyssey THEMIS Infrared map-projected .CUB.gz files (IRGEO2) need to be un-zipped before using gdal_translate. These data are similar to the visible data files in "inputfilepath" syntax (.CUB, not .LBL).

More details about the gdal_translate command can be found at http://www.gdal.org/gdal_translate.html.

For data users · June 26, 2017

No problem. Just let me know if you experience any further problems.

Thanks,

Dan

For data users · June 22, 2017

Hi Phil,

I believe I have fixed the problem. Please give it a try with Chrome.

Thank you very much for bringing the problem to our attention!

Dan

For data users · June 22, 2017

Hi Phil,

Thank you for contacting us about the problem you experienced.

I ran some tests. I can recreate the problem with Chrome and Edge, but files download properly with Firefox. I will need to research this further. Which browser and OS are you using?

Thanks,

Dan

June 2, 2016

The following ODE website enhancements were deployed May 31, 2016:

Functionality to search by a list of PDS product ids, including wildcards
Mars ODE - ability to filter search results by Mars Year
Mars ODE - ability to download non-PDS processed versions of MRO CTX images from ASU's Mars Space Flight Facility's Image Explorer from product detail pages or at cart checkout
Mars ODE - ability to download MRO CRISM TRDR's corresponding DDR files at cart checkout, for improved interface usability
Deployed help documentation updates for website features and existing data sets available through ODE

May 5, 2016

May 5th, 2016 – MRO HiRISE Updates

- MRO HiRISE EDR, RDR, DTM and Anaglyph data products released through April 2nd, 2016 (Orbit 45,399)

See https://wufs.wustl.edu/ode/odeholdings/Mars_holdings.html for a full list of holdings in Mars ODE.

April 27, 2016

April 27th, 2016 - MEX HRSC New Version 3 map projected data (REFDR3) loaded into ODE

The third release of HRSC new Version 3 of map projected RDR data (REFDR3) have been loaded into ODE, with coverage through November 25, 2005. Additional version 3 data will be released incrementally until it completely replaces the version 2 data set. Six months after that the version 2 data will be taken offline.

See https://wufs.wustl.edu/ode/odeholdings/Mars_holdings.html for a full list of data holdings in ODE.

April 22, 2016

April 21, 2016 - Revised Diviner GDR Level-3 data has been loaded into ODE

Review the data set's errata file for details of the changes.
http://pds-geosciences.wustl.edu/lro/lro-l-dlre-4-rdr-v1/lrodlr_1001/errata.txt

See https://wufs.wustl.edu/ode/odeholdings/Moon_holdings.html for a full list of data holdings in ODE.

April 13, 2016

April 13th, 2016 - MEX HRSC New Version 3 map projected data (REFDR3) loaded into ODE

The second release of HRSC new Version 3 of map projected RDR data (REFDR3) have been loaded into ODE, with coverage through March 18, 2005 (through Orbit 1500). Additional version 3 data will be released incrementally until it completely replaces the version 2 data set. Six months after that the version 2 data will be taken offline.

See https://wufs.wustl.edu/ode/odeholdings/Mars_holdings.html for a full list of data holdings in ODE.

Sign In

Dan Scholes

Posts

Joined

Last visited

Content Type

Profiles

Forums

Downloads

Blogs

Posts posted by Dan Scholes

LOLA RDR Query tool - data since 2014

ODE Map Search Update

Faster LROC data downloads through ODE cart

New product location search using freehand polygon

Map Projection for ENVI Users

Smart Sorting of Data Query Results

Opening file formats

ODE - The October 2018 PDS release of HiRISE Data Loaded into ODE

ODE - Mars Odyssey Release 65 data loaded into ODE

ODE - LRO Release 35 Loaded into ODE

Python sample for downloading PDS Geosciences Node archive directories or ODE cart requests

Using Wget for downloading PDS Geosciences Node archive directories or ODE cart requests

New - ODE Cart Request Download Page

New - Download All Product & Derived Files Button

MakeLabels progam for generating PDS4 Labels

ODE REST Python examples

Map Projection for ENVI Users

Lola altitude data problem

Lola altitude data problem

Lola altitude data problem

ODE Website enhancements - May 31, 2016

ODE - The May 2016 PDS release of MRO HiRISE Data Loaded into ODE

ODE - The Third Release of MEX HRSC New Version 3 Map Projected Data Loaded into ODE

Revised Diviner GDR Level-3 data has been loaded

ODE - The Second Release of MEX HRSC New Version 3 Map Projected Data Loaded into ODE