Jump to content
PDS Geosciences Node Community

Dan Scholes

Geo Staff
  • Posts

    46
  • Joined

  • Last visited

Posts posted by Dan Scholes

  1. An optional user login has been added to ODE. It's free and simple to use! Through this feature, users can save product searches, map searches, and product details pages! Past cart history and viewed product detail pages are accessible, as well. Click the anonymous user icon  anonymousUserIcon.jpg in the ODE banner to create an account or sign in. Additional user account information is found in the ODE help or email us at ode@wunder.wustl.edu with questions.

    userLoginBanner.jpg

    Links to ODE versions

    Lunar ODE
    Mars ODE
    Mercury ODE
    Venus ODE
     

    For a quick example, we will display saving a product search result list as a bookmark.

    Example of Saving a product search result list (must be logged in):

    searchResultSave1.jpg

     

    Click the "Bookmark Search" link. Then enter an optional description and save the bookmark.

    searchResultSave2.jpg

    Saved bookmarks are accessed from the user icon menu in the ODE banner. From that location, bookmarks, history, and previous cart orders can be accessed. Here we are listing the saved bookmarks. Click a bookmark to open the saved location to a new browser tab.

    searchResultSave3.jpg

     

  2. We are pleased to announce that the USGS Unified Geologic Map of the Moon is now available through the Lunar ODE map search.

    The map is displayed as two map layers in ODE.

    1. The first layer contains unit contacts, geologic unit polygons, linear features, and unit and feature nomenclature annotation.

    2. The second layer, which we call the base layer is a shaded-relief product derived from SELENE Kaguya terrain camera stereo (equatorial, ~60 m/pix) and LOLA altimetry (north and south polar, 100 m/pix).

     

    screenExample.jpg

  3. We have released a new version of our MakeLabels tool. The new 6.2 version includes additional features to help with the creation of PDS labels.

    The program and documentation can be found on the PDS Geosciences Node's MakeLabels page.

    Version 6.2 updates include:

    • The output label destination directory is created, if it does not already exist and the user has appropriate permissions.
    • Any template label tags that are not replaced (due to syntax errors or missing columns in the spreadsheet) are listed in the post-processing summary report.
    • The MakeLabel's GUI interface now has a sizeable report window.
    • The report summary has been moved to the end of the processing output report.
    • The option to upper or lower case fields from the source spreadsheet has been added to the template tags.
    • A new template tag allows a single line to be hidden if it's corresponding field is empty in the spreadsheet.  (example: <name><!-- |specimen_id| OR HIDE-THIS-LINE --></name>)
    • Show if and hide if template tags no longer need to be left aligned in the template. These tags can be indented and spaced with the surrounding XML.
  4. We are pleased to announce the release of the new PDS Geosciences Node Spectral Library website.

    The PDS Geosciences Node Spectral Library is a database of laboratory spectra submitted by various data providers. It currently includes spectra from the Reflectance Experiment Laboratory (RELAB) at Brown University. Additional data sets will be added to the website in the coming months.

    The website allows users to search the catalog of specimen and measurements using a facet search. Results can be viewed in quick view summary or full detail pages. Measurement data can be downloaded individually or through a cart system.

    forumPost.jpg

  5. Hi Christy,

    I just tried the Windows version of the software on my Windows 10 machine. It would not run properly, either. I think we can assume the software has not kept up with the latest operating systems.

    I do have a couple of alternative options. I have confirmed that the files can be viewed with the PDS4 viewer: https://sbnwiki.astro.umd.edu/wiki/PDS4_Viewer
    Using this tool, the data files can be exported as csv, tab delimited, and additional formats.

    Keep in mind that the archive you mentioned is both PDS3 and PD4 compatible. To use PDS3 tools, use the .lbl label, while PDS4 tools will use the .xml label. The XML label will be needed for the PDS4 Viewer.

    You could also write a script with IDL, MATLAB, Python, or another programming language to parse the data files. The column specifications are found in the label files. The PDS3 label file refers to format files which are found in the archive's label directory (https://pds-geosciences.wustl.edu/messenger/mess-e_v_h-grns-3-grs-cdr-v1/messgrs_2001/label/). The product you mentioned refers to https://pds-geosciences.wustl.edu/messenger/mess-e_v_h-grns-3-grs-cdr-v1/messgrs_2001/label/grs_cal_sh3.fmt. The PDS4 label contains all of the column definitions.

    Let me know if you have additional questions.

    Best wishes,

    Dan

     

     

  6. Hi There,

    I am one of the developers who maintains the ODE website. I ran the same query in the ODE LOLA RDR Query tool, and I received the same response. I believe it is reasonable. Remember, you are just limiting the output to one day. The tool is not setup very well for global coverage searches, but it can be done for narrow time windows. If you wish to query 2014-2019, I would suggest doing a region of interest. That is more of the objective of the query tool.

    The limited areas of coverage for 2014 and later is expected. The orbiter was in a different orbit and the instrument issues you mentioned. To see the different in coverage for later years, you can use the ODE map search page to filter LRO LOLA RDR coverage to specific time ranges.  Link to map search interface:  https://ode.rsl.wustl.edu/moon/indexMapSearch.aspx

    Map interface limited to LOLA RDR (image mapSearch1.jpg)

    LOLA RDR filtered to 2014 observations. (image mapSearch2.jpg)

    Thanks,

    Dan

     

    mapSearch1.jpg

    mapSearch2.jpg

  7. New functionality has been added to the ODE product search page to allow users to select a freehand polygon location for coverage searches. Similar freehand polygon search filtering has been added to the map search page.

    Mars ODE Product Search - https://ode.rsl.wustl.edu/mars/indexProductSearch.aspx

    Mercury ODE Product Search -  https://ode.rsl.wustl.edu/mercury/indexProductSearch.aspx

    Lunar ODE Product Search - https://ode.rsl.wustl.edu/moon/indexProductSearch.aspx

    Venus ODE Product Search -  https://ode.rsl.wustl.edu/venus/indexProductSearch.aspx

    productSearchCustomLocation.jpg

  8. Hi Raj Patel,

    Thank you for contacting us about your question. 

    First, I confirmed there are no .QUB files in the THEMIS IRBTR (Infrared Brightness Temperature Record) data set. 

    Newer versions of GDAL support the conversion of a .QUB file type into GeoTiff (.tif).

    Here is an example of the command:

    Gdal_translate -of GTiff D:\test\data\I00818001RDR.QUB D:\test\data\I00818001RDR.tif

    The first link on the following FAQ describes the tools ASU's Mars Space Flight Facility recommends for opening THEMIS images.
    http://viewer.mars.asu.edu/faq#t6n18

    Let me know if you have further questions.

    Best wishes,
    Dan

     

  9. Hi Deepak,

    Thank you for the valuable suggestions and feedback. Our staff will be discussing these suggestions over the next couple of weeks to determine our current capabilities and prioritizing future ODE updates. I will be able to provide you with more details after these discussions occur. Also, we will have a booth at LPSC where your presentation ideas can be discussed. Please stop by the PDS booth during LPSC.

    Best wishes,

    Dan

  10. Hi Jairo,

    Thank you for emailing us with your question.

    You are receiving the “corrupted file” response because your computer does not have a native application for opening the PDS IMG files. Also, sometimes a cd/dvd image file is called an IMG, so an application used for mounting disk images tries to open the file in a different expected format.

    A typical PDS image is stored as a simple binary array. It is described by a PDS label (a set of ASCII-text keyword=value statements), which can be either attached or detached. An attached label is embedded at the beginning of the image file; a detached label is in a separate file with the same name, extension LBL. The label tells you everything you need to know to read the image -- the number of lines, number of pixels (samples) per line, the size and data type of a sample, and map projection information if the image is a map. (PDS deliberately does not rely on any commercial or proprietary formats to archive image data, in order to ensure the long-term viability of the data.)

    Typically IMG files are opened with programs including ENVI, NASA View, or GDAL. ArcMap might support IMG files, but I don’t remember. The IMG files can also be parsed and manipulated with code (Python, IDL, Matlab, C++, C#).

    In the future, I hope to add functionality into ODE to allow a user to request the files in various formats, but it is not on the near list. In the meantime, GDAL is a standard application for changing formats.
    http://www.gdal.org/ (general information and the link to download)
    http://www.gdal.org/gdal_translate.html   (basic GDAL commands)

    Here is an example commands that can be used to convert an img to tif:
    Gdal_translate -of GTiff D:\test\data\MGN\fmap\fl13s309.img D:\test\data\MGN\fmap\fl13s309.tif

    NASA View is better for just viewing images:
    https://pds.nasa.gov/tools/about/pds3-tools/nasa-view.shtml

    Let us know if you have any further questions.
    Thanks,
    Dan

     

  11. October 8th, 2018 - Mar Odyssey Release 65 Data Updates

    Updated THEMIS standard data products (IREDR, IRRDR, IRBTR, VISEDR, VISRDR and VISABR data) from February 19, 2002 to March 8, 2018

    Updated THEMIS geometry data products (IRGEO2, IRPBT2, VGEO2, and VISALB data) from September 28, 2013 to March 8, 2018

    Updated GRS data products (CGS, SGS, DND, AHD, ELEMTS) from February 19, 2002 to January 9, 2018

    See https://wufs.wustl.edu/ode/odeholdings/Mars_holdings.html

  12. September 18, 2018 - LRO Release 35 has been loaded into ODE.

    Updated LOLA EDR products from June 18th, 2009 to July 7th, 2018
    Updated LOLA RDR products from July 13th, 2009 to August 8th, 2018
    Updated LAMP EDR and RDR products from July 6th, 2009 to June 14th, 2018
    Updated LEND EDR_SCI, EDR_HK, RDR_RSCI, and RDR_CHK products from June 18th, 2009 to June 14th, 2018
    Updated LEND RDR_DLD and RDR_DLX products from June 27th, 2009 to March 14th, 2018
    Updated Diviner EDR and RDR products from July 5th, 2009 to June 15th, 2018
    Updated Mini-RF Bistatic data products (EDR, RDR, and DDR) from April 6th, 2011 to December 16th, 2017.
    Updated LROC EDR, CDR and RDR products from June 30th, 2009 to June 15th, 2018

    Updated Diviner RDR query tool database from July 5th, 2009 to June 15th, 2018.
    Updated LOLA RDR query tool database from July 13th, 2009 to August 8th, 2018

    Please see ODE Lunar Holdings - https://wufs.wustl.edu/ode/odeholdings/Moon_holdings.html

  13. Below I have included a Python 3.6 sample script for downloading files from the PDS Geosciences Node. The sample includes a configuration for downloading data files from both a PDS Geosciences Node archive and files from an Orbital Data Explorer (ODE) cart request. The script supports multiple levels of sub directories, as well. The script includes variables that should be set by the user for his or her environment. The example PDS data set and ODE cart request both exist, and they are available for test executions of the script. Python 3.6 is required for the script to function. This script is also available for download in the downloads section of the forum.

    # PDSGeosciencesNode_FileDownload.py
    # Dan Scholes 2/19/18
    # Pypthon 3.6 compatible version
    # Example of downloading data files using 
    # links from HTTP PDS Geosciences Node Data Archive
    # or Orbital Data Explorer (ODE) Cart location
    # Note: One drawback of this script is that it downloads one file at a time, rather than multiple streams.
    # Additional Note: In the future, changes to the PDS Geosciences Node website and Orbital Data Explorer website may cause this example to no longer function.
    # Disclaimer: This sample code is provided "as is", without warranty of any kind, express or implied. In no event shall the author be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the sample code or the use or other dealings with the sample code.
    
    # Phython download website: https://www.python.org/downloads/
    
    import urllib.request
    import re
    import time
    from pathlib import Path
    
    # Variables for user to populate----------
    saveFilesToThisDirectory = 'c:/temp/data/'  # local destination path to save files
    
    #next two lines are for downloading from the PDS Geosciences Node archive
    url = "http://pds-geosciences.wustl.edu/mro/mro-m-rss-5-sdp-v1/mrors_1xxx/"  #enter the directory you would like to download
    relativeLinkPathBase = "http://pds-geosciences.wustl.edu"										#this is the default location for the relative paths on the website (just leave this value) 		
    
    #next two lines are for downloading an ODE cart request
    #url = "http://ode.rsl.wustl.edu/cartdownload/data/sample/" 			#enter the directory you would like to download
    #relativeLinkPathBase = "http://ode.rsl.wustl.edu/"							#this is the default location for the relative paths on the ode cart website (just leave this value) 		
    
    recursiveVal = True					# True/False whether to download files in subdirectories of the specified location in the url variable
    verboseMessages = False			# True/False whether to display verbose messages during the script processing
    
    # End of variables for user to populate----------
    
    relativeLinkPathBase = relativeLinkPathBase.rstrip('/')
    maxDownloadAttempts = 3
    filesToDownloadList = []
    
    def get_pageLinks(inUrl,inRecursive):
    	if verboseMessages: print("Cataloging Directory: ",inUrl)  #directory to process
    	
    	myURLReader = urllib.request.urlopen(inUrl.rstrip('/'))
    	myResults = myURLReader.read().decode('utf-8').replace("<a href=","<A HREF=").replace("</a>","</A>")
    	myURLReader.close()
    		
    	data=myResults.split("</A>")
    	tag="<A HREF=\""
    	endtag="\">"
    	for item in data:
    	
    	    if "<A HREF" in item:
    	        try:
    	            ind = item.index(tag)
    	            item=item[ind+len(tag):]
    	            end=item.index(endtag)
    	        except: pass
    	        else:
    	            #The link is found
    	            itemToDownload = item[:end]
    	            if "." in itemToDownload:
    	            	#the link is to a file
    	            	if relativeLinkPathBase not in itemToDownload:
    	            		#is the path relative, so we add the base url		            	
    	            		itemToDownload = relativeLinkPathBase + itemToDownload
    	            	filesToDownloadList.append(itemToDownload)
    	            else:
    	            	# it's a directory, so let's go into it if recursive is chosen
    	            	if inRecursive:
    		            	if itemToDownload not in inUrl:  #we make sure it isn't a link to parent directory
    		            		if relativeLinkPathBase not in itemToDownload:
    		            			itemToDownload = relativeLinkPathBase + itemToDownload
    		            		# the directory is a subdirectory, so we will follow it
    		            		if verboseMessages: print("subdirectory to process ", itemToDownload)
    		            		get_pageLinks(itemToDownload,inRecursive)
    
    def download_files():
    	# download the files that were identified
    	# this is refering to the global list of files to download
    	localSuccessfulDownloads = 0
    
    	print("==Downloads starting ==============")
    	for link in filesToDownloadList:
    		downloadAttempts = 0		
    		fileDownloaded = False
    		if verboseMessages: print("downloading file: ",link)
    		local_link = link;
    		local_link = saveFilesToThisDirectory + local_link.replace(relativeLinkPathBase,"")
    		local_filename = link.split('/')[-1]	
    
    		#make sure the local directory stucture has been created
    		path = Path(local_link.replace(local_filename,"")) 
    		path.mkdir(parents=True, exist_ok=True)
    		
    		while not fileDownloaded and downloadAttempts < maxDownloadAttempts:			
    			try: 
    				urllib.request.urlretrieve(link,local_link)	
    				localSuccessfulDownloads += 1	
    				fileDownloaded = True
    			except urllib.error.URLError as e:
    				downloadAttempts += 1 #we will retry the download the number of times allowed by maxDownloadAttempts variable
    				if verboseMessages: print("downloadError: ",e.reason)
    				if verboseMessages: print("downloadErrorFile: ",link," attempt:",downloadAttempts)
    				if downloadAttempts < maxDownloadAttempts:
    					time.sleep(15)  #wait 15 seconds before the next attempt
    				else:
    					print("Could not successfully download: ",link," after ",downloadAttempts," download attempts")
    				
    	print("==Downloads complete ==============")
    	print("SuccessfulDownloads: ",localSuccessfulDownloads," out of ",len(filesToDownloadList))
    	
    	
    print('==Process is starting ===================')
    #get the file links
    get_pageLinks(url, recursiveVal)
    print("==Collected ", len(filesToDownloadList), " file links ======")
    #now download the files
    download_files()
    

     

  14. Below I have included example Wget commands for downloading files from the PDS Geosciences Node. The first example demonstrates downloading a PDS data set from the PDS Geosciences Node archive. The second example demonstrates using Wget to download an Orbital Data Explorer (ODE) cart request.

    Dan Scholes 2/20/18
    Example of downloading data files using 
    links from HTTP PDS Geosciences Node Data Archive
    or Orbital Data Explorer (ODE) Cart location
    
    Note: In the future, changes to the PDS Geosciences Node website and Orbital Data Explorer website may cause this example to no longer function.
    Disclaimer: This sample code is provided "as is", without warranty of any kind, express or implied. In no event shall the author be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the sample code or the use or other dealings with the sample code.
    
    Wget website: https://www.gnu.org/software/wget/
    
    Example of downloading a PDS Geosciences Node archive subdirectory
    wget -rkpN -P c:\temp\data -nH --cut-dirs 2 --level=15 --no-parent --reject "index.html*" -e robots=off http://pds-geosciences.wustl.edu/mro/mro-m-crism-4-typespec-v1/mrocr_8001/
    
    Example of downloading ODE Cart Request
    wget -rkpN -P c:\temp\data -nH --cut-dirs 2 --level=15 --no-parent --reject "index.html*" -e robots=off http://ode.rsl.wustl.edu/cartdownload/data/sample
    
    -r means recursively download files
    
    -k means convert links. Links on the webpage will be localhost instead of example.com/path.
    
    -p means get all webpage resources, so wget will obtain images and javascript files to make website work properly.
    
    -N is to retrieve timestamps, so if local files are newer than files on remote website, the remote files will be skipped.
    
    -P sets the local destination directory for the downloaded files.
    
    -e is a flag option that must be set for the robots=off to work.
    
    robots=off means ignore robots file.
    
    -c allows the command to pick up where it left off if the connection is dropped and the command I re-run.
    
    --no-parent keeps the command from downloading all the files in the directories above the requested level.
    
    --reject "index.html*" keeps wget from downloading every directory's default index.html.
    
    -nH  will disable the generation of the host-prefixed directories. 
    	In the example above, a directory ode.rsl.wustl.edu will not be created locally.
    
    --cut-dirs 2  Ignore the count of directory components. Basically, this example will omit the first 2 directory levels from the path it creates locally for the files that are downloaded.
    	Example: http://ode.rsl.wustl.edu/cartdownload/data/sample  The first directory in the destination directory will be "sample".
    
    --level=depth
    --level=15  Levels to recursively search. The default is 5, but we will need to go farther with ODE cart and PDS Geosciences Archive.
    
    ---------------------------------------------------------------------------------------------------------------------------------------------------------
    -nd or --no-directories  it is used to put all the requested files in one directory. We are not using this feature, but a user may prefer this option.
    
    
    

     

  15. ODE has recently released a new ODE cart request download page. This is a new option for accessing a requested ODE cart request. A link to the page is provided in the order fulfillment email that a user receives after their cart request is ready for download. The new ODE cart request download page supports multiple download methods and offers faster options for users to retrieve requested ODS files.

    The  ODE cart request download page contains a single click link to download the entire user cart request using the Aspera Connect web browser plug-in. This option provides a convenient and fast download method for most users. The Aspera Connect browser plug-in employs the FASP data transfer technology to better utilize the user’s available bandwidth. HTTP and FTP links are provided on the page for users who prefer to download the files through a client application, a different web browser plug-in, a local script, or directly through the web browser.

    Additionally, the page supports browsing and downloading specific directories and files of the cart request through the Aspera Connect web browser plug-in or using direct HTTP links. Multiple files can be downloaded simultaneously through this page.

    Try the new page the next time you request PDS products through the ODE cart!

     

    odeCartRequestDownloadPageForum.jpg

  16. ODE has recently added a new option to download all of a PDS product's files through a single click. The new option is available from the product detail page of PDS products hosted by the PDS Geosciences Node. This option uses the Aspera Connect browser plug-in. Aspera technology more fully utilizes one's available bandwidth than traditional TCP connections, such as HTTP and FTP, which will improve the speed the file download speed.

    Give the new feature a try and let us know how it works for you.

    Example page with the option:

    http://ode.rsl.wustl.edu/mars/indexproductpage.aspx?product_id=FRT0000B278_07_IF164S_TRR3&product_idGeo=17880484

    Help page further describing the feature and installation of the Aspera Connect browser plug-in:

    http://ode.rsl.wustl.edu/mars/pagehelp/quickstartguide/index.html?product_detail_page.htm#aspera

     

    detailPageAsperaDownload1.jpg

×
×
  • Create New...