Jump to content
PDS Geosciences Node Community

Dan Scholes

Geo Staff
  • Content Count

    39
  • Joined

  • Last visited

Everything posted by Dan Scholes

  1. We have released a new version of our MakeLabels tool. The new 6.2 version includes additional features to help with the creation of PDS labels. The program and documentation can be found on the PDS Geosciences Node's MakeLabels page. Version 6.2 updates include: The output label destination directory is created, if it does not already exist and the user has appropriate permissions. Any template label tags that are not replaced (due to syntax errors or missing columns in the spreadsheet) are listed in the post-processing summary report. The MakeLabel's GUI interface now has a sizeable report window. The report summary has been moved to the end of the processing output report. The option to upper or lower case fields from the source spreadsheet has been added to the template tags. A new template tag allows a single line to be hidden if it's corresponding field is empty in the spreadsheet. (example: <name><!-- |specimen_id| OR HIDE-THIS-LINE --></name>) Show if and hide if template tags no longer need to be left aligned in the template. These tags can be indented and spaced with the surrounding XML.
  2. We are pleased to announce the release of the PDS Geosciences Node Spectral Library website. See the release announcement under the main forum announcements. Feel free to ask questions and provide feature requests on this forum. Thanks!
  3. We are pleased to announce the release of the new PDS Geosciences Node Spectral Library website. The PDS Geosciences Node Spectral Library is a database of laboratory spectra submitted by various data providers. It currently includes spectra from the Reflectance Experiment Laboratory (RELAB) at Brown University. Additional data sets will be added to the website in the coming months. The website allows users to search the catalog of specimen and measurements using a facet search. Results can be viewed in quick view summary or full detail pages. Measurement data can be downloaded individually or through a cart system.
  4. Hi Christy, I just tried the Windows version of the software on my Windows 10 machine. It would not run properly, either. I think we can assume the software has not kept up with the latest operating systems. I do have a couple of alternative options. I have confirmed that the files can be viewed with the PDS4 viewer: https://sbnwiki.astro.umd.edu/wiki/PDS4_Viewer Using this tool, the data files can be exported as csv, tab delimited, and additional formats. Keep in mind that the archive you mentioned is both PDS3 and PD4 compatible. To use PDS3 tools, use the .lbl label, while PDS4 tools will use the .xml label. The XML label will be needed for the PDS4 Viewer. You could also write a script with IDL, MATLAB, Python, or another programming language to parse the data files. The column specifications are found in the label files. The PDS3 label file refers to format files which are found in the archive's label directory (https://pds-geosciences.wustl.edu/messenger/mess-e_v_h-grns-3-grs-cdr-v1/messgrs_2001/label/). The product you mentioned refers to https://pds-geosciences.wustl.edu/messenger/mess-e_v_h-grns-3-grs-cdr-v1/messgrs_2001/label/grs_cal_sh3.fmt. The PDS4 label contains all of the column definitions. Let me know if you have additional questions. Best wishes, Dan
  5. Hi There, I am one of the developers who maintains the ODE website. I ran the same query in the ODE LOLA RDR Query tool, and I received the same response. I believe it is reasonable. Remember, you are just limiting the output to one day. The tool is not setup very well for global coverage searches, but it can be done for narrow time windows. If you wish to query 2014-2019, I would suggest doing a region of interest. That is more of the objective of the query tool. The limited areas of coverage for 2014 and later is expected. The orbiter was in a different orbit and the instrument issues you mentioned. To see the different in coverage for later years, you can use the ODE map search page to filter LRO LOLA RDR coverage to specific time ranges. Link to map search interface: https://ode.rsl.wustl.edu/moon/indexMapSearch.aspx Map interface limited to LOLA RDR (image mapSearch1.jpg) LOLA RDR filtered to 2014 observations. (image mapSearch2.jpg) Thanks, Dan
  6. The ODE map search has been updated to have product coverage map layers grouped by mission/instrument/processing level. This map layer organization matches the product search page. Mars Orbital Data Explorer Map Search Lunar Orbital Data Explorer Map Search Mercury Orbital Data Explorer Map Search Venus Orbital Data Explorer Map Search Mars ODE Map Search Example:
  7. The PDS Geosciences Node and the LROC Data Node have established a faster data transfer method. As a result, ODE cart downloads of LROC data will be fulfilled noticeably faster than in the past. We encourage users to give it a try through the Lunar ODE website.
  8. New functionality has been added to the ODE product search page to allow users to select a freehand polygon location for coverage searches. Similar freehand polygon search filtering has been added to the map search page.
  9. Hi Raj Patel, Thank you for contacting us about your question. First, I confirmed there are no .QUB files in the THEMIS IRBTR (Infrared Brightness Temperature Record) data set. Newer versions of GDAL support the conversion of a .QUB file type into GeoTiff (.tif). Here is an example of the command: Gdal_translate -of GTiff D:\test\data\I00818001RDR.QUB D:\test\data\I00818001RDR.tif The first link on the following FAQ describes the tools ASU's Mars Space Flight Facility recommends for opening THEMIS images. http://viewer.mars.asu.edu/faq#t6n18 Let me know if you have further questions. Best wishes, Dan
  10. Hi Deepak, Thank you for the valuable suggestions and feedback. Our staff will be discussing these suggestions over the next couple of weeks to determine our current capabilities and prioritizing future ODE updates. I will be able to provide you with more details after these discussions occur. Also, we will have a booth at LPSC where your presentation ideas can be discussed. Please stop by the PDS booth during LPSC. Best wishes, Dan
  11. Hi Jairo, Thank you for emailing us with your question. You are receiving the “corrupted file” response because your computer does not have a native application for opening the PDS IMG files. Also, sometimes a cd/dvd image file is called an IMG, so an application used for mounting disk images tries to open the file in a different expected format. A typical PDS image is stored as a simple binary array. It is described by a PDS label (a set of ASCII-text keyword=value statements), which can be either attached or detached. An attached label is embedded at the beginning of the image file; a detached label is in a separate file with the same name, extension LBL. The label tells you everything you need to know to read the image -- the number of lines, number of pixels (samples) per line, the size and data type of a sample, and map projection information if the image is a map. (PDS deliberately does not rely on any commercial or proprietary formats to archive image data, in order to ensure the long-term viability of the data.) Typically IMG files are opened with programs including ENVI, NASA View, or GDAL. ArcMap might support IMG files, but I don’t remember. The IMG files can also be parsed and manipulated with code (Python, IDL, Matlab, C++, C#). In the future, I hope to add functionality into ODE to allow a user to request the files in various formats, but it is not on the near list. In the meantime, GDAL is a standard application for changing formats. http://www.gdal.org/ (general information and the link to download) http://www.gdal.org/gdal_translate.html (basic GDAL commands) Here is an example commands that can be used to convert an img to tif: Gdal_translate -of GTiff D:\test\data\MGN\fmap\fl13s309.img D:\test\data\MGN\fmap\fl13s309.tif NASA View is better for just viewing images: https://pds.nasa.gov/tools/about/pds3-tools/nasa-view.shtml Let us know if you have any further questions. Thanks, Dan
  12. October 10th, 2018 – MRO HiRISE Updates HiRISE EDR, RDR, DTM and Anaglyph data products released through August 31, 2018 (Orbit 56,699) See https://wufs.wustl.edu/ode/odeholdings/Mars_holdings.html
  13. October 8th, 2018 - Mar Odyssey Release 65 Data Updates Updated THEMIS standard data products (IREDR, IRRDR, IRBTR, VISEDR, VISRDR and VISABR data) from February 19, 2002 to March 8, 2018 Updated THEMIS geometry data products (IRGEO2, IRPBT2, VGEO2, and VISALB data) from September 28, 2013 to March 8, 2018 Updated GRS data products (CGS, SGS, DND, AHD, ELEMTS) from February 19, 2002 to January 9, 2018 See https://wufs.wustl.edu/ode/odeholdings/Mars_holdings.html
  14. September 18, 2018 - LRO Release 35 has been loaded into ODE. Updated LOLA EDR products from June 18th, 2009 to July 7th, 2018 Updated LOLA RDR products from July 13th, 2009 to August 8th, 2018 Updated LAMP EDR and RDR products from July 6th, 2009 to June 14th, 2018 Updated LEND EDR_SCI, EDR_HK, RDR_RSCI, and RDR_CHK products from June 18th, 2009 to June 14th, 2018 Updated LEND RDR_DLD and RDR_DLX products from June 27th, 2009 to March 14th, 2018 Updated Diviner EDR and RDR products from July 5th, 2009 to June 15th, 2018 Updated Mini-RF Bistatic data products (EDR, RDR, and DDR) from April 6th, 2011 to December 16th, 2017. Updated LROC EDR, CDR and RDR products from June 30th, 2009 to June 15th, 2018 Updated Diviner RDR query tool database from July 5th, 2009 to June 15th, 2018. Updated LOLA RDR query tool database from July 13th, 2009 to August 8th, 2018 Please see ODE Lunar Holdings - https://wufs.wustl.edu/ode/odeholdings/Moon_holdings.html
  15. Version 1.0.0

    499 downloads

    This zip file includes example Wget commands for downloading files from the PDS Geosciences Node. The first example demonstrates downloading a PDS data set from the PDS Geosciences Node archive. The second example demonstrates using Wget to download an Orbital Data Explorer (ODE) cart request.
  16. Version 1.0.0

    486 downloads

    This zip file contains a sample Python 3.6 script for downloading files from the PDS Geosciences Node. The sample includes a configuration for downloading data files from both a PDS Geosciences Node archive and files from an Orbital Data Explorer (ODE) cart request. The script supports multiple levels of sub directories, as well. The script includes variables that should be set by the user for his or her environment. The example PDS data set and ODE cart request both exist, and they are available for test executions of the script. Python 3.6 is required for the script to function.
  17. Below I have included a Python 3.6 sample script for downloading files from the PDS Geosciences Node. The sample includes a configuration for downloading data files from both a PDS Geosciences Node archive and files from an Orbital Data Explorer (ODE) cart request. The script supports multiple levels of sub directories, as well. The script includes variables that should be set by the user for his or her environment. The example PDS data set and ODE cart request both exist, and they are available for test executions of the script. Python 3.6 is required for the script to function. This script is also available for download in the downloads section of the forum. # PDSGeosciencesNode_FileDownload.py # Dan Scholes 2/19/18 # Pypthon 3.6 compatible version # Example of downloading data files using # links from HTTP PDS Geosciences Node Data Archive # or Orbital Data Explorer (ODE) Cart location # Note: One drawback of this script is that it downloads one file at a time, rather than multiple streams. # Additional Note: In the future, changes to the PDS Geosciences Node website and Orbital Data Explorer website may cause this example to no longer function. # Disclaimer: This sample code is provided "as is", without warranty of any kind, express or implied. In no event shall the author be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the sample code or the use or other dealings with the sample code. # Phython download website: https://www.python.org/downloads/ import urllib.request import re import time from pathlib import Path # Variables for user to populate---------- saveFilesToThisDirectory = 'c:/temp/data/' # local destination path to save files #next two lines are for downloading from the PDS Geosciences Node archive url = "http://pds-geosciences.wustl.edu/mro/mro-m-rss-5-sdp-v1/mrors_1xxx/" #enter the directory you would like to download relativeLinkPathBase = "http://pds-geosciences.wustl.edu" #this is the default location for the relative paths on the website (just leave this value) #next two lines are for downloading an ODE cart request #url = "http://ode.rsl.wustl.edu/cartdownload/data/sample/" #enter the directory you would like to download #relativeLinkPathBase = "http://ode.rsl.wustl.edu/" #this is the default location for the relative paths on the ode cart website (just leave this value) recursiveVal = True # True/False whether to download files in subdirectories of the specified location in the url variable verboseMessages = False # True/False whether to display verbose messages during the script processing # End of variables for user to populate---------- relativeLinkPathBase = relativeLinkPathBase.rstrip('/') maxDownloadAttempts = 3 filesToDownloadList = [] def get_pageLinks(inUrl,inRecursive): if verboseMessages: print("Cataloging Directory: ",inUrl) #directory to process myURLReader = urllib.request.urlopen(inUrl.rstrip('/')) myResults = myURLReader.read().decode('utf-8').replace("<a href=","<A HREF=").replace("</a>","</A>") myURLReader.close() data=myResults.split("</A>") tag="<A HREF=\"" endtag="\">" for item in data: if "<A HREF" in item: try: ind = item.index(tag) item=item[ind+len(tag):] end=item.index(endtag) except: pass else: #The link is found itemToDownload = item[:end] if "." in itemToDownload: #the link is to a file if relativeLinkPathBase not in itemToDownload: #is the path relative, so we add the base url itemToDownload = relativeLinkPathBase + itemToDownload filesToDownloadList.append(itemToDownload) else: # it's a directory, so let's go into it if recursive is chosen if inRecursive: if itemToDownload not in inUrl: #we make sure it isn't a link to parent directory if relativeLinkPathBase not in itemToDownload: itemToDownload = relativeLinkPathBase + itemToDownload # the directory is a subdirectory, so we will follow it if verboseMessages: print("subdirectory to process ", itemToDownload) get_pageLinks(itemToDownload,inRecursive) def download_files(): # download the files that were identified # this is refering to the global list of files to download localSuccessfulDownloads = 0 print("==Downloads starting ==============") for link in filesToDownloadList: downloadAttempts = 0 fileDownloaded = False if verboseMessages: print("downloading file: ",link) local_link = link; local_link = saveFilesToThisDirectory + local_link.replace(relativeLinkPathBase,"") local_filename = link.split('/')[-1] #make sure the local directory stucture has been created path = Path(local_link.replace(local_filename,"")) path.mkdir(parents=True, exist_ok=True) while not fileDownloaded and downloadAttempts < maxDownloadAttempts: try: urllib.request.urlretrieve(link,local_link) localSuccessfulDownloads += 1 fileDownloaded = True except urllib.error.URLError as e: downloadAttempts += 1 #we will retry the download the number of times allowed by maxDownloadAttempts variable if verboseMessages: print("downloadError: ",e.reason) if verboseMessages: print("downloadErrorFile: ",link," attempt:",downloadAttempts) if downloadAttempts < maxDownloadAttempts: time.sleep(15) #wait 15 seconds before the next attempt else: print("Could not successfully download: ",link," after ",downloadAttempts," download attempts") print("==Downloads complete ==============") print("SuccessfulDownloads: ",localSuccessfulDownloads," out of ",len(filesToDownloadList)) print('==Process is starting ===================') #get the file links get_pageLinks(url, recursiveVal) print("==Collected ", len(filesToDownloadList), " file links ======") #now download the files download_files()
  18. Below I have included example Wget commands for downloading files from the PDS Geosciences Node. The first example demonstrates downloading a PDS data set from the PDS Geosciences Node archive. The second example demonstrates using Wget to download an Orbital Data Explorer (ODE) cart request. Dan Scholes 2/20/18 Example of downloading data files using links from HTTP PDS Geosciences Node Data Archive or Orbital Data Explorer (ODE) Cart location Note: In the future, changes to the PDS Geosciences Node website and Orbital Data Explorer website may cause this example to no longer function. Disclaimer: This sample code is provided "as is", without warranty of any kind, express or implied. In no event shall the author be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the sample code or the use or other dealings with the sample code. Wget website: https://www.gnu.org/software/wget/ Example of downloading a PDS Geosciences Node archive subdirectory wget -rkpN -P c:\temp\data -nH --cut-dirs 2 --level=15 --no-parent --reject "index.html*" -e robots=off http://pds-geosciences.wustl.edu/mro/mro-m-crism-4-typespec-v1/mrocr_8001/ Example of downloading ODE Cart Request wget -rkpN -P c:\temp\data -nH --cut-dirs 2 --level=15 --no-parent --reject "index.html*" -e robots=off http://ode.rsl.wustl.edu/cartdownload/data/sample -r means recursively download files -k means convert links. Links on the webpage will be localhost instead of example.com/path. -p means get all webpage resources, so wget will obtain images and javascript files to make website work properly. -N is to retrieve timestamps, so if local files are newer than files on remote website, the remote files will be skipped. -P sets the local destination directory for the downloaded files. -e is a flag option that must be set for the robots=off to work. robots=off means ignore robots file. -c allows the command to pick up where it left off if the connection is dropped and the command I re-run. --no-parent keeps the command from downloading all the files in the directories above the requested level. --reject "index.html*" keeps wget from downloading every directory's default index.html. -nH will disable the generation of the host-prefixed directories. In the example above, a directory ode.rsl.wustl.edu will not be created locally. --cut-dirs 2 Ignore the count of directory components. Basically, this example will omit the first 2 directory levels from the path it creates locally for the files that are downloaded. Example: http://ode.rsl.wustl.edu/cartdownload/data/sample The first directory in the destination directory will be "sample". --level=depth --level=15 Levels to recursively search. The default is 5, but we will need to go farther with ODE cart and PDS Geosciences Archive. --------------------------------------------------------------------------------------------------------------------------------------------------------- -nd or --no-directories it is used to put all the requested files in one directory. We are not using this feature, but a user may prefer this option.
  19. ODE has recently released a new ODE cart request download page. This is a new option for accessing a requested ODE cart request. A link to the page is provided in the order fulfillment email that a user receives after their cart request is ready for download. The new ODE cart request download page supports multiple download methods and offers faster options for users to retrieve requested ODS files. The ODE cart request download page contains a single click link to download the entire user cart request using the Aspera Connect web browser plug-in. This option provides a convenient and fast download method for most users. The Aspera Connect browser plug-in employs the FASP data transfer technology to better utilize the user’s available bandwidth. HTTP and FTP links are provided on the page for users who prefer to download the files through a client application, a different web browser plug-in, a local script, or directly through the web browser. Additionally, the page supports browsing and downloading specific directories and files of the cart request through the Aspera Connect web browser plug-in or using direct HTTP links. Multiple files can be downloaded simultaneously through this page. Try the new page the next time you request PDS products through the ODE cart!
  20. ODE has recently added a new option to download all of a PDS product's files through a single click. The new option is available from the product detail page of PDS products hosted by the PDS Geosciences Node. This option uses the Aspera Connect browser plug-in. Aspera technology more fully utilizes one's available bandwidth than traditional TCP connections, such as HTTP and FTP, which will improve the speed the file download speed. Give the new feature a try and let us know how it works for you. Example page with the option: http://ode.rsl.wustl.edu/mars/indexproductpage.aspx?product_id=FRT0000B278_07_IF164S_TRR3&product_idGeo=17880484 Help page further describing the feature and installation of the Aspera Connect browser plug-in: http://ode.rsl.wustl.edu/mars/pagehelp/quickstartguide/index.html?product_detail_page.htm#aspera
  21. The PDS Geosciences Node released the MakeLabels tool this July. The program can be used to generate PDS4 labels using a label template and one or two Excel spreadsheets. It replaces placeholders in the template with values from the spreadsheets. In fact it can be used to output any type of ASCII file. More information about the program and it's downloadable file can be found on the PDS Geosciences Node website. http://pds-geosciences.wustl.edu/tools/makelabels.html
  22. The ODE team has posted three simple Python examples of querying the ODE REST interface. These scripts are found in the forum's download section. https://geoweb.rsl.wustl.edu/community/index.php?/files/file/26-ode-rest-python-examples/
  23. Version 1.0.0

    1435 downloads

    Attached are three simple Python examples of querying the ODE REST interface. odeProductCount_Python27.py - A Python 2.7 example of querying the ODE REST interface for the count of products of particular product type. The results are written to the command prompt interface. odeProductCount_Python36.py - A Python 3.6 example of querying the ODE REST interface for the count of products of particular product type. The results are written to the command prompt interface. odeREST_FileDownload.py - A Python 3.6 example of querying ODE for products of a particular mission/instrument/product type. The product files of the first 10 query results will be downloaded to the local machine.
  24. Note for ENVI users of map-projected data: Some Geosciences Node hosted map-projected data sets (.IMG files) are not read by ENVI as projected data sets. Using the Geospatial Data Abstraction Library (GDAL) gdal_translate command, users can easily convert most of these data products into GeoTIFF or ENVI file formats which are read by ENVI as projected data. GDAL packages for various computer systems can be found here: https://trac.osgeo.org/gdal/wiki/DownloadingGdalBinaries. The Windows package used in this example is the OSGEO4W 64-bit version. A MacOS X build is available, as well. Download and install a GDAL package that has gdal_translate.exe. gdal_translate.exe runs from the command line. Make the folder that contains gdal_translate.exe your current directoryTo do this, type "cd path" without the quotes into the command line, where "path" is the full path to the folder that contains gdal_translate.exe. gdal_translate.exe is executed with the following syntax: gdal_translate –of outputtype inputfilepath outputfilepath "outputtype" is GTiff for a GeoTiff (.tif) and ENVI for an ENVI file (.dat) Both file paths are the full path to the file. In the case of a file with a detached PDS label, the "inputfilepath" points to the label, not the data file. Open the file as normal in ENVI. Example using OSGeo4W 64-bit (Windows): C:\>cd OSGeo4W64\bin C:\OSGeo4W64\bin>gdal_translate -of GTiff C:\hrsc\h2064_0000_dt4.img C:\hrsc\h2064_0000_dt4.tif Example using GDAL 2.1 Complete (MacOS X): :/ cd Library/Frameworks/GDAL.framework/Programs :Programs ./gdal_translate -of ENVI /Desktop/hrsc/h2064_0000_dt4.img /Desktop/hrsc/h2064_0000_dt4.dat Examples of data sets that benefit from the GDAL translate command: MRO - HiRISE DTM Mars Express - all HRSC, OMEGA DDRGM Odyssey - THEMIS IRGEO2, VGEO1, VGEO2 MGS - MOLA MEGDR, TES TIMAP Additional instructions for specific data sets: Mars Odyssey THEMIS Visible map-projected .CUB files (VGEO1 and VGEO2) have detached PDS labels, but the map-projection data is in the .CUB file, so the "inputfilepath" should be the data file itself. Mars Odyssey THEMIS Infrared map-projected .CUB.gz files (IRGEO2) need to be un-zipped before using gdal_translate. These data are similar to the visible data files in "inputfilepath" syntax (.CUB, not .LBL). More details about the gdal_translate command can be found at http://www.gdal.org/gdal_translate.html.
  25. No problem. Just let me know if you experience any further problems. Thanks, Dan
×
×
  • Create New...