Convert LP NS data from ASCII format to Geotif with a ArcGIS Pro Tool

[June Wang]

A user want to use the LP NS special data products for hydrogen abundance at ~15 km/pixel for his research. He want to import it into ArcGIS Pro and run a tool to show only data point with hydrogen signatures > 100ppm since water-ice should be present close to the surface.

The link to the archived LPNS data: https://pds-geosciences.wustl.edu/missions/lunarp/reduced_special.html

Below is the steps to access the data and do the conversion from an ASCII to a Geotiff in ArcGIS
1)    Choose the data.
The radius of the Moon is about 1737 km, which means the Moon scale is about 30 km/deg at the equator.
scale = radius *2 *pi / 360
     So for gridded data at 1/2 deg corresponds to about 15 km/pix.
2)    Convert from an ASCII to a Geotiff in ArcGIS Pro. ‘ASCII To Raster’ is a deprecated tool in the ArcGIS Pro. The Copy Raster tool can be used in the ArcGIS Pro to convert an ASCII file representing raster data to a raster dataset. First prepare an ASCII file for ArcGIS Pro according to the information in https://pro.arcgis.com/en/pro-app/latest/tool-reference/conversion/ascii-to-raster.htm. See attached example ASCII files. Suggest to change the file extension to *.asc to get better performance in ArcGIS Pro. Other extensions might have problem in the conversion. Then run the Copy Raster tool to make the conversion.   

 

hydrogenhd_mod.txt ironhd_mod.txt

[June Wang]

There is a long list of discussion on the reading of the LP NS special data products in binary format at openplanetary.slack.com (https://openplanetary.slack.com/archives/C3RHA4HFX/p1636384638008600)
The description of the binary *.dat data is copied below (https://pds-geosciences.wustl.edu/missions/lunarp/reduced_special.html). Basically, they are binary image arrays as 2-byte MSB_INTEGER, 720 samples * 360 lines.
"Each binary file is formatted as an image covering the entire planet with an array of 720 samples and 360 lines. Data values are scaled as integers in the image array with two bytes per pixel in IEEE order. For all binary files except the potassium, radon, and polonium, image values are ten times the values given in the ASCII files. The binary files for potassium abundances have integer values that are the same as values given in the corresponding ASCII file. The binary files for the radon and polonium data have integer values that are 1000 times the values given in the corresponding ASCII file. The binary image is a simple cylindrical map projection with 0.5° pixel size. The first image line corresponds to the south pole and the last image line corresponds to the north pole of the moon. The first sample corresponds to -180° East longitude and the last sample corresponds to 180° East longitude. Binary file names have the form <name>.dat and are about 0.5 MB in size."

Some sample Python codes to read the binary data are copied below.
From Chance,

import numpy as np
import struct
import matplotlib.pyplot as plt

pds_lp_file = './thoriumhd.dat'
pds_lp_data_binread = []
with open(pds_lp_file, 'rb') as file:
    file.seek(0,2)
    num_bytes = file.tell()
    file.seek(0)
    i = 0 
    while i < num_bytes:
        binary_data = file.read(2) 
        i += 2 
        unpacked = struct.unpack("h", binary_data) 
        pds_lp_data_binread.append(tuple(unpacked)[0])
        
magic_num = 720

pds_lp_arr = np.array(pds_lp_data_binread).reshape((int(len(pds_lp_data_binread)/magic_num), magic_num))
print("Data shape: {}".format(pds_lp_arr.shape))
print("Data min: {}".format(np.amin(pds_lp_arr)))
print("Data max: {}".format(np.amax(pds_lp_arr)))


plt.figure(figsize=(6,6))
plt.imshow(pds_lp_arr)
plt.show()

From Mario D'Amore, please see more details at https://gist.github.com/kidpixo/549fd1c95956166855431cc81e141626

import numpy as np
import pandas as pd
import matplotlib
from matplotlib import pyplot as plt
%matplotlib inline

# read using numpy
data = (np.fromfile('therms.dat',dtype=np.dtype('>h'),)/10)
# 10 scaling  is from the documentation, changes for each file.
# 360,720 is from the documentation, changes for each file.
plt.figure(figsize=[8,3.5])
plt.imshow(data.reshape([360,720]),origin='lower',cmap=plt.cm.inferno_r)
plt.colorbar(fraction=0.025)
plt.title('therms with numpy')

# read using pandas
df = pd.read_csv('therms.txt',skiprows=12,sep=',',
            names=['Lat_min','Lat_max',
                   'Lon_min','Lon_max',
                   'Counts' ,'Std'     ]
                )
df.plot.scatter(x='Lon_min',y='Lat_min',c='Counts',
                cmap=plt.cm.inferno_r,figsize=[8,4],
               title='therms with pandas')

But one thing need to notice is that all the binary data values are scaled to integers. After reading the data and scaling back to its original value, you will find there is slightly difference in the the accuracy of decimal points between ASCII and binary for all the data products if they have non-integer values in the ASCII files initially. For example,

ASCII Data                                Min                         Max
thoriumhd.txt                            0.32075                12.935
hydrogenhd.txt                      -56.197                  195.55
ironhd.txt                                   2.6220                 26.645

Binary Data                 Min      Min/10       Max     Max/10
thoriumhd.dat                 3         0.3           129       12.9
hydrogenhd.dat          -561    -56.1           1955    195.5
ironhd.dat                       26        2.6          266       26.6