Feng

Array scanning spectrometers are becoming an important element in remote sensing, particularly for in-situ studies. Often times the data are provided as spectral libraries and not a spectral image cubes. To support conversion of individual spectra to spectral image cubes we provide an IDL script that takes individual spectra and creates the cube. To run the script download the attached file and in ENVI/IDL (standard ENVI, not Classic) run create_library_cube.pro in IDL, then run it in the IDL prompt by typing create_library_cube, param1, param2, param3 (for example: create_library_cube, 10, 10, 0) or just create_library_cube if you don’t want to specify any input parameters. Explanations of Param1 to Param3 can be found in comments included in the IDL pro file. IDL script: ; Created: Apr. 18, 2023 ; By: Feng Zhou ; Edited: May 3, 2023 ; input parameters: ; cubeCol: int, designed output cube column ; cubeRow: int, designed output cube row ; zzFlag: int, 1 or 0, if 1 then convert spectra to zig-zag matter, ; for example we are using 10X10 cube as an example ; Each image cube is 10 spectra by 10 spectra taken as a raster across the surface. ; The first row has spectra 0 thru 9, the second row has spectra 19 thru 10, and so on (the spectra are ordered in a zig-zag fashion). ; if 0, The first row has spectra 0 thru 9, the second row has spectra 10 thru 19, and so on pro create_library_cube, cubeCol, cubeRow, zzFlag compile_opt idl2 e = ENVI(/CURRENT) lib=envi_pickfile(title='Select the spectral library to convert') if (lib eq '') then return envi_open_file, lib, r_fid=fid envi_file_query,fid, dims=dims, wl=wl, ns=ns, nl=nl spec=envi_get_data(fid=fid, dims=dims, pos=0) ; check if there are input parameters if(cubeCol > 0) then begin newNs = cubeCol endif else begin ;default column value newNs = 10 endelse if(cubeRow > 0) then begin newNl = cubeRow endif else begin ;default row value newNl = 10 endelse data = Fltarr(newNl,newNs,ns) data[*,*,*] =0.0 if(zzFlag eq 1 ) then begin for i=0,ns-1 do begin for j=0, nl-1 do begin reminder = j MOD newNs newrow = (j-reminder)/newNs oddOReven = newrow MOD 2 if(oddOReven eq 1) then begin newcol = newNs -1 - reminder endif else begin newcol = reminder endelse data[newrow, newcol, i] = spec[i,j] endfor endfor endif else begin for i=0,ns-1 do begin for j=0, nl-1 do begin reminder = j MOD newNs newrow = (j-reminder)/newNs newcol = reminder data[newrow, newcol, i] = spec[i,j] endfor endfor endelse newfile = lib.replace(".sli", "_s2c.img") p=[1,0,2] spec_out = transpose(data, p) raster=ENVIRaster(spec_out, URI=newfile, NBANDS = ns, NCOLUMNS = newNs, NROWS = newNl, DATA_TYPE = 4) raster.Save Task = ENVITask('SetRasterMetadata') Task.INPUT_RASTER = raster Task.NBANDS = ns Task.NCOLUMNS = newNs Task.NROWS = newNl Task.DATA_TYPE = 'Float' Task.BYTE_ORDER = 'Host (Intel)' Task.INTERLEAVE = 'bsq' Task.WAVELENGTH_UNITS = 'Micrometers' if( n_elements(wl) gt 1) then Task.WAVELENGTH = wl Task.Execute newRaster =e.OpenRaster(newfile) View1 = e.GetView() Layer1 = View1.CreateLayer(newRaster) end

Hi Emran, After communicating with the CRISM team, we got an answer from Frank Morgan, the developer of CAT: "One possibility is to call the IDL function directly. The ATP stuff is in routines atp_nontile.pro and atp_tile.pro. Assuming they’re working with non-tile (like FRS, FRT, HRL etc) data use atp_nontile.pro. The header describes the inputs. It takes a big structure with all the input parameters. Set the batch keyword to avoid halts on trapped errors (that mode is more lightly tested than non-batch, but hopefully will work.) A lot of the apparent complexity in that routine is handling different data format cases, trapping errors, reading ancillary data etc. The real work is fairly straightforward. If they want to maximize efficiency they may want to pull from crism_photom_corr for photometric correction, and scaleatm_pcm and patch_vs_artifact for atmospheric correction." Thanks, Feng

Hi Rob, Sorry for confusing you. I was thinking to build a Geographic lookup table from DDR in ENVI, then reference the TRDR to GLT to generate a TRDR_GLT image, and finally re-project it in ArcGIS. I did a quick test but unfortunately, it failed in the last step. Maybe you can try the ISIS which is open source software to do the projection. ISIS has the pixel2Map function: https://isis.astrogeology.usgs.gov/Application/presentation/Tabbed/pixel2map/pixel2map.html Thanks, Feng

Hi Rob, CAT does need the IDL. Do you try using the CRISM MTRDR product? Meanwhile are you familiar to ESRI GIS software? If you have the ArcGIS installed, you may use the map projection tool in ArcTool box to do the map projection. Thanks, Feng

Hi Sehaj, Please try the latest version 5.6.2. It is knowing issue for ENVI 5.6 that the map projection engine used in 5.6 cannot parse some strings correctly. In 5.6.2 release, ENVI fixed this problem. We had tested CAT map projection in ENVI 5.6.2 from several PCs. The performance is about within one minute. If you dodn't have access to 5.6.2, I would suggest you to switch back to 5.5.3 version. CAT map projection in 5.5.3 is also fast. Thanks, Feng

Hi Nathan, We emailed the question to the data producer one week ago but didn't get his response. On the other hand, we think your result and code are correct. We did get the same result as yours through different method. There is some difference between SU TER and summary parameters from IF TER. I suggest you can continue your way for the further steps. Thanks, Feng

Hi Nathan, We are contacting the creator of the SU_TER products to see if there is any additional steps involved. We will let you know once we get answers. Thanks, Feng

Hi Nathan, I am unable to run your Matlab code. I got some errors. One error is: I use CAT function to compute these summary parameters from IF_TER and then compare to the SU_TER. I also got the similar results like yours. I wonder if it is caused by calculation rounding error. Thanks, Feng

Hi Nathan, I will follow your step to see if we can have the same result. Meanwhile would you please post your result in the forum? Thanks, Feng Zhou

Hi Sehaj, Thanks for you update. I will help you to fix these problems. To better assist you, can we go to a zoom meeting to fix these problems? You can send me (feng.zhou@wustl.edu) an email with your available time slots so that I can send you a zoom link. BTW, would you please try the standard interface of ENVI 5.6? It has a lots of new features than classic. Thanks, Feng

Hi Sehaj, The header error is because the customized map projection engine data for CAT is not set up correctly in your computer. You can following the instruction document (custom_pedata_instructions.docx) in CAT_ENVI\aux_files\pe_user_defined folder to set up correctly. Please let me know if you still have a problem. BTW what ENVI version are you using? I saw you use ENVI classic. Is it possible for you to upgrade ENVI to 5+ version and use the standard interface? Thanks, Feng

Hi Holger, Thanks for bringing this to us. The AN list you mentioned is taken directly from the mission supplied document (Mars2020_Camera_SIS) and we will check with the mission team to see what changes should be made. Best, Feng BTW the areo.info link in your post seems not working.

hi Rochdi, My suggestion is that you merge the S (VNIR) and L(IR) cubes into one and then slice them to what you want. The S cubes and L cubes have minor offsets. To merge them, you have to register one another. And then use attached IDL code to merge them. Following are detailed steps: 1, download the CTX covers the S/L cubes area in ODE (using Geotiff format CTX product) for example using crism trdr (frt0001176e_07) as example: first search frt0001176e_07 product then show result on map, you can see frt0001176e_07 area on the map; select CTX layer; on map display, draw a rectangle to cover this area using "Select Products with Rectangle" tool; download the CTX product in Geotiff format for exmaple (B07_012228_1427_XN_37S039W.tiff) 2, in CAT doing map projection for both S and L cubes 3. register map projected S and L cubes to the CTX product by using "Image Registration Workflow" Manually select tie points (I select 9 tie points ) between CTX and TRDR: 4, rename the attached SL_alignment_v5.txt to SL_alignment_v5.pro and open it in IDL, In IDL interface, click "Run', it will ask you select L cubes (results from step 3) , then select S cubes (results from step 3) and output the merged data. Now you have the data covers from 0.3646 to 4. You can select bands you need to export a subset data. SL_alignment_v5.txt

Hi Rochdi, It shows your CAT is successfully installed (it is under Display Tab, CAT 7.4 not under the toolbox extensions) in the first recording you sent to me. You shall be able to use all CAT function. Please let me know if you have any problems. Thanks, Feng

Hi Rochdi: My timezone is Central Daylight Time (GMT -5). It's fine if we cannot go through virtual meeting. We can fix this issue via email communication. First I want to know several things: What's ENVI version you are using? Are you using Windows or Linux? Would you please check the idl.pref file described in the instruction document? Thanks, Feng

Hi: This is Feng, I am glad to help you on the CAT issue. Based on your description, it seems ENVI doesn't recognized the CAT path. Once CAT is installed successfully, it won't show on ENVI toolbox but will be under the Display tab. We can start a zoom if you have a prefer time to solve this issue. Thanks

Hi: This is Feng, I am glad to help you on the CAT issue. Based on your description, it seems ENVI doesn't recognized the CAT path. Once CAT installed successfully, it won't show on ENVI toolbox but will be under the Display tab. We can start a zoom if you have a prefer time to solve this issue. Thanks, Feng

Hi Ralph, I checked the same frame I processed two weeks ago. It seems I don't have the black strip. What is the value at the black strip when you move mouse on the black strip using cursor value in ENVI? How big is the result file? Would you please resample it to small one and then send it to me with the header file? Thanks, Feng

Hi Ralph, This is caused by ENVI displaying the CRISM 65535 data_ignore_value. The warp image is displaying the big span than the valid data range. To display it correctly, you can modify the histogram stretch. Here are the steps: 1, first check the histogram for three bands value range of the trr3: 2, use "save as" tool under File to output the warped result to specify the data ignore value: 3, then change the output image accordingly: From to: Please let me know if you have any questions. Thanks, Feng

Hi Ralph, i saw the miss-match feature when overlapping CRISM on CTX even projecting them to the same map projection. This is because CTX and CRISM are from two instruments and processed differently. So you may use ENVI image registration to co-register CRISM to CTX to reduce the difference (see image registration workflow in ENVI: https://www.harrisgeospatial.com/docs/ImageRegistration.html). Please let me know if you have any question. Thanks, Feng

Hi Ralph, This is Feng. I will assistant you on this issue.These definitions are in CAT folder: CAT_ENVI -> aux_files -> crism_projection. These are three files in the folder. And you shall copy them to ENVI folder like: C:\Program Files\Harris\ENVI55\IDL87\resource\pedata\user_defined You can check CAT installation guide for more details: step 5 in https://pds-geosciences.wustl.edu/missions/mro/CAT_74_ENVI_5x_Setup_Windows.pdf Please let me know if you have any question. Thanks, Feng

Hi Vidyesh, A data ignore value is a designated pixel value that ENVI should ignore when processing an image or computing statistics. When CAT processing these image, it sets the noData as 65535. If the data ignore value is not set as 65535, when ENVI display these images, it displays the whole data range (i.e. from 0 to 65535). So the input histogram for each color (RGB) is from 0 to 65535, the output display will arrange them from 0 to 255. Since most values data are close to zero end not the 65535 end, it returns 0 (for each color i.e. RGB) on the output display instead of a value between 0 and 255. So ENVI now displayes a image which has R=0, G=0, B=0. which is actual a black image. Hope this helps. Thanks, Feng

Hi Vidyesh, I download these two data and processed the exact same way as you described but didn't find any problem. Your processing looks fine. One thing you may look at is the header file of the stacking image. The blank in your image could be the data_ignore_value in your header. Would you please compare the header file with mines: ENVI description = { Create Layer File Result [Tue Dec 17 09:47:59 2019]} samples = 751 lines = 634 bands = 518 header offset = 0 file type = ENVI Standard data type = 4 interleave = bsq sensor type = Unknown byte order = 0 map info = {MRO Mars Equirectangular [IAU 2000] [-35.00N; -39.88E], 1.0000, 1.0000, -6787.3851, -2187402.8323, 1.8000000000e+001, 1.8000000000e+001, D_Mars_2000_IAU_IAG_custom_sphere_lat35, units=Meters} projection info = {17, 3389570.0, -35.000000, -39.880000, 0.0, 0.0, D_Mars_2000_IAU_IAG_custom_sphere_lat35, MRO Mars Equirectangular [IAU 2000] [-35.00N; -39.88E], units=Meters} coordinate system string = {PROJCS["MRO Mars Equirectangular [IAU 2000] [-35.00N; -39.88E]",GEOGCS["GCS_Mars_2000_IAU_IAG_custom_sphere_lat35",DATUM["D_Mars_2000_IAU_IAG_custom_sphere_lat35",SPHEROID["Mars_2000_IAU_IAG_custom_sphere_lat35",3389570.0,0.0]],PRIMEM["Reference_Meridian",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Equidistant_Cylindrical"],PARAMETER["False_Easting",0.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",-39.88],PARAMETER["Standard_Parallel_1",-35.0],UNIT["Meter",1.0]]} wavelength units = Nanometers data ignore value = 6.55350000e+004 band names = { Layer (Band 1:frt0001176e_07_if164s_trr3_corr_p.img), Layer (Band 2:frt0001176e_07_if164s_trr3_corr_p.img), ..... Also I am using CAT 7.4 in ENVI 5.5 standard (not classic version) to process data. This may be another difference between us. Please let me know if the problem is still there after you check the header or reprocessed in ENVI 5.5 standard. Thanks, Feng

Hi Wladimir, This is Feng, I will try to answer your questions: Should I obtain the summary parameters and then apply the map projection or it is the opposite? Basically they are identical. For my opinion, I will do map projection then summary parameters since it reserves the summary parameters band name. It is necessary the "ratioing" for the spectra and use the "Flatten summary parameters" option?. I've seen a post where its says that is enough the "Continuum Removed" option Sorry I don't understand this question. I will take the team scientist about this. - How can i obtain a better resolution for the summary parameters image?. I've seen very good resolution images from papers that doesn't look like what I have obtaining (img_1) The summary parameters doesn't change the resolution. For this product, it is 18m/pixel. Here is all CRISM EDR resolution: https://ode.rsl.wustl.edu/venus/pagehelp/quickstartguide/index.html?crism_edr.htm For some Along-track oversampled (ATO) Spatial pixels unbinned for target – 18 m/pixel cross-track, up to ~3 m/pixel downtrack, requires special processing for increased resolution; half swath length as above. For img-1, if you only select one band (BDI1000IR), you can see When I create the summary parameters image the spectra doesn't look like the TRDR pure image because the data values change. Does that mean that I have to use the spectra from the TRDR and the summary parameters image is just like a "color guide" fot the mineral phases? (img_2) Yes. For your attached Img-2, the index is not the wavelength but the summary parameters band: Finally, how can I use the summary parameters properly and elaborate mineralogical maps from they? For the summary parameters, you can read the following paper: Revised CRISM spectral parameters and summary products based on the currently detected mineral diversity on Mars https://agupubs.onlinelibrary.wiley.com/doi/10.1002/2014JE004627 Hope this helps. Thanks, Feng

Hi Wladimir, Would you please provide the scenes you worked on so that we can better go through from there? Thanks, Feng