Jump to content
PDS Geosciences Node Community
Sign in to follow this  
Andy

SHARAD EDR data problems

Recommended Posts

Dear all,

I have a query that the SHARAD EDR echo data are saved by 8-bit signed integer or 8-bit unsigned integer? such as SHARAD 1689 orbit EDR data.

for instance, when we read the EDR data of SHARAD 1689 orbit EDR data, which has two .data files named as e_0168901_002_ss19_700_a_a.dat  and e_0168901_002_ss19_700_a_s.dat. We think that e_0168901_002_ss19_700_a_s.dat. file contains the expected echo data and the file e_0168901_002_ss19_700_a_a.dat only just contains the system information as auxiliary description of the echo data. Do we have the right understanding of this EDR data?

Therefore we only emphasize on file  e_0168901_002_ss19_700_a_s.dat to retrieve the expected echo data. When we extract the echo data from the file e_0168901_002_ss19_700_a_s.dat, we obtain the information that the SHARAD 1689 have 3786 bytes of each single-observations as A-scope and 4556 observations in along-track. The each observation has 3786 bytes which has last 3600 bytes contained the expected echo data, and the 3600 bytes echo data are stored as 8-bit signed integer while the header of each A--scope single observations are stored as 8-bit unsigned integer. The 186 bytes before the expected echo EDR data which has length of 3600 bytes illustrates the header system informations of each A-scope single observation data. Do we have the right understandings of the way of the data stored in .dat file?

 

In the edrsis.pdf file, we find that there should include file E_0123405_001_SS07_700_A.data, but we can not find this file in the data storage website http://ode.rsl.wustl.edu/mars/indexDatasets.aspx. In the  edrsis.pdf file, it describes that the e_0168901_002_ss19_700_a_s.dat. file is the Instrument data binary file and the e_0168901_002_ss19_700_a_a.dat represents Geometric information binary file, which can be found in page 15 in edrsis.pdf file. Can you explain these descriptions in edrsis.pdf file? 

Thanks you very much. We are looking forward to receiving answers from you.

 

Best Regards,

Andy

 

 

 

 

Share this post


Link to post
Share on other sites

Dear Andy,

I will try to answer all your questions.

 

> I have a query that the SHARAD EDR echo data are saved by 8-bit signed integer or 8-bit unsigned integer?

 

The echo data are stored as signed 8-bit integers.

 

> We think that e_0168901_002_ss19_700_a_s.dat. file contains the expected echo data and the file e_0168901_002_ss19_700_a_a.dat only just contains the system information as auxiliary description of the echo data. Do we have the right understanding of this EDR data?

 

Yes.

 

> When we extract the echo data from the file e_0168901_002_ss19_700_a_s.dat, we obtain the information that the SHARAD 1689 have 3786 bytes of each single-observations as A-scope and 4556 observations in along-track.

 

Almost right. There are 3786 bytes in each observation and 4551 (not 4556) observations in the file. This information comes from these keywords in the label file e_0168901_002_ss19_700_a.lbl:

    ROW_BYTES              = 3786
    ROWS                         = 4551
 

>Each observation has 3786 bytes which has last 3600 bytes contained the expected echo data, and the 3600 bytes echo data are stored as 8-bit signed integer while the header of each A--scope single observations are stored as 8-bit unsigned integer.

 

Almost right. It is true that the last 3600 bytes in the observation contain the echo data, and they are stored as 8-bit signed integers. This part of the file is described by a label fragment (called a format file) SCIENCE8BIT.FMT in the LABEL directory of the archive. In this format file, the keyword BIT_DATA_TYPE = MSB_INTEGER means that each 8-bit value is a signed integer. (If it were unsigned, the keyword value would be MSB_UNSIGNED_INTEGER.)

 

But the first 186 bytes in each observation are not all 8-bit unsigned integer values. Some are 8 bits, some are 32 bits (4 bytes), some are 16 bits (2 bytes), some are 4 bits. This part of the data file is described by the format file SCIENCE_ANCILLARY.FMT in the LABEL directory. These 186 bytes contain the header information for each observation.

 

> In the edrsis.pdf file, we find that there should include file E_0123405_001_SS07_700_A.data, but we can not find this file in the data storage website.

 

That is not a real file name; it is just an example. The SIS was written before any actual data were received, so the author made up a file name for the example.

 

> In the  edrsis.pdf file, it describes that the e_0168901_002_ss19_700_a_s.dat. file is the Instrument data binary file and the e_0168901_002_ss19_700_a_a.dat represents Geometric information binary file, which can be found in page 15 in edrsis.pdf file. Can you explain these descriptions in edrsis.pdf file?

 

I see these statements on page 9 of EDRSIS.PDF. The actual descriptions of the binary files are found in the format files in the LABEL directory, which are reproduced in section 7.5 of EDRSIS.PDF. The instrument data (the *a_s.dat file) is described by the format files SCIENCE8BIT.FMT and SCIENCE_ANCILLARY.FMT as I mentioned before. The geometric file (the *a_a.dat file) is described the format file AUXILIARY.FMT. There is one record in the geometric file for each record in the science data file; therefore, e_0168901_002_ss19_700_a_a.dat contains 4551 records. Each record has 267 bytes, as described in AUXILIARY.FMT. Some are 2-byte or 4-byte unsigned integers; some are 4-byte or 8-byte real numbers, and one value "GEOMETRY_EPOCH" is the UTC date and time in ASCII text.

 

I hope this helps you read the SHARAD EDR files.

 

Susan Slavney

PDS Geosciences Node

 

Share this post


Link to post
Share on other sites

Dear Andy,

I will try to answer all your questions.

 

> I have a query that the SHARAD EDR echo data are saved by 8-bit signed integer or 8-bit unsigned integer?

 

The echo data are stored as signed 8-bit integers.

 

> We think that e_0168901_002_ss19_700_a_s.dat. file contains the expected echo data and the file e_0168901_002_ss19_700_a_a.dat only just contains the system information as auxiliary description of the echo data. Do we have the right understanding of this EDR data?

 

Yes.

 

> When we extract the echo data from the file e_0168901_002_ss19_700_a_s.dat, we obtain the information that the SHARAD 1689 have 3786 bytes of each single-observations as A-scope and 4556 observations in along-track.

 

Almost right. There are 3786 bytes in each observation and 4551 (not 4556) observations in the file. This information comes from these keywords in the label file e_0168901_002_ss19_700_a.lbl:

    ROW_BYTES              = 3786

    ROWS                         = 4551

 

>Each observation has 3786 bytes which has last 3600 bytes contained the expected echo data, and the 3600 bytes echo data are stored as 8-bit signed integer while the header of each A--scope single observations are stored as 8-bit unsigned integer.

 

Almost right. It is true that the last 3600 bytes in the observation contain the echo data, and they are stored as 8-bit signed integers. This part of the file is described by a label fragment (called a format file) SCIENCE8BIT.FMT in the LABEL directory of the archive. In this format file, the keyword BIT_DATA_TYPE = MSB_INTEGER means that each 8-bit value is a signed integer. (If it were unsigned, the keyword value would be MSB_UNSIGNED_INTEGER.)

 

But the first 186 bytes in each observation are not all 8-bit unsigned integer values. Some are 8 bits, some are 32 bits (4 bytes), some are 16 bits (2 bytes), some are 4 bits. This part of the data file is described by the format file SCIENCE_ANCILLARY.FMT in the LABEL directory. These 186 bytes contain the header information for each observation.

 

> In the edrsis.pdf file, we find that there should include file E_0123405_001_SS07_700_A.data, but we can not find this file in the data storage website.

 

That is not a real file name; it is just an example. The SIS was written before any actual data were received, so the author made up a file name for the example.

 

> In the  edrsis.pdf file, it describes that the e_0168901_002_ss19_700_a_s.dat. file is the Instrument data binary file and the e_0168901_002_ss19_700_a_a.dat represents Geometric information binary file, which can be found in page 15 in edrsis.pdf file. Can you explain these descriptions in edrsis.pdf file?

 

I see these statements on page 9 of EDRSIS.PDF. The actual descriptions of the binary files are found in the format files in the LABEL directory, which are reproduced in section 7.5 of EDRSIS.PDF. The instrument data (the *a_s.dat file) is described by the format files SCIENCE8BIT.FMT and SCIENCE_ANCILLARY.FMT as I mentioned before. The geometric file (the *a_a.dat file) is described the format file AUXILIARY.FMT. There is one record in the geometric file for each record in the science data file; therefore, e_0168901_002_ss19_700_a_a.dat contains 4551 records. Each record has 267 bytes, as described in AUXILIARY.FMT. Some are 2-byte or 4-byte unsigned integers; some are 4-byte or 8-byte real numbers, and one value "GEOMETRY_EPOCH" is the UTC date and time in ASCII text.

 

I hope this helps you read the SHARAD EDR files.

 

Susan Slavney

PDS Geosciences Node

Dear Susan Slavney,

Many thanks for your appreciate, kind and helpful answers, these issues have been clearly resolved. I have some other questions that how the EDR and RDR data are generated. I give you my understanding that how the EDR and RDR data are generated. i think that firstly the 4 sequential echoes are summed to convert to single-observation A-scope data, whether there is average or not? then the echoes data are processed on board of spacecraft such as Doppler processing, the processed data become float 32-bit, then the processed float 32-bit data are converted  to signed 8-bit integer data, and the signed 8-bit data are transferred and download to earth ground. Do i have correct understanding about the origin and how the EDR data which are stored in website as we see are generated?

In addition, we give you my understanding that how the RDR data are generated. firstly, the data obtained on earth ground are signed 8-bit data, then the 8-bit data are processed on ground according to the requirements, then the 8-bit data are changed to float 32-bit again,  finally the again processed float 32-bit data are stored as RDR data. Do i have correct understanding about the origin of RDR data and the how the RDR data are generated?

Finally, if the 8-bit EDR data are generated from processed float 32-bit data on board, then i have a question that how the float 32-bit data are converted to signed 8-bit, and why not choose 16-bit for transferring to earth ground? 

Can you tell me the sampled bits digits of the applied analog to digital (AD) converter on board of SHARAD system?

 

Thanks you very much. We are looking forward to receiving answers from you.

 

 

Best Regards.

Andy

Share this post


Link to post
Share on other sites

Dear Susan Slavney,

Many thanks for your appreciate, kind and helpful answers, these issues have been clearly resolved. I have some other questions that how the EDR and RDR data are generated. I give you my understanding that how the EDR and RDR data are generated. i think that firstly the 4 sequential echoes are summed to convert to single-observation A-scope data, whether there is average or not? then the echoes data are processed on board of spacecraft such as Doppler processing, the processed data become float 32-bit, then the processed float 32-bit data are converted  to signed 8-bit integer data, and the signed 8-bit data are transferred and download to earth ground. Do i have correct understanding about the origin and how the EDR data which are stored in website as we see are generated?

In addition, we give you my understanding that how the RDR data are generated. firstly, the data obtained on earth ground are signed 8-bit data, then the 8-bit data are processed on ground according to the requirements, then the 8-bit data are changed to float 32-bit again,  finally the again processed float 32-bit data are stored as RDR data. Do i have correct understanding about the origin of RDR data and the how the RDR data are generated?

Finally, if the 8-bit EDR data are generated from processed float 32-bit data on board, then i have a question that how the float 32-bit data are converted to signed 8-bit, and why not choose 16-bit for transferring to earth ground? 

Can you tell me the sampled bits digits of the applied analog to digital (AD) converter on board of SHARAD system?

 

Thanks you very much. We are looking forward to receiving answers from you.

 

 

Best Regards.

Andy

Share this post


Link to post
Share on other sites

Andy,

Your questions are a little bit beyond my knowledge of the SHARAD data. Have you read section 4.1.3.4, Pre-Summing and Data Compression, in the SHARAD EDR Software Interface Specification? (http://pds-geosciences.wustl.edu/mro/mro-m-sharad-3-edr-v1/mrosh_0002/document/edrsis.pdf) Also, section 4.3.2, Data Product Generation, in the SHARAD RDR Software Interface Specification (http://pds-geosciences.wustl.edu/mro/mro-m-sharad-4-rdr-v1/mrosh_1002/document/rdrsis.pdf). I cannot tell you any more than what you find in those documents. If you have read them and still have questions, I will forward your questions to the SHARAD data providers.

Susan

Share this post


Link to post
Share on other sites

Andy,

Your questions are a little bit beyond my knowledge of the SHARAD data. Have you read section 4.1.3.4, Pre-Summing and Data Compression, in the SHARAD EDR Software Interface Specification? (http://pds-geosciences.wustl.edu/mro/mro-m-sharad-3-edr-v1/mrosh_0002/document/edrsis.pdf) Also, section 4.3.2, Data Product Generation, in the SHARAD RDR Software Interface Specification (http://pds-geosciences.wustl.edu/mro/mro-m-sharad-4-rdr-v1/mrosh_1002/document/rdrsis.pdf). I cannot tell you any more than what you find in those documents. If you have read them and still have questions, I will forward your questions to the SHARAD data providers.

Susan

Dear Susan Slavney,

I have read carefully these two pdf files (edrsis.pdf, rdrsis.pdf), but there still have some questions.

(1) on board processing, it includes the pre-summing up to 32 echoes, and the AD is the 8-bit samples, but the SHARAD processor works in 32-bit precision, then the 32-bit samples echos data are conversed to 8-bit by compressed process called fixed scaling and dynamic scaling before transferring and downloading to earth ground. finally, the converted and compressed samples echo data are transferred to ground using the CFDP protocol. The EDR data are also expressed by 8-bit integer. so i have a question that whether the EDR data stored as 8-bit are the compressed data through employing the fixed scaling or dynamic scaling to convert the 32-bit to 8-bit on board processing or not? If it is so, the EDR data should be decompressed by reversing scaling. it means that the EDR data download from PDS website should not be utilized directly, because the decompressed  process should be done before using the EDR data for observing the real echoes. Do i have correct understanding?

In a word, whether the EDR data are compressed data or not? 

 

The answers are important for us, can you provide us the detailed description about EDR data including the compressed process mentioned in edrsis.pdf file ? and whether there should need the decompressed process of EDR data for upcoming data use or not ?

Thanks you very much. We are looking forward to receiving answers from you.

 

Best Regards,

Andy

Share this post


Link to post
Share on other sites

Dear Susan Slavney,

I have read carefully these two pdf files (edrsis.pdf, rdrsis.pdf), but there still have some questions.

(1) on board processing, it includes the pre-summing up to 32 echoes, and the AD is the 8-bit samples, but the SHARAD processor works in 32-bit precision, then the 32-bit samples echos data are conversed to 8-bit by compressed process called fixed scaling and dynamic scaling before transferring and downloading to earth ground. finally, the converted and compressed samples echo data are transferred to ground using the CFDP protocol. The EDR data are also expressed by 8-bit integer. so i have a question that whether the EDR data stored as 8-bit are the compressed data through employing the fixed scaling or dynamic scaling to convert the 32-bit to 8-bit on board processing or not? If it is so, the EDR data should be decompressed by reversing scaling. it means that the EDR data download from PDS website should not be utilized directly, because the decompressed  process should be done before using the EDR data for observing the real echoes. Do i have correct understanding?

In a word, whether the EDR data are compressed data or not? 

 

The answers are important for us, can you provide us the detailed description about EDR data including the compressed process mentioned in edrsis.pdf file ? and whether there should need the decompressed process of EDR data for upcoming data use or not ?

Thanks you very much. We are looking forward to receiving answers from you.

 

Best Regards,

Andy

Share this post


Link to post
Share on other sites

Andy,

Please refer to the SHARAD instrument paper for details about the data processing, including compression. I'm afraid that is all I can tell you.

Seu, Roberto, et al., SHARAD sounding radar on the Mars Reconnaissance Orbiter, Journal of Geophysical Research (112): doi 10.1029/2006JE002745, 2007.

Susan

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×