3 Data Quality Flag Description - Chuyên Trang Chia Sẻ Kiến Thức Thời Trang Mới Nhất

Next: 4 Raw Image Screening
Up: No Title
Previous: 2.3.2.2 Resolution Perpendicular to

For the creation of the IUE Final Archive, data quality ( $\nu$ ) flags
are provided on a pixel-by-pixel basis for the two-dimensional (2-D)
photometrically corrected (LI) and geometrically resampled (SI) output
files, and as a function of wavelength in the one-dimensional merged
extracted image (MX) file. These quality flags denote exceptional
conditions in the data which can range from fairly minor situations of
telemetry dropouts in the background regions to quite serious conditions
of telemetry dropouts in extracted spectral regions. The $\nu$ flag
values have been apportioned such that the more serious conditions have
more negative values in order to provide an immediate indication as to
the severity of the problem condition. In contrast to the method of
“epsilon” flagging as implemented in IUESIPS, where only one problem
condition is noted (i.e., the most severe), the $\nu$ flags are encoded
to indicate all problem conditions associated with each pixel or
wavelength bin.

The flexibility of the $\nu$ flags is derived from the bit-encoding of
the individual problem conditions. Using a total of 16 bits, the flags
are stored as negative values in two’s complement form (bit 16 contains
the sign), and the remaining 15 bits are utilized to represent each
defined problem condition. Table 3.1 describes the problem
conditions and defines the corresponding $\nu$ flag values. Once all
problem conditions are identified for each pixel or wavelength bin, the
individual flag values are added together to produce a total value which
is a unique combination of its components. Since additional $\nu$ flags
are available for the Final Archive dataset, and some problem conditions
may have an altered definition when compared to the IUESIPS $\epsilon$ flags, a more detailed definition of the $\nu$ flags is in order. The
descriptions of the $\nu$ flags in this chapter are ordered according to
their generation in NEWSIPS processing rather than by severity. Flags
produced by the raw image screening process
( RAW_ SCREEN )
are mentioned first, while the flags
introduced during the extraction phase (i.e., SWET or
BOXCAR )
are discussed last.

Table 3.1:
$\nu$ Flag Values
Condition
$\nu$ Flag Value
BIT

Pixels not photometrically corrected
-16384
15

Missing minor frame in extracted spectrum
-8192
14

Reseau (in the ITF)
-4096
13

Permanent ITF artifact
-2048
12

Saturated pixel (w.r.t. ITF pixel DN)
-1024
11

Warning track (near edge of PHOTOM region)
-512
10

Positively extrapolated ITF
-256
9

Negatively extrapolated ITF – far below ITF level 1
-128
8

RAW_ SCREEN
cosmic ray/bright spot
-64
7

SWET cosmic ray (low disp. only)
-32
6

Microphonics (LWR only)
-16
5

Potential DMU corrupted pixel
-8
4

Missing minor frame in extracted background
-4
3

Uncalibrated data point (MX data only)
-2
2

No known problem condition
0
1

The

RAW_ SCREEN
cosmic ray/bright spot detection algorithm
(Chapter 4.1
) is an application of a median filtering technique and
identification of the spots is based upon their limited spatial extent
and unusual brightness. This flag (-64) is differentiated from the
low-dispersion extraction (SWET)
cosmic ray flag (-32) which
is generated by a sigma rejection criterion discussed below.

While the NEWSIPS algorithms used to detect cosmic rays/bright spots and
microphonics (-16) are similar to the IUESIPS versions, the flagging of
missing minor frames (MMFs) is a new implementation. Clearly, it is
important to know of the existence of a MMF in the vicinity of spectral
information in order to evaluate the extent to which the MMF will affect
the data. All telemetry dropouts are flagged in 2-D images as MMFs, but
it is important to translate this information into how these dropouts
affect the extracted spectral data and background. As a result, MMF
pixels can be assigned either of two $\nu$ flag values depending on
their proximity to the spectral data. The first flag (-8192) is
utilized to label missing pixels in the spectral data. The second flag
(-4) denotes missing pixels in the extracted background. Since the
background is fit with a Chebyshev polynomial, missing pixels in the
extracted background potentially play a very small role in the
computation of the net flux. The much smaller absolute value of the
second flag reflects the less crucial nature of the condition. The MMF
detection algorithm is described in
Chapter 4.4.

A recent addition to the $\nu$ flags is the one that specifies pixels
which are most likely affected by DMU corruption. This flag (-8) is
allotted to pixels via a statistical process in
( RAW_ SCREEN )
and only applies to images taken after October of
1994. Information related to the detection of DMU corrupted pixels can
be found in
Chapter 4.5.

During the photometric correction (PHOTOM)
stage of
processing, extrapolations required for the conversion from data number
(DN) to flux number (FN) are appropriately flagged as being either
positive (DN above ITF Level 12) or negative (DN far below ITF Level 1).
Positively extrapolated pixels are flagged with -256 while pixels with
excessive negative extrapolations are given a flag of -128. Refer to
Chapter 6.3.2
for details concerning the definition of what constitutes
extrapolated data.

Some of the flags, or more properly, the pixel locations of the flags,
have been pre-defined based upon the location of the condition in the
ITF images. For example, the positions of all permanent artifacts
(-2048) and reseaux (-4096) are defined according to their positions
in the ITF images and are flagged after the science image and the ITF
are properly registered. The PHOTOM region is based upon the
shape of the illuminated target region of the ITF; this region is nearly
a circle for the SWP camera, but resembles more of a flattened circle
for the long wavelength cameras. Pixels located outside the
PHOTOM region are given a $\nu$ flag value of -16384. The
low-dispersion PHOTOM swath is a band contained within this
pseudo-circular region, optimally positioned about the low-dispersion
spectrum. In most instances, the photometric correction of an individual
pixel depends somewhat upon its neighbors. As the edge of the
PHOTOM region is approached, there can be wild fluctuations in the DN
(and hence the resulting FN) values, so that the photometric correction
may be less reliable in these regions. Therefore, a warning track flag
(-512) is defined to designate these less reliable flux values. The
warning track is an approximately five-pixel-deep buffer zone on the
inside edge of the PHOTOM region.

The saturation level for each pixel within the PHOTOM region has
been obtained by examining the ITF curve of each pixel and determining
the DN associated with any substantial leveling off of the ITF curve (DN
vs. exposure time) with increased intensity. According to the analysis
of the ITF images, there are a number of pixels which “saturate” at
values much less than 255 DN. Ergo, many more pixels may be
considered saturated by the NEWSIPS system than IUESIPS, since in
IUESIPS only pixels with a DN of more than 250 are deemed to be
saturated. Saturated pixels are issued a $\nu$ flag value of -1024.

SWET examines the fitted profile values in the cross-dispersion
direction in each wavelength bin and compares the actual data to the
profile, rejecting those values deviating by more than “N” sigma and
assigning a flag value of -32. The SWET cosmic ray detection is
not applicable for high-dispersion data, as a boxcar extraction is
applied in that case. A description of the SWET cosmic ray removal
process is given in
Chapter 9.5.

The uncalibrated data flag (-2) is found only in the final extracted
spectrum and indicates wavelength regions which have been extracted from
the resampled image, but whose absolute flux calibration has not been
defined.

Because the $\nu$ flags are stored as negative integers, a warning on
the correct interpretation of these data is necessary. Negative integers
are stored in two’s complement form by most computer systems. This means
that the internal bit settings for negative values are calculated by
subtracting the desired value from 215 or 231, for 2 or 4 byte
integers respectively. Accordingly, many more bits are turned on (set to
1) to represent the negative integer than just the bits for the actual
digit. Consequently, if intrinsic bit decoding functions are used to
decode the $\nu$ flag values as is, the wrong answer will be derived. It
is most straightforward to first take the absolute value of the $\nu$ flag image or vector before attempting to interpret the data.

Next: 4 Raw Image Screening
Up: No Title
Previous: 2.3.2.2 Resolution Perpendicular to
Karen Levay

12/4/1997