Biases and Flats with MPO Canopus

Introduction

Software developers have produced lots of software packages that build and use bias and flat frames for astronomical CCD data frames. In this short tutorial, we review why astronomers require such frames and we describe how one software package, MPO Canopus, builds and uses these frames.

Definitions

We must be careful with our language while describing the "how and why" of flat and bias usage. So here we will list some definitions.

Overview

A CCD camera taking pictures of the sky can change the way the sky looks in several ways. For example, CCD picture has unwanted signal called "bias". This is signal that is always present, even in an exposure of zero time. We want to subtract it away. The CCD will add "dark current" to the picture. Dark current is signal that appears in the pixels because the CCD is warm. It increases linearly with time and is a function of temperature. The warmer the CCD, the faster the dark current accumulates. We want to subtract dark current too. Another source of contamination is the fact that all the pixels do not respond to light in exactly the same way. Some are more efficient at counting photons and they will appear brighter than another less efficient pixel receiving exactly the same number of photons. We want to get rid of this effect by multiplying by a flat frame that records the efficiency of each pixel in the CCD. Multiplying will brighten the less efficient pixels and dim the more efficient pixels.

For the mathematically inclined among the readers, we will call the bias frame B, the dark frame D, the flat frame F the data frame I. the exposure time for the dark frame tD, the exposure time for the data frame tI, and the resulting data frame is I'. Then our processing scheme will be

Each time we perform one of these subtractions or multiplications, we increase the noise in the final image. (The noise increases as the square root of the sum of the squares of the noise in each of the image, flat, dark, and bias frames.) So we must reduce the noise in the bias, dark, and flat as much as reasonable. We do that by combining several frames to create the bias, dark, and flat as described in the next section.

Creating Low-Noise Bias, Dark, and Flat Frames

Our first step is to create a low-noise bias frame. We do this by taking the median of all our zero time frames. We can think of this technique as stacking all our zero time frames so their pixel coordinates match. Then we take the median of the stack of pixels at each pixel coordinate. We create the bias frame from the resulting median at each coordinate. The noise of the resulting frame goes down just as if we were taking a mean. (one over square root of n where n is the number of pictures in the mean). For example, if we use four frames, our noise will be 1/2 the original. If we use 16 frames, our noise will be 1/4 the original. If we use 100 frames, our noise will be 1/10 the original.

The bias frame represents unwanted signal present in every frame. So the next step is to subtract the bias frame from every dark and flat exposure. The subtraction is a pixel by pixel subtraction. We subtract a bias frame pixel from the corresponding pixel in target frames.

The dark frame represents unwanted signal that is proportional to the time of an exposure. It is somewhat like the bias. Because of the time component, we have to be more careful how we process it. Three key things to remember are 1) the shutter is always closed while the CCD is integrating, 2) we should have exactly the same integration time in each of the frames used to build the dark frame, 3) the integration time should be long enough that we have significant signal beyond the bias level. Then we create the dark frame by finding the median value of each pixel in the same way as we created the bias. Again the noise goes down as if we were taking a mean of the dark frames. (One over square root of n where n is the number of pictures.) A problem with dark frames is that we generally assume a constant temperature for both the flat and data frames. This assumption is never exactly satisfied but sometimes it it is completely unjustified.

After creating the dark frame, we perform a pixel by pixel subtraction of the "corrected" value of the dark frame. That is, we divide the value of the dark frame pixel by its exposure time, multiply by the exposure time of our bias pictures and then subtract. Using multiple images reduces the noise.

The flat pictures are more complicated. They often have different exposure times, or if they are twilight exposures, the brightness of the sky changes with every picture. So we find the average brightness in every picture and multiply or divide each pixel in a picture by a constant so that all the pictures have the same average value. Then we select the median value of every pixel to form a version of the flat frame. There are some normalizations and inversions remaining to make the equation above work, but we are essentially done. And because we used multiple images, we reduced the noise of the flat.

How Many Calibration Frames are Required?

How many exposures do we need to create adequate bias, dark, and flat frames? Unfortunately, there is no pat answer. I advocate using plenty. Assume we have 16 of each and each frame we take has a characteristic noise of N. Then with 16 exposures to build a bias, the bias noise will be N/4. Similarly for the dark and flat. (We have made 48 exposures before we take a single data frame.)

When we flatten the data frames using the operations as indicated in the equation above, the noise adds in "quadrature". That is, for each operation (addition and multiplication) we get

adding noise in quadrature

Although the noise in the bias, flat, dark, and data is never the same, we will find it instructive to assume that it is the same. Then we find that the total noise in a processed data frame becomes

total noise for sets of 16 calibration frames

For comparison, let's look at the numbers when we use a four exposure set for each of the bias, dark, and flat.

adding noise in quadrature again
total noise for sets of 4 calibration frames

So when we use 16 exposures to generate flat, dark, and bias frames, we increase the noise in the data frame by 9%. When we use only 4 frames, we increase the noise in the data frame by 32%. If we use only one exposure for the bias, dark and flat, we increase the noise in the data frame by 100%.

One final note about the number of flat exposures. We need to have a complete set of flat exposures for each filter in which we will get data. The bias and dark exposures are independent of the color because the shutter is not open.

There are lots of ways to get flat exposures. I recommend you perform a web search on "creating flat frames". You will find lots of techniques and warnings. Each method has its advantages and drawbacks. Select the one most effective for your instrument, and observing/reduction techniques.

MPO Canopus Procedure

The following sections describe how to create a bias, dark, and flat master frames using MPO Canopus. The following sections also describe how to use the master bias, dark and flat to flatten data frames. There are many commercial and public domain software products that will perform these operations but we often use MPO Canopus so we provide the detailed instructions for its use here.

Sometimes, we have frames created with a camera that is cooled with liquid nitrogen or some special cooler that keeps the temperatures below -100°C. For these frames, we can ignore dark current and dark frames. So at the very bottom of this page is a set of instructions for that case. But, those instructions are a modification of the general instructions, so read through those first.

We begin with the assumption that the observer has all the necessary calibration frames for bias, dark and flat generation. The first procedure is creating a master bias.

A video tutorial for this section is available here.

Procedure Step Expected Result
From the main MPO Canopus window, select the "Utilities" menu, mouse over the "Image Processing..." option and select the "Batch Process" menu item. The "Batch Image Processing" dialog box appears.
Depress the "Select Files" button. The "Select Files for Processing" dialog box appears.
From the "Select Files for Processing" dialog box, select the set of files needed to build the bias frame. This selection uses the normal MS Windows selection techniques. Depress the "Open" button. The "Select Files for Processing" dialog box disappears. The "Batch Image Processing" dialog box now indicates the number of files selected.
From the "Batch Image Processing" dialog box, depress the button that has a "book" icon on its face. The selected list of files appears in the "Selected Files" dialog box.
Normally all files in the list are selected. The user may select a subset of the list by clicking the box to the left of the file name to toggle it into or out of this selection list. When satisfied with the selection, depress the "OK" button. The "Selected Files" dialog box disappears.
In the "Batch Image Processing" dialog box, find the "Save to" field and enter the directory name where flat frame file is to be written. From the Action pull down menu, select "Median". Depress the "Process" button. MPO Canopus processes the selected images and calculates the pixel values for master bias frame. This processing can take several minutes. At the end of the processing, Canopus produces a "Save Image" dialog box.
Within the "Save Image" dialog box, write the name of a file into which the bias frame is written. Then depress the "Save" button. Canopus writes the bias frame to the selected file and creates a viewing window that displays the bias frame.
Use the scroll bars of the viewing window to examine the bias. The bias frame should be featureless except for CCD defects.

We now subtract the bias from all the frames except the bias frames.

A video tutorial for this section is available here.

Procedure Step Expected Result
Assuming the "Batch Image Processing" dialog box is still available, depress the "Select Files" button. The "Select Files for Processing" dialog box appears.
Select all the frames except the bias integrations. That is, select the dark, flat, and data integrations. (There may be exceptions to this selection rule but it is generally the right thing to do.) Use the standard Windows file selection mechanism. Then depress the "Open" button. The "Select Files for Processing" dialog box disappears. The "Batch Image Processing" dialog box indicates the number of frames selected.
Within the "Batch Image Processing" dialog box, depress the button with the "book" icon on its face. A list of the selected files appears in the "Select Files" dialog box.
All the selected files have a red "X" indicating they are selected. The user may toggle any of the files into or out of the list by clicking on the box to the left of the file. Normally, all the files should remain as selected. Then depress the "OK" button. The "Select Files" dialog box disappears.
From the "Action" field in the "Batch Image Processing" dialog box, select "Merge Dark". The "Dark" field becomes enabled.
Within the "Dark" field, type the name of the master bias frame created previously. Within the "Save to" field, type the name of the directory into which the processed frames are to be written. (The "folder" icon to the right of these fields will make the selection process easier.) Then depress the "Process" button. Canopus subtracts the bias frame from each of the frames selected and writes the resulting frame in the selected output directory. The resulting file names have a "_P" appended to the original file name.

Now we can create the master dark frame. Brian Warner provided the following explanation of how dark frame scaling works.

"If the exposure time can be read from the FITS header of the dark, then the program computes the required scaling between the science frame and dark (assuming the exposure time can be read from from the science frame) before subtracting the dark."

"If the dark is already pre-scaled and it has a keyword for exposure that Canopus recognizes, then the exposure time in that master dark must be set to the same as the science images."

"If the exposure time cannot be found in the dark and/or science frame, or is 0, then the scaling ratio is set to 1.0."

In my opinion, the safest options are to make the dark integrations the same length of time as the data integrations. If the data integrations are variable length, the dark integrations should all be the same length and at least as long as the longest data integration.

The following table indicates the steps for creating a master dark frame using MPO Canopus.

A video tutorial for this section is available here.

Procedure Step Expected Result
Assuming that the "Batch Image Processing" dialog box is still available, depress the "Select Files" button. The "Select Files for Processing" dialog box appears.
From the "Select Files for Processing" dialog box, select the set of files needed to build the master dark frame. This selection uses the normal MS Windows selection techniques. Depress the "Open" button. The "Select Files for Processing" dialog box disappears. The "Batch Image Processing" dialog box now indicates the number of files selected.
From the "Batch Image Processing" dialog box, depress the button that has a "book" icon on its face. The selected list of files appears in the "Selected Files" dialog box.
Normally all files are selected. The user may select a subset of the list toggling the box to the left of each file name. When satisfied with the selection, depress the "OK" button. The "Selected Files" dialog box disappears.
In the "Batch Image Processing" dialog box, find the "Save to" field and enter the directory name where the dark frame file is to be written. From the Action pull down menu, select "Median". Depress the "Process" button. MPO Canopus processes the selected images and calculates the pixel values for master dark frame. This processing can take several minutes. At the end of the processing, Canopus produces a "Save Image" dialog box.
Within the "Save Image" dialog box, write the name of a file into which the master dark frame is written. Then depress the "Save" button. Canopus writes the master dark frame to the selected file and creates a viewing window that displays the dark frame.
Use the scroll bars of the viewing window to examine the dark. The dark frame should be relatively featureless except for CCD defects.

It is good policy to examine the FITS header of the new master dark frame. It must have an integration time specified or it will not scale properly. Typically, the integration time is defined by a FITS keyword called "EXPTIME". Canopus provides a FITS header editor so that the keyword can be inserted if necessary but you will have to look in the documentation to find how to do it.

On to the next major step. We have to subtract the scaled dark frame from the frames used to create the flat. Canopus will do the scaling automatically if the FITS header of the dark frame contains the integration time. The steps will be very similar to those used when we subtracted the master bias from all the other frames.

A video tutorial for this section is available here.

Procedure Step Expected Result
Assuming the "Batch Image Processing" dialog box is still available, depress the "Select Files" button. The "Select Files for Processing" dialog box appears.
Select all the frames to be used to create the flat. (There may be exceptions to this selection rule but it is generally the right thing to do.) Use the standard Windows file selection mechanism. Then depress the "Open" button. The "Select Files for Processing" dialog box disappears. The "Batch Image Processing" dialog box indicates the number of frames selected.
Depress the button with the "book" icon on its face. A list of the selected files appears in the "Select Files" dialog box.
All the selected files have a red "X" indicating they are selected. The user may toggle the selection of any of the files in the selection list by clicking on the box to the left of the file name. Normally, all the files should remain as selected. Then depress the "OK" button. The "Select Files" dialog box disappears.
From the "Action" field in the "Batch Image Processing" dialog box, select "Merge Dark". The "Dark" field becomes enabled.
Within the "Dark" field, type the name of the master dark frame created previously. Within the "Save to" field, type the name of the directory into which the processed frames are to be written. (The "folder" icon to the right of these fields will make the selection process easier.) Then depress the "Process" button.

A word of caution: Canopus will append an "_P" to the file name of the processed file. If the original file already has an "_P" appended, Canopus will write the new results using the old file name and the user will loose the original file. It is often best to create a new destination directory to avoid this problem.
Canopus subtracts the scaled master dark frame from each of the frames selected and writes the resulting frame in the selected output directory. The resulting file names have a "_P" appended to the original file name.

We are now ready to create the master flat frame. Recall that we have already subtracted the bias frame and dark frame from the flat integrations.

A video tutorial for this section is available here.

Procedure Step Expected Result
Assuming the "Batch Image Processing" dialog box is still available, depress the "Select Files" button. The "Select Files for Processing" dialog box appears.
Select all the frames to be used to create the master flat. Use the standard Windows file selection mechanism. Then depress the "Open" button. The "Select Files for Processing" dialog box disappears. The "Batch Image Processing" dialog box indicates the number of frames selected.
Depress the button with the "book" icon on its face. A list of the selected files appears in the "Select Files" dialog box.
Within the "Select Files" dialog box, the user may toggle the selection of any of the files in the selection list by clicking on the box to the left of the file name. Normally, all the files should remain as selected. Then depress the "OK" button. The "Select Files" dialog box disappears.
From the "Action" field in the "Batch Image Processing" dialog box, select "Median Normalized". Within the "Save to" field, type the name of the directory into which the master flat frame is to be written. (The "folder" icon to the right of these fields will make the selection process easier.) Then depress the "Process" button. Canopus calculates the normalized median from each of the frames selected and then presents a dialog box so the user can name the file.
Select a name for the flat and type it into the name field. Depress the "Save" button. Canopus saves the master flat frame with the chosen name. The file naming dialog box disappears. Canopus also presents an image viewing window so the user can examine the master flat frame.
Use the scroll bars in the viewing window to examine the flat. By right clicking on the image, the user can adjust the stretch of the frame to see more details. The flat should show lots of features like dust donuts, CCD defects, vignetting, and so on.

Finally, we can create the flattened data frames. We will use the "Merge Dark and Flat" option of the "Batch Image Processing" dialog box. This option will first subtract the time normalized master dark frame from the data. It then normalizes the master bias frame so its average pixel value is one. It then divides the data by the bias. The effect is to brighten pixels that have lower gain and dim pixels that have higher gain.

A video tutorial for this section is available here.

Procedure Step Expected Result
Assuming the "Batch Image Processing" dialog box is still available, depress the "Select Files" button. The "Select Files for Processing" dialog box appears.
Select all the data or "science" frames. Use the standard Windows file selection mechanism. Then depress the "Open" button. The "Select Files for Processing" dialog box disappears. The "Batch Image Processing" dialog box indicates the number of frames selected.
Depress the button with the "book" icon on its face. A list of the selected files appears in the "Select Files" dialog box.
The user may toggle the selection of any of the files in the selection list by clicking on the box to the left of the file name. Normally, all the files should remain as selected. Then depress the "OK" button. The "Select Files" dialog box disappears.
From the "Action" field in the "Batch Image Processing" dialog box, select "Merge Dark and Flat". The "Dark" and the "Flat" file name fields become enabled.
Within the "Dark" field, type the name of the master dark frame created previously. Within the "Flat" field, type the name of the master flat frame created previously. Within the "Save to" field, type the name of the directory into which the processed frames are to be written. (The "folder" icon to the right of these fields will make the selection process easier.) Then depress the "Process" button. Canopus subtracts the scaled master dark frame and then divides by the normalized flat for each of the frames selected and writes the resulting frame in the selected output directory. The resulting file names have a "_P" appended to the original file name.

Congratulations! You now have a set of flattened data files suitable for further processing.

Omitting the Dark Frame

We do not need a master dark frame when we obtain data with a CCD camera cooled below -100°C. Here are some hand waving instructions for that case.

A video tutorial for this section is available here.

Procedure Step Expected Result
Users will still need a master bias frame so generate the bias just as described above. The user will get a master bias frame.
Subtract the master bias frame from all the flat integrations and all the data (or science) integrations. Use the instructions above. The user will get a set of bias subtracted frames.
Create the master flat frame just as described above. (We have specifically omitted the master dark generation and subtracting the master dark frame from the flat and data integrations.) The user will get a master flat frame.
Perform the "Merge Dark and Flat" step as described above. However, the master bias frame will now serve as the master dark frame. That is, put the name of the master bias in the place where the name of the master dark normally goes. The user gets a set of flattened data frames.