Chapter 2, Calibration -- Darks and Flats
Now that you have a hard disk full of raw images after a successful imaging session, it's time to start putting the final
image together. This chapter covers the first steps of image processing: Calibration involves utilitizing any
dark frames and/or flat field frames to prepare the images for further processing. Adding and stacking multiple images
lets you put together the shorter exposures you may have taken to produce one "final" image for that part of the image,
whether it's for luminance or one of the color channels.
I hope you've noticed that, aside from the stern warning in the previous chapter, I'm covering calibration and stacking BEFORE talking about stretching or histogram manipulation --
there's a reason for that! You should NOT manipulate the raw image histogram by stretching, brightness or contrast adjustment,
or any other operations until after calibration and/or stacking. Any image data changes you make before doing these basic
operations will affect the results (almost always negatively) from calibration and stacking operations, so don't do it!
Of course it's OK to do what some programs call a "screen stretch" (that is, it affects the way the raw image is displayed
on the computer screen, but does NOT affect the actual data in the saved file) to see how well you've captured the object
or how well your tracking is going, but do not do any operations that change the raw data in the saved file until after
calibration and image stacking is finished! MaximDl, for example, lets you manipulate a clearly-labeled "Screen Stretch" histogram while an image is
displayed, so that you can play with the brightness ranges and see what data is in your raw image. Changing the screeen stretch parameters does not
affect the data in the image file in any way, only how it's displayed. Some other image processing programs will actually auto-stretch the
image to be displayed, but they do so by changing the pixel values in the image itself. If you then manipulate that image, your results will not
be as good as they would have been had you used un-stretched data. I can't emphasize this enough: LEAVE THE BRIGHTNESS VALUES ALONE FOR NOW!
Example Files
From this point on throughout the tutorial, you can follow along with the examples given by downloading a set of raw files from
here. To use the
files, which are contained in a ZIP archive format, un-zip them to a specific directory you've created on your PC to hold the example files (for
example, create a folder called "ImageTutorialFiles" and UNZIP all the files to that folder). In the archive are all the files used in the examples,
and are the same ones used to create the images you see throughout the tutorial. They include luminance frames, Red, Green, and Blue filtered color
frames, dark frames, and flat-field frames. The files are all in standard FITS format, and should be loadable by any of the popular astro-image processing
programs including MaximDL/CCD, AstroArt, IRIS, AIP4WIN, etc. Please notify me (e-mail) if you have a problem opening the files in a particular software
package. The ZIP archive is rather large -- about 6 Megabytes -- because it has to contain all of the necessary files. I'll apologize in advance to those
reading this with slow internet connections, but if you like you can skip the download and just follow along without actually working with the files.
Calibration -- Dark Frames
Calibration means to remove artifacts from a raw image that are there artificially, and will affect the final data. The first of these
artifacts that may have to be dealt with is dark current. Your CCD camera does a good job of accumulating photons from the sky into each
of those little pixel wells on the chip, and converting those photon hits to a count indicating the brightness of the object at that pixel.
But it's not perfect, and the underlying electronics in the camera can add some electrons to some of the pixels, and the camera, being just
a collector, doesn't know if that charge came from photons coming through the telescope or electrons sneaking in through the chip's substrate.
This extra charge shows up as bright spots in an image, because the pixel counts the photons that hit it as well as the dark current electrons,
making those pixels brighter than surrounding pixels. Fortunately, for any given exposure time and temperature, the camera will always produce
the same number of dark current electrons in each pixel, so the effect can be removed from the image by the subtraction of a dark frame. A dark
frame is an exposure that is the same length as the real image's exposure, but with no light allowed to hit the CCD detector. Since no light comes
from outside, the dark frame records only the dark current generated by the camera/chip, and subtracting the dark frame from the regular exposure
removes the dark current from the image.
A dark frame from a camera with a fairly high dark-current chip -- that dark current noise will be in your images unless you subtract a dark frame!
Some cameras, notably those from StarlightXpress with low-noise Sony chips, don't need dark frames because of their low dark current...this is
a dark frame from an HX916 camera, there is so little dark current you can get away without subtracting darks much of the time.
Subtracting dark frames from your images involves three basic steps:
-- Take a series of dark frame images (at least 5, 10 is better!) the same exposure length as your light frames
-- Median combine (or average) the individual dark frames to make a "master" dark frame
-- subtract the master dark frame from each raw, individual light image
Keep in mind first of all that despite removing any dark current present in your images, which is an artifact and not noise, subtracting
dark frames from your images will ALWAYS add noise to your images, not reduce it. You can minimize the amount of noise that is added
to your images by dark frame subtraction by taking a large number of dark frames and median combining them -- doing a median combine on the
dark frame images will help remove noise from the "master" dark and insure that there is not much more than the dark current in the master
that you want to subtract, and the more images you median-combine the lower the noise component of the master dark frame will be. There is
a point of diminishing returns, however, so usually around 10 to 20 dark frames is considered completely sufficient to make a master dark frame.
You can also average the individual dark frames together, but averaging them together will not reduce spurious noise artifacts in the master
nearly as well as a median combine will. Refer to the stacking methods part of this chapter for more information on diferent ways to combine
images. Note that you CANNOT sum (add) your individual dark frames together -- if you had 10 individual dark frames, and you add them together, you
have a master dark frame with 10 times the dark current of a single frame, not exactly what you want to subtract from a light frame, is it?
Once you have a master dark frame made, save it to disk. Make sure you don't alter the data in the master dark frame in any way
after you've created the median/average master file -- no histogram stretching, no auto-stretch, no blurring, sharpening, nothing. For
the dark frame to work properly, it has to be completely raw, just the average or median combination of all of the individual dark frames.
The actual procedure of subtracting a dark frame varies depending on the software you're using to process your images. In Maxim DL, for example,
you can simply select "Set Calibration" from the "Process" menu, and select the master dark frame you just made as the dark frame to subtract
from your images. You can then either load all of the images into Maxim, select "Calibrate All" from the "Process" menu, and all of your
light images will have the dark frame subtracted from them. There are easier ways, including using a simple script to batch process
all of your files, which will be explained a little later. Once all of your light frames have had the master dark frame subtracted from them,
you should save them back to the hard drive, but with a different name than they originally had or in a different directory! This is actually a key
tip to remember as we proceed through the tutorial: each time you open a file and do some kind of operation on it, save the results of the new operation
as a new file. Not only does this keep the version you had just before doing the operation intact, it lets you go back to any step in the process later
on and try a slightly different approach.
Calibration -- Flat Fielding
Calibration also includes flat-field images. What is a flat-field image? It's basically an image of what your optical system and camera chip look like without any sky objects. This is important because nearly every optical system has "defects" in it that can show up in your images. These defects include bits of dust on optical surfaces, which show up in sky images as round spots or donuts, like below:
"Dust donuts" in a raw sky image, from an image shot through an 8" SCT
Many optical systems also don't quite fully illuminate the full chip of a CCD camera -- because of the design of the system, or the use of focal reducers, the image is brighter towards the center with light falloff towards the outer parts of the image. This is known as "vignetting," and produces an image like the one below:
Vignetting in an image taken with a 10" SCT and f/6.3 focal reducer,
notice how the background is more black towards the outer edges of the image...
Finally, most CCD chips themselves have a slightly uneven response to light across the surface of the chip. This can be due to very slight manufacturing defects, a slightly longer exposure time on different parts of the chip as the image is read out, etc. This uneven chip response will often show up as a gradient from light to dark across an image. The image below, stretched to greatly exaggerate the appearance of the gradient, is a "camera flat" (an image taken with the camera only, NOT through a telescope) that shows the chip's actual response to an evenly illuminated, plain white surface:
"Camera flat" image showing non-linear response of a CCD chip
Stretched to exaggerate the effect, in reality the values from top
to bottom differ only by a few hundred counts
Flat field calibration can magically remove all of these artifacts from your raw images! So, what do you need to do to get this magic flat-field image that fixes up all of these problems? In simple terms, you need an image of the optical system. You can get such an image by pointing the telescope at some perfectly evenly illuminated surface, with the focuser in the same position as your sky images, and with all the same filters, reducers, etc. in place, and taking an image. It does get a little more complicate than that, however. Here are some steps to take to insure good flat-field images that will do what they are supposed to do:
1. How "bright" your flat-field images are is VERY important. You want the average pixel values in a flat-field image to be around one-half the maximum brightness value that your camera can produce. For the most common 16-bit cameras, that means an average value of 32,767. You can go a little higher or lower than this value, so the actual range you should aim for with a 16-bit camera is 20,000 to 35,000.
2. You should take at least 10 individual flat-field images, and median combine them to produce a "master flat" image. Median combining a large number of flats will reduce the noise in the master flat, remove any one-time anomalies from the images such as cosmic-ray hits, and produce a smoother master flat that will work better. Ten images combined is the minimum. Using twenty or thirty produces better results. Much more than 30 won't show any difference.
3. It's VERY important that flats be shot with the EXACT SAME optical setup as your sky images. So, if you're doing LRGB imaging for color shots, that means you'll need seperate master flats that match the imaging setup for each of the L,R,G,B images -- with the proper filter (or lack of one) in place, the same focus point, etc. That's a LOT of images to shoot!
4. You MUST image a very evenly illuminated surface or object to get good flats. If the illumination is not perfectly even across the entire field of view (say, for example, you pointed your scope at the wall below your porch light to shoot flats, and it's brighter at the top of the filed of view than at the bottom), you'll be ADDING artifacts of uneven illumination to your images as much as removing it.
That last point -- getting an evenly illuminated flat -- is the subject of much debate and ingenuity among CCD imagers. There are lots of ways to get an evenly illuminated surface for your flat field images, here are some of the most popular:
1. Use a diffuser on the front of the scope. This can be as simple as a white t-shirt stretched across the front of the scope and held in place with rubber bands, or as elaborate as a self-built "light box." Here are some web resources to give you some ideas for making a diffuser/light box for your own telescopes:
Chuck Shaw's light box construction page
Dean Salman's light box for a TV-76 refractor
2. "Sky Flats" -- you can leave the front of the telescope uncovered, point it at a fairly evenly illuminated part of the sky just before or after twilight, turn off your mount's drive, and shoot your flats. Each of the flats will have star trails in it since the mount is not tracking, but when median-combined the star trails disappear and you're left with a nice, even flat field image. The biggest downside of this technique is that you need a LOT of flats that you can median-combine to get rid of all the stars (very important, or you'll have streaks from trailed stars in your calibrated images!), and that the sky brightness changes fairly rapidly just before/after twilight, so you'll have to constantly adjust your exposure times to get average pixel values in the right range. Still, with no additional equipment needed, it's a simple method.
As I said, there are lots of ways to shoot flats. My own method uses the t-shirt trick. Here's what I do:
1. I stretch a clean, white t-shirt across the aperture of my scope, using both the front and back of the shirt for two layers of white cloth, and fix it there with a rubber band. Make sure to smooth out any wrinkles in the material INSIDE the scope's field of view.
2. If I'm at home, I have a "remote lightbox" that I point the scope at -- this is just a foam-core box 2-feet square, with four layers of semi-translucent milk plastic over the front, that I hang on my porch light in my backyard. The diffusion of the milk plastic creates a nice, evenly illuminated white target. with my porch light about 20 feet from my scope, even my widest-field scopes can only see the front illuminated surface. With this constant-brightness source, I take test images to get the proper pixel count, then shoot all the flats in one big bunch. If I'm in the field, I still use the t-shirt, but use the twilight sky to illuminate it. This requires either thinking ahead and shooting flats before a night of imaging, or staying up all night and shooting them in the morning. It also requires watching exposure times carefully to keep the pixel values in the proper range.
You also need to shoot flat-dark exposures -- dark frames, with the scope or camera covered, that are the same exposure time as your flat-field images. As with normal dark frames, you really want to take at least 10 dark-flats and median combine them to keep noise down. Once you have all the raw flats and dark-flats shot, here's how to process everything:
1. Median-combine all of the dark-flats for a particular exposure time into a master dark.
2. Load all the raw flats, and subtract the master dark frame just as for normal images.
3. Median-combine all of the calibrated flats into a master flat. If the software you're using to median-combine the flats has the option to "normalize" the raw flat images (like MaximDL does), use it. More on this later.
4. Use this master flat in your software's "flat-field" routine. In MaximDL, you can use one dialog box to set up the dark frames for dark subtraction, the flat field images for flat calibration, and more. Set it up once, apply the calibration to all of your raw images, then save the calibrated images under new file names.
Whew. Calibration is a lot of work, isn't it? You've got to shoot all of those dark frames, flat field frames, flat-dark frames, etc. for L,R,G,B, Ha, etc., then process all of the files, then calibrate all of the images...is it worth it?
An image of M51 shot with an 8" SCT and f/6.3 focal reducer.The image on the left was made up of LRGB images that were all
uncalibrated -- there are hot pixels from no dark frames, dust donuts, and vignetting in the image. The image on the right had
all of the individual LRGB frames calibrated with darks and flats. The white point at the center of the galaxy and the black point
at the top-right of the frame are identical. Is calibration worth it? It was in this case!
In other cases, it might not be worth the time and effort if doing full calibration makes little difference to the final image. For example:
An image of the Horsehead nebula region shot through a Takahashi Sky90 refractor. The image on the left is a single
raw frame, the image in the middle is a Sigma Sum of 20 raw frames, and the final color image is on the right. In this case,
there are no dust donuts or spots in the raw images, the Sky90 doesn't vignette, and there are only some hot pixels in the raw
image that need to be dealt with. The summed image in the middle was the sum of uncalibrated frames, and is in great shape without
calibration. The final color image was made from all uncalibrated LRGB frames. In this case, it wasn't worth it!
If there's no vignetting or dust donut problems in the raw image, and you have a camera (like my StarlightXpress HX916) with very low dark noise so that all you really have to deal with in the raw images are a few hot pixels, you can easily skip the full calibration steps and still wind up with a very fine image. In this case, to deal with the hot pixels, the raw images were summed together using a "Sigma Sum" method (you can read about that in the next chapter!), which effectively eliminated the only artifacts in the image. Such uncalibrated results are much easier to get with non-vignetting optical systems, small clean refractors, and clear dark skies.
In the long run, you will almost ALWAYS get better results from calibrating your raw images with both dark and flat frames than from NOT doing so. With some optical setups, most notably SCTs with focal reducers, you have little choice but to at least do flat field calibration, as the amount of vignetting will be very objectionable without it. In the end you have to make the choice based on your knowledge of your own setup, the quality of your raw images, and the number and kinds of artifacts you have.
A couple of final notes about calibration...First, if you're using a one-shot color camera such as the StarlightXpress MX7-C or SXV-H9C, lucky you! You only have to do flats and darks for ONE set of raw images instead of one for each of the L,R,G,B filtered exposures like the poor monochrome camera guys. Remember, though, that dark subtraction and flat fielding for you one-shot guys must be done BEFORE color conversion of either the raw files or the flats and darks. Treat the one-shot-color raw files as if they were monochrome luminance files at this point. Save the raw images, do the darks and median-combine them, do the flats and dark-subtract and median combine them. Then calibrate your raw one-shot files, and re-save them. Only then can you color-convert them and begin final processing. Second, whether you're using a monochrome or one-shot camera, once you've calibrated your raw files and are ready to save the calibrated files back out to your hard drive, remember the lessons from chapter one -- DON'T TOUCH THAT HISTOGRAM! Turn off any auto-stretch functions in your software before you do calibration, and save the raw files just as they are after calibration. This is not the time to stretch the images so they look better and you can see more detail, just leave the histogram alone. We'll get to stretching later (Chapter 4).
At this point, you should have a set of images (completely raw or properly calibrated) that you will use to start building the final image. We'll cover starting to put those images together in the next chapter...
Tutorial Introduction
Chapter 1: Gathering and Preparing the Image Data
Chapter 2: Calibration -- Darks and Flats
Chapter 3: Aligning and Stacking Images
Chapter 4: Maximizing Your Data -- Histograms, Stretching, Contrast, Brightness, Gamma
Chapter 5: Dealing with Imperfections and Artifacts
Chapter 6: Basics of Color Images
Chapter 7: Advanced Color Image Processing
Chapter 8: Advanced Techniques -- Masking and other tricks
Chapter 9: Summary and Final Thoughts
All text and images Copyright (c) 2003, Paul LeFevre
Mail me with comments & criticisms!