Last issue, we talked about image sensors and the role they play in creating digital images. This issue, we’ll be discussing the next block in the imaging system which immediately follows the image sensor and is arguably the most important part of the overall camera system. That very important part called the image signal processor (ISP). An ISP can be thought of as the brains of the imaging system in a camera as it’s in charge of controlling many of the individual parts of a camera that work together in order to produce a usable image.
Within this ISP, there are hundreds of embedded algorithms and features, but the main three that have the largest effect on image quality are known as the three A’s: auto focus (AF), auto exposure (AE), and auto white balance (AWB).
Auto Focus
Depending on the camera type and auto focus system employed, the details of the specific algorithm will be different, but the overall idea is the same. Without getting into specifics about optics, many of you are probably aware that a camera cannot keep all depths of a scene in focus at the same time, especially when taking a close-up photo of an object. In order to remedy this problem, auto focus was created to change the focus point of an image by moving the lens element closer or further away from the sensor. This seems pretty easy until you remember that the ISP needs to guess what aspect of the scene you want to focus on, figure out how it moves the lens to properly focus on that area, and how to make this process as smooth and transparent as possible. In common imaging systems, the ISP may complete the following to find proper focus: The ISP will send commands to the lens actuator to shift the lens in uniform steps as it measures the contrast of a scene. After measuring the contrast, it will find the maximum value of this contrast and set the lens to that position to achieve focus lock.
Auto Exposure
This function of the ISP also varies based on the type of camera used, but the general principle applied in order to control exposure is constant across all camera platforms. Auto exposure is basically the ISP changing settings within the camera in order to control the amount of light which will eventually be collected by the sensor. Some main ways of accomplishing this is to adjust the aperture (size of the opening which allows the light through) of the lens, changing the shutter speed or changing the gain applied to the signal (also referred to as ISO). Changing various components of the camera have various effects on the resultant image and therefore, the ISP must first determine the proper exposure to set and then decide the trade-offs required when selecting the camera settings to use. The ISP is designed to detect the current shooting conditions and make a smart trade-off based on the settings.
Auto White Balance
The spectral power distribution of light is very different depending on the source from which it is produced, causing the colour of the light to vary. You may notice that light from candles or incandescent bulbs produce a “warm” or yellow light while fluorescent bulbs and daylight are more “cool” or blue. This colour is measured as a correlated colour temperature (CCT) which is based on the light emitted by a black body at a certain temperature and is therefore measured as a colour temperature with the unit of Kelvin. This little bit of background knowledge is necessary to understand why AWB is necessary in a camera system and why it is an integral part of the overall image quality. The ISP needs to establish the CCT of the light illuminating the specific scene and then apply an adjustment in order for it to look as though our eyes see it. Our eyes are capable of making this adjustment on their own, which is why we normally don’t notice how orange or blue light is in many cases. An incorrect white balance setting can typically be seen when a resultant image is very blue or very orange in hue.
This is a very high level overview of some of the main aspects of the ISP which occur prior to taking an image. Once the image has been captured, the image will travel through what is normally called the “ISP pipeline.” The image will first be converted from the Bayer pattern discussed in our last article to an image that resembles something we would expect out of a camera. After that, the exact steps are variable but the most common steps deal with applying dead pixel correction, noise reduction, saturation, contrast, and any other adjustments that a manufacturer would like to include in their pipeline.
The biggest thing that people don’t realize when thinking about a camera is that every sensor and every ISP is slightly different and therefore must be “tuned” in order for them to work together. The development process on tuning an ISP and sensor combination is very involved and requires a great deal of decision-making and troubleshooting in order to give the customer the best image possible. The biggest problem with this is that everyone has a different opinion on aspects of image quality: While some may like saturated and vibrant colours that are not very close to real life, others would prefer to see the same colour as real life despite it not being as vibrant and saturated.
These decisions require trade-offs to be made which, in turn, shapes the look and feel of a specific camera. This is the main reason why images from various cameras can look so different from each other despite having very similar hardware specifications. A quick tip when buying a camera is to find some sample images online before buying in order to see if you like the tuning set for that camera.
The last thing we’ll touch on in regards to the ISP is the wide range of extra capabilities that are being included in modern ISPs in order to make processing more accurate, faster and of better quality. These features include video image stabilization which use other components like a gyro in order to stabilize video (can be seen in the iPhone 4S), high dynamic range photos, the ability to detect a scene and then apply trade-offs specific to that setting, as well as many more that are constantly being added.
The image signal processor is the brain of any imaging operation and is an essential part in telling the rest of the camera system what to do, as well as providing a final image that is of higher image quality then what would come out of a raw sensor. I hope this taught you a little about the intricacies of an imaging system and I would encourage you to read more about any specific topic in this article as we have just given you a very, very high-level explanation.
Leave a Reply