Roger Cicala: why I don’t use an MTF bench to test my own lenses

I have a complete testing lab at my disposal: MTF benches, lens test projectors, spectrometers, lasers, an Imatest setup gathering dust in a back room; everything all the cool kids have. A lot of people assume I test the hell out of my own shiny new personal lenses after I buy them. (Yes, I buy my own stuff). I do test them, but not in the lab. I go out and take pictures with them.

It’s not because I’m such a great photographer that my practiced eye can tell more about the lens through photographs than any lab test could. I’m a mediocre photographer. Years ago I tried making a living as a photographer. I sold some prints once, made enough to pay for maybe half a lens, and after another six months without a sale I decided to explore other methods of supporting my extravagant lifestyle.

The lab is faster, gives tons of information, and makes cool graphs. But I still don’t use it to test my personal lenses

It’s not because the lab stuff doesn’t give useful information. The lab gives a LOT of useful information. Most people don’t have time to learn how to interpret it, or learn its value and limitations, but it’s useful information nonetheless. And the lab is fast; I can test a lens about 32 different ways in a couple of hours. My ‘test a lens with photography’ time is a half a day or more. So the lab is faster, gives tons of information, and makes cool graphs. But I still don’t use it to test my personal lenses.

Lab tests give a ton of precise information. Understanding and interpreting it is, I’ll admit, not completely intuitive.

That’s because all lab tests have some major limitations. The biggest one is this: real images are 3-dimensional, they are focused at a variety of distances, and almost always contain foregrounds and backgrounds. Optical tests are two-dimensional slices taken at a fixed focusing distance with no background or foreground. The focusing distance is infinity for an optical bench. It’s a single, close distance for Imatest / DxO / and other computer image analysis methods.

So, the lab tests tell me everything I want to know about the plane of exact best focus at one focusing distance. That’s really useful information, especially if you want to find out if a lens is optically maladjusted, want to know what kind of aberrations it has, or are interested in its maximum resolution. And it gives people numbers – the ammunition of choice in many a Forum War.

Even a three-dimensional standard comparison image, such as the kind that DPReview and other sites use, is basically limited to one focusing distance. That distance is different for different focal lengths but it’s always fairly close up. And, if it’s an indoor target, the depth of those targets is usually only a few feet at most; it’s not going to show you what the out of focus area 30 feet behind the image plane looks like.

What I actually do to test a new lens

Photographs give me far more information than the lab, even if it’s less exact. I don’t recommend brick wall or side-of-building photographs. Those are just 2-dimensional slices like the lab gives, but with more variables and less information. I want photographs of 3-dimensional subjects.

With the right background (I prefer a field or yard of grass) you can quickly compare resolution at a half-dozen focusing distances. Sure, some lenses are about the same at all distances, but many are not. No zoom lens is equally sharp at all focal lengths. My favorite grass field is a hill behind my office that slopes up away from me. I focus on the mower tracks and quickly get images at several focusing distances.

Simple grass slope image taken with a Canon 50mm F1.2 lens at F1.4. Grass (or pebbles or concrete or all manner of things that make fairly uniform photographs filled with fine detail) are great for figuring out the zone of acceptable sharpness (for you) of a lens.

Repeating this set of images at several apertures lets me see at what aperture maximum center, middle, and edge sharpness occur (those are almost always different). It’s good to know things like there’s maximum center sharpness at F4 and the edges are at maximal sharpness at F6.3 or F8 or that they never get very sharp.

Grass is also great because it gives you a nice sharpness comparison as you leave the area of best focus. I also recommend looking at what you consider the depth of field at each aperture and focusing distance. Depth of field is not an area of maximal sharpness. It is an area of acceptable sharpness; there is greater and lesser sharpness within the depth of field. Your definitions of ‘acceptable sharpness’ in your images may be greater, or less, than the calculated depth of field.

You rarely see dramatic changes in a prime lens’ field curvature at different focusing distances, but you will usually see a dramatic change in a zoom’s field curvature at different focal lengths

More importantly, some lenses fall off of the sharpness cliff as they exit their area of maximal sharpness, others drift so slowly down the gentle sharpness slope that it really does seem as if the entire depth of field is maximally sharp. Also, that sharpness slope often changes at different apertures. Those are all good things to know.

The other thing I do is to take some of my grass images and run them through a Photoshop ‘Find Edges’ filter or equivalent. This will let you visualize the field curvature of your lens and see how it varies at different focal lengths or focusing distances. (Pro tip: you rarely see dramatic change in a prime lens’ field curvature at different focusing distances. You will, however, usually see a dramatic change in a zoom’s field curvature at different focal lengths.) That’s really useful information that few people know about their lenses. The find edges type filters are also a good way to look at depth of field at various apertures or with different lenses.

	Same image as above (Canon 50mm F1.2) run through a find edges filter – the field curvature is obvious.
	Field curvature of Canon 50mm F1.2 as measured on an optical bench. You get about the same information from the grass photo and find edges filter as you would from the $250,000 optical bench.

Grass shots also give you a superb way to see if your lens is softer in one area or if the field is tilted. The grass image above is very slightly tilted, an amount that’s about normal for a good prime lens. A more dramatic field curvature might look as though you’d rotated the dark area 15 or 20 degrees in Photoshop.

About half the people who take building or brick wall images and think their lens is ‘decentered’ actually have a lens with a field tilt; the lens is equally sharp on both sides, but not at the same distance as center focus. It’s actually very hard to detect a field tilt by shooting a chart and evaluating a two-dimensional image.

A large field tilt in a prime lens is unusual while a field tilt at some focal lengths of a zoom is pretty common. (I’ve seen 45 degree field tilts in zooms, but 10 degrees or so is routine.) If you return your zoom lens to the store for exchange, the replacement will probably have a different field tilt at another focal length.

People like to talk about a lens’ bokeh like it’s one thing, but bokeh often varies

If the lens is one for which I consider bokeh important, I use the a Bokelizer. Basically, this is a couple of strings of tiny Christmas lights hung in a three-dimensional pattern. I take some images at various focusing distances and evaluate the foreground and background in-focus highlights, as well as the in-focus lights. People like to talk about a lens’ bokeh like it’s one thing, but bokeh often varies in the foreground vs the background, at different focusing distances, and depending on how far off-center the object is for many lenses.

Why do I look at in-focus lights, since they have nothing to do with out-of-focus highlights? Because comparing pinpoint light sources is a superb way to see if the lens is optically maladjusted. ‘Optically maladjusted’ means a lens that has a decentered, tilted or poorly spaced element. On the forums, people often refer to all of these issues as ‘decentering’ but that’s less than correct.

Illustrations of the various types of optical maladjustments. In reality, a given lens usually has several small errors, rather than one single large one.

Each of those optical maladjustments causes different optical problems and often they’re apparent when looking at pinpoint light sources. Looking at pinpoint light sources also gives you an idea of the coma and other aberrations that the lens displays by design.

This image was created from equipment in the repair department that basically just projects pinhole lights. You can easily see the difference between a good lens (upper half) and one that is slightly decentered (bottom half).

Once I’m done with the stuff above, I go out and take the kinds of pictures that I bought the lens for. But the hour or two needed for the checks above gave me a lot of information about how to best use the lens’ strengths and weaknesses before I set off to shoot. It also shows me if the lens is optically maladjusted, and there’s no sense taking a bunch of photographs if I already know I’m going to return the lens.

Will taking pictures tell me if I got a copy that’s every bit as sharp as the copy Reviewer Guy got? Absolutely not. Does it let me spout numbers in ‘my lens is better than your lens’ Forum Wars? Again, no. But it certainly does tell me if the lens meets my expectations and will do the job I want it to do. Lab tests give me all manner of information, but they can’t tell me whether I’m going to like the images from the lens.

It doesn’t matter to me at all if I have the sharpest copy of a lens or not. I just want to know if it’s acceptable for the purposes I want to use it for

To be completely honest, if I think the lens isn’t as sharp as I expect, then I may actually take it to the lab and measure it on the bench. I’ve done that maybe twice in the last ten years out of a few dozen lenses I’ve purchased, and both times it turned out that the lens wasn’t up to spec. So, really, I knew the answer without using the bench.

Photographic testing won’t tell you if your lens is among the sharpest copies of that lens, or if it’s in the top half of the variation range or things like that. If you want to know that, then really you need to pay someone to test the lens on a test bench. Why don’t I do that? Because it doesn’t matter to me at all if I have the sharpest copy or not. I just want to know if it’s acceptable to me for my purposes.

Roger Cicala is the founder of Lensrentals.com. He started by writing about the history of photography a decade ago, but now mostly writes about the testing, construction and repair of lenses and cameras. He follows Josh Billings’ philosophy: “It’s better to know nothing than to know what ain’t so.”