Computational photography: what ‘format’ is it?

Gratuitous header; moment of enlightenment.

One of the unavoidable buzzwords of the last couple of years has been ‘computational photography’. Besides sounding slightly oxymoronic and insulting to the ‘real’ photographer who presumably represents what they see and doesn’t attempt to manipulate objects into (or out of) being that aren’t physically there, the reality is that it’s unavoidable and has been unavoidable since the start of the digital era. Everything that requires photons to be converted into electrical signals and back to photons again (whether off a display or reflected off a print) – must be mathematically interpreted and altered in some form before output. It is not possible to avoid this: the Bayer interpolation, in-camera JPEG conversions, any file format saving, conversion to print color space – a ‘computation’ has to be performed to translate the data. Hell, there’s already an implicit computation in the analog to digital stage (although arguably photons are already ‘digital’ since they represent discrete quanta of energy, but that’s another discussion for another time). However, what I’d like to discuss today* is something one step further down that road, and following on from the previous posts on format illusions: in light of the broader possibilities of computational photography, what does ‘format’ even mean?

*I.e. excluding things like subject recognition for tracking, depth mapping and simulated shallow DOF transitions etc. for the time being; we’ll revisit that later.

Beyond the obvious erasure of memory cards, the concept of format refers to the size (and relative size) of the recording medium. Different sizes of recording medium have different tradeoffs, strengths and weaknesses; but generally, the more the merrier. The reason this becomes largely academic (and possibly highly confusing from a characteristic property standpoint) is that computational photography allows you to effectively change the size of the recording medium on the fly, without necessarily any accompanying physical changes. And precisely therein lies the first aspect of confusion: optical properties are hardwired to physical ones, meaning that no matter how much you pano-stitch on your iPhone, the lens still retains the physical and optical properties of a 4.25mm f1.8 (or whatever it happens to be) – even if the angle of view is different, and the dynamic range/ color/ tonal expectations don’t equate to a 1/2.3″ sensor. Though the particular example of panoramic stitching changes the effective overall angle of view but not the local properties of the recording medium in each small portion of the image, there are other examples like HDR stacking or super resolution stacking that change the effective photosite properties but not the angle of view. And we haven’t even talked about depth mapping yet, which is a bit more complex and we’ll leave for later.

Bottom line: the ‘traditional’ expectations of this sensor size and this angle of view equals this kind of look (or maximum range of looks due to the latitude of the medium) is simply no longer true. Tonally, at any rate, there has been improvement enough that the capabilities of most sensors in most situations outweigh what you actually need to represent both the scene, and more importantly, the photographer’s idea of how the scene should look. There’s more resolution than most output media. This means that we have headroom to spare, and sufficient flexibility to manipulate the tonal map of output to whatever we wish. In short: the imagination now has to catch up with the technological possibilities afforded by capably-used equipment.

Important diversion: resolution is NOT independent of tonality. More resolution doesn’t mean just more ability to count hairs, but the ability to create fine tonal gradations even across areas of subject that don’t necessarily have any high frequency detail. The stacking of images shot from slightly different points of view to create a super resolution image does just this – providing of course your subject doesn’t move between captures. But make the capture speed fast enough, and you effectively have a static situation most of the time – even if the camera is handheld. You just need enough difference to provide at least a sub-pixel shift between captures, and this is what most of the pixel shift and super resolution cameraphone modes do. Something similar (or even combined) can be achieved tonally by varying exposure between captures. In an ideal world, you’d be able to do both: shoot a lot of pictures very quickly at different exposures and then blend the whole in a natural way to create an image that’s both high resolution and with input dynamic range precisely matching the scene.

That takes care of tonality, dynamic range and resolution. Even the preserve of larger formats – focal plane transitions – is eroding. This is where the depth mapping comes in: with cameras in two locations it’s possible to calculate the distance of each subject element in the scene that’s visible to both cameras; it’s a matter of trigonometry to figure out the physical distance to subject if you already know the focal length and physical separation between the cameras (which presumably does not change). The greater the angular difference, the better this depth map will be – either by bringing the subject closer, increasing focal length, increasing camera separation – or all three. Or you could go a separate route and split the beam further down the road to record two separate images on sensor (Lytro’s approach). There are degrees to this implementation, of course – most of us are familiar with face detection and faux-bokeh modes on smartphones, but they’re limited by both physical size (camera separation and focal length) and the amount of time consumers are willing to wait for computation and rendering. The most flexible kind of depth map would have to record the whole scene in focus and with sufficient resolution and dynamic range from two fairly different vantage points – from here we could change composition within a fairly wide latitude after the fact. From here, we are only one step away from having ray tracing ability and the ability to change light after the fact. Oddly enough, there doesn’t yet seem to be that much overlap between VR content generation and photography – this may be due to practical hardware limitations, or perhaps it’s just a step too far for most hobbyists; after all, we already see a sharp decline in interest as effort increases even in conventional photography.

I wouldn’t leave this property up to the viewer, though: firstly, the amount of data required would be colossal, and secondly, we would lose the most important part of photography: the creative interpretation of the photographer. Without this conscious exclusion of elements in the scene that they do not find interesting or relevant to their vision, then we might as well just be creating a VR tour. I’m also not sure that this much choice is a good thing for photographers: it means you are not forced to make a decision at the time of capture; later reflection on the scene may yield very different feelings or no feelings at all, and the transience of that moment becomes lost. There’s also the distinct possibility of paralysis by choice: it’s already bad enough most of the time with a zoom lens.

I suspect the reason we haven’t really seen any of this take off yet isn’t a technological one, or a cost-related one. It’s more to do with imagination: it already requires a lot of practice and training to effectively visualise how three dimensions reduces to two for the scene in front of you; it’s a step further to then apply a different perspective to the ones naturally afforded by your eyes; then compound this by attempting to imagine the scene from a different vantage point from the one you are physically at. And we still need to add post processing to that, which if done properly, isn’t a ‘try the filters at random’ approach but has repercussions on how you need to shoot to capture the exposure data you need later for a smooth tonal manipulation. All of that is already difficult, and we haven’t even left traditional single-capture photography yet. The concept of variable depth of field already didn’t take off – even with the post-focus correction ability that’s come with it, despite the latter being genuinely useful for fast situations where your camera might have missed focus slightly and that minor tweak can make the difference between a near miss and ‘nailed it’.

In essence, a transition to fully computational photography is one step away from asking people to become animated film directors – you control camera angle, light, field of view, depth of field etc. after capturing the scene data. It shifts the creative process from being in a place, seeing an idea and capturing it to capturing data and then trying to extract an idea afterwards; these are very different cognitive processes. On top of that, it complicates the process too much: we go even further down the road of being button pushers rather than being simply seers. I know even the best of us have a fixed latitude of concentration: there’s only so much awareness of the scene you can have if you’ve also got complex machinery to operate. In a way, that’s perhaps why some of the modern cameraphones really deserve recognition: not because of what they can do, but what we’re not aware of that they are doing in the background: I know most of the time the result looks as I’d expect at the output sizes I’d use something like that for, and perhaps a bit beyond (don’t look too closely, it’s ugly at the pixel level; perhaps almost deliberately so to trick the eyes into believing there’s more continuity of texture than was actually captured) – but I don’t know how it’s doing it for any given frame. You only notice it when the result isn’t as you expect, and even then, it happens less and less. Personally, I like this vision of computational photography because of the freedom it affords to focus on the picture, not the process. MT

__________________

Visit the Teaching Store to up your photographic game – including workshop videos, and the individual Email School of Photography. You can also support the site by purchasing from B&H and Amazon – thanks!

We are also on Facebook and there is a curated reader Flickr pool.

Images and content copyright Ming Thein | mingthein.com 2012 onwards unless otherwise stated. All rights reserved