Saturday, 26 March 2016

Gestalt - Limitations of CPA and 3D Modelling

In my two most recent postings I have showcased two novel and potentially powerful techniques for exploring the thorny subject of gestalt from digital images.  Both techniques attempt to tease apart questions about the subtle shape or structure of our subjects, but in slightly different ways.   In those postings I merely presented these concepts.  Now it's time to critique them. 

Comparative Photographic Analysis (CPA) techniques use real digital images, presented together perhaps in the form of a collage or photo montage to try and convey some of the overall gestalt characteristics of different candidate species, hopefully making it easier to choose the correct identification for our subject.  But there are potential pitfalls with these methods, some of which I will explore here.

3D modelling takes a rather different approach.  By immersing us in a three dimensional space we might perhaps explore the proportions and dimensions of our subjects from different perspectives.  Once again there are pitfalls but the analysis provides different clues to those obtainable using the CPA technique.  Hence the two techniques are somewhat complementary.

Once again I continue to work on a really challenging subject, that of the Dublin Bay Swift, found by Sture Persson on Christmas Day, 2002.

3D Modelling - The Pros and Cons
While no doubt heavily time-consuming I think the potential for 3D modelling to aid in bird identification is massive.  I have learnt an awful lot about the techniques and the potential short-comings of 3D modelling in just a few short weeks.  And no doubt I have a hell of a lot to learn.

The big advantage with 3D is of course the ability to immerse oneself inside an image and explore detail from any angle.  Working in 3D I guess one begins to appreciate what it's like to pilot a helicopter!  The task of simply trying to locate a point in 3D space requires constant rotation of the model for reference because there are three axes to consider.  Point positioning in 2D is straightforward.  In  3D it seems next to impossible!  One quickly starts to realise a digital image in reality offers a very narrow window onto our subject.   Any assumptions we think we can make about the true shape, size and relative proportions of our subject based on single images are patently false!

But of course the major flaw with the 3D models which I have generated is that they are purely artists impressions.   In order to make truly accurate 3D representations of different birds one would have to either scan subjects in 3D or painstakingly measure and reconstruct them.  For my experiments I didn't set out to prove that the Dublin Bay Swift was one species or another.  This was not a process of elimination but a process of inclusion.  I wanted to explore the possibility that different species could be ruled in based on the evidence and I think I have proved that.

Taking three of the five available images I hope I have shown in the video below that it is just as possible the Dublin Bay Swift could be a White-rumped as a Pacific, based solely on relative proportions, posture and viewing angles.  While one might imagine the rump patch to be overly large and extensive for a White-rumped Swift, the pattern could theoretically be explained entirely by posture and angle relative to the camera.  We don't even have to resort to theories about complex moult or wear to explain the pattern.


CPA, The Pro's and Cons
Having considered the extraordinary power of 3D modelling as illustrated above, surely 2D photo montage is a poor substitute.  There are some similarities between these two techniques.  For instance it was possible to find a number of images of different species taken from very similar angles which hinted at the possibility these birds could be indistinguishable when viewed from certain angles.  Note for example the left hand images of Pacific and White-rumped Swifts below.

3D modelling has allowed me to explore that problem far more closely.  It has 'rescued me' from one of the traps I had fallen into during my analysis of the Dublin Bay Swift.  In the bottom row second and third from left the Dublin Bay Swift is flying towards the camera with what appears to be forward-swept wings.  From such an angle might one risk making the assumption that foreshortening is not such a problem?  The wings appear quite short.  In fact, compared with many similar images of Pacific Swift, the wings might appear too short for Pacific.  Had I found the clue that would clinch the identification of the Dublin Bay Swift?  Sadly it was not to be.  The 3D analysis above confirms that a Pacific Swift could carry it's wings in a manner that would match the Dublin Bay images.  

The best lesson for me from this whole exercise is that foreshortening plays a role in EVERY bird image.

Notably, when one studies the biometrics of these Swifts an interesting pattern emerges.  The body length of many Apus Swifts more or less matches the wing length (measured on the closed wing).  What differs between them is mainly their body and wing proportions, tail length and shape etc, but not their relative body to wing lengths.  Another potential problem with this photo montage method is that there is a high dependency on maintaining a common scale in all these images.  The benchmark which I used was body proportionality.  One of the most puzzling things about the Dublin Bay Swift is it's rather barrel-chested, bull-headed appearance.  While few would probably doubt this bird is either A. caffer or A. pacificus, its body proportions don't seem to match either species perfectly.  This got me thinking about the circumstances of this record.  A powerful aerial insectivorous bird with a high energy intake requirement, in Northern Europe, on Christmas Day!  Surely this bird was barely surviving.  Such a bird might fluff up its body feathers and withdraw its head and wings towards it's centre to try and conserve heat.  On that basis the images chosen below for comparison may not really match the gestalt of a bird lost and struggling to survive!

Perhaps as the arguments above suggest I didn't pick the best subject to showcase two novel techniques for exploring the challenges of gestalt from bird images.  

Or then again, perhaps I did!  I have persevered with the development of tools having resisted the notion that this mystery Swift could not be identified.  Along the way I have been forced to admit defeat.  I now believe it cannot be safely identified with any certainty.  While the circumstances including prolonged southerly winds, the presence of a Common Swift (A.apus) on Guernsey at the same time and the fact the observer described 'a small swift' flying just a couple of meters above head height all tend to support the conclusion the Dublin Bay Swift was a White-rumped Swift.  On the other hand, that niggling rump pattern, the fact pacificus remains the most likely white-rumped species in NW Europe and the potential a cold, lethargic and puffed up pacificus might easily masquerade as a smaller species all creates more than sufficient doubt for acceptance as a 1st national record.  It would be a terrible shame were such a star record to remain shrouded under a permanent 'pend' categorisation.  Provided there is sufficient evidence maybe the best compromise would be to accept it onto the Irish list as a White-rumped/Pacific Swift.

As for these techniques.  The challenge to put a name to this bird has forced me to look far more deeply into the subject of gestalt than I had anticipated when I set out.  And I have developed new skills and insights along the road.  I wouldn't rule out to possibility that these techniques could be used very effectively to aid a tricky identification but at the moment I am coming away from these experiments with the niggling doubt that for every clue these tools might reveal there are probably many more false hopes.  That before one finds the path that leads to a firm identification one must travel down many blind alleys or cul-de-sacs.

Saturday, 12 March 2016

Gestalt - 3D Modelling

The software programme Blender is often recommended as the best free 3D modelling software on the Internet.  I have long held an interest in trying my hand at 3D modelling and have downloaded Blender or other programmes in the hopes of giving it a go.  However, it's quickly evident when one opens these software programmes that a steep learning curve lies directly ahead!  Attempting 3D modelling from scratch is perhaps not for the faint-hearted.  But, last weekend I was determined to finally give it a try.  After watching a few excellent tutorials by Jacob Lewis (start HERE) I finally got off the starting blocks and managed to produce my first ever 3D model which I have affectionately called "Swifty".

The Inevitable Progression to 3D
Over the last year or so the subject of 3D modelling repeatedly crept into this blog under a number of guises.  It has started to dawn on me that this is going to be a valuable and necessary tool as this blog develops.  For instance, while trying to ascertain the direction of a light source on a subject in this posting HERE I came across a really interesting experiment performed by Prof. Hany Farid and his team from Dartmouth College.  They had generated a carefully constructed 3D world and 3D model in order to test theories about lighting and shadow.  More specifically they were testing a theory that a certain image showing a gun-toting Lee Harvey Oswald (Time Magazine cover Feb 21st, 1964) was a forgery.  This opened up the possibility in my mind of using 3D modelling and lighting environments to aid in bird identification from tricky photographs.  

The scientific value of 3D modelling is very neatly highlighted by an ongoing research project being undertaken in University of Sheffield.  The "Mark My Bird" project harnesses the power of citizen science to mark up and analyse the bills of all 10,000 bird species.  For more see HERE.  

3D modelling again came up while I was exploring the question of taking accurate 3D measurements from 2D images.  I used other open source software from the Internet to test a theory I had about a possible technique to make 3D models from 2D images - with some surprising results!  For more see HERE.

It was my most recent posting on the concept of Comparative Photographic Analysis (CPA) HERE that really sparked my interest in getting to grips with 3D modelling for once and for all.  By gathering together a collection of images of candidate species and making direct comparisons with our mystery subject it might be possible to glean some clues to it's identity.  Thats the premise behind CPA.  The next logical step following on from CPA would seem to me to be the generation of a 3D model.  With a 3D representation we might be able to actually replicate our photos using 2D renders from the 3D environment.  And that is what this posting is all about.

Meet 'Swifty' - my first 3D model
For my first effort I'm quite proud of 'Swifty', my first 3D model.  Through some structural modifications I can turn this model into a White-rumped Swift (Apus caffer), Pacific Swift (Apus pacificus) or any other Apus Swift.  Swifts make a pretty good subject for my first 3D effort, being fairly uniform in body structure, with fairly stiff wings.  This has allowed me to keep the number of 'moving parts' to a minimum.  While this is a fairly rudimentary design I don't need too much detail for the purposes of the experiment I am undertaking here.

Rough Versus Accurate 3D Modelling
Making 3D models from scratch is no different to any other form of art and design work.  Completed work may serve any number of functions.  A critically and painstakingly accurate, photo realistic 3D model might allow for the taking of very accurate measurements.  But the time and energy involved in making such a model might not be worth all the effort involved.  A model with that level of complexity would be more easily created using 3D scanning of real specimens rather than building a model from scratch.  However 3D modelling has uses at a more simple level.  We may be happy in some cases to work with a reasonably accurate representation like 'Swifty'.  If for example our goal is not to carefully measure the distance between two points but to test theories about structure and perspective, our simpler models might do the trick.  At least, that is my aspiration here.

The Dublin Bay Swift - another perspective
The purpose of my posting on a concept I call CPA (HERE) was to demonstrate how direct comparison between images can help generate clues about a bird's shape, or perhaps it's plumage field marks, to hopefully unlock a tricky identification.  In this posting I am approaching the problem from a different perspective - or rather, from infinite perspectives!  Here I have attempted to reproduce a number of Sture Persson's intriguing photographs by rendering 2D images of 'Swifty'.  The purpose is to try and understand what might be going on in each of the images.

Image 1
Above, once again I have presented a collage of the four key images photographed by Sture Persson.  Taking the left hand image first, one might initially assume this provides the best clues to the identification of this bird.  Afterall, it's a nice profile image.  However, as the CPA posting highlighted, this bird's wings are foreshortened in this image.  We really can't get any clues as to the length of this bird's wings from this image so I am not going to do any more with this image for now.

Image 2
The second image from the left is the sharpest.  What I find most intriguing about this (and the third image) is that the wings appear to be held in the forward-swept position.  For me this is key to trying to assess the length of this bird's wings relative to it's body.  In the examples below I have placed Swifty's wings in the forward position and tilted the bird slightly to match image 2.

Image 4
I have found this image particularly difficult to interpret.  Clearly the bird is flying at an angle somewhat towards the camera, but it's proportions appear odd from this perspective.  The key question is how much of the rump are we seeing in this image?  If we are mainly seeing the side of the rump then this would be inconsistent with A.caffer and more in line with A.pacificus.  However, just because a bird appears to be flying above head height does not rule out the possibility that the rump could be tilted down, towards the observer.  Perspective can play strange tricks on the brain.

Here is a short video demonstrating how Image 4 looks in this 3D model.  For more examples and a critique of 3D modelling and a related technique, CPA see HERE.