Möbius Transformations on Spherical Photos and Videos
Spherical Video presents some interesting challenges, for instance, how do you zoom? Or how do you rotate on any arbitrary axis? As the mathematician Henry Segerman pointed out in a post for EleVR, you can achieve both of the above using Möbius transformations. The transformations are conformal (they preserve angles), and they map circles to circles (considering a line to be a circle of infinite radius). Professor Segerman’s work is the inspiration for the project I am describing here.
I wrote an implementation using Möbius transformations to manipulate spherical images in WebGL/Three.js. The implementation is here. Click on the ‘?’ for help or watch this video.
So what is going on here?
The mathematical procedure:
Spherical cameras such as the Ricoh Theta save images in an equirectangular format. This is then wrapped around a sphere to give the spherical effect. (With three.js you create a material whose texture is the saved image from the Theta and then create a mesh using this material and a sphere geometry.) The X and Y of the equirectangular format map to the longitude and latitude of the sphere.
The Riemann sphere is a representation of the complex plane as a sphere using reverse stereographic projection. Möbius transformations are transformations of the complex plane. Our three.js sphere is a normal sphere in 3-d cartesian space (R3); each point on its surface has an (x,y,z) coordinate where x, y and z are real numbers. We can convert our x,y,z point to a point in the complex plane if we take our sphere to be a Riemann sphere. Or in jargon we can say equivalently: P¹(C) is diffeomorphic to the sphere S². The formulas are here (and elsewhere).
So for instance the south pole is (0,0,-1) in cartesian/R3 space and (0,0i) in complex space. The north pole is (0,0,1) in cartesian space/R3 and (∞, ∞i) in complex space. (The Reimann sphere is really the complex plane plus one point which is the point at the north pole, but I digress).
The Riemann Sphere
CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=1357613
The technical procedure:
To implement our transformations, we are going to move pixels around on the texture. We can only do this in the shader since doing it in javascript would be way to slow. Specifically we are going to do it in the fragment shader. Our vertex shader will be short and boring, doing only what is necessary to build the sphere’s vertices. Nothing special there. The fragment shader is where the work is done.
The fragment shader is called when the graphics layer needs to know what color to put at a given UV coordinate. We will pluck that pixel from our transformed texture. The steps to execute on each call to the fragment shader are:
1 – Get the UV coordinates. ‘UV’ is industry nomenclature for the X and Y coordinates of the texture that the graphics layer is trying to draw.
2 – We know that the rectangular texture must map to a sphere. The top of the rectangle maps to the north pole, the bottom to the south pole. So we can calculate the X, Y, and Z coordinate of the sphere corresponding to the UV coordinate. That is, we know which point of the sphere we are drawing.
3 – This sphere is also a Riemann sphere as mentioned above, so we calculate the point (a + bi) on the complex plane corresponding to the cartesian point (X,Y,Z) on the sphere, using the formulas mentioned above.
4 – We now know our location on the complex plane and we apply as many transformations as we’d like. When we are done we have our new complex point (c + di).
5 – We reverse the above steps and calculate the cartesian X,Y,Z corresponding to (c + di). Then we calculate the UV corresponding to this X,Y,Z and we use the color at that pixel, UV.
The only Möbius transformations I’ve implemented for now are those to do rotation and zoom. By default the fixed points are antipodal though you can explicitly set the fixed points using the Epsilon 1 and Epsilon 2 buttons. The video linked to above gives a good overview of what all the different buttons do.
This is a rotation around 2 non-antipodal fixed points:
In addition to Möbius transforms to do rotation and zoom, there are some other complex transformation options.
You may have noticed that we start with the point on the complex plane (a + bi) for which we need a pixel. We then transform (a + bi) into (c + di) to get the pixel we will draw. Thus the transformation is really a reverse transformation.
Infinities of volleyball players on North Avenue Beach (Works at 60FPS on video!):
This article by Henry Segerman gives a comprehensive overview of the math behind these transformations and includes more examples of other transformations you can do.
