Having calibrated the camera, we can proceed with implementing our AR application. We will make a very simple application that only shows a plain 3D box on top of the marker, using the jMonkeyEngine (JME) 3D rendering suite. JME is very feature-rich, and full-blown games are implemented using it (such as Rising World); we could extend our AR application into a real AR game with additional work. When looking over this chapter, the code needed to create a JME application is much more extensive than what we will see here, and the full code is available in the book's code repository.
To start, we need to provision JME to show the view from the camera behind the overlaid 3D graphics. We will create a texture to store the RGB image pixels, and a quad to show the texture. The quad will be rendered by an orthographic camera (without perspective), since it's a simple 2D image without depth.
The following code will create a Quad, a simple, flat, four-vertex 3D object that will hold the camera view texture and stretch it to cover the whole screen. Then, a Texture2D object will be attached to the Quad, so we can replace it with new images as they arrive. Lastly, we will create a Camera with orthographic projection and attach the textured Quad to it:
// A quad to show the background texture
Quad videoBGQuad = new Quad(1, 1, true);
mBGQuad = new Geometry("quad", videoBGQuad);
final float newWidth = (float)screenWidth / (float)screenHeight;
final float sizeFactor = 0.825f;
// Center the Quad in the middle of the screen.
mBGQuad.setLocalTranslation(-sizeFactor / 2.0f * newWidth, -sizeFactor / 2.0f, 0.f);
// Scale (stretch) the width of the Quad to cover the wide screen.
mBGQuad.setLocalScale(sizeFactor * newWidth, sizeFactor, 1);
// Create a new texture which will hold the Android camera preview frame pixels.
Material BGMat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");
mCameraTexture = new Texture2D();
BGMat.setTexture("ColorMap", mCameraTexture);
mBGQuad.setMaterial(BGMat);
// Create a custom virtual camera with orthographic projection
Camera videoBGCam = cam.clone();
videoBGCam.setParallelProjection(true);
// Create a custom viewport and attach the quad
ViewPort videoBGVP = renderManager.createMainView("VideoBGView", videoBGCam);
videoBGVP.attachScene(mBGQuad);
Next, we set up a virtual perspective Camera to show the graphic augmentation. It's important to use the calibration parameters we obtained earlier so that the virtual and real cameras align. We use the focal length parameter from the calibration to set the frustum (view trapezoid) of the new Camera object, by converting it to the field-of-view (FOV) angle in degrees:
Camera fgCam = new Camera(settings.getWidth(), settings.getHeight());
fgCam.setLocation(new Vector3f(0f, 0f, 0f));
fgCam.lookAtDirection(Vector3f.UNIT_Z.negateLocal(), Vector3f.UNIT_Y);
// intrinsic parameters
final float f = getCalibrationFocalLength();
// set up a perspective camera using the calibration parameter
final float fovy = (float)Math.toDegrees(2.0f * (float)Math.atan2(mHeightPx, 2.0f * f));
final float aspect = (float) mWidthPx / (float) mHeightPx;
fgCam.setFrustumPerspective(fovy, aspect, fgCamNear, fgCamFar);
The camera is situated at the origin, facing the -z direction, and pointing up the y-axis, to match the coordinate frame from OpenCV's pose estimation algorithms.
Finally, the running demo shows the virtual cube over the background image, covering the AR marker preciesly:
