Integrating OpenCV into the Android Studio

In this section, we will show you the steps to integrate OpenCV in the Android Studio with the Native Development Kit and use the OpenCV stitching module in C++ to create the final panorama image. We will also do some computations with OpenCV Android SDK Java to show how the interaction goes about between Java and C++ interfaces.

Compiling OpenCV Android SDK to the Android Studio project

Officially, the OpenCV Android SDK is an Eclipse project, which means we can't simply use it in our Android Studio project. We need to convert the OpenCV Android SDK to an Android Studio project and import it as a module to our application.

Note

We assume that you have downloaded the latest OpenCV for Android from http://opencv.org/downloads.html. At the time of writing, we now have OpenCV for Android 3.0.0.

Let's extract the downloaded file to your favorite path, for example, /Volumes/Data/OpenCV/OpenCV-android-sdk.

Then, we need to open a new Android Studio window and select Import project (Eclipse ADT, Gradle, and so on). In the popup window, you should select the java folder at OpenCV-android-sdk/sdk/java and click on OK.

Compiling OpenCV Android SDK to the Android Studio project

Import project visualization

In the next window, we will choose a path to store the new OpenCV SDK project. In our case, we choose /Volumes/Data/OpenCV/opencv-java and click on Next.

Compiling OpenCV Android SDK to the Android Studio project

Select import destination visualization

In the last window, we will simply click on Finish and wait until Android Studio completes the Gradle build process. Basically, Gradle is the default build system of Android Studio. At this step, we want to make sure that the OpenCV SDK can be compiled successfully. One of the common errors is that you haven't downloaded the required Android SDK. The error message is very straightforward. You can follow the message to solve the problem. In our case, there is no problem as in the following screenshot.

Compiling OpenCV Android SDK to the Android Studio project

Build competition visualization

At this time, we can close this project and open our Panorama project.

Setting up the Android Studio to work with OpenCV

In order to use OpenCV in our project, we need to import the OpenCV Android SDK to our project. With this SDK, we can use the OpenCV Java API and perform image processing tasks easily. Moreover, we must make a further step to tell Android Studio to compile OpenCV C++ code to use OpenCV in the Native Development Kit (NDK). We will split this section into three subsections: Importing the Android SDK, Creating a Java-C++ interaction, and Compiling OpenCV C++.

Importing the OpenCV Android SDK

We assume that you have opened the Panorama project. We need to import the converted OpenCV Android SDK in the previous section as follows:

File | New | Import Module

In the New Module window, we will select the source directory to the converted project. In our case, we will choose /Volumes/Data/OpenCV/opencv-java. Then, we'll check the import checkbox, change the module name to :opencv-java, as shown in the following screenshot and click Finish:

Importing the OpenCV Android SDK

A new module window

Next, we need to modify build.gradle in the app folder to add one line to the dependencies section:

dependencies {
    compile fileTree(dir: 'libs', include: ['*.jar'])
    compile 'com.android.support:appcompat-v7:22.1.1'

    compile project(":opencv-java")
}

Finally, we must sync the project with the button Sync Project with Gradle Files.

Note

If you only need the OpenCV Java Interface and don't want to use OpenCV C++, you must copy the libs folder at OpenCV-android-sdk/sdk/native/libs to your app/src/main folder. Then, you must add the following loadLibrary code to your class file:

static {
    //If you use OpenCV 2.*, use "opencv_java"
    System.loadLibrary("opencv_java3");
}

Creating a Java and C++ interaction with Java Native Interface (JNI)

Before we start the compile process, we will create a class file named NativePanorama.java and add a method to the NativePanorama class:

public class NativePanorama {
    public native static void processPanorama(long[] imageAddressArray, long outputAddress);
}

The processPanorama method will receive an array of long addresses of each image and a long address of an output image.

You must rebuild the project in order to follow the ensuing steps. The detailed explanation is in the next paragraph:

  • Use the javah command line to create a C++ header
  • Create a .cpp file for the newly created header in the jni folder to implement the function in C++

You may notice the keyword native before the processPanorama method. This means that we will use this method to interact between Java and C++ in our application. Therefore, we need to create some headers and source files to implement our C++ code. We must follow Java Native Interface (JNI) to use C++ code, so the process may be a bit complex and out of the scope of this book.

In the following parts, we will show you the steps to use OpenCV C++.

Note

If you want to understand JNI, you may want to take a look at the JNI documentation found at:

http://docs.oracle.com/javase/7/docs/technotes/guides/jni/

Also, have a look at the JNI tips from API guides found at:

http://developer.android.com/training/articles/perf-jni.html

First, we will use the javah command in the terminal to create the corresponding C++ header for the processPanorama method. In order to do this, you need to open the terminal on your machine, then change the directory to the folder app/src/main in your Android application and run the following command:

javah -d jni -classpath ../../build/intermediates/classes/debug/ com.example.panorama.NativePanorama

You only need to verify the package name and the name of the class file, NativePanorama. The command will not display anything on the terminal, as shown in the following figure. You may want to rebuild the project if you encounter the following error: Error: Could not find class file for 'com.example.panorama.NativePanorama'.

Creating a Java and C++ interaction with Java Native Interface (JNI)

The terminal after using the javah command

As the result of the javah command, we now have a folder named jni in our app/src/main folder with a file com_example_panorama_NativePanorama.h. This header contains a function to work with Java Interface. When processPanorama is called, this function will run in C++.

Next, we will create a source file named com_example_panorama_NativePanorama.cpp in the jni folder. We recommend that you should copy the function declaration from the header file to the source file and add the parameter names as follows:

#include "com_example_panorama_NativePanorama.h"
JNIEXPORT void JNICALL Java_com_example_panorama_NativePanorama_processPanorama
  (JNIEnv * env, jclass clazz, jlongArray imageAddressArray, jlong outputAddress){
}

The only thing left is that we need to compile OpenCV C++ SDK to use it in the preceding source file.

Compiling OpenCV C++ with NDK/JNI

In order to use OpenCV in C++ code, we need to compile OpenCV C++ and use an Android.mk file as the make file to build and link our C++ file with OpenCV library. However, Android Studio doesn't support Android.mk out of the box. We need to do lots of things to make this happen.

First, we will open the local.properties file and set ndk.dir to be your path to the Android NDK folder. In our case, the local.properties will look like this:

sdk.dir=/Users/quanhua92/Library/Android/sdk
ndk.dir=/Users/quanhua92/Software/android-ndk-r10e

Note

You can get the Android NDK at: https://developer.android.com/ndk/index.html

Secondly, we open the build.gradle file in our app folder and add this line at the top:

import org.apache.tools.ant.taskdefs.condition.Os

Then, we need to add the following code between the defaultConfig tag and buildType tag to create a new Gradle task to build C++ code.

// begin NDK OPENCV
sourceSets.main {
    jni.srcDirs = [] //disable automatic ndk-build call
}
task ndkBuild(type: Exec, description: 'Compile JNI source via NDK') {
    def rootDir = project.rootDir
    def localProperties = new File(rootDir, "local.properties")
    Properties properties = new Properties()
    localProperties.withInputStream { instr ->
        properties.load(instr)
    }
    def ndkDir = properties.getProperty('ndk.dir')
    if (Os.isFamily(Os.FAMILY_WINDOWS)) {
        commandLine "$ndkDir\ndk-build.cmd",
                'NDK_PROJECT_PATH=build/intermediates/ndk',
                'NDK_LIBS_OUT=src/main/jniLibs',
                'APP_BUILD_SCRIPT=src/main/jni/Android.mk',
                'NDK_APPLICATION_MK=src/main/jni/Application.mk'
    } else {
        commandLine "$ndkDir/ndk-build",
                'NDK_PROJECT_PATH=build/intermediates/ndk',
                'NDK_LIBS_OUT=src/main/jniLibs',
                'APP_BUILD_SCRIPT=src/main/jni/Android.mk',
                'NDK_APPLICATION_MK=src/main/jni/Application.mk'
    }
}
tasks.withType(JavaCompile) {
    compileTask -> compileTask.dependsOn ndkBuild
}
//end

You may want to look at the following figure for a screenshot of our build.gradle.

Compiling OpenCV C++ with NDK/JNI

A screenshot of our build.gradle

Next, we create a file named Application.mk in the jni folder and put the following lines in it:

APP_STL := gnustl_static
APP_CPPFLAGS := -frtti -fexceptions
APP_ABI := all
APP_PLATFORM := android-16

Finally, we create a file named Android.mk in the jni folder and set up this file as below to use OpenCV in our C++ code. You may need to change the OPENCVROOT variable to the location of OpenCV-android-sdk in your machine:

LOCAL_PATH := $(call my-dir)

include $(CLEAR_VARS)
#opencv
OPENCVROOT:= /Volumes/Data/OpenCV/OpenCV-android-sdk
OPENCV_CAMERA_MODULES:=on
OPENCV_INSTALL_MODULES:=on
OPENCV_LIB_TYPE:=SHARED
include ${OPENCVROOT}/sdk/native/jni/OpenCV.mk

LOCAL_SRC_FILES := com_example_panorama_NativePanorama.cpp
LOCAL_LDLIBS += -llog
LOCAL_MODULE := MyLib

include $(BUILD_SHARED_LIBRARY)

With the preceding Android.mk, Android Studio will build OpenCV into libopencv_java3.so and build our C++ code into libMyLib.so in the folder app/src/main/jniLibs. We have to open our MainActivity.java and load this library to use in our application as follows:

public class MainActivity extends ActionBarActivity {
    static{
        System.loadLibrary("opencv_java3");
        System.loadLibrary("MyLib");
    }

Note

If you use OpenCV Android SDK Version 2.*, you should load opencv_java instead of opencv_java3.

Implementing the OpenCV Java code

In this section, we will show you OpenCV in Java side to prepare the data for the stitching module in the OpenCV C++ side.

First, we will create a list to store all of the captured images when the user presses the Capture button:

private List<Mat> listImage = new ArrayList<>();

Then, in the onPictureTaken method of the jpegCallback variable, we want to convert the captured Bitmap into an OpenCV Mat and store in this listImage list. You need to add these lines before the drawing parts with Canvas:

Mat mat = new Mat();
Utils.bitmapToMat(bitmap, mat);
listImage.add(mat);

Finally, when the user clicks the Save button, we would want to send the address of the images in listImage to the OpenCV C++ code to perform the stitching process.

In imageProcessingRunnable, we will add these codes after the showProcessingDialog function call:

try {
    // Create a long array to store all image address
    int elems=  listImage.size();
    long[] tempobjadr = new long[elems];
    for (int i=0;i<elems;i++){
        tempobjadr[i]=  listImage.get(i).getNativeObjAddr();
    }
    // Create a Mat to store the final panorama image
    Mat result = new Mat();
    // Call the OpenCV C++ Code to perform stitching process
    NativePanorama.processPanorama(tempobjadr, result.getNativeObjAddr());

    // Save the image to external storage
    File sdcard = Environment.getExternalStorageDirectory();
    final String fileName = sdcard.getAbsolutePath() + "/opencv_" + System.currentTimeMillis() + ".png";
    Imgcodecs.imwrite(fileName, result);

    runOnUiThread(new Runnable() {
        @Override
        public void run() {
            Toast.makeText(getApplicationContext(), "File saved at: " + fileName, Toast.LENGTH_LONG).show();
        }
    });

    listImage.clear();
} catch (Exception e) {
    e.printStackTrace();
}

In the preceding code, we will create a long array to store all the native addresses of each Mat image. Then, we will pass this long array and the native address of a Mat result, to store the panorama image. The OpenCV C++ code will run to perform stitching with the stitching module. After this, we save the result into the external storage and make a simple toast to indicate to the user that the panorama is saved. Finally, we clear the listImage list to start a new section.

Implementing the OpenCV C++ code

At this moment, we want to implement the processPanorama in OpenCV C++. The implementation is really simple; we will only edit the com_example_panorama_NativePanorama.cpp file as follows:

#include "com_example_panorama_NativePanorama.h"
#include "opencv2/opencv.hpp"
#include "opencv2/stitching.hpp"

using namespace std;
using namespace cv;

JNIEXPORT void JNICALL Java_com_example_panorama_NativePanorama_processPanorama
  (JNIEnv * env, jclass clazz, jlongArray imageAddressArray, jlong outputAddress){
  // Get the length of the long array
  jsize a_len = env->GetArrayLength(imageAddressArray);
  // Convert the jlongArray to an array of jlong
  jlong *imgAddressArr = env->GetLongArrayElements(imageAddressArray,0);
  // Create a vector to store all the image
  vector< Mat > imgVec;
  for(int k=0;k<a_len;k++)
  {
    // Get the image
    Mat & curimage=*(Mat*)imgAddressArr[k];
    Mat newimage;
    // Convert to a 3 channel Mat to use with Stitcher module
    cvtColor(curimage, newimage, CV_BGRA2RGB);
    // Reduce the resolution for fast computation
    float scale = 1000.0f / curimage.rows;
    resize(newimage, newimage, Size(scale * curimage.rows, scale * curimage.cols));
    imgVec.push_back(newimage);
  }
  Mat & result  = *(Mat*) outputAddress;
  Stitcher stitcher = Stitcher::createDefault();
  stitcher.stitch(imgVec, result);
  // Release the jlong array 
  env->ReleaseLongArrayElements(imageAddressArray, imgAddressArr ,0);
}

In the preceding code, we converted the long array of image addresses into images and pushed into a vector called imgVec. We also resized the image for fast computation. The stitching module is really easy to use.

First, we will create an instance of Stitcher.

  Stitcher stitcher = Stitcher::createDefault();

Then, we use this stitcher to stitch our vector image of Mat. The panorama image will be saved into a resultant Mat.

The following screenshot shows an example of a panorama image processed with the default configuration:

Implementing the OpenCV C++ code

A sample image captured in the corridor

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset