Tag: cv

Categories
Android OpenCV Programming

Use OpenCV to show camera on android App with correct orientation

Return to OpenCV 4 Android Tutorials List

In this series of tutorials, you are learning OpenCV for Android using Android Studio. So far, you have made a working OpenCV Android Studio Project containing OpenCV module.

In Here is the overview of what we are going to achieve:

  • Create an OpenCV for Android Project (As in previous tutorial in this series) (Here)
  • Show Camera on the Screen
    1. Edit Manifest
      1. Add permission to allow camera.
      2. Support various screen sizes and adaptability of camera.
      3. Use specific features of the camera.
    2. Add a custom Layout show_camera.xml to display camera in it.
    3. Edit MainActivity_show_camera.java 
      1. Import required android classes
      2.  Import OpenCV classes
      3. Connect to OpenCV manager
      4. Initiate LogCat to log events of out app
      5. SHOW CAMERA ON LAYOUT
      6. Corrected Orientation (portrait)

Trust me, it is very simple to do. Lets do it!

 

Manifest.xml

We need to edit the AndroidManifest.xml file like following:

  1. Allow permission to use camera 
    <uses-permission android:name="android.permission.CAMERA"/>
  2. Allow camera to optimize to the dimensions of the device in use 
    <supports-screens android:resizeable="true"
        android:smallScreens="true"
        android:normalScreens="true"
        android:largeScreens="true"
        android:anyDensity="true" />
  3. Use Front camera and autofocus of camera :S 
     <uses-feature android:name="android.hardware.camera" android:required="false"/>
    <uses-feature android:name="android.hardware.camera.autofocus" android:required="false"/>
    <uses-feature android:name="android.hardware.camera.front" android:required="false"/>
    <uses-feature android:name="android.hardware.camera.front.autofocus" android:required="false"/>

The Manifest.xml file in its entirety is given here

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="opencv.codeonion.com.opencv_test">

    <application
        android:allowBackup="true"
        android:icon="@mipmap/ic_launcher"
        android:label="@string/app_name"
        android:supportsRtl="true"
        android:theme="@style/AppTheme"
        >
        <activity
            android:name=".MainActivity_show_camera"
            android:label="@string/app_name"
            android:theme="@style/AppTheme.NoActionBar"
            android:screenOrientation="portrait">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />

                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
        </activity>
    </application>

    <supports-screens   android:resizeable="true"
                        android:smallScreens="true"
                        android:normalScreens="true"
                        android:largeScreens="true"
                        android:anyDensity="true" />

    <uses-permission android:name="android.permission.CAMERA"/>
    <uses-feature android:name="android.hardware.camera" android:required="false"/>
    <uses-feature android:name="android.hardware.camera.autofocus" android:required="false"/>
    <uses-feature android:name="android.hardware.camera.front" android:required="false"/>
    <uses-feature android:name="android.hardware.camera.front.autofocus" android:required="false"/>

</manifest>

 

This entire process is given here:

Show camera on android App using OpenCV for Android - Edit Manifest

This is all you need to do in AndroidManifest.xml

Create a new layout (show_camera.xml )

Now in your project, do the following:

  1. Create a new layout called show_camera.xml
  2. Add the following code it to: 
    <?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:orientation="vertical" android:layout_width="match_parent"
    xmlns:opencv="http://schemas.android.com/apk/res-auto"
    android:layout_height="match_parent">

    <org.opencv.android.JavaCameraView
        android:layout_width="fill_parent"
        android:layout_height="fill_parent"
        android:visibility="gone"
        android:id="@+id/show_camera_activity_java_surface_view"
        opencv:show_fps="true"
        opencv:camera_id="any" />

</LinearLayout>

    Note that the JavaCameraView has the id of show_camera_activity_java_surface_view

This is given below:

 

MainActivity_show_camera.java

Import the following Android Classes in your MainActivity_show_camera:

import android.os.Bundle;
import android.support.v7.app.AppCompatActivity;
import android.util.Log;
import android.view.MenuItem;
import android.view.SurfaceView;
import android.view.WindowManager;

Import the following OpenCV classes in your MainActivity_show_camera:

import org.opencv.android.JavaCameraView;
import org.opencv.android.BaseLoaderCallback;
import org.opencv.android.CameraBridgeViewBase;
import org.opencv.android.CameraBridgeViewBase.CvCameraViewFrame;
import org.opencv.android.CameraBridgeViewBase.CvCameraViewListener2;
import org.opencv.android.LoaderCallbackInterface;
import org.opencv.android.OpenCVLoader;
import org.opencv.core.Core;
import org.opencv.core.CvType;
import org.opencv.core.Mat;
import org.opencv.imgproc.Imgproc;

Now, lets work with the MainActivity_show_camera Class 🙂 (Code in the end). I will explain every line of OpenCV code in detail.

  • Implement the OpenCV class CvCameraViewListener2 to allow OpenCV to communicate with android camera functionalities.
public class MainActivity extends AppCompatActivity implements CvCameraViewListener2 {
  • Declare the following items in the class 
    // Used for logging success or failure messages
    private static final String TAG = "OCVSample::Activity";

    // Loads camera view of OpenCV for us to use. This lets us see using OpenCV
    private CameraBridgeViewBase mOpenCvCameraView;

    // Used in Camera selection from menu (when implemented)
    private boolean              mIsJavaCamera = true;
    private MenuItem             mItemSwitchCamera = null;

    // These variables are used (at the moment) to fix camera orientation from 270degree to 0degree
    Mat mRgba;
    Mat mRgbaF;
    Mat mRgbaT;
  • Now, lets call OpenCV manager to help our app communicate with android phone to make OpenCV work 
    private BaseLoaderCallback mLoaderCallback = new BaseLoaderCallback(this) {
        @Override
        public void onManagerConnected(int status) {
            switch (status) {
                case LoaderCallbackInterface.SUCCESS:
                {
                    Log.i(TAG, "OpenCV loaded successfully");
                    mOpenCvCameraView.enableView();
                } break;
                default:
                {
                    super.onManagerConnected(status);
                } break;
            }
        }
    };
    • CameraBridgeViewBase mOpenCvCameraView
      This variable acts as a bridge between camera and OpenCV library.
    • BaseLoaderCallback mLoaderCallback = …..
      Well, once OpenCV library is loaded, you may want to perform some actions. For example, displaying a success or failure message.
  • Initiate the LogCat 
    public MainActivity_show_camera() {
        Log.i(TAG, "Instantiated new " + this.getClass());
    }
  • Now, when the activity is created, display the OpenCV camera in the layout. show_camera.xml. 
    @Override
    protected void onCreate(Bundle savedInstanceState) {
        Log.i(TAG, "called onCreate");
        super.onCreate(savedInstanceState);
        getWindow().addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);

        setContentView(R.layout.show_camera);

        mOpenCvCameraView = (JavaCameraView) findViewById(R.id.show_camera_activity_java_surface_view);

        mOpenCvCameraView.setVisibility(SurfaceView.VISIBLE);

        mOpenCvCameraView.setCvCameraViewListener(this);
    }
  •  The following three functions handle the events when the app is Paused, Resumed and Closed/Destroyed 
    @Override
    public void onPause()
    {
        super.onPause();
        if (mOpenCvCameraView != null)
            mOpenCvCameraView.disableView();
    }

    @Override
    public void onResume()
    {
        super.onResume();
        if (!OpenCVLoader.initDebug()) {
            Log.d(TAG, "Internal OpenCV library not found. Using OpenCV Manager for initialization");
            OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_3_0_0, this, mLoaderCallback);
        } else {
            Log.d(TAG, "OpenCV library found inside package. Using it!");
            mLoaderCallback.onManagerConnected(LoaderCallbackInterface.SUCCESS);
        }
    }

    public void onDestroy() {
        super.onDestroy();
        if (mOpenCvCameraView != null)
            mOpenCvCameraView.disableView();
    }
  • Now, we will do two main things:
    • Receive Image Data when the camera preview starts on your screen 😀 
      public void onCameraViewStarted(int width, int height) {

        mRgba = new Mat(height, width, CvType.CV_8UC4);
        mRgbaF = new Mat(height, width, CvType.CV_8UC4);
        mRgbaT = new Mat(width, width, CvType.CV_8UC4);
    }
    • Destroy image data when you stop camera preview on your phone screen 
      public void onCameraViewStopped() {
        mRgba.release();
    }
    • Now, this one is interesting! OpenCV orients the camera to left by 90 degrees. So if the app is in portrait more, camera will be in -90 or 270 degrees orientation. We fix that in the next and the most important function. There you go! 
          public Mat onCameraFrame(CvCameraViewFrame inputFrame) {

        // TODO Auto-generated method stub
        mRgba = inputFrame.rgba();
        // Rotate mRgba 90 degrees
        Core.transpose(mRgba, mRgbaT);
        Imgproc.resize(mRgbaT, mRgbaF, mRgbaF.size(), 0,0, 0);
        Core.flip(mRgbaF, mRgba, 1 );

        return mRgba; // This function must return
    }

The entire code is given below:

package opencv.codeonion.com.opencv_test;

import android.os.Bundle;
import android.support.v7.app.AppCompatActivity;
import android.util.Log;
import android.view.MenuItem;
import android.view.SurfaceView;
import android.view.WindowManager;

import org.opencv.android.JavaCameraView;
import org.opencv.android.BaseLoaderCallback;
import org.opencv.android.CameraBridgeViewBase;
import org.opencv.android.CameraBridgeViewBase.CvCameraViewFrame;
import org.opencv.android.CameraBridgeViewBase.CvCameraViewListener2;
import org.opencv.android.LoaderCallbackInterface;
import org.opencv.android.OpenCVLoader;
import org.opencv.core.Core;
import org.opencv.core.CvType;
import org.opencv.core.Mat;
import org.opencv.imgproc.Imgproc;

// OpenCV Classes

public class MainActivity_show_camera extends AppCompatActivity implements CvCameraViewListener2 {

    // Used for logging success or failure messages
    private static final String TAG = "OCVSample::Activity";

    // Loads camera view of OpenCV for us to use. This lets us see using OpenCV
    private CameraBridgeViewBase mOpenCvCameraView;

    // Used in Camera selection from menu (when implemented)
    private boolean              mIsJavaCamera = true;
    private MenuItem             mItemSwitchCamera = null;

    // These variables are used (at the moment) to fix camera orientation from 270degree to 0degree
    Mat mRgba;
    Mat mRgbaF;
    Mat mRgbaT;

    private BaseLoaderCallback mLoaderCallback = new BaseLoaderCallback(this) {
        @Override
        public void onManagerConnected(int status) {
            switch (status) {
                case LoaderCallbackInterface.SUCCESS:
                {
                    Log.i(TAG, "OpenCV loaded successfully");
                    mOpenCvCameraView.enableView();
                } break;
                default:
                {
                    super.onManagerConnected(status);
                } break;
            }
        }
    };

    public MainActivity_show_camera() {
        Log.i(TAG, "Instantiated new " + this.getClass());
    }



    @Override
    protected void onCreate(Bundle savedInstanceState) {
        Log.i(TAG, "called onCreate");
        super.onCreate(savedInstanceState);
        getWindow().addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);

        setContentView(R.layout.show_camera);

        mOpenCvCameraView = (JavaCameraView) findViewById(R.id.show_camera_activity_java_surface_view);

        mOpenCvCameraView.setVisibility(SurfaceView.VISIBLE);

        mOpenCvCameraView.setCvCameraViewListener(this);
    }

    @Override
    public void onPause()
    {
        super.onPause();
        if (mOpenCvCameraView != null)
            mOpenCvCameraView.disableView();
    }

    @Override
    public void onResume()
    {
        super.onResume();
        if (!OpenCVLoader.initDebug()) {
            Log.d(TAG, "Internal OpenCV library not found. Using OpenCV Manager for initialization");
            OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_3_0_0, this, mLoaderCallback);
        } else {
            Log.d(TAG, "OpenCV library found inside package. Using it!");
            mLoaderCallback.onManagerConnected(LoaderCallbackInterface.SUCCESS);
        }
    }

    public void onDestroy() {
        super.onDestroy();
        if (mOpenCvCameraView != null)
            mOpenCvCameraView.disableView();
    }

    public void onCameraViewStarted(int width, int height) {

        mRgba = new Mat(height, width, CvType.CV_8UC4);
        mRgbaF = new Mat(height, width, CvType.CV_8UC4);
        mRgbaT = new Mat(width, width, CvType.CV_8UC4);
    }

    public void onCameraViewStopped() {
        mRgba.release();
    }

    public Mat onCameraFrame(CvCameraViewFrame inputFrame) {

        // TODO Auto-generated method stub
        mRgba = inputFrame.rgba();
        // Rotate mRgba 90 degrees
        Core.transpose(mRgba, mRgbaT);
        Imgproc.resize(mRgbaT, mRgbaF, mRgbaF.size(), 0,0, 0);
        Core.flip(mRgbaF, mRgba, 1 );

        return mRgba; // This function must return
    }
}

The entire process of editing MainActivity_show_camera.java is shown here:

Show camera on android App using OpenCV for Android - MainActivity_show_camera

Note: The camera disorientation problem is a bitch an issue not to be takien lightheartedly. Thanks for noting.

This concludes this tutorial, now that you have created a camera app, it is now time to continue this and perform more Computer Vision! In the next tutorial, I will draw a simple object on the screen. Be sure to check out the list of OpenCV for Android tutorials for latest tutorials.

Thanks for reading, and please share and comment if this helped and to share your ideas and opinions.

Return to OpenCV 4 Android Tutorials List