Saturday, March 22, 2014

Displaying OpenCV Mat object in a JavaFX Image

Introduction

This post describes a way to display an OpenCV Mat object within a JavaFX Image.


Java part - Input part

  1. A Mat object is created in Java and populated with an image file using Highgui.imread()
  2. A ByteBuffer object is created using ByteBuffer.allocateDirect(). This buffer will contain the image after it has been operated upon by native code. allocateDirect() allows native code to directly access the buffer without needing expensive buffer copies.
  3. Native code is passed the address of the Mat object  along with the ByteBuffer created previously.
Example code is shown below (along with relevant comments) -


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
/**
 * Extends JavaFX Application class
 */
public class Main extends Application {

    final String filePath = "test65mp.jpg";

    // Output image pixel format can be BGRA or RGB
    final boolean isBGRA = true;

    static {
        // Load the OpenCV DLLs
        System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
        // Load custom DLL that inverts pixels in native code
        System.load("D:\\PROJECTS\\VisualStudio2013\\OpenCVJavaFXImageIO\\x64\\Release\\OpenCVJavaFXImageIO.dll");
    }

    /** Declaration of native method that will invert colours of each pixel.
     */
    public native void invert(long matAddr, ByteBuffer byteBuff, int buffChannels);

    /** Entry point for JavaFX application.
     * @param primaryStage passed by the system to this application.
     */
    @Override
    public void start(Stage primaryStage) {

        // Image file is loaded into an OpenCV Mat.
        // Default pixel format is BGR, but if 2nd arg is -1 as shown below then
        // format is BGRA in case input image contains Alpha channel.
        Mat mat = Highgui.imread(filePath, -1);

        int width = mat.width();
        int height = mat.height();
        int buffChannels = isBGRA ? 4 : 3;
        int len = width * height * buffChannels;
        
        // Will hold processed image
        ByteBuffer byteBuff = ByteBuffer.allocateDirect(len);

        // The address of the Mat object (containing image) is passed to native code,
        // which then operates directly on it. The byteBuff will hold the processed image
        invert(mat.getNativeObjAddr(), byteBuff, buffChannels);

        // Take image from byteBuff and display it in a JavaFX ImageView Node in a JavaFX Stage
 displayImage(byteBuff, width, height, buffChannels, isBGRA, primaryStage);
        
    }

// Image display code ...

}


C++ Native part - to process Mat object

  1. The code which processes the Mat image is inside a native C++ method, which is invoked by Java code using JNI (Java Native Interface). This method is passed the address of the Mat object, the ByteBuffer to which it should return the processed image, and the number of channels in the output. 
  2. Default pixel format in a Mat object is BGR while JavaFx Image display format is BGRA. Image can also accept RGB format. So after the native code has finished processing the Mat object, it should convert it either to BGRA (preferred, as that's one less conversion on the Java side) or RGB, using cv::cvtColor(). In case BGR is converted to BGRA, a new destination Mat is required but in case of BGR to RGB conversion, in-place conversion can be done on the source Mat.
  3. Finally, the data member (which points to image array) of the processed/converted Mat image is copied to the previously passed-in ByteBuffer.
Example code is shown below -


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
// Header file containing JNI method declarations. Generated by javah tool.
#include "D:\Documents\NetBeansProjects\JavaFXOpenCVImageIO\hdr.h"

#include <opencv2\core\core.hpp>
#include <opencv2\highgui\highgui.hpp>
#include <opencv2\imgproc\imgproc.hpp>

using namespace cv;

JNIEXPORT void JNICALL Java_io_Main_invert(JNIEnv * env, jobject obj, jlong matAddr, jobject byteBuff, jint buffChannels) {
 // In the code below, the input image format is assumed to be BGR. If it could also be BGRA then code will
 // need to be modified accordingly.

 // Reference (alias) to Mat object created by Java code.
 Mat& image = *(Mat*)matAddr;

 // Invert the pixel colours.
 for (int y = 0; y < image.rows; y++) {
  for (int x = 0; x < image.cols; x++) {
   Vec3b& pixel = image.at<Vec3b>(y, x);
   pixel[0] = ~pixel[0];
   pixel[1] = ~pixel[1];
   pixel[2] = ~pixel[2];
  }
 }


 // opencv Mat created from Highgui.imread() uses BGR format while 
 // the JavaFx Image formats are BGRA and RGB.

 // Converting from BGR to RGB is faster than BGR -> BGRA, but, on the other hand,
 // loading BGRA ByteBuffer into JavaFx image is faster than loading any other format as JavaFx uses BGRA format for display.

 // If Java code expects 3 channel RGB then do in-place conversion of BGR to RGB in 'image' Mat object.
 if (buffChannels == 3) {
  cvtColor(image, image, CV_BGR2RGB);
 }

 // If Java code expects 4 channel BGRA then copy inverted image to 4 ch BGRA 'dest' Mat object.
 Mat dest;
 if (buffChannels == 4) {
  dest = Mat(image.rows, image.cols, CV_8UC4);
  cvtColor(image, dest, CV_BGR2BGRA);
 }

 // Copy Mat image data to the direct ByteBuffer allocated by Java code.

 jbyte *buff = (jbyte*)env->GetDirectBufferAddress(byteBuff);
 jlong len = env->GetDirectBufferCapacity(byteBuff);

 if (buffChannels == 3) {
  memcpy(buff, image.data, len);
 }

 if (buffChannels == 4) {
  memcpy(buff, dest.data, len);
 }

}


Java part - Image display

  1. The PixelFormat (BGRA or RGB) of the native-code processed image in the ByteBuffer is set appropriately then the ByteBuffer is loaded into a WritableImage.
  2. The WritableImage is loaded into an ImageView node and displayed on a JavaFX Stage (window) as usual.
Example code is shown below -


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
/**
 * Extends JavaFX Application class
 */
public class Main extends Application {

    //Image input code shown previously here...//

    /**
     * Takes image from ByteBuffer and displays it.
     * @param byteBuff
     * @param width
     * @param height
     * @param isBGRA 
     */
    void displayImage(ByteBuffer byteBuff, int width, int height, int buffChannels, boolean isBGRA, Stage stage) {
        // This is the image which will contain the processed image in the ByteBuffer
        WritableImage dest = new WritableImage(width, height);
        
        // This is the pixel format of the image in the ByteBuffer (which is populated by native code).
        PixelFormat<ByteBuffer> buffFormat = null;

        if(isBGRA) {
            // use getByteBgraPREInstance instead of getByteBgraInstance so that
            // dest image need not waste time multiplying each colour by alpha channel A,
            // as A (set in native code) is a dummy value anyway.
            buffFormat = PixelFormat.getByteBgraPreInstance();
        } else {
            buffFormat = PixelFormat.getByteRgbInstance();
        }
        
        dest.getPixelWriter().setPixels(0, 0, width, height, buffFormat, byteBuff, width * buffChannels);
        
        // The JavaFX Image pixel format for display is always pre-multiplied BGRA. If the backing ByteBuffer or byte array
        // has any other format, it is converted to this format before display.
        System.out.println("Dest type: " + dest.getPixelReader().getPixelFormat().getType());
        
        showWindow(dest, stage);
    }
    
    void showWindow(Image image, Stage stage) {
        ImageView imageView = new ImageView(image);
        StackPane root = new StackPane();
        root.getChildren().add(imageView);
        Scene scene = new Scene(root);
        stage.setScene(scene);
        stage.show();
    }
}


Summary

  1. In Java, load an image file into a Mat object.
  2. Pass the address of the Mat object and a ByteBuffer to native code.
  3. Native code processes the image and puts processed image in previously passed ByteBuffer.
  4. Java takes the processed image from the ByteBuffer and displays it.

Conclusion

The objective of this post has been to show a way to efficiently display an OpenCV Mat object within a JavaFX Image. I haven't yet found a way to pass the Mat.data member from native code to Java without copying, hence the need to pass the direct ByteBuffer. If you have any ideas for more efficient code, please add a comment.

References

JavaFX Javadocs for Image, WritableImage, PixelReader, PixelWriter, PixelFormat.
OpenCV Javadocs for Mat, Highgui.
Java Javadocs for ByteBuffer.

Appendix

  • The Java bit was coded in NetBeans IDE 8.0. 
    • Add opencv-248.jar as a dependency.
  • Java platform is 1.8.0 64-bit.
  • C++ bit was coded in Visual Studio Express 2013.
    • Project mode is Release x64.
    • Add Include paths for OpenCV includes and Java JNI includes.
    • Add Library paths for OpenCV x64 VC12 libs.
    • Add Linker additional dependencies - opencv_core248.lib, opencv_highgui248.lib and opencv_imgproc248.lib.
  • OpenCV version 2.4.8.
  • OS is Windows 7 Home Premium.
Add paths to OpenCV DLLs and OpenCV Java DLL to system PATH environment variable. For my machine and setup they are like -
D:\PROJECTS\LIBRARIES\c\opencv\build.org\x64\vc12\bin
D:\PROJECTS\LIBRARIES\c\opencv\build.org\java\x64














  

3 comments:

  1. this is the most detailed answer I could find!! Thanks, I am going to try it today :)

    ReplyDelete
  2. You make so many great points here that I read your article a couple of times. Your views are in accordance with my own for the most part. This is great content for your readers. image typing data entry

    ReplyDelete