April 21, 2020 | Engineering Tips

How to add Smart IDReader to your Android app

Smart IDReader SDK package consists of the following parts:

bin – libjniSmartIdEngine.so native library for the 32-bit ARMv7 architecture

bin-64 – libjniSmartIdEngine.so native library for the 64-bit ARMv8 architecture

bin-x86 – libjniSmartIdEngine.so native library for the 32-bit x86 architecture

bin-x86_64 – libjniSmartIdEngine.so native library for x86_64 architecture.

bindings – JNI wrapper jniSmartIdEngineJar.jar for the libjniSmartIdEngine.so library

data-zip – directory with engine configuration archive

doc – SDK documentation

sample – Sample app for Android studio which implements code described below

A couple of points for clarification:

· There are 4 native libraries for different architectures. Assemblies for ARMv7 and ARMv8 are the main ones, whereas the x86 and x86_64 are usually used for specific devices based on Intel mobile processors.

· Java Native Interface wrapper jniSmartIdEngineJar.jar is required for the C++ code to be executed from Java applications. 

The main steps to add recognition functionality provided Smart IDReader are the following:

  •  1. Adding the necessary files to the project
  • 2. Data preparation and engine initialization
  • 3. Adding camera support to the application
  • 4. Starting recognition, transferring data and receiving result

Adding the necessary files to the project

Android Studio creates a libs folder from which the Gradle collector picks up and adds JAR files to the project by default. It is necessary to copy into JNI wrapper jniSmartIdEngineJar.jar and native-libs.jar. File native-libs.jar is the easiest way to add native libraries into your project by creating inside subfolders with the architecture’s names armeabi-v7a, arm64-v8a, x86 and x86_64 and putting inside corresponding libraries.

After installing the application the Android OS will automatically deploy the right library version for the device. Also you need to add the accompanying engine configuration bundle files to the project’s assets folder. If the folder is missing, you can create it manually or using the File | New | Folder | Assets Folder command. By default, we use the “data” folder in Assets to store bundle files.

Data preparation and engine initialization

After the necessary files have been added to the application, we need to do the following:

• Read configuration data from bundle file stored in assets

    private byte[] loadBundle(Context context, String assetsPath) throws Exception
   {
       AssetManager assetManager = context.getAssets();
       String[] bundleNames = assetManager.list(assetsPath);
       if (bundleNames.length == 0)
           throw new Exception("No bundles in assets");
       String bundlePath = null;
       final String bundleExt = ".zip"; // bundle extension filter
       for(String bundleName : bundleNames) {
           if(bundleName.endsWith(bundleExt)) {
               bundlePath = assetsPath + File.separator + bundleName;
               break;
           }
       }
       if(bundlePath == null)
           throw new Exception("No zip bundle found");
       InputStream inputStream = assetManager.open(bundlePath);
       ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
       byte[] buffer = new byte[0xFFFF];
       for (int len = inputStream.read(buffer); len != -1; len = inputStream.read(buffer))
           outputStream.write(buffer, 0, len);
       inputStream.close();
       return outputStream.toByteArray();
    }

• Load the library and initialize the engine

        private RecognitionEngine engine; // engine instance
       Private SessionSettings settings; // settings instance
   private void initEngine(Context context) throws Exception{
       final String libraryName = "jniSmartIdEngine";
       System.loadLibrary(libraryName);
       final String assetsPath = "data"; // default bundle folder
       // create engine instance
       engine = new RecognitionEngine(loadBundle(context, assetsPath));
       // create settings instance
       settings = engine.CreateSessionSettings();
    }

Adding camera support to your application

If your application already uses a camera, please proceed to the next section. 

Although Camera class has been declared as deprecated since the API version 21 (Android 5.0), we are still using it and not the Camera2, for the following reasons:

• The Camera class is much easier to use and it provides the necessary functionality

• The support for older devices on Android 4.x.x is still relevant

To add camera support to the application, you need to register the following lines in the manifest:

<uses-permission android:name="android.permission.CAMERA" />
<uses-feature android:name="android.hardware.camera" />

For Android 6.x and higher it is necessary to request permission to use the camera. The users of these systems can always revoke the permission in their settings anyway, so you still need to perform the check.

    private final int REQUEST_CAMERA_PERMISSION = 1;
   private boolean cameraGranted = false;
   protected void onCreate(Bundle savedInstanceState) {
       super.onCreate(savedInstanceState);
// check whether camera access granted before or not
boolean cameraGranted = ContextCompat.checkSelfPermission(this,
Manifest.permission.CAMERA) == PackageManager.PERMISSION_GRANTED;
if(!cameraGranted) {
ActivityCompat.requestPermissions(this,new String[]{Manifest.permission.CAMERA}, REQUEST_CAMERA_PERMISSION);
}
   @Override
   public void onRequestPermissionsResult(int requestCode, String permissions[], int[] grantResults)
   {
       switch (requestCode) {
           case REQUEST_CAMERA_PERMISSION: {
               for (int grantResult : grantResults) {
                   if (grantResult == PackageManager.PERMISSION_GRANTED)
                       cameraGranted = true;
               }
               if (!cameraGranted)
                   showMessage("Please enable Camera permission."); // show message that user declined camera access
           }
           default: {
               super.onRequestPermissionsResult(requestCode, permissions, grantResults);
           }
       }
    }

First of all we need to open the camera. Then it is crucial to set main camera parameters – the focus mode and preview resolution. Due to the wide variety of devices and different characteristics of their cameras, this issue should be given special attention. In case the camera does not have support for the autofocus, then you would need to work with a fixed or aimed at infinity focus, and get the images from the camera “as-is”. Otherwise, if the focus is available, we need to check if FOCUS_MODE_CONTINUOUS_PICTURE or FOCUS_MODE_CONTINUOUS_VIDEO modes are supported, which means the camera will be automatically focused on the document during the whole scanning process. If these modes are supported, then we configure them in the parameters. 

private Camera camera;

……

camera = Camera.open();

Camera.Parameters params = camera.getParameters();
// List of supported focus modes
List<String> focus_modes = params.getSupportedFocusModes();
String focus_mode = Camera.Parameters.FOCUS_MODE_AUTO;
boolean isAutoFocus = focus_modes.contains(focus_mode);
if (isAutoFocus) {
if (focus_modes.contains(Camera.Parameters.FOCUS_MODE_CONTINUOUS_PICTURE))
    focus_mode = Camera.Parameters.FOCUS_MODE_CONTINUOUS_PICTURE;
else if (focus_modes.contains(Camera.Parameters.FOCUS_MODE_CONTINUOUS_VIDEO))
    focus_mode = Camera.Parameters.FOCUS_MODE_CONTINUOUS_VIDEO;
} else {
// In case when autofocus is not supported, take the first available focus mode
focus_mode = focus_modes.get(0);
}
// Setting the focus mode
params.setFocusMode(focus_mode);
camera.setParameters(params);

Setting the resolution preview is quite simple, the basic requirements are that the aspect ratios of the preview camera correspond to the sides of the display area so that there is no distortion when viewing, and the resolution should be as high as possible since the quality of document recognition depends on it. In our example, the application displays preview in full screen, so we select the maximum resolution that corresponds to the aspect ratios of the screen.

DisplayMetrics metrics = new DisplayMetrics();
getWindowManager().getDefaultDisplay().getMetrics(metrics);
// the screen sides ratio
float best_ratio=(float)metrics.heightPixels / (float)metrics.widthPixels;
List<Camera.Size> sizes = params.getSupportedPreviewSizes();
Camera.Size preview_size = sizes.get(0);
// allowable deviation from the optimal ratio
final float tolerance = 0.1f;
float preview_ratio_diff = Math.abs( (float) preview_size.width / (float) preview_size.height - best_ratio);
// choosing the optimal recognition resolution preview of the camera according to the screen sides size
for (int i = 1; i < sizes.size() ; i++)
{
Camera.Size tmp_size = sizes.get(i);
float tmp_ratio_diff = Math.abs( (float) tmp_size.width / (float) tmp_size.height - best_ratio);
if( Math.abs(tmp_ratio_diff - preview_ratio_diff) < tolerance && tmp_size.width > preview_size.width || tmp_ratio_diff < preview_ratio_diff)
{
    preview_size = tmp_size;
    preview_ratio_diff = tmp_ratio_diff;
}
}
// setting the preview size in camera settings
params.setPreviewSize(preview_size.width, preview_size.height);
camera.setParameters(params);

The next step is to set the camera orientation and display the preview on the surface of the Activity. The 0 degrees angle by default imposes the landscape orientation of the device, thus when you rotate the screen, the orientation has to be changed accordingly.

SurfaceView surface = (SurfaceView) findViewById(R.id.preview);
...

// portrait orientation
camera.setDisplayOrientation(90); // use portrait orientation
// displaying preview on the application surface
camera.setPreviewDisplay(surface.getHolder());
// initiating the preview process
camera.startPreview();

Starting recognition, transferring data and receiving result

The camera is now connected and showing preview, the most interesting thing remains – to engage the recognition engine and get the results. To launch the recognition process we need to start a new recognition session, set up a document mask (which documents we are going to recognize), desirable time out (maximum session length time in seconds) and set a callback to receive frames from the camera in preview mode.

private RecognitionSession session;
……………………..
void startSession(String documentMask, String timeOut)
{
       try {
             // setting session parameters
            sessionSettings.SetOption("common.sessionTimeout", timeOut);
            sessionSettings.RemoveEnabledDocumentTypes("*");
            sessionSettings.AddEnabledDocumentTypes(documentMask);
             // creating the recognition session
session = engine.SpawnSession(sessionSettings);
        // semaphores indication the readiness of the frame for the processing and awaiting the frame
        frame_waiting = new Semaphore(1, true);
        frame_ready = new Semaphore(0, true);
        // launching the recognition stream in a separate AsyncTask
        new EngineTask().execute();
             // setting the callback to receive the images from the camera
             camera.setPreviewCallback(this);
             // processing flag
           processing = true;
        } catch (RuntimeException e) {
        ...
        }
}

The onPreviewFrame () function receives the current image from the camera as a sequence of bytes in YUV NV21 format. Since it can only be invoked in the main thread, to not slow it down the engine calls for image processing are placed in a separate thread using AsyncTask, the process is synchronized using semaphores. After receiving the image from the camera, we give a signal to the work thread to launch the processing, and a signal to receive a new image by the end.

// current image from camera
private static volatile byte[] data;
...
@Override
public void onPreviewFrame(byte[] data_, Camera camera)
{
if(frame_waiting.tryAcquire())
{
    data = data_;
    // semaphore indicating that image is ready for recognizing
    frame_ready.release();
}
}

class EngineTask extends AsyncTask<Void, RecognitionResult, Void>
{
@Override
protected Void doInBackground(Void... unused) {
    while (processing) {
        try {
            Camera.Size size = camera.getParameters().getPreviewSize();
            // sending the frame to the recognition engine and receiving the result
            RecognitionResult result = session.ProcessYUVSnapshot(data, size.width, size.height, ImageOrientation.Portrait);
             // publish result to UI thread
              publishProgress(result);  
        }catch(RuntimeException e)
        {
            ... }
        catch(InterruptedException e)
        {
            ...
        }
    }
    return null;

       @Override
       protected void onProgressUpdate(RecognitionResult... res) {
          RecognitionResult result = res[0];
          // check and show result
           showResult(result);
           // semaphore indicating the awaiting of the new frame
           frame_waiting.release();
        }

After processing each image, the engine returns the current recognition result, which includes: detected document zones (areas), text fields with confidence values ​and flags, and graphic fields such as photos or signatures. If the data is recognized correctly or a timeout has occurred, the IsTerminal flag will notify about the completion of the process. For intermediate results the visualization of the detected zones and fields can be performed, along with some additional information.

void showResult(RecognitionResult result)
{
if(result.IsTerminal() == true) {
// stop session if timeOut reached or result is ready
stopSession();
// receiving the recognized fields of the document
StringVector texts = result.GetStringFieldNames();
// receiving the images from the document (photos, signatures, etc)
StringVector images = result.GetImageFieldNames();
for (int i = 0; i < texts.size(); i++) // document text fields
{
    StringField field = result.GetStringField(texts.get(i));
    String value = field.GetUtf8Value(); // fields data
    boolean is_accepted = field.IsAccepted(); // status of the field
    ...
}
for (int i = 0; i < images.size(); i++) // graphic field of the document
{
    ImageField field = result.GetImageField(images.get(i));
           String value = field.GetValue().GetBase64String();
           final byte[] bytes = Base64.decode(value, Base64.DEFAULT);
           Bitmap bmp = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
    ...
}
             }
}
void stopSession()
{
   processing = false;
   data = null;
   frame_waiting.release();
   frame_ready.release();
   // ending the process of receiving images from the camera
   camera.setPreviewCallback(null); ...
}

 

More Engineering Tips posts

Smart IDReader Library Reference

Smart IDReader Library Reference

The Smart IDReader Library allows recognizing various ID documents on images or video data obtained either from cameras or from scanners. Here we place a brief description of classes and members of the Library.

Test Drive Our Smart Engines

Free demo apps allow you to experience the power of Smart Engines software for intelligent document scanning in a real-world context.

Why not experience the power of Smart Engines for yourself? Our demo apps allow you to test the capabilities of our identity document recognition software on mobile devices in videostream or in a single image (photo, scan).

Simply display any document to the camera in real-time or choose a photo from the gallery, and the app will recognize and capture the necessary data.

Demo apps Privacy Policy

id documents enginge by Smart Engines
Apple App Store Badge
Google Play Badge
id documents enginge by Smart Engines

Get in Touch

For questions about our products, research, people or project proposals, please get in touch.

Contact Form
Warning before submitting your request:

Smart Engines is fully committed to provide an answer within 2 working days. However, it is your responsibility that your IT infrastructure does not block our reply or redirect it into your spams. If you haven’t received any answer from us within 2 working days, please check your spams or simply call us.

Smart Engines guarantees that the provided information will not be made public and will be used only internally.