Custom Face Detector with Quality Function¶

You can create your own face detector with a quality function if you don't want to use the default LocalFaceDetector and wish to use it together with IadCameraController.

Creating a Custom Face Detector¶

Note

You need to synchronize the state of your implementation to ensure it is safe for use in a multithreaded environment. The code below demonstrates how to synchronize the detector's state in the detect() and close() methods.

Danger

If the detector's state is not properly synchronized, it may result in undefined behavior.

A custom face detector should implement the AutocaptureFaceDetector<Jpeg, *> interface. In this example, we use GoogleFaceDetector as the underlying detector.

import android.graphics.Bitmap
import android.graphics.BitmapFactory
import android.os.Handler
import android.os.Looper
import androidx.core.graphics.createBitmap
import com.google.android.gms.tasks.Task
import com.google.mlkit.vision.common.InputImage
import com.google.mlkit.vision.face.Face
import com.google.mlkit.vision.face.FaceDetection
import com.google.mlkit.vision.face.FaceDetectorOptions
import java.lang.AutoCloseable
import java.util.concurrent.CountDownLatch
import net.idrnd.android.idlive.face.camera.images.Jpeg
import net.idrnd.android.idlive.face.detection.interfaces.AutocaptureFaceDetector

// Copied from the official Google repository:
// https://github.com/googlesamples/mlkit/blob/master/android/vision-quickstart/app/src/main/java/com/google/mlkit/vision/demo/kotlin/facedetector/FaceDetectorProcessor.kt
class GoogleFaceDetector : AutocaptureFaceDetector<Jpeg, Face?>, AutoCloseable {

    var onDetectionResultListener: OnDetectionResultListener? = null

    private val uiHandler = Handler(Looper.getMainLooper())
    private val detector = FaceDetection.getClient(
        FaceDetectorOptions.Builder()
            .setClassificationMode(FaceDetectorOptions.CLASSIFICATION_MODE_ALL)
            .build()
    )

    private var isLastImageSuitableForCapturing = false
    private var cachedBitmap: Bitmap? = null

    private val detectorLock = Any()
    private val isLastImageSuitableForCapturingLock = Any()

    private var isClosed = false

    override fun detect(image: Jpeg): Face? {
        if (isClosed) return null

        val image = jpegToInputImage(image)

        val task = synchronized(detectorLock) {
            if (isClosed) return null
            detector.process(image)
        }

        val foundFace = task.await().firstOrNull()

        synchronized(isLastImageSuitableForCapturingLock) {
            isLastImageSuitableForCapturing = if (foundFace == null) {
                false
            } else {
                val smilingThreshold = 0.5f
                (foundFace.smilingProbability ?: 0f) >= smilingThreshold
            }
        }

        uiHandler.post {
            onDetectionResultListener?.onDetectionResult(foundFace)
        }

        return foundFace
    }

    override fun isLastImageSuitableForCapturing(): Boolean {
        return synchronized(isLastImageSuitableForCapturingLock) {
            // Determine whether the latest image satisfies the conditions for automatic capturing.
            // This will be used by IadCameraController to trigger photo capture.
            // Important note: There is no guarantee that returning `true` will result in an actual capture.
            // The IadCameraController will attempt to capture only if the other internal conditions allow it.
            isLastImageSuitableForCapturing
        }
    }

    override fun close() {
        if (isClosed) return
        synchronized(detectorLock) {
            isClosed = true
            detector.close()
        }
    }

    private fun jpegToInputImage(jpeg: Jpeg): InputImage {
        if (cachedBitmap == null) {
            cachedBitmap = createBitmap(jpeg.size.width, jpeg.size.height)
        }

        val options = BitmapFactory.Options().apply {
            inBitmap = cachedBitmap
        }

        val bitmap = BitmapFactory.decodeByteArray(
            jpeg.content,
            0,
            jpeg.content.size,
            options
        )

        return InputImage.fromBitmap(bitmap, jpeg.imageInfo.rotationDegrees)
    }

    private fun Task<List<Face>>.await(): List<Face> {
        var output = listOf<Face>()
        val latch = CountDownLatch(1)

        addOnSuccessListener { facesList ->
            output = facesList
            latch.countDown()
        }

        addOnFailureListener { error ->
            throw error
        }

        addOnCanceledListener {
            latch.countDown()
        }

        latch.await()
        return output
    }
}

Integrating the Custom Face Detector¶

Simply replace LocalFaceDetector with your custom face detector in the constructor of IadCameraController:

val faceDetector = GoogleFaceDetector(),
val cameraController = IadCameraController(faceDetector, previewView, lifecycleOwner)

More Information¶

You can find an example of this functionality in the idlive-face-capture-android-X.X.X-release/iad-example folder.