Integrating a custom AutoML tflite model with flutter app

Question

I am new to Flutter, basically, I followed a tutorial online to train a custom image labeling model with Google's AutoML API then downloaded the model as three files(dict.txt, manifest.json, model.tflite), and now I am trying to integrate it with my flutter application.

Here is my code to load and run the TFlite model:

Future loadModel() async {
    try{
      res = await Tflite.loadModel(
          model: "assets/models/model.tflite",
          labels: "assets/models/dict.txt",
      );
      print("loading tf model...");
      print(res);
    }on PlatformException{
      print ("Failed to load model");
    }
  }

Future recognizeImageBinary(File image) async {
    var imageBytes = await image.readAsBytesSync();
    var bytes = imageBytes.buffer.asUint8List();
    img.Image oriImage = img.decodeJpg(bytes);
    img.Image resizedImage = img.copyResize(oriImage, height: 112, width: 112);

    var recognitions = await Tflite.runModelOnBinary(
      binary: imageToByteListUint8(resizedImage, 112),
      numResults: 2,
      threshold: 0.4,
      asynch: true
    );
    setState(() {
      _recognitions = recognitions;
    });
  }

According to the tutorial, AutoML custom trained model is with the type Uint8, so I used the function below to convert it:

Uint8List imageToByteListUint8(img.Image image, int inputSize) {
    var convertedBytes = Uint8List(4 * inputSize * inputSize * 3);
    var buffer = Uint8List.view(convertedBytes.buffer);
    int pixelIndex = 0;
    for (var i = 0; i < inputSize; i++) {
      for (var j = 0; j < inputSize; j++) {
        var pixel = image.getPixel(j, i);
        buffer[pixelIndex++] = img.getRed(pixel);
        buffer[pixelIndex++] = img.getGreen(pixel);
        buffer[pixelIndex++] = img.getBlue(pixel);
      }
    }
    return convertedBytes.buffer.asUint8List();
  }

And I got exceptions like this:

E/AndroidRuntime( 6372): FATAL EXCEPTION: AsyncTask #2
E/AndroidRuntime( 6372): Process: com.soton.gca_app, PID: 6372
E/AndroidRuntime( 6372): java.lang.RuntimeException: An error occurred while executing doInBackground()
E/AndroidRuntime( 6372):    at android.os.AsyncTask$3.done(AsyncTask.java:318)
E/AndroidRuntime( 6372):    at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:354)
E/AndroidRuntime( 6372):    at java.util.concurrent.FutureTask.setException(FutureTask.java:223)
E/AndroidRuntime( 6372):    at java.util.concurrent.FutureTask.run(FutureTask.java:242)
E/AndroidRuntime( 6372):    at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:243)
E/AndroidRuntime( 6372):    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
E/AndroidRuntime( 6372):    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
E/AndroidRuntime( 6372):    at java.lang.Thread.run(Thread.java:760)
E/AndroidRuntime( 6372): Caused by: java.lang.IllegalArgumentException: Cannot convert between a TensorFlowLite tensor with type UINT8 and a Java object of type [[F (which is compatible with the TensorFlowLite type FLOAT32).
E/AndroidRuntime( 6372):    at org.tensorflow.lite.Tensor.throwIfTypeIsIncompatible(Tensor.java:316)
E/AndroidRuntime( 6372):    at org.tensorflow.lite.Tensor.throwIfDataIsIncompatible(Tensor.java:304)
E/AndroidRuntime( 6372):    at org.tensorflow.lite.Tensor.copyTo(Tensor.java:183)
E/AndroidRuntime( 6372):    at org.tensorflow.lite.NativeInterpreterWrapper.run(NativeInterpreterWrapper.java:166)
E/AndroidRuntime( 6372):    at org.tensorflow.lite.Interpreter.runForMultipleInputsOutputs(Interpreter.java:311)
E/AndroidRuntime( 6372):    at org.tensorflow.lite.Interpreter.run(Interpreter.java:272)
E/AndroidRuntime( 6372):    at sq.flutter.tflite.TflitePlugin$RunModelOnBinary.runTflite(TflitePlugin.java:478)
E/AndroidRuntime( 6372):    at sq.flutter.tflite.TflitePlugin$TfliteTask.doInBackground(TflitePlugin.java:419)
E/AndroidRuntime( 6372):    at sq.flutter.tflite.TflitePlugin$TfliteTask.doInBackground(TflitePlugin.java:393)
E/AndroidRuntime( 6372):    at android.os.AsyncTask$2.call(AsyncTask.java:304)
E/AndroidRuntime( 6372):    at java.util.concurrent.FutureTask.run(FutureTask.java:237)
E/AndroidRuntime( 6372):    ... 4 more

I got really confused now, anyone can please help here?

The TFLite model requires the inputs to have a dtype of float32 whereas you are providing inputs with dtype=uint8. Make sure you cast the img.getRed( pixel ) and other two values to float. — Shubham Panchal
– Shubham Panchal, Commented Mar 29, 2020 at 1:33
@ShubhamPanchal I tried to convert the image to float32list as you suggest, it still gives me the same error above — Hazel Wang
– Hazel Wang, Commented Mar 31, 2020 at 10:11

Hazel Wang · Accepted Answer · 2020-03-31 10:09:33Z

1

@Shubham It appears that the exceptions exist anyhow, even I use the method:

Uint8List imageToByteListFloat32(img.Image image, int inputSize, double mean, double std) {
    var convertedBytes = Float32List(1 * inputSize * inputSize * 3 );
    var buffer = Float32List.view(convertedBytes.buffer);
    int pixelIndex = 0;
    for (var i = 0; i < inputSize; i++) {
      for (var j = 0; j < inputSize; j++) {
        var pixel = image.getPixel(j, i);
        buffer[pixelIndex++] = ((img.getRed(pixel) - mean) / std).toDouble();
        buffer[pixelIndex++] = ((img.getGreen(pixel) - mean) / std).toDouble();
        buffer[pixelIndex++] = ((img.getBlue(pixel) - mean) / std).toDouble();
      }
    }
    return convertedBytes.buffer.asUint8List();
  }

answered Mar 31, 2020 at 10:09

Hazel Wang

1112 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

L.Goyal Over a year ago

Can anyone help me? The newer versions of the image library do not have the img.getRed methods. What should I use in place of those?

Collectives™ on Stack Overflow

Integrating a custom AutoML tflite model with flutter app

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related