Tensorflow in Ionic

I am working on a project where I am using Ionic and TensorFlow for machine learning. I have converted my TensorFlow model to a tensorflowjs model. I have put the model.json file and shard files of the tensorflowjs model in the assets folder in Ionic. Basically, I have put my tensorflowjs model in the assets folder of ionic. I am wanting to use Capacitor to access the camera and allow users to take photos. Then, the photos will be passed to the tensorflowjs model in assets to get and display a prediction for that user.

Here is my typescript code:

import { Component, OnInit, ViewChild, ElementRef, Renderer2 } from ‘@angular/core’;

import { Plugins, CameraResultType, CameraSource} from ‘@capacitor/core’;

import { DomSanitizer, SafeResourceUrl} from ‘@angular/platform-browser’;

import { Platform } from ‘@ionic/angular’;

import * as tf from ‘@tensorflow/tfjs’;

import { rendererTypeName } from ‘@angular/compiler’;

import { Base64 } from ‘@ionic-native/base64/ngx’;

import { defineCustomElements } from ‘@ionic/pwa-elements/loader’;

const { Camera } = Plugins;

@Component({

selector: ‘app-predict’,

templateUrl: ‘./predict.page.html’,

styleUrls: [’./predict.page.scss’],

})

export class PredictPage{

linearModel : tf.Sequential;

prediction : any;

InputTaken : any;

ctx: CanvasRenderingContext2D;

pos = { x: 0, y: 0 };

canvasElement : any;

photo: SafeResourceUrl;

model: tf.LayersModel;

constructor(public el : ElementRef , public renderer : Renderer2 , public platform : Platform, private base64: Base64,

private sanitizer: DomSanitizer) 

{}

async takePicture() {

const image = await Camera.getPhoto({

    quality: 90,

    allowEditing: true,

    resultType: CameraResultType.DataUrl,

    source: CameraSource.Camera});

  const model = await tf.loadLayersModel('src/app/assets/model.json');

  this.photo = this.sanitizer.bypassSecurityTrustResourceUrl(image.base64String);

  defineCustomElements(window);

const pred = await tf.tidy(() => {

      // Make and format the predications

    const output = this.model.predict((this.photo)) as any;

                            

      // Save predictions on the component

    this.prediction = Array.from(output.dataSync()); 

    });

}

}

Here, in the above code, I have imported the necessary tools. Then, I have my constructor function and a takepicture() function. In the takepicture function, I have included functionality for the user to take pictures. However, I am having trouble with passing the pictures taken to the tensorflowjs model to get a prediction. I am passing the picture taken to the tensorflowjs model in this line of code:

const output = this.model.predict((this.photo)) as any;

However, I am getting an error stating that:

“Argument of type ‘SafeResourceUrl’ is not assignable to parameter of type ‘Tensor | Tensor’.\n Type ‘SafeResourceUrl’ is missing the following properties from type ‘Tensor’: length, pop, push, concat, and 26 more.”,

It would be appreciated if I could receive some guidance regarding this topic as there are not many resources online that could help me with this issue.

2 posts - 2 participants

Read full topic