Camera video doesn't show up - iPhone iOS 14.3 - All browsers

[bureauberg]

Hi all, we use your Jeeliz FaceFilter library for a while now and it works great!
But recently, when using an iPhone (iOS 14.3), after approval at the alert pop up to use the camera, it doesn't show the camera feed anymore on iPhone 12, Safari, iOS 14.3 (nor at Chrome Mobile 77 and Firefox using iOS 14.3, same iPhone 12).
webglreport.com shows WebGL1 supported, WebGL2 not supported.

Please note that this issue suddenly occurred without any code changes so it might be due to an update of iOS. Before, it used to work fine on the same phones. The problem I'm facing is similar to issue 198.

Additionally, I tested it with iPhone 7+, 8, 8+ and 11, all with Safari and the same result.
With Android and at desktop no problems occur. It looks like the issue occurred after the latest iOS update.

Any insights? Please note that I'm not a developer myself but I"ll pass any findings thankfully through.

Thanks!
Jacco

[xavierjs]

Hi @bureauberg

Thank you for this bug report.
Are you sure the camera is not used by other apps?
Which jeeliz FaceFilter demo are you trying? Does the problem occur with the boilerplate:
https://jeeliz.com/demos/faceFilter/demos/threejs/cube2cv/
?

The deployed demos were not updated, I have updated them. Please open this boirlerplate link in private browsing mode to avoid any browser cache issue.

Can you give me the output of https://jeeliz.com/demos/WebGLCoreLogger/index.html (the content of the textarea field)?

I tested the stuffs on Iphone11 / IOS14.4 and it works (Safari, Chrome, with and without WebGL2 enabled).

Best,
Xavier

[bureauberg]

Hi Xavier,

Thanks for your very, quick reply!

Are you sure the camera is not used by other apps?

Yes. I tested the web app with 5 different iPhones (iPhone 7plus, iPhone 8, iPhone plus, iPhone 11 and iPhone 12)
At all of them, I first closed all apps and cleared the browser cache. The very same issue occurred on all the above iPhones, running the latest Safari, Chrome and Firefox browser. So it looks like it is due to an update of iOS (to 14.4).

It used to work flawlessly on all these iPhones before and it still does work great on Android (i.e. Galaxy S10, Chrome) and desktop (W10, W8.1, even on W8 and W7).

Which jeeliz FaceFilter demo are you trying?

I didn't check the FaceFilter demo's yet on the above iPhones, I'll do that as well, just like the Boilerplate you mentioned.

However, I'm wondering though if we need to update the Jeeliz lib to fix this issue?

Please forgive me that I'm not a developer myself, but a creative strategist and therefore my questions might not be appropriate here on Github. If so, please let me know.

Some details about our web app:
We developed a web application and took the Tiger demo as an example of what we would like to accomplish.

The web application is a personality test to find out what kind of 'Dragon' you are (a Red, Blue, Green or Yellow Dragon).
After answering 24 questions, you could approve to use the camera to see yourself with a Dragon FaceMask, matching the outcome of your personality test. Then you can take a screenshot and share it, along with your personal landing page.

In case you'd like to have a look, I can send you some test links through email.
Currently, it is still at staging so not yet at the production server. At staging, we have a test mode of only 4 questions.

Thanks again and best regards,
Jacco van den Berg

[xavierjs]

Hi,

Thank you for these details.
You need to update FaceFilter before testing, I have fixed a couple of bugs these later weeks. And I regularly fix new bugs.
I also need a log of https://jeeliz.com/demos/WebGLCoreLogger/index.html
Otherwise, I can do nothing to fix it.

Once your web app is published, feel free to add a link on the Third Party section of the README of this repository: https://github.com/jeeliz/jeelizFaceFilter#third-party

Best regards,
Xavier

[bureauberg]

Hi Xavier,

Thanks for your reply. We have updated FaceFilter and now the camera feed works again.
However, now the camera feed is upside down, just as described in issue 197, 200 and 208.
See the screenshot:

I also noticed that the Tiger demo shows the feed correctly but the face behind the mask, is upside down as well. Please note my eyes and eyebrows in the screenshot:

Any suggestions on how to fix?

See the attachments for the WebGLCoreLogger at iOS14.4, Android 11 and W10, all using Chrome after we updated FaceFilter.

Logger-Safari-iOS14_4.txt

logger-Chrome-Android-GalaxyS10.txt

Logger-W10-Chrome88.txt

Thanks again,

Jacco

[xavierjs]

Hi,

Thank you for the logs.
You need to apply the videoTransformMat2 matrix to put the video in the right position and scale for the latest releases of FaceFilter (see #200)
It will also put the video in the correct flip mode.

This commit is fixing the bug in the werewolf demo: bf6bfaf
The modifications you have to do are likely very similar.

Best,
Xavier

[bureauberg]

Thanks for the feedback, that was helpful. :)

The only thing left now, is that the head behind the mask is still upside down. This is the very same as I already noticed in your demo of the tiger, see my former post above and screenshot if the tiger demo.
Do you have any fix available to this?

Thanks,
Jacco

[bureauberg]

Hi Xavier,

Just to be sure; do you mean by your last reply that the face showing upside-down behind the mask could be fixed by the commit of the werewolf demo as well? Or by #200?

I might have misunderstood your reply partly because the Tiger demo shows the face behind the mask upside-down while the rest of the camera stream is normal.

Thanks again for your suggestions and your efforts on the Jeeliz lib!

Best regards,
Jacco

[xavierjs]

Hi @bureauberg

You need to apply the videoTransform Matrix otherwise the video will be upside down.
Even if your video is normal.

Indeed, when transferred to a WebGL texture, an HTML element like a video, an image, or a canvas is flipped down along the vertical axis (Y). There is an option to flip the pixels along theY axis to avoid this ( gl.pixelStorei(gl.UNPACK_FLIP_Y_WEBGL, true) ) and it was implemented in the previous versions of Jeeliz FaceFilter, but it is not optimized and there were some compatibility issues with some Chinese low-cost mobile phones.

So now the video texture is flipped down. I provide a transform matrix to flip it up, scale it according to the aspect ratio of the canvas and also rotate it (if a video rotation is required).

From a user perspective, you just need to get the matrix from Jeeliz FaceFilter initialization, and apply it to UV video coordinates in the shaders. In #200 I did it for the werewolf demo.

This change was still not applied properly to the tiger mask demo, I am doing this right now :)

Best regards,
Xavier

[xavierjs]

The tiger mask should be fixed now :)

[bureauberg]

Hi Xavier,

Thanks a lot for your insights, guidance and fix of the demo. It gave us a better insight into why it didn't work anymore at production and how to fix it, which has been done successfully.

Thanks again and keep up the great work with Jeeliz!

Best regards,
Jacco

[xavierjs]

Hi @bureauberg

You are welcome
I would be glad to add a link to your face filter web app (s) from the readme of this repo ( https://github.com/jeeliz/jeelizFaceFilter#third-party ). Feel free to submit a pull request to add it, or to post the link here.

Best regards,
Xavier

[bureauberg]

Thanks, Xavier.
One final question; I noticed that the demos, like the werewolf, don't seem to work on Firefox.

I've tested it with

• W10, FF86.0, desktop
• Android 11, FF 86.0

The issue is the same at both; after giving approval to the browser to use the camera, no cam feed shows up.
See the attachments for both issues.

Do you know of any fix for this or is Firefox not supported (yet) on W10 nor Android?

Thanks a lot!
Jacco