The Fedora Mobility SIG is working to get Fedora Linux working on mobile device. This includes the PinePhone, a smartphone built with open hardware. Many distros are working to add support for the features that smartphone users expect, including video calls. I was able to use Firefox as a fully-featured Matrix client to make video calls.
As you may know, Matrix is a chat protocol which can be used with various clients, even as a web application. As a web application you can use web features like WebRTC to open audio & video chats with end-to-end encryption (“E2EC” in Matrix terms ).
All apps failing?
As of this writing, no tested application on the normal Fedora aarch64 repo is capable of accessing the video cameras on the PinePhone, besides the specially-crafted Megapixels. Megapixels just takes pictures at the moment, so we had no working video solution.
Firefox makes a good Matrix webclient UI and I knew from earlier tests, that in a WebRTC session audio works fine. Firefox can easily be made a clickable “Matrix” icon on the desktop by creating a matching desktopfile:
[Desktop Entry] Version=1.0 Name=Matrix Web Comment=Directlink GenericName=Matrix Web Exec=env LIBVA_DRIVER_NAME=v4l2_request LIBVA_V4L2_REQUEST_VIDEO_PATH=/dev/video0 LIBVA_V4L2_REQUEST_MEDIA_PATH=/dev/media0 MOZ_ENABLE_WAYLAND=1 firefox-wayland %U https://HOMESERVERNAME/web/ Terminal=false X-MultipleArgs=false Type=Application Icon=/home/pine/images/matrix-logo.png Categories=Network;WebBrowser; MimeType=text/html; Keywords=web;browser;internet; X-Desktop-File-Install-Version=0.26
Unfortunately, the preferred –headless argument crashes current Firefox, so we don’t get the so called kiosk-mode, where the webpage takes the entire available space and removes all controls from the view.
It does not only sound like an accident, it was. To our very great surprise, the PinePhone cameras went on and presented a live YUYV image from the ov5640 camera chip. I have a theory how Firefox did it, but that needs further investigations.
You can mark the 4th of March, 2021 at 11:07 CET as the time and date, that first PinePhone video call (that we know of) was made. \o/
“Houston, we have video!”
The other side of the call looked like this, and it was a surprise too, as we got full colors for the image. This is something Megapixels doesn’t do by default.
The image format YUYV is post processed on the camera chip itself, which gives it a more colorful debrayering in opposite to Megapixels’ preferred way, where post processing is done with dcraw after the image is taken. It also means post-processing is way faster done in hardware.
You may notice that the image is rotated by 90 degrees. This is the native cam image. Firefox simply opened the camera, but was unaware that it had to rotate the image. A minor setback, I’m sure, will be fixed in no time, now that the world knows about it.
This video shows the actual speed things happen on the Pinephone, when it is under heavy load. Hardware acceleration should improve it.
The first reaction on the global PinePhone chat on Matrix was amazing:
aluizioneto: 11:19Very nice!!! That would solve everything for me. Where is my PP CE?? 😃
Marius: 11:11 Breaking News .. Video Chat via Matrix from Pinephone WORKING 😆
A great moment for PinePhone users!
But there is still much to be done here. Firefox on PinePhone does not have hardware acceleration in use yet, so it took all 4 CPU cores on the phone to do that call. It was still not enough processing power, as the audio part was interrupted and lagging behind. At the moment, Mr. Stransky is helping out to fully enable the Allwinner GPU for Firefox. We already have a working libva driver for the Allwinner system-on-a-chip, called “libva-request”.
MPV is already using this lib to decode FullHD MP4 movies on my PinePhone with next to zero frame drops. The amount of power consumption has dropped significantly, too. It’s possible to watch a full-length movie without running out of juice. The PinePhone is using 3.6 Watts when all CPU cores are running, which results in approximately two hours until the battery is drained. With the help of the GPU this is no longer the case, as the power consumption got down to around 2 Watts, doubling the life time of a full battery charge.
When Firefox is using the GPU and we have a better performance, it will be possible to make a videocall and leave the house without a mobile power bank.
We have to thank so many contributors to made this happen, in special the Fedora SIG Mobility team members who are massively testing and updating packages for Pinephone and PineTab.
I hope, that we all can see us in a PinePhone driven smooth Video-conference soon.