Open immersive experiences directly #1171

felipeerias · 2024-01-10T08:16:00Z

This PR implements support for opening immersive experiences directly as soon as the application is launched.

We do this by adding a built-in extension which will activate a particular element in the page.

We need to set the preference dom.vr.require-gesture to false so this script can launch immersive WebXR experiences.

The information required to identify the element to activate is passed as parameters to the Intent:

open_in_immersive : enables automatic immersive mode
open_in_immersive_parent_xpath : the XPath of the parent element (usually an iframe); if null, we assume the root document
open_in_immersive_element_xpath : within that parent, the XPath to identify the element to activate
the Intent must also contain a target URL to open

felipeerias · 2024-01-10T08:19:56Z

This PR can be tested from the command line with:

adb shell am start -n "com.igalia.wolvic/com.igalia.wolvic.VRBrowserActivity" \
    -a android.intent.action.MAIN                                             \
    -c android.intent.category.LAUNCHER                                       \
    -a android.intent.action.VIEW                                             \
    -d "https://heyvr.io/arcade/games/barista-express"                        \
    --ez open_in_immersive true                                               \
    --ez hide_webxr_interstitial true                                         \
    -e open_in_immersive_parent_xpath '//*[@id="game-frame"]'                 \
    -e open_in_immersive_element_xpath '/html/body/a-scene/div[2]/button'

felipeerias · 2024-01-11T02:23:34Z

I have updated the PR so that it uses the XPath of the parent element, which is a more flexible solution.

svillar

I think I told @felipeerias about this in private, but I'll add it here for everybody interested to know. The idea of using an extension is nice, but the problem is that the Chromium backend won't have extensions support initially as it's a lot of work. So provided that we don't have a solution at the moment I'd prioritize one that does not involve extensions.

That said I agree that having an extension that inspects the DOM is likely the best we could do, so I indeed have mixed feelings.

felipeerias · 2024-07-10T08:03:30Z

This implementation might be the best that we can do, but it still has several issues.

First of all, the command to launch Wolvic in immersive mode needs to know the exact element in the page which will open the WebXR experience. If the page changes, it will be necessary to update that command.

Secondly, the approach of using a Web extension to trigger specific elements in the page will not be possible on Chromium, because we don't support extensions there just yet.

Third, in some cases our extension is not able to reach the immersive button because the WebXR experience is inside of an iframe from a different origin, and security rules prevent us from accessing a cross-origin object from the embedding page. It might be possible to work around this issue by passing messages between the content scripts running on the page and on the iframe.

This last problem is particularly painful because it affects HeyVR, a large repository of WebXR games: the pages are in https://heyvr.io but their iframes point towards https://games.heyvr.io.

Finally, some pages might have specific quirks that we have to consider. For example, Moon Rider requires two clicks to open the immersive experience, and Magical Reflections requires the user to accept cookies before anything else.

With all that in mind, the implementation in this PR does work reasonably well. Check out the following video:

ScreenRecording_2024.07.10-16.57.13.mp4

Command to launch A-Painter:

adb shell am start -n "com.igalia.wolvic.dev/com.igalia.wolvic.VRBrowserActivity" \
    -a android.intent.action.VIEW                                                 \
    -c android.intent.category.LAUNCHER                                           \
    -d "https://aframe.io/a-painter/"                                             \
    --ez open_in_immersive true                                                   \
    --ez hide_webxr_interstitial true                                             \
    -e open_in_immersive_element_xpath '\/html\/body\/a-scene\/div[2]\/button'

Command to launch Moon Rider:

adb shell am start -n "com.igalia.wolvic.dev/com.igalia.wolvic.VRBrowserActivity" \
    -a android.intent.action.VIEW                                                 \
    -c android.intent.category.LAUNCHER                                           \
    -d "https://moonrider.xyz/"                                                   \
    --ez open_in_immersive true                                                   \
    --ez hide_webxr_interstitial true                                             \
    -e open_in_immersive_element_xpath '\/\/\*[@id=\"vrButton\"]'

app/src/main/assets/extensions/wolvic_autowebxr/content.js

svillar · 2024-08-07T15:06:40Z

app/src/main/assets/extensions/wolvic_autowebxr/content.js

+    if (targetElement) {
+        logDebug('Target element found, calling click()');
+        targetElement.click();
+        targetElement.click();


There was a case where one click() didn't work. I think that I will remove this and test again different experiences, to check if it is still needed.

So, we still have the two clicks... does it mean it's still required?, I still don't get why

svillar · 2024-08-07T15:09:59Z

app/src/common/shared/com/igalia/wolvic/VRBrowserActivity.java

@@ -128,6 +128,12 @@ public class VRBrowserActivity extends PlatformActivity implements WidgetManager
    public static final String EXTRA_KIOSK = "kiosk";
    private static final long BATTERY_UPDATE_INTERVAL = 60 * 1_000_000_000L; // 60 seconds

+    boolean mOpenInImmersive = false;


I think this should be private

Yes, of course.

app/src/common/shared/com/igalia/wolvic/ui/widgets/Windows.java

felipeerias · 2024-08-14T10:02:59Z

@svillar I have updated the PR following your comments. I also had to allow autoplay when we start in this mode to remove an error with Moon Rider, although for some reason it is still not working properly (it was working fine before).

Reference: https://wiki.mozilla.org/Media/block-autoplay

felipeerias · 2024-08-14T10:26:20Z

Access Mars:

 adb shell am start -n "com.igalia.wolvic/com.igalia.wolvic.VRBrowserActivity" \
    -a android.intent.action.VIEW                                                 \
    -c android.intent.category.LAUNCHER                                           \
    -d "https://accessmars.withgoogle.com/"                                                   \
    --ez open_in_immersive true                                                   \
    --ez hide_webxr_interstitial true                                             \
    -e open_in_immersive_element_xpath '\/html\/body\/div\[1\]\/header\/div\/button'

(Note that depending on your setup you might have to add .dev to the package name)

svillar · 2024-08-14T14:20:12Z

app/src/common/shared/com/igalia/wolvic/VRBrowserActivity.java

+    public static final String EXTRA_OPEN_IN_IMMERSIVE = "open_in_immersive";
+    // Element where a click would be simulated to launch the WebXR experience.
+    public static final String EXTRA_OPEN_IN_IMMERSIVE_PARENT_XPATH = "open_in_immersive_parent_xpath";
+    public static final String EXTRA_OPEN_IN_IMMERSIVE_ELEMENT_XPATH = "open_in_immersive_element_xpath";


The other Intent parameters in that file are public.

I think it's a good idea for documentation purposes, as a way to make clear that these strings are supposed to be used outside of this class.

svillar · 2024-08-14T14:23:14Z

app/src/main/assets/extensions/wolvic_autowebxr/content.js

+    if (targetElement) {
+        logDebug('Target element found, calling click()');
+        targetElement.click();
+        targetElement.click();


So, we still have the two clicks... does it mean it's still required?, I still don't get why

svillar · 2024-08-14T14:26:41Z

app/src/main/assets/extensions/wolvic_autowebxr/content.js

+
+// Limit the number of times that we can try to launch the experience, to avoid an infinite loop.
+var retryCounter = 0;
+const RETRY_LIMIT = 20;


I know this is kind of arbitrary but a maximum 20" retry seems to much to me. Maybe good for poor connections though...? In that case the user would have some other problems. My point is that users won't get any idea of what's going on for a lot of time. The feeling after 4-5" is that nothing works, nobody will wait for 20" for an app to launch.

Yes, but that's a problem in general with this feature. Some experiences just take a long time to load from the network.

In this initial implementation, we show the browser window before launching the immersive experience. Eventually my plan is to go directly from the loading screen to the immersive experience.

svillar · 2024-08-14T14:29:59Z

app/src/main/res/raw/fxr_config.yaml

-  apz.one_touch_pinch.enabled: false
+  apz.one_touch_pinch.enabled: false,
+  # allows scripts to open WebXR immersive sessions
+  dom.vr.require-gesture: false


I feel a bit uneasy having this. Would it be possible not to add this here and manually add it to the runtime settings builder in EngineProvider.kt somehow? We don't need to spend much time on that, but if it's possible to add it just whenever is needed it'd be much better.

One possibility could be to do it in SessionUtils.prepareConfigurationPath(), which copies the configuration file from the assets to the device's filesystem. Basically, the idea would be to append an extra line to the file if needed:

final String REQUIRE_GESTURE_CONFIG = "dom.vr.require-gesture: false"; WidgetManagerDelegate widgetManager = (WidgetManagerDelegate) aContext; if (widgetManager.isOpenInImmersive()) { outputStream.write(REQUIRE_GESTURE_CONFIG.getBytes()); }

Another approach could be to have two files in the assets, and copy one or the other depending on the current configuration. Maybe that would be better?

This PR implements support for opening immersive experiences directly as soon as the application is launched. We do this by adding a built-in extension which will activate a particular element in the page. We need to set the preference dom.vr.require-gesture to false so this script can launch immersive WebXR experiences. Media autoplay is also allowed in this mode. The information required to identify the element to activate is passed as parameters to the Intent: - open_in_immersive - open_in_immersive_parent_xpath - open_in_immersive_element_xpath - a target URL to open

felipeerias requested review from svillar, javifernandez and HollowMan6 January 10, 2024 08:16

felipeerias force-pushed the felipeerias/startInImmersive branch from 15751a8 to 9f956d4 Compare January 11, 2024 02:22

HollowMan6 linked an issue Jan 30, 2024 that may be closed by this pull request

Automatic VR 360 WebXR Open #1211

Open

HollowMan6 mentioned this pull request Jan 30, 2024

Automatic VR 360 WebXR Open #1211

Open

svillar mentioned this pull request Mar 1, 2024

Immersive start page, no Wolvic UI shown #1279

Open

svillar reviewed Mar 6, 2024

View reviewed changes

felipeerias marked this pull request as draft July 3, 2024 08:59

felipeerias force-pushed the felipeerias/startInImmersive branch 2 times, most recently from 09a9140 to 8aecce9 Compare July 10, 2024 07:59

felipeerias requested review from svillar and removed request for HollowMan6 July 10, 2024 08:04

felipeerias marked this pull request as ready for review July 10, 2024 08:05

felipeerias force-pushed the felipeerias/startInImmersive branch from 8aecce9 to 701c442 Compare July 25, 2024 08:42

svillar mentioned this pull request Jul 29, 2024

Auto Enter WebXR #1085

Closed

svillar suggested changes Aug 7, 2024

View reviewed changes

svillar linked an issue Aug 13, 2024 that may be closed by this pull request

Immersive start page, no Wolvic UI shown #1279

Open

felipeerias force-pushed the felipeerias/startInImmersive branch 2 times, most recently from 7aad147 to f0f25fb Compare August 14, 2024 09:56

felipeerias force-pushed the felipeerias/startInImmersive branch from f0f25fb to a2b14df Compare September 3, 2024 07:06

felipeerias requested a review from svillar September 5, 2024 11:37

svillar suggested changes Sep 9, 2024

View reviewed changes

felipeerias force-pushed the felipeerias/startInImmersive branch from a2b14df to 77692ff Compare September 11, 2024 07:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Open immersive experiences directly #1171

Open immersive experiences directly #1171

felipeerias commented Jan 10, 2024 •

edited

Loading

felipeerias commented Jan 10, 2024 •

edited

Loading

felipeerias commented Jan 11, 2024

svillar left a comment

felipeerias commented Jul 10, 2024

svillar Aug 7, 2024

felipeerias Aug 7, 2024

svillar Aug 14, 2024

svillar Aug 7, 2024

felipeerias Aug 7, 2024

felipeerias commented Aug 14, 2024

felipeerias commented Aug 14, 2024

svillar Aug 14, 2024

felipeerias Sep 11, 2024

svillar Aug 14, 2024

svillar Aug 14, 2024

felipeerias Sep 11, 2024

svillar Aug 14, 2024

felipeerias Sep 11, 2024

Open immersive experiences directly #1171

Are you sure you want to change the base?

Open immersive experiences directly #1171

Conversation

felipeerias commented Jan 10, 2024 • edited Loading

felipeerias commented Jan 10, 2024 • edited Loading

felipeerias commented Jan 11, 2024

svillar left a comment

Choose a reason for hiding this comment

felipeerias commented Jul 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

felipeerias commented Aug 14, 2024

felipeerias commented Aug 14, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

felipeerias commented Jan 10, 2024 •

edited

Loading

felipeerias commented Jan 10, 2024 •

edited

Loading