PocketStudio for mobile

Recording Scenes using AR

PocketStudio’s mobile app allows you to access your 3D scenes and record them (create virtual cameras) using augmented reality (AR). Animated cameras recorded in AR are stored on the cloud and become instantly available for you to use in your projects. You can even live stream your AR camera to your team.

Changes made to the 3D scene in the desktop app will be visible on the phone in real-time and vice versa. The two applications are connected via the server and the data is synchronized automatically and instantaneously between all devices. You can add and move objects around in the desktop application and they will pop up and change position on the phone in real-time.

To install the app see Installing PocketStudio for mobile.

To record a camera using PocketStudio’s augmented reality app, do as follows:

  1. Tap on the icon to launch the application.

  2. Sign in with your usual username and password.

  3. Once you have signed in, the list of your projects will be listed on the screen. Open the project you want to record by tapping on the project’s card.

  4. You will then be displayed with a list of the 3D scenes contained in that project.

  5. Tap on the scene you wish to create a camera for. It is recommended to not draw scenes on the phone that contain more than 1M triangles. See our tips for preparing your scenes for AR for more details.

    About data synchronization: if the data for the scene you wish to work with is not yet on the phone, the app will first download the scene’s data from the server. This can take some time depending on your internet connection speed and scene complexity. Once up to date, the scene will be automatically updated if a user changes it remotely. Data synchronisation is done automatically by PocketStudio for you.

  6. The ARCamera view will show up. Track a horizontal surface by pointing the phone to a horizontal surface while moving it around. See tips for tracking real-world surfaces for further details. As soon as the app has detected a stable surface, a wireframe will be overlaid on top of that real-world surface in the camera view. If you now move the camera around, the grid should feel like it is sticking to the real-world surface.

  1. Tap on the grid to drop the 3D scene into the view. The scene origin will be placed at the touch point. The 3D scene will be rendered on top of the camera view without textures and lights (on mobile devices, these features are not supported at the moment).

    If you open the Scene Editor in PocketStudio for desktop, you will be able to see your AR camera moving live in the viewport. For further details see Locating users in the scene and Looking through their eyes

    You can rotate, scale and move using gesture:

    • Dollying in/out: press your thumb and a finger together on the screen and pull them apart to dolly in (get closer to the scene - make the scene larger). Pinch the two fingers together to dolly out (get further away from the scene, make the scene smaller).
    • Rotate: swip a finger across the screen to rotate the 3D scene around.
    • Translate: press your thumb and a finger together on the screen and move them around to move the scene.

    The 3D scene is not visible? The data of the scene need to be entirely downloaded before being drawn to the screen. If you tap on the screen to drop the 3D scene and that nothing happens, check that the data of the scene have been downloaded in full first. The second reason can be the size of the scene. It might not be visible at first if it is too big or too small.

  2. If your scene contains animations you can play this animation before recording the camera. Tap on the play button in the middle bottom of the screen. The animation will loop. Only skinned/skeletal animations are currently played on the phone (imported via FBX). Alembic caches and objects with animated transforms are not supported yet.

  3. To record the camera, tap on the record button located to the left of the screen. When the camera is being recorded, the record button turns green and blinks . To stop recording tap on the record button again. When the camera is rolling, the animation will loop. You can thus create takes that are longer than the animation itself. As soon as you stop recording, the camera is stored on the cloud and can be used in your project right away.

  4. Go to PocketStudio’s desktop application. Open the scene you have created a camera for. Open the camera popover window. Cameras created with the phone have their names starting with AR and have a small timer icon in the lower left corner on the camera card, indicating the camera is animated.

The PocketStudio mobile app streams the position of the device to the cloud server in real-time. This means that you can view the perspective of any mobile device in the desktop app live.

To view AR cameras live from the desktop application do as follows:

  • In the desktop app, open the same scene that you are viewing in the mobile app.
  • Let’s say Saira is operating her phone to frame a scene. Click on the icon to open the Scene Graph and Users panel. The user list contains all the users editing the scene from either a desktop or a phone. Then double-click on Saira’s name in this users list; this will switch your viewport to Saira’s view. For more information on this feature, see Locating users in the scene and looking through their eyes.

Reviewing AR cameras and using them in a project

To review and use your AR cameras in the project do as follows:

  • From PocketStudio for desktop, open the scene for which you created AR cameras in the Scene Editor.
  • Click on the camera icon on the toolbar to display the camera popover.
  • Cameras created with the mobile app using augmented reality have their names starting with the prefix AR; to indicate that they are animated, a timer icon is displayed in the bottom left corner of their card.
  • To look through an AR camera, double-click on its thumbnail as for any camera.

Smooth your AR camera

If the raw AR camera motion is too shaky you can smooth it out using the Smoothing camera property. The higher the values the smoother the camera path. Set the value to 1000 as a starting point and adjust the value until you get a result you like.

  • Select the AR camera to display the camera properties in the Property Editor panel
  • Increase the value of the Smoothness parameter (located under the Camera Motion (AR)** group) to smooth the motion of the camera.
To smooth the path of a moving camera.
The higher the value (expressed in millisecond) the smoother. Values between 1000 and 5000 give best results.

Exporting cameras

To export cameras produced with the mobile phone see Physical cameras: Exporting cameras. When AR cameras are exported, PocketStudio doesn’t export the raw data but the filtered, smoothed data. To export the raw data set the AR camera Smoothing property back to 0.

Stabilizing the camera: cameras made with “hand-held” phones are shakier than professional cameras which are heavier. You can either use a phone/camera stabilizer while shooting and/or smooth the camera after you have recorded the scene, using the smoothing camera property. See editing and Exporting your AR cameras.

AR tracking instability: when getting too close on an object or when making the scene very large in order to be able to be close to an object, the phone tracking might become unstable: the camera makes random jumps and as result the scene jitters. The problem is somehow similar to the floating point precision error rendering artifacts you eventually get when dealing with scenes that are extremely large (in terms of distances between objects). Scaling up the scene (if you need to get close to a small object for example) can help solve the problem partially: by scaling the scene up you increase the scale in which the phone will be able to record and track small movements.

Tips for tracking real-world surfaces

If you find it difficult to track a real-world surface, try one of the following:

  • Make the real-world scene as bright as you can. When the scene is dark, it is much harder for ARCore to spot and track surfaces. Add more light to the scene.
  • ARCore will have a very hard time tracking uniform surfaces. For example it will be hard for ARCore to track the floor of a room that is completely white. ARCore needs variations of colors and shapes in order to reconstruct its environment in 3D and from there, compute the position of the phone in 3D space. While working in a clean bright environment is a good idea, be sure that your working environment is populated with a variety of objects or patterns the phone can track.
  • The shinier the surfaces the harder the track. Choose matte surface preferentially.

Tips for preparing your 3D scenes for AR

If you experience some stuttering while shooting a scene, the scene is probably too heavy to be rendered in real-time. See our tips for preparing your scenes for AR for info on how to prepare your scenes.