PocketStudio for mobile
Recording Scenes using Augmented Reality
PocketStudio native mobile app allows you to access all your projects’ 3D scenes and record them (create virtual cameras) using augmented reality. It is fun, easy and useful!
By trade, filmmakers master the art of cinematography, which includes choosing camera angles to record scenes. However, directors have generally had to rely on animators to control virtual cameras, because 3D packages are vast and complex to dip into. Using a mobile phone to record 3D scenes finally removes this limitation.
How does it work?
Project data in PocketStudio is centralized in the cloud. This means projects built in PocketStudio can be accessed and edited from anywhere in the world, and from any device, whether it be a computer or a mobile phone. All you need is the PocketStudio app. PocketStudio’s proprietary collaborative engine allows you to access and edit the project’s data seamlessly, wherever you are.
PocketStudio’s cloud infrastructure can thus be leveraged on different devices to take advantage of their own specific features, such as augmented reality on mobile devices.
All cameras recorded in AR are stored on the cloud as well and are instantly available for you to use in your projects. The process is entirely transparent to users thanks to PocketStudio’s built-in real-time collaborative editing technology.
To install the app see Installing PocketStudio for mobile.
To record a camera using PocketStudio’s augmented reality app, do as follows:
Edits made to the project from the phone will be visible in the desktop application in real-time and vice versa. The two applications are connected to the server and the project’s data are synchronized automatically and instantaneously between all devices. See Editing Scenes on Mobile Devices for further details.
Tap on the icon to launch the application.
Sign in with your usual username and password.
Once you have signed in, the list of your projects will be listed on the screen. Open the project you want to record by tapping on the project card.
You will then be displayed with a list of the 3D scenes contained in that project.
Tap on the scene you wish to create a camera for. It is recommended to not draw scenes on the phone that contain more than 1M triangles. See our tips for preparing your scenes for AR for more details.
About data synchronization: if the data for the scene you wish to work with is not yet on the phone, the app will first download the scene’s data from the server. This can take some time depending on your internet connection speed and scene complexity. Once up to date, the scene will be automatically updated if a user changes it remotely. Data synchronisation happens automatically for you.
The ARCamera view will show up. Track a horizontal surface by pointing the phone to a horizontal surface while moving it around. See tips for tracking real-world surfaces for further details. As soon as the app has detected a stable surface, a wireframe will be overlaid on top of that real-world surface in the camera view. If you now move the camera around, the grid should feel like it is sticking to the real-world surface. Does it work?
Tap on the grid to drop the 3D scene into the view. The scene origin will be placed at the touch point. The 3D scene will be rendered on top of the camera view without textures and lights (on mobile devices, these features are not supported at the moment).
If you open the Scene Editor in PocketStudio for desktop, you will be able to see your AR camera moving live in the viewport. For further details see Live AR cameras in PocketStudio desktop
You can rotate, scale and move using gesture:
- Dollying in/out: press your thumb and a finger together on the screen and pull them apart to dolly in (get closer to the scene - make the scene larger). Pinch the two fingers together to dolly out (get further away from the scene, make the scene smaller).
- Rotate: swip a finger across the screen to rotate the 3D scene around.
- Translate: press your thumb and a finger together on the screen and move them around to move the scene.
The 3D scene is not visible? The data of the scene need to be entirely downloaded before being drawn to the screen. If you tap on the screen to drop the 3D scene and that nothing happens, check that the data of the scene have been downloaded in full first. The second reason can be the size of the scene. It might not be visible at first if it is too big or too small.
If your scene contains animations (see below to see what type of animation is supported by the app) you can play this animation before recording the camera. See animation support in AR for further details. Tap on the play button in the middle bottom of the screen. The animation will loop.
To record the camera, tap on the record buttonlocated to the left of the screen. When the camera is being recorded, the record button turns green and blinks . To stop recording tap on the record button again. When the camera is rolling, the animation will loop. You can thus create takes that are longer than the animation itself. As soon as you stop recording, the camera is stored on the cloud and can be used in your project right away.
Go to PocketStudio’s desktop application. Open the scene you have created a camera for. Open the camera popover window. Cameras created with the phone have their names starting with AR and have a small timer iconin the lower left corner on the camera card, indicating the camera is animated.
Stabilizing the camera: cameras made with “hand-held” phones are shakier than professional cameras which are heavier. You can either use a phone/camera stabilizer while shooting and/or smooth the camera after you have recorded the scene, using the smoothing camera property. See editing and Exporting your AR cameras.
AR tracking instability: when getting too close on an object or when making the scene very large in order to be able to be close to an object, the phone tracking might become unstable: the camera makes random jumps and as result the scene jitters. The problem is somehow similar to the floating point precision error rendering artifacts you eventually get when dealing with scenes that are extremely large (in terms of distances between objects). Scaling up the scene (if you need to get close to a small object for example) can help solve the problem partially: by scaling the scene up you increase the scale in which the phone will be able to record and track small movements.
Tips for tracking real-world surfaces
If you find it difficult to track a real-world surface, try one of the following:
- Make the real-world scene as bright as you can. When the scene is dark, it is much harder for ARCore to spot and track surfaces. Add more light to the scene.
- ARCore will have a very hard time tracking uniform surfaces. For example it will be hard for ARCore to track the floor of a room that is completely white. ARCore needs variations of colors and shapes in order to reconstruct its environment in 3D and from there, compute the position of the phone in 3D space. While working in a clean bright environment is a good idea, be sure that your working environment is populated with a variety of objects or patterns the phone can track.
- The shinier the surfaces the harder the track. Choose matte surface preferentially.
Tips for preparing your 3D scenes for AR
If you experience some stuttering while shooting a scene, the scene is probably too heavy to be rendered in real-time. See our tips for preparing your scenes for AR for info on how to prepare your scenes.
At this point in time, the mobile app only supports skinned animations (bones + skinning weight).
The PocketStudio mobile app streams the position of your device to the cloud server in real-time. This means that you can view the perspective of your mobile device in the desktop app live.
To view AR cameras live from the desktop application you can do the following:
- In the desktop app, open the same scene that you are viewing in the mobile app.
- One of the cameras in the scene will be moving freely as you move the phone around. This is the live AR camera.
- Open the camera panel by clicking on the camera icon on the toolbar. In the camera list, look for the camera named
Android_<username>. To look through the camera, double click on its thumbnail.
Reviewing AR cameras and using them in a project
To review and use your AR cameras in the project do as follows:
- From PocketStudio for desktop, open the scene for which you created AR cameras in the Scene Editor.
- Click on the camera icon on the toolbar to display the camera popover.
- Cameras created with the mobile app using augmented reality have their names starting with the prefix AR; to indicate that they are animated, a timer icon is displayed in the bottom left corner of their card.
- To look through an AR camera, double-click on its thumbnail as for any camera.
Smooth your AR camera
If the raw AR camera motion is too shaky you can smooth it out using the Smoothing camera property. The higher the values the smoother the camera path. Set the value to 1000 as a starting point and adjust the value until you get a result you like.
To export cameras produced with the mobile phone see Physical cameras: Exporting cameras. When AR cameras are exported, PocketStudio doesn’t export the raw data but the filtered, smoothed data. To export the raw data set the AR camera Smoothing property back to 0.