Extended Reality Collaboration (ERC)
Virtual and Mixed Reality System for Collaborative Design and Holographic-Assisted On-site Fabrication
Our ERC system aims to combine design and fabrication functionalities in a collabo-rative virtual environment and enhance communication between two geographically separated stakeholders. Consequently, the system not only converges on- and off-site activities but also integrates the processes of design development and physical fabri-cation into one virtual shared environment.
Master Thesis & Scientific Paper: MAS ETH in Architecture and Digital Fabrication, 2021
Evgenia Angelaki, Foteini Salveridou
Tutors: Daniela Mitterberger, Romana Rust, Lauren Vasey
Duration: 3 months
Devices:
AR : Hololens Device
VR: Oculus Device
Software:
Unity , C#, Python, Rhino, Grasshopper
Communication: ROS (Robot Operating System)
Presented at the Design Modeling Symposium, Berlin, 2022
Abstract:
Most augmented and virtual applications in architecture, engineering, and construction focus on structured and predictable manual activities and routine cases of information exchange such as quality assurance or design review systems. However, collaborative design activities such as negotiation, task specification, and interaction are not yet sufficiently explored. This paper presents a mixed-reality immersive collaboration system that enables bi-directional communication and data exchange between on-site and off-site users, mutually accessing a digital twin.
Extended Reality Collaboration (ERC) allows building site information to inform design decisions and new design iterations to be momentarily visualized and evaluated on-site. Additionally, the system allows the developed design model to be fabricated with holographic instructions. In this paper, we present the concept and workflow of the developed system, as well as its deployment and evaluation through an experimental case study. The outlook questions how such systems could be transferred to current design and building tasks and how such a system could reduce delays, avoid misunderstandings and eventually increase building quality by closing the gap between the digital model and the built architecture.
User scenario showing the on- and off-site scenario with two distinct stakeholders
ERC involves at least two different stakeholders with different expertise that are in different locations; one user is on-site, and the other is off-site.
The on-site user, "MR-User," is equipped with a MR headset, whereas the off-site user, "VR-User," utilizes a virtual-reality (VR) headset. The MR-User represents an expert con-struction worker, craftsperson, or construction site manager. The role of the MR-User is to provide site-specific data, insight knowledge, and instruct manual fabrication. The VR-User represents a stakeholder such as an architect or planner who navigates in a digital twin of the construction site. The role of the VR-User is to request and receive on-site information and feedback on the design to adjust the design according-ly. Furthermore, the VR-User provides different design options and supervises fabri-cation. Both users meet in the virtual space collaborating synchronously.
System Walkthrough
ERC is designed around two distinct phases:
1. Collaborative design phase
2. Augmented fabrication phase.
In phase one, the VR-User (architect) and the MR-User (expert) evaluate the design options collaboratively. The MR-User localizes and cre-ates the digital twin, and then both users can meet in virtual space. The MR-User sees the design options as holographs on-site, while the VR-User sees them in the digital twin of the construction site. The collaborative design phase has two distinct features: 3D sketching and annotating, and collaborative design on-the-fly. Phase two allows the users to plan and fabri-cate the design and has two features: holographic fabrication and fabrication supervision.
The users follow a linear sequence of interactive design and fabrication sessions.
A - Creation of a digital twin
B - Localization and meeting in virtual space
C - 3D sketching and annotating
D - Collaborative design on-the-fly
E - Holographic fabrication
F - Fabrication supervision
System Architecture
The system architecture consists of three main parts: (1) an on-site MR setup with a scanning system, (2) an online server, and (3) an off-site VR setup. The on-site MR setup consists of a laser scanning device (Leica RTC 360) providing high-resolution on-site scans, an MR-headset (Microsoft Hololens2), a lap-top, and a WIFI router. The off-site hardware consists of a VR headset (Oculus Quest 2), a laptop, and a WIFI router.
The software setup is structured as follows. Two autonomous Unity3D applications were developed, one for MR and one for VR. The MR application uses the Mixed reality toolkit (MRTK) and OpenXR library to enable spatial awareness scanning and QR-code detection. The VR application is developed using the OpenXR library. Fur-thermore, Rhinoceros3D, Grasshopper, and Python are used to create algorithmic designs. Rhino.Inside, enables compatibility and bidirectional communication be-tween external Unity processes and Grasshopper. The online communication is based on the Robot Operating System (ROS) (Quigley et al. 2009). The rosbridge package is used to access the publish-and-subscribe architecture of ROS and ROS# for the Uni-ty3D applications.
Interface
2. Collaborative synchronous design
3. Features: Sketching and Annotation
4. Holographic instructions - Fabrication Interface
5. Live feedback and communication during fabrication
Case Study
To validate the feasibility of the proposed method and demonstrate the potential for a concrete fabrication system such as façade panels, we focused on one full-scale exper-imental implementation. For a user-friendly experience, the user interface (UI) design was based on each user's different roles and work packages. We used sticky notes as placeholders to showcase the various and complex types of infor-mation that can be exchanged between two geographically separated users. This in-formation includes the position (P), rotation (a), size (f1), geometry (folding type) (f2), and color of each unit
The total fabrication time was 26 hours, whereas the interactive design was around 1 hour. The final design was fabricated using two MR headsets, and a total of 4000 sticky notes were placed.
The final design was split into distinct fabrication sessions of 60-90 minutes. We used an attractor-based approach for the computational design, which influenced the design depending on its location in space and its distance from physical boundaries. Specifically, the attractor's location (CP) changed the position, rotation, color, size, and folding type of the sticky notes. In our case study, the sticky note's location was projected onto the spatial mesh data (M) scanned by the MR User. This projection resulted in a precise position for each sticky note on the as-built data of the installation site.
During the design phase, the VR-User moved the attractor as an interactive 3D prism in virtual space to control the number of pro-jections. The VR-User could adjust the design parameters collaboratively with the MR-User while the MR-User saw the different results as holographs in-situ. Further-more, the MR-User could interact with the design via sketching to adjust the outline of the design. After agreeing on a final design, the MR-User fabricated the full-scale experimental implementation (see Fig. 11) while the VR-User supervised and in-formed the process.
Our ERC system allowed for an intuitive and real-time design interaction for users in different physical locations. The users had access to a full-scale impression of the architectural model augmented and contextualized by site-specific information. Both users collaboratively designed and fabricated a complex and full-scale architectural installation (see Fig. 12). Furthermore, personalized communication was achieved by creating avatars for all users. Implementing the ERC system and the case study pro-vided us with insights into the hardware and software limitations.
