基于增强现实的多线索远程工业协同系统

doi:10.11871/jfdc.issn.2096-742X.2023.06.013

数据与计算发展前沿 ›› 2023, Vol. 5 ›› Issue (6): 138-152.

CSTR: 32002.14.jfdc.CN10-1649/TP.2023.06.013

doi: 10.11871/jfdc.issn.2096-742X.2023.06.013

基于增强现实的多线索远程工业协同系统

赵青¹(),张越^1,²,张崇斌³,田东^1,²,崔文凯⁴,单桂华^1,^2,^*()

1.中国科学院计算机网络信息中心，北京 100083
2.中国科学院大学，北京 100049
3.中铁工程设计咨询集团有限公司，北京 100055
4.中铁上海工程局集团第七工程有限公司，陕西西安 710016

收稿日期:2023-06-15 出版日期:2023-12-20 发布日期:2023-12-25
通讯作者: *单桂华（E-mail: sgh@sccas.cn）
作者简介:赵青，中国科学院计算机网络信息中心，高级工程师，中国计算机学会会员，硕士，主要研究方向：人机交互技术、大数据智能交互与混合现实。
本文承担的工作为：文章总体框架把握，论文撰写。
ZHAO Qing is an senior engineer in Computer Network Information Center, Chinese Academy of Science. CCF member. Her research interests include human-computer interaction techniques, Big data intelligent interaction and mixed reality.
In this paper, she undertakes the following tasks: the overall framework grasping of the article and paper writing.
E-mail: zhaoqing@sccas.cn|单桂华，中国科学院计算机网络信息中心，研究员，博士，主要研究方向：大数据可视化与可视分析、智能交互。
本文承担的工作为：文章总体方向把握。
SHAN Guihua is a professor in Computer Network Information Center, Chinese Academy of Science. Her research interests include big data visualization and visual analysis, intelligent interaction.
In this paper, she undertakes the following tasks: the overall direction grasping of the article.
E-mail: sgh@sccas.cn
基金资助:
中国国家铁路集团有限公司重大科研开发计划项目(K2023G003);中国中铁科研开发计划项目(2020-重大-07);中国中铁科研开发计划项目(2021-重大-05-01);中国中铁科研开发计划项目(2021-重大-10);中铁工程设计咨询集团科研开发计划项目(研2020-12);中铁工程设计咨询集团科研开发计划项目(研2022-5);中铁工程设计咨询集团科研开发计划项目(研2023-5)

Augmented Reality-Based Multi-Cues Remote Industrial Collaboration System

ZHAO Qing¹(),ZHANG Yue^1,²,ZHANG Chongbin³,TIAN Dong^1,²,CUI Wenkai⁴,SHAN Guihua^1,^2,^*()

1. Computer Network Information Center, Chinese Academy of Sciences, Beijing 100083, China
2. University of Chinese Academy of Sciences, Beijing 100049, China
3. China Railway Engineering Design and Consulting Group Co., Ltd, Beijing 100055, China
4. The Seventh Engineering Co., LTD., China Railway Shanghai Engineering Bureau Group, Xi'an, Shanxi 710016, China

Received:2023-06-15 Online:2023-12-20 Published:2023-12-25

摘要/Abstract

摘要：

【目的】 为解决工业传统巡检方式因工作效率低下、巡检不到位造成人员伤亡和财产损失等问题，提出一种基于增强现实技术的远程工业协同系统。【方法】 该系统设计了双向线索的交互模式，在远程专家和本地工作人员之间创建丰富多样的线索和双向注释来增强双方协作的效率，更高效地完成远程协作任务。为保证专家和工作人员在各自场景中手绘的二维线索信息实时映射为三维线索信息，采用一种基于深度相机焦距的线索信息的扩散虚实算法，该算法可实时获取手势绘制信息，工作人员的目光注视可实时呈现在专家端。为验证该系统的可用性与优势，选择在真实复杂的场景下模拟了远程异地协同巡检任务和设备操控任务，并设计完备的实验和用户问卷调研。【结果】 对比实验结果，表明该系统的新型交互模式具有更高的用户认可度，相比于传统纸质维护说明、音视频电话指导可大幅降低工作人员任务负荷，并提高任务完成效率。【结论】 最后研究了多种形式下用户认知的主观反馈，提出在工业场景下引导用户行为的建议。

关键词: 增强现实, 多线索, 远程协同, 可视分析, 人机交互

Abstract:

[Objective] In order to solve the problems of casualties and property losses caused by low work efficiency and inadequate inspection in the traditional industrial inspection, this paper proposes a remote industrial collaboration system based on augmented reality technology. [Methods] The system enables remote experts and local workers to interact with rich bidirectional clues to complete remote collaboration tasks. In order to ensure that 2D clues drawn by experts and local workers in their respective scenes are mapped to 3D clues in real time, a diffusion virtual reality algorithm based on depth of camera focal length is adopted, which can obtain gesture rendering information in real time. And in order to verify the usability and advantages of the system, this paper simulates remote inspection tasks and equipment control tasks, and then designs comparative experiment and complete user surveys. [Results] The experimental results demonstrate the effectiveness of the augmented reality remote industrial collaboration system in reducing workers' workload. Additionally, it enhances the social presence of both remote parties and improves task efficiency compared to traditional audio and video conferences. Furthermore, this paper utilizesd the outcomes of usability tests to validate the authors' suggested ideas and implementesd improvements for future application scenarios. [Conclusions] Finally, this paper also studiesd the subjective feedback of user cognition in various forms, and puts forward suggestions to guide user behavior in industrial scenarios.

Key words: augmented reality, multiple clues, remote collaboration, visual analysis, human-computer interaction

赵青, 张越, 张崇斌, 田东, 崔文凯, 单桂华. 基于增强现实的多线索远程工业协同系统[J]. 数据与计算发展前沿, 2023, 5(6): 138-152.

ZHAO Qing, ZHANG Yue, ZHANG Chongbin, TIAN Dong, CUI Wenkai, SHAN Guihua. Augmented Reality-Based Multi-Cues Remote Industrial Collaboration System[J]. Frontiers of Data and Computing, 2023, 5(6): 138-152, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2023.06.013.

图/表 11

图1

图2

图3

图4

图5

图6

图7

图8

图9

图10

图11

参考文献 29

[1]	EDER M, HULLA M, MAST F, et al. On the application of Augmented Reality in a learning factory working environment[J/OL]. Procedia Manufacturing, 2020, 45: 7-12.
[2]	OBERMAIR F, ALTHALER J, SEILER U, et al. Maintenance with Augmented Reality Remote Support in Comparison to Paper-Based Instructions: Experiment and Analysis[C]// 2020 IEEE 7th International Conference on Industrial Engineering and Applications (ICIEA). Bangkok, Thailand: IEEE, 2020: 942-947.
[3]	MOURTZIS D, ZOGOPOULOS V, XANTHI F. Augmented reality application to support the assembly of highly customized products and to adapt to production re-scheduling[J]. The International Journal of Advanced Manufacturing Technology, 2019, 105(9): 3899-3910. doi: 10.1007/s00170-019-03941-6
[4]	MARRIOTT K, SCHREIBER F, DWYER T, et al. Immersive Analytics: 11190[M/OL]. Cham: Springer International Publishing, 2018[2022-09-08]. http://link.springer.com/10.1007/978-3-030-01388-2.
[5]	SERENO M, WANG X, BESANCON L, et al. Collaborative Work in Augmented Reality: A Survey[J]. IEEE Transactions on Visualization and Computer Graphics, 2020: 1-1.
[6]	KIM S, LEE G, SAKATA N, et al. Improving co-presence with augmented visual communication cues for sharing experience through video conference[C]// 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Munich, Germany: IEEE, 2014: 83-92.
[7]	Rasmussen T A, Huang W. SceneCam: Using AR to improve Multi-Camera Remote Collaboration[C]// SIGGRAPH Asia 2019 XR. 2019: 36-37.
[8]	JONES B, WITCRAFT A, BATEMAN S, et al. Mechanics of Camera Work in Mobile Video Collaboration[C]// Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. Seoul Republic of Korea: ACM, 2015: 957-966.
[9]	GAO L, BAI H, HE W, et al. Real-time visual representations for mobile mixed reality remote collaboration[C]// SIGGRAPH Asia 2018 Virtual & Augmented Reality on - SA '18. Tokyo, Japan: ACM Press, 2018: 1-2.
[10]	GAUGLITZ S, LEE C, TURK M, et al. Integrating the physical environment into mobile remote collaboration[C]// Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services - MobileHCI '12. San Francisco, California, USA: ACM Press, 2012: 241.
[11]	BAI H, SASIKUMAR P, YANG J, et al. A User Study on Mixed Reality Remote Collaboration with Eye Gaze and Hand Gesture Sharing[C]// Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Honolulu HI USA: ACM, 2020: 1-13.
[12]	GAO L, BAI H, LINDEMAN R, et al. Static local environment capturing and sharing for MR remote collaboration[C]// SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications on - SA '17. Bangkok, Thailand: ACM Press, 2017:1-6.
[13]	TEO T, LEE G A, BILLINGHURST M, et al. 360Drops: Mixed Reality Remote Collaboration using 360 Panoramas within the 3D Scene*[C]// SIGGRAPH Asia 2019 Emerging Technologies. Brisbane QLD Australia: ACM, 2019: 1-2.
[14]	TEO T, LAWRENCE L, LEE G A, et al. Mixed Reality Remote Collaboration Combining 360 Video and 3D Reconstruction[C/OL]// Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Glasgow Scotland Uk: ACM, 2019: 1-14[2022-08-30].
[15]	YANG J, SASIKUMAR P, BAI H, et al. The effects of spatial auditory and visual cues on mixed reality remote collaboration[J]. Journal on Multimodal User Interfaces, 2020, 14(4): 337-352. doi: 10.1007/s12193-020-00331-1
[16]	WANG P, BAI X, BILLINGHURST M, et al. 3DGAM: using 3D gesture and CAD models for training on mixed reality remote collaboration[J]. Multimedia Tools and Applications, 2021, 80(20): 31059-31084. doi: 10.1007/s11042-020-09731-7
[17]	CHANG Y C, WANG H C, CHU H kuo, et al. AlphaRead: Support Unambiguous Referencing in Remote Collaboration with Readable Object Annotation[C]// Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. Portland Oregon USA: ACM, 2017: 2246-2259.
[18]	THORAVI KUMARAVEL B, ANDERSON F, FITZMAURICE G, et al. Loki: Facilitating Remote Instruction of Physical Tasks Using Bi-Directional Mixed-Reality Telepresence[C]// Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. New Orleans LA USA: ACM, 2019: 161-174.
[19]	FUSSELL S R, SETLOCK L D, YANG J, et al. Gestures Over Video Streams to Support Remote Collaboration on Physical Tasks[J]. Human-Computer Interaction, 2004, 19(3): 273-309. doi: 10.1207/s15327051hci1903_3
[20]	TECCHIA F, ALEM L, HUANG W. 3D helping hands: a gesture based MR system for remote collaboration[C]// Proceedings of the 11th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry - VRCAI'12. Singapore, Singapore: ACM Press, 2012: 323.
[21]	BENKO H, ISHAK E W, FEINER S. Collaborative Mixed Reality Visualization of an Archaeological Excavation[C]// Third IEEE and ACM International Symposium on Mixed and Augmented Reality. Arlington, VA, USA: IEEE, 2004: 132-140.
[22]	JANG S, STUERZLINGER W, AMBIKE S, et al. Modeling Cumulative Arm Fatigue in Mid-Air Interaction based on Perceived Exertion and Kinetics of Arm Motion[C]// Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. Denver Colorado USA: ACM, 2017: 3328-3339.
[23]	Piumsomboon T, Dey A, Ens B, et al. The effects of sharing awareness cues in collaborative mixed reality[J]. Frontiers in Robotics and AI, 2019, 6: 5. doi: 10.3389/frobt.2019.00005 pmid: 33501022
[24]	HIGUCH K, YONETANI R, SATO Y. Can Eye Help You?: Effects of Visualizing Eye Fixations on Remote Collaboration Scenarios for Physical Tasks[C]// Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. San Jose California USA: ACM, 2016: 5180-5190.
[25]	PIUMSOMBOON T, DAY A, ENS B, et al. Exploring enhancements for remote mixed reality collaboration[C]// SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications on - SA '17. Bangkok, Thailand: ACM Press, 2017: 1-5.
[26]	Yuan Liao, Zhonghua Wang and Yanfen Luo. The design and implementation of a WebRTC based online video teaching system[C]// 2016 IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC), Xi'an, China, 2016, 137-140.
[27]	LORETO S, ROMANO S P. Real-time communications in the web: Issues, achievements, and ongoing standardization efforts[J]. IEEE Internet Computing, 2012, 16(5): 68-73.
[28]	HARMS C, BIOCCA F. Internal consistency and reliability of the networked minds measure of social presence[C]// Seventh annual international workshop: Presence. Universidad Politecnica de Valencia Valencia, Spain, 2004: 246-251.
[29]	HART S G, STAVELAND L E. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research[M]// Advances in Psychology: 52. Elsevier, 1988: 139-183.

基于增强现实的多线索远程工业协同系统

Augmented Reality-Based Multi-Cues Remote Industrial Collaboration System

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 29

相关文章 4

编辑推荐

Metrics

本文评价

[1]	王海涛,宋丽华. 情景感知：基本概念、关键技术与应用系统[J]. 数据与计算发展前沿, 2022, 4(3): 110-123.
[2]	李玥,杨波,芦旭熠,单桂华. 面向SWF日志事件流数据的可视分析系统[J]. 数据与计算发展前沿, 2021, 3(4): 3-17.
[3]	赵凡,马小东,任芃锟. 网吧记录数据可视分析系统SRDVis[J]. 数据与计算发展前沿, 2021, 3(4): 70-80.
[4]	蒲剑苏,朱正国,邵慧,高博洋,朱焱麟,闫宗楷,向勇. 基于可视化的固态电解质材料机器学习筛选与预测[J]. 数据与计算发展前沿, 2021, 3(4): 18-29.