Advanced
Quality of Experience Experiment Method and Statistical Analysis for 360-degree Video with Sensory Effect
Quality of Experience Experiment Method and Statistical Analysis for 360-degree Video with Sensory Effect
Journal of Broadcast Engineering. 2020. Dec, 25(7): 1063-1072
Copyright © 2020, The Korean Institute of Broadcast and Media Engineers
  • Received : October 21, 2020
  • Accepted : December 14, 2020
  • Published : December 30, 2020
Download
PDF
e-PUB
PubReader
PPT
Export by style
Article
Author
Metrics
Cited by
About the Authors
Hoe-Yong, Jin
Department of Convergent Software, Myongji University
Sang-Kyun, Kim
Department of Convergent Software, Myongji University
goldmunt@gmail.com

Abstract
This paper proposes an experimental method for measuring the quality of experience to measure the influence of the participants' immersion, satisfaction, and presence according to the application of sensory effects to 360-degree video. Participants of the experiment watch 360-degree videos using HMD and receive sensory effects using scent diffusing devices and wind devices. Subsequently, a questionnaire was conducted on the degree of immersion, satisfaction, and present feelings for the video you watched. By analyzing the correlation of the survey results, we found that the provision of sensory effects satisfies the 360-degree video viewing, and the experimental method was appropriate. In addition, using the P.910 method, a result was derived that was not suitable for measuring the quality of the immersion and presence of 360-degree video according to the provision of sensory effects.
Keywords
I. Introduction
With the recent advances in technology including 5G, virtual reality (VR) technology has begun to attract attention. In addition, various sensory devices such as fragrance, wind, and motion chairs used in 4D movie theaters have begun to be used with media such as general games and broadcasting contents. Experiments were conducted to measure the quality of user experience to find out how these sensory effects affect the media. There was a result of a 77% increase in user's enjoyment in multi-sensory media with haptic and wind effects [1] . There has been an experiment showing that it helps improve the user's senses when watching 360-degree videos accompanied by senses other than audiovisual [2] . In addition, various sensory effects such as wind, vibration, and fragrance were provided and applied to VR-related research [3] . On the other hand, there has been a study that users feel a sense of reality more when tactile and auditory stimuli are given using VR graphics [4] . There has been a study to analyze how the stimuli of the sense of touch and the sense of smell affect the emotion and the sense of existence in a virtual reality environment [5] . The experiment which performed subjective image quality evaluation of 3D TV according to whether MPEG-V based sensory effect using DCR method was provided [6] , and the effect on the sense of presence and satisfaction when providing multiple sensory effects based on 2D image [7] . There is an experiment that measured the influence of immersion and the sense of presence when a fragrance effect was provided in a VR-based graphic virtual environment [8] .
This paper proposes an experimental method that can measure the Quality of Experience (QoE) when sensory effects are provided during a 360-degree video experience, and derives statistical analysis results of the experimental results. In the Quality of Experience experiment on the 360-degree video experience with sensory effects previously conducted, the experiment was conducted by consisting of three groups as a control group providing only the wind effect, only the scent effect, and the scent and wind effect at the same time [9] . The validity of the experiment is verified when the experimental group is composed of a single group to provide sensory effects other than audiovisual. In addition, through correlation analysis, we examine how the DCR method recommended in ITU-T P.910 [10] affects immersion and sense of reality when applied to a 360-degree video experiment with sensory effects.
The structure of this paper is as follows. Section 2 introduces the video quality measurement experimental method recommended through ITU-T P.910, and Section 3 presents the experimental tools and experimental design proposed in this paper. In Section 4, the results of the Quality of Experience measurement according to the presence or absence of sensory effects and the appropriateness of the sensory effect experiment of 360-degree video using the DCR method are determined through correlation analysis of the experimental results followed by the conclusions in Section 5.
Ⅱ. ITU-T P.910 Test Method
ITU-T P.910 is a standard recommendation that defines an experimental method for measuring video quality in multimedia applications. P.910 describes a method of subjective evaluation of video quality for users. Two experimental methods for measuring image quality proposed in P.910 are the Absolute Category Rating (ACR) method and the Degradation Category Rating (DCR) method. The ACR method provides one video at a time with a single stimulus method, and responses to it are independently evaluated and scored. To this end, one image is provided for 10 seconds, and a subjective evaluation of the video quality for 10 seconds is performed immediately after that. The DCR method is the Double Stimulus Method, and a single video is divided into two as a reference image stimulus and one for comparison, and the comparative evaluation score is assigned as one. Figure 1 below is a diagram showing the procedure for the DCR experiment method. After providing a reference image for 10 seconds, rest for 2 seconds, a comparison image for 10 seconds, and evaluation of the video quality for 10 seconds, then another image is displayed. The reason for using a gray screen between experiments is to separate different types of image stimuli and to perform a subjective evaluation of image quality at that time. The evaluation is performed by providing it in the same way.
PPT Slide
Lager Image
DCR experimental method procedure [10]
Ⅲ. Experimental Design
- 1. Experiment environment and participants
Participants in the experiment were 100 undergraduate and graduate students between the ages of 20 and 31. Participants of the experiment consisted of those who had no abnormalities in the senses of sight, smell, and touch, and no abnormalities in the senses due to diseases, smoking, and environmental factors. The experimental environment was composed of only a chair in a room wide enough to remove the factors that would affect the surrounding obstacles during the virtual reality experience and only one desk where the PC and HMD would be installed. Items that emit scent were removed so as not to affect the sense of smell, and to eliminate the influence of natural wind, the experiment was conducted with the window closed and the air conditioner stopped.
- 2. Experimental Equipment
The equipment used in the experiment used an HMD, a scent device, and a wind effect device. HTC’s Vive was mounted on the participants’ head and used as an HMD device that plays 360-degree video, and Aromajoin's aroma shooter and three aroma cartridges (Banana, Chocolate fondue, Coffee) were used as the scent device. Dyson's AM07 model was used as a wind effect device.
- 3. Experiment methods
In the experiment, the difference in subjective immersion and presence of the experiment participant in the 360-degree video according to the sensory effect is provided. In addition, an experiment is conducted to understand how the experiment is performed using the DCR method of ITU-T P.910 on the sense of immersion and presence. The first experiment is to experience a 360-degree video that guides you to look at an object as directed by the subtitle without providing sensory effect, and a video that provides sensory effect when looking at an object as directed by the subtitle. The sensory effects provided in the 360-degree video are three scent effects and wind effects. The three aromas are coffee, banana, and cocoa. This was verified and adopted through previous studies [11] . In the 360-degree video, when the participants recognized each object on the screen viewed through the HMD, the scent device was activated to recognize the fragrance. In order to make the participant look at the object, a caption was used to induce the participant's gaze. In the case of the wind device, when the participant looked at the window and the shaking blind, the experimenter operated the wind device so that the participant could feel the wind.
PPT Slide
Lager Image
Experimental equipment (left: HMD, middle: aroma shooter, right: fan)
Figure 3 is a panoramic image of a 360-degree video, showing coffee on the left, cocoa in the center, and a banana on the right. Also, the blinds of the windows are shaken by the wind in the back.
PPT Slide
Lager Image
Panorama image of 360 degree video
Figure 4 shows the order of the sensory effect experiment, in the order of coffee, cocoa, banana, and blind, to look at without providing sensory effect for 10 seconds, and to experience the scent and wind effects when looking at objects in the same order after taking a break for 1 minute. The video used in both experiments is the same video and the playback time is the same. After the experiment is over, a Quality of Experience questionnaire is prepared to measure the subjective experience that users feel according to the presence or absence of sensory effects. Participants of the experiment watched a video with/without sensory effect and then immediately evaluated the questionnaire to compare the effect on immersion and present.
PPT Slide
Lager Image
The experiment sequence of influence according to whether the sensory effect is provided.
Figure 5 shows the sequence of the sensory effect experiment applying the P.910 method. The sensory effect experiment using the P.910 method induces the participant's gaze through subtitles, initially looking at an object for 10 seconds without providing a sensory effect, and then showing a gray screen for 2 seconds. After that, it shows a gray screen for 10 seconds and looks at an object for 10 seconds while providing a sensory effect, and then shows a gray screen for 2 seconds. The participant of the experiment conducts a sensory effect experiment using the P.910 method as described above, conducts a Quality of Experience questionnaire for the P.910 method, and ends the experiment.
PPT Slide
Lager Image
The experimental sequence of sensorial effect using the P.910 method
Ⅳ. Experiment Results
- 1. Questionnaire of QoE
The items to be measured in the experiment are as follows. First, the degree of immersion measures the degree of interest and concentration in VR images depending on whether sensory effects are provided. Second, satisfaction measures the degree of enjoyment and interest in a 360-degree video experience. Third, the sense of presence measures the degree to which a VR image feels like the real thing. Finally, the sensitivity to sensory effects measures the degree to which the sensory effect it provides is felt.
The survey contents used in the experiment consist of a total of 36 items, including 9 questions for immersion ( Table 1 ), 7 questions for satisfaction ( Table 2 ), 5 questions for a sense of presence ( Table 3 ), and 7 questions related to sensitivity to sensory effect ( Table 4 ). After the P.910 experiment, it consisted of 5 questions of immersion and presence ( Table 5 ) and 3 questions of satisfaction level by each experiment ( Table 6 ). The first experiment is an experiment that watched a 360-degree video without providing sensory effects, the second experiment is an experiment that watched the video accompanied by sensory effects, and the third experiment means a sensory effect experiment conducted in the P.910 method.
Tables 1 to 4 are evaluation questions for sensory effect experiments. When the participants of the experiment looked at each object using HMD, the effect on the immersion and the sense of presence depending on the presence or absence of the sensory effect was measured on a Likert 5-point scale: (1) Strongly disagree; (2) Disagree; (3) Neither agree nor disagree; (4) Agree; (5) Strongly agree.
Evaluation items of immersion
PPT Slide
Lager Image
Evaluation items of immersion
Evaluation items of satisfaction
PPT Slide
Lager Image
Evaluation items of satisfaction
Evaluation items about the sense of presence
PPT Slide
Lager Image
Evaluation items about the sense of presence
Evaluation items of sensibility
PPT Slide
Lager Image
Evaluation items of sensibility
Table 5 is an evaluation question for the P.910 experiment. The comparison between the P.910 method and the sensory effect experiment conducted previously was evaluated on the Likert 5-point scale: (1) Strongly disagree; (2) Disagree; (3) Neither agree nor disagree; (4) Agree; (5) Strongly agree. Table 6 shows the satisfaction level for each experiment, and the 1st, 2nd, and 3rd experiments are ranked in the order of 1~3 for each question according to the satisfaction level.
Immersion and sense of presence according to the experiment of the P.910 method
PPT Slide
Lager Image
Immersion and sense of presence according to the experiment of the P.910 method
Satisfaction level by experiment
PPT Slide
Lager Image
Satisfaction level by experiment
- 2. Comparison between multi-group and single group experiments
In the previous experiment [1] , a multi-group Quality of Experience experiment divided into a group that did not provide sensory effect, a scent group, a wind group, and a fragrance and wind group was conducted. In this paper, we examine the differences in the quality of experience measurement results when a single group presented with a video that does not provide sensory effects and a video with sensory effects are experienced. The items used in the questionnaire in the previous experiment were Q1 to Q5 in Table 1 , Q10 to Q15 in Table 2 , and Q17 to 21 in Table 3 , for a total of 15 questions. The two sets of Pearson correlation coefficients obtained by correlating the evaluation values of the 15 questions of the scent group, the wind group, and the wind and scent groups with the 15 questions of this experiment were 0.58. The values derived from the single group experiment and the values derived from the multi-group experiment have a linear quantitative relationship. It means that the quality of experience measurement according to the experimental method proposed in this paper is not significantly different from the previous experiment, the result of the experiment through the control composition between multiple groups.
- 3. Statistical analysis for sensorial effect experiment
Correlation analysis was performed to see how immersion, satisfaction, presence, and sensitivity to sensory effects affect the quality of experience results according to sensory effects.
Table 7 is a chart of correlation analysis of immersion, satisfaction, presence, and sensitivity to sensory effects according to sensory effects. The relationship between the two sets means the correlation between the level of immersion, satisfaction, presence, and sensitivity to sensory effects evaluated on the Likert 5-point scale and the preference ranking according to the type of experiment. The higher the negative correlation result of the Pearson correlation coefficient, the higher the preference for the experiment. It can be seen that when both immersion, satisfaction, and presence are accompanied by sensory effects when viewing a 360-degree video, Pearson's correlation coefficient has a negative linear relationship and has a higher preference than when sensory effects are not provided. Sensitivity to sensory effects also had a strong negative linear relationship, which means that the similarity to reality is high when 360-degree video and sensory effects are accompanied.
Correlation analysis result of sensorial effect experiment
PPT Slide
Lager Image
Correlation analysis result of sensorial effect experiment
- 4. Statistical analysis for verification of the suitability of the DCR test method
The experiment was conducted to confirm whether the DCR method was suitable for the immersion and sense of presence experiment for 360-degree video with sensory effects.
Table 8 is the result of the survey of Table 6 for 100 people. Each questionnaire item was ranked 1st, 2nd, and 3rd in the order of preference. The survey result showed that the average of the preferences of 100 participants was 1.2 for a 360-degree video that provided sensory effect, 2.11 for a 360-degree video that did not provide sensory effect, and 2.69 for the DCR type experiment. Through this, it can be seen that the sensory effect experiment of 360-degree video using the DCR method is less satisfactory than the 360-degree video experiment proposed in this paper.
Preference survey results by the experimental method
PPT Slide
Lager Image
Preference survey results by the experimental method
Table 9 shows the results of the Quality of Experience questionnaire of test participants for the DCR test method. In the survey, the first experiment was an experiment that did not provide sensory effects when watching a 360-degree video, and the second experiment was an experiment that provided sensory effects when watching a 360-degree video, and the third experiment was a 360-degree using DCR This is a VR video sensory effect experiment. The score for each item was calculated using the “Top and Bottom box scoring” method. For each question, Top2 was selected based on “5 – Strongly agree” and “4 – Agree” based on the Liked 5-point scale, and “1 – Strongly disagree” and “2 – Disagree” were selected as Bottom2. From questions 1 and 2, it can be seen that the DCR-type gray screen interferes with the sense of immersion and presence. From questions 4 and 5, it can be seen that many of the participants experienced a higher sense of immersion and presence in the experimental method proposed in this paper than in the DCR method. In addition, through question 3, it can be seen that the provision of sensory effects when watching a 360-degree video enhances the sense of immersion and presence. As a result, it was found that applying the DCR method to the 360-degree video sensory effect experiment reduces the sense of immersion and presence in the video.
DCR test method survey results
PPT Slide
Lager Image
DCR test method survey results
Table 10 is a result of the evaluation of the sense of the presence of the sensory effect experiment. It can be seen that the sense of the presense felt by the test participant in the five items is (3) neither agree nor disagree or higher. On the other hand, there is a high opinion that items 2 and 5 in the DCR evaluation method that is the sense of presence is impaired. In other words, it can be seen that the DCR-based experimental method degrades providing a sense of presence compared to the proposed sensory effect experiment method.
Survey results of the sense of the presence of sensory effect experiment
PPT Slide
Lager Image
Survey results of the sense of the presence of sensory effect experiment
Ⅴ. Conclusion
In this paper, a QoE experiment was conducted to measure the effect of providing sensory effects when viewing 360-degree videos in a virtual environment using HMD in four categories: immersion, the sense of presence, satisfaction, and sensory sensitivity.
The immersion and presence of the 360-degree video sensory effect experiment for a single group were compared with the previous experiment [1] , in which a control group was formed through multiple groups. As a result of the correlation analysis of the response of the experiment, it was found that there was no significant difference between the two experiments due to a strong linear relationship. In addition, by analyzing immersion, satisfaction, sense of presence, and sensitivity to sensory effects according to the presence or absence of sensory effects, it was found that users' preferences increased when sensory effects were provided. It was concluded that the sensory effect experiment applying the DCR method proposed in P.910 caused a decrease in the sense of immersion and presence due to the gray screen used in the DCR method. The DCR method is not suitable for the 360-degree video Quality of Experience experiment with sensory effects, and the method proposed in this paper is considered to be suitable for the QoE experiment.
This work has supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT)(No.1711107567).
BIO
Hoe-Yong Jin
- 2019. 02 : Dept. of Computer Engineering, Myongji University B.S
- 2019. 03 ~ Current : Dept. of Data Technology, Myongji University, M.S
- Research interests : 4D media, Blockchain, Internet of Things and VR
Sang-Kyun Kim
- 1997. : Computer Science, Univ. of Iowa, B.S.(1991), M.S(1995), Ph.D(1997)
- 1997. 03. ~ 2007. 02. : Professional Researcher, Multimedia Lab. of Samsung Advanced Institute of Technology
- 2007. 03. ~ 2016. 02. : Professor of Computer Engineering, Myongji University
- 2016. 03. ~ Current : Professor of Software Convergent, Myongji University
- Research interests : Digital Content(image, video and audio) analysis and management, 4D media, Blockchain, VR, Internet of Things and multimedia standardization
References
Yuan Zhenhui , Chen Shengyang , Ghinea Gheorghita , Muntean Gabriel-Miro 2014 User Quality of Experience of Mulsemedia Applications ACM Transactions on Multimedia Computing, Communications, and Applications 15
Guedes Álan L. V. , de A. Azevedo Roberto G. , Frossard Pascal , Colcher Sérgio , Junquerira Barbosa Simone Diniz 2019 Subjective Evaluation of 360-degree Sensory Experiences IEEE 21st Internatonal Workshop on Multimedia Signal Processing
Kim Sang-Kyun , Joo Yong Soo 2014 Sensorial Information Extraction and Mapping to Generate Temperature Sensory Effects ETRI Journal 36 (2) 224 - 231    DOI : 10.4218/etrij.14.2113.0065
Cooper Natalia , Milella Ferdinando , Pinto Carlo , Cant Iain , White Mark , Meyer Gerog 2018 The effects of substitute multisensory feedback on task performance and the sense of presence in a virtual reality environment PLoS ONE 13 (2)
Serrano Berenice , Barios Rosa M. , Botella Cristina 2016 Virtual reality and stimulation of touch and smell for inducing relaxation: A randomized controlled trial Computers in Human Behavior 55 (part A) 1 - 8    DOI : 10.1016/j.chb.2015.08.007
Timmerer Christian , Waltl Markus , Rainer Benjamin , Hellwagner Hermann 2012 Assessing the quality of sensory experience for multimedia presentations Signal Processing: Image Communication 27 909 - 916    DOI : 10.1016/j.image.2012.01.016
Jalal Lana , Anedda Matteo , Popescu Vlad , Murroni Maurizio 2018 QoE Assessment for IoT based Multi Sensorial Media Broadcasting IEEE Transactions on Broadcasting 552 - 560
Egan Darragh , Keighrey Conor , Barrett John , Quaio Yuasong , Brennan Sean , Timmerer Christian , Murray Nial 2017 Subjective Evaluation of an Olfaction Enhanced Immersive Virtual Reality Environment Proceeding of the 2nd International Workshop on Multimedia Alternate Realities 15 - 18
Lim Y. 2018 Study on wind and scent effect on 360° VR video content, Master’s thesis of Myongji University Retrieved from
1999 ITU-T P.910, Subjective Video Quality Assessment Methods for Multimedia Applications International Communication Union
Lim Yong-Chul , Kim Sang-Kyun , Lee YoungMi 2017 A Preliminary Study of MPEG-V Scent effects on Virtual Reality Content The Korean Institute of Broadcast and Media Engineers 22 (5) 589 - 599