The purpose of this study is to design an authoring tool and a corresponding device for an olfactory display that can augment the immersion and reality in broadcasting services. The developed authoring tool allows an olfactory display to be properly synchronized with the existing video service by applying the standardized format using ISO/IEC 23005 (MPEG-V) and the corresponding developed scent display device. To propose the proper data format for the olfactory display, we have analyzed both the multimodal combination and the cross-modality related to the olfactory display. From the results of the analysis, we derived a set of olfactory parameters for the olfactory display that are related to emotion. The analyzed parameters related to emotion in an olfactory display are synchronization, scent intensity, scent persistence, and hedonic tone. These parameters should be controlled so that the olfactory display can be in harmony with the existing media to augment emotion. In addition, we developed a scent display device that can generate many kinds of scents and that satisfies design conditions for olfactory parameters that are for use with broadcasting services. Smell output is one of the most exciting and explorative research areas in the field of human–computer interaction and is used in virtual reality, augmented reality, and computer-mediated communication [1] – [4] . The olfactory sense stimulates the human memory and helps build up the human senses in a positive manner [5] – [7] . However, the lack of an effective scent display device to generate various odors has hampered numerous practical attempts to combine smell with other media. Recently, there has been a great increase in interest in how to manufacture a scent display device, and indeed, a commercial device has recently appeared on the market [1] – [8] . Many researchers have tried to apply olfactory displays to existing multimedia services. In particular, Prof. Takamichi Nakamoto of the Tokyo Institute of Technology attempted to produce controlled smells through valves using an inkjet-based technology, and in fact, he has developed a technology to blend diverse scents by combining dozens of scent ingredients. In addition, he has developed a new odor recorder using an electronic nose that is based on quartz crystal microbalance (QCM) sensor arrays to measure an odor and convert it into digital information. In this way, he has been able to finally represent odors in a more tangible way [9] – [10] . His odor recording technology was applied to virtual reality to show cooking processes and may be extended to animation and video clip applications. By not accounting for the olfactory modality that comes from the nature of a smell, despite this effort, his method can’t spatially and temporally match up with existing multimedia information, such as visual and auditory information. The purpose of this study is to design an authoring tool and corresponding scent display device that can augment immersion and reality by using olfactory display in realistic broadcasting services. Realistic broadcasting is defined as a broadcasting service system that uses multimodal immersive media to provide its end users with a greater sense of realism. A number of researchers have developed core technologies for realistic broadcasting. As a result, 3D display, 3D sound, and haptic or touch feedback technologies have been competitively developed by many research groups [11] – [15] . However, despite all of these technological advances, realistic broadcasting using olfactory display has not yet been implemented. Recently, ISO/IEC 23005 (MPEG-V) [16] – [19] has defined metadata for interactions that take place between virtual worlds and real worlds through the use of a scent display device, but it only standardized a scent’s intensity and duration as scent parameters. However, many studies have shown that the control of a scent’s intensity and duration alone is not enough to produce a successful outcome [20] – [28] . For a successful application of olfactory displays in realistic broadcasting, we have analyzed both the modality combination and the cross modality between olfactory displays and video content [22] – [28] , and we derived a set of scent parameters for olfactory displays that are related to human emotion. The first parameter is the synchronization of odors between different media objects that need to be presented together simultaneously. If the release of odors is not properly synchronized with an existing media, then, unfortunately, this may reduce the emotional effects of such odors. In addition, we considered scent intensity, scent persistence, and hedonic tone as other parameters. These parameters should be controlled so that they can be in harmony with an existing media so as to augment emotion. We defined and designed the following items on the basis of these parameters: In addition, we developed an olfactory media broadcasting system based on MPEG-V for emotion augmentation of video services. The developed device and authoring tool allows an olfactory display to be properly synchronized and emotionally harmonized with existing video services. Along with the sensation of the 3D film industry, the development of emotion-media broadcasting has received a lot of public attention. Emotion-media broadcasting, in general, involves the addition of sensory effects to 3D films, immersing audiences more deeply into a movie viewing experience. Along with the two human senses of sight and hearing, such sensory effects as wind, vibration, and scent can stimulate other human senses, such as the tactile and olfactory senses. This paper focuses more on olfactory senses and develops an olfactory media broadcasting system that makes use of a scent display device. To synchronize scent effects with video content, an olfactory media authoring tool, based on MPEG-V, has been produced experimentally. Figure 1 shows an olfactory media broadcasting system comprising the following three parts: an emotion-media authoring tool, an emotion-media transmission server, and an emotion-media terminal. The emotion-media transmission server encapsulates scent effect metadata through a digital storage media command and control (DSM-CC) object carousel or section, and multiplexes with an MPEG-2 system–based video stream [29] – [30] . In this process, the transport stream ID for the scent effect metadata should be defined, and the program specific information indicating the relation between the TV program and the metadata should be transmitted through the broadcasting channel. In addition, synchronization of the scent-effect presentation between the scent effect and the video content should be considered. For this, trigger information on the progress time of a TV program should be transmitted periodically, and this may need to be sent through a bidirectional channel. The emotion-media terminal receives an MPEG-2 AV stream multiplexed with emotion effect metadata through either a broadcasting or bidirectional channel. It should be capable of controlling various kinds of emotion effect devices and of directly gathering user preferences from the emotion effect metadata. For this, a core engine module based on MPEG-V is being implemented to represent various emotion effects, such as scent, wind, vibration, and temperature information. The emotion-media authoring tool synchronizes the scent effect metadata with the TV program, based on MPEG-V. In addition, it has a control interface that can be used to control the olfactory display to augment the emotions of users — an augmentation that is based on the system’s internal authoring guide that takes into consideration the human factors of olfactory. The above-mentioned authoring guideline defines the minimum interval display time for a scent; lists and classifies the primary scents used in a broadcasting environment; gives a psychological analysis of olfactory based on human factors; [29] and so on. In this paper, we will focus on the design and implementation of a scent display device that can augment immersion and reality through the use of an emotion-media authoring tool. There have been many researches on the integration of olfactory media with other media, such as video and audio, to produce a richer sensory experience in multimedia applications. These researches have helped serve to highlight the fact that the most important factors for olfactory media are smell intensity, duration, and synchronization with other media [22] – [25] . For a more successful application of olfactory display in realistic broadcasting, we have analyzed the modality combination and cross modality between olfactory displays and video content [23] – [28] and derived a set of scent parameters and design conditions for an olfactory display device for human emotions. Olfactory perception is much slower than visual perception, sound perception, and haptic feedback. Oloffson reported that odor perception was carried out at a slow speed of around 1,000 ms after sniff onset and that odor valence (feeling whether the odor was pleasant or unpleasant) was carried out at a slower speed of around 1,200 ms [31] . In particular, if you consider the near-simultaneous use of scent and haptic displays, then these effects will produce an inevitable disharmony; thus, greater consideration should be given to their effects when attempting to use them in combination with each other. Sense of touch is a more direct stimulus — one that does not leave lingering feelings. It is received as quickly as visual stimuli, and it is more strongly and directly connected with emotion control than is visual, auditory, or olfactory stimuli. For the proposed olfactory display, which makes use of the proposed scent display device, a new issue has arisen — that being that we need to consider how best to deliver a smell to the nose of a user after the command for scent display has been given. In this paper, we will define the time that it takes for this to happen as the “delivery time” or “secondary order delay time.” This factor will inevitably give rise to the issue of total odor perception time, since the delivery time of a scent display device will need to be added to the odor perception time of 1,200 ms after sniff onset. Hence, the delivery time of a scent display device becomes an important factor in the design of an olfactory display [1] . Despite the slow response time, odors can improve association by triggering the memories of our past. R.S. Herz [32] showed that odors evoke more emotional memories than other stimuli, without a person necessarily having to consciously be able to identify the aroma. R.S. Herz and G.C. Cupchik [33] additionally showed that if the cue for recall was hedonically congruent with the item to be remembered (for example, a painting), then the memory relating to the original emotional experience was enhanced. Therefore, it is better for olfactory display to act as a background media that can help complement those feelings felt through auditory and visual stimuli. In addition, an odor is intrinsically an ambient media and peripheral medium, and as such, it is appropriate for displaying ambient information [1] . Based on an earlier research finding, the modalities of an olfactory display should include perception ability, perception speed, associative memory, and ambient media. The perception ability of odors involves the following four basic tasks: intensity, persistence, qualitative description (character description), and hedonic tone [34] – [35] . The perception speed depends on the delivery time (velocity) of the scent display device. As we already know, the associative ability of odors is significantly associated with their intensity, persistence, and hedonic tone. The ability of an odor to act as an ambient media is linked to its intensity and persistence. Table 1 shows the modalities and their respective olfactory parameters for an odor display. When information presented to two modalities differs, what will happen in human perception and emotion? The conflict of modalities between smell and other senses can have influence on one's judgment or decisions [26] – [27] . Synesthesia by olfactory. A condition in which one type of stimulation evokes the sensation of another, as when the smelling of a scent produces the visualization of a scene, is known as synesthesia. Commonly, synesthesia cannot be treated as information, but it is related to scent intensity, hedonic tone, and familiarity [36] . A person’s familiarity with an odor can be associated with a qualitative description. Discord between modalities due to olfactory display. Discord may exist between modalities. Commonly, in movies, a visual stimulus lasts longer than an auditory stimulus, because they have different degrees of persistency, even if both stimuli were given at the same time, whereby they would cause modality discord. As odors have very strong persistency, the problem of discord could be a critical issue. Disharmony between modalities due to olfactory display. A visual image of a lemon given with a sweet odor may conflict with a person’s memory of a sour taste. Similarly, a pleasant forest odor given off during an urgent battle scene in a video game would conflict with a person’s feelings of tension. In these cases, the two modalities create a negative effect, where one sensation contrasts and interferes with another; thus, the respective modalities cause discordance. Such discordance is related to hedonic tone, intensity, and qualitative description [37] . As indicated in the above, an inappropriate scent display can cause serious cross modalities. Table 2 outlines the olfactory parameters that can adjust the aforementioned cases of cross modality. For realistic broadcasting services using olfactory displays, the most important thing is to be able to adequately control the olfactory parameters. We defined the design conditions of the scent display device so that it can control the olfactory parameters. These design conditions are described in Table 3 . The scent display device for multimedia services has to be able to generate many kinds of scents and satisfy design conditions for realistic broadcasting services. An olfactory display device for realistic broadcasting services must have the following: Hedonic assessment is the process of rating odors on a scale ranging from extremely unpleasant to extremely pleasant, via neutral. The scent display device developed for this research does not completely control the scent hedonic, because the hedonic control requires a complex chemical process and a large size. Our device is capable of only a few adjustments, by controlling scent intensity and using a diluent. Figure 2 shows the physical structure of the designed scent display device. Figure 3 shows the designed scent display device, which is of size 300 mm × 250 mm × 100 mm. The scent display device developed for this paper is connected with the control server via Wi-Fi. Figure 4 shows the timing diagram for controlling the scent intensity, persistence, and odor blending. Two small-sized pumps are used for controlling the scent intensity and persistence by adjusting the flow rate through the control of pulse width modulation (PWM) duty cycles. Pump 1 can control the emission rate per second of scents, and pump 2 can control the intensity by adjusting the amount of fresh air that is supplied for diluting scent concentration. PWM control of the blower is used for adjusting the propagation rate of scents (scent speed). In addition, the switching time of solenoid valves can be adjusted for a simple odor blending. Additionally, the scent persistence can be chemically adjusted by controlling the operation time of the valves connected to a diluent. The intensity and persistence of scents should be controlled so as to match the emotions of the images being used. Table 4 shows the changes in scent intensity according to the PWM duty cycles of the two pumps under the condition that the flow rate of the outlet is fixed at about 900 ml/min. We measured the scent concentration at the outlet of the scent display device by using TGS2600 and 2,611 gas sensors. The relationship between sensor resistance and scent concentration can be expressed by R _o / R _s , where R _s is the sensor resistance in displayed gases at various concentrations and R _o is the sensor resistance in fresh air. We use two pumps to control the intensity of a scent while maintaining a constant flow-speed at the outlet of the device. If you wish to increase the scent intensity, then you have only to increase the PWM duty cycle of pump 1 and reduce the PWM duty cycle of pump 2 to the same extent as the required increment in intensity. On the contrary, if you wish to decrease the scent intensity, then you have only to reduce the PWM duty cycle of pump 1 and increase the PWM duty cycle of pump 2 to the same extent as the required decrement. To verify the control of a scent’s intensity in a scent display device, we use ethanol, instead of a scent, in our experiments. Table 5 shows the experimental results for verifying the control possibility of the flow rate under the condition that there is a steady scent concentration. It is very important to control the flow rate under this condition. If we increase the flow speed under this condition, then scent persistence can be increased because the total amount of displayed scent is increased by a faster flow speed. From Table 5 , we could verify the control possibility of scent persistence. From Tables 4 and 5 , we can derive the following relation: (1) $$D_1=P-D_2\text{ ,}$$ where D ₁ ≥ D ₂ and P ≤ 200. The variables D ₁ and D ₂ denote the PWM duty cycles (%) of pumps 1 and 2, respectively. The predetermined value for persistence control is denoted by P . If you want to increase the scent intensity, then you have only to increase D ₁ and to decrease D ₂ . And, if you want to increase persistence, then you have only to increase P . Table 6 shows the experimental results on the scent release velocity at the outlet. To synchronize a scent with images, it is very important to understand just how the scent release velocity control operates. The scent release velocity can be controlled by varying the PWM duty cycle of the blower. To measure the scent release velocity, a gas sensor was installed at 40 cm from the outlet of the scent display device. We measured the arrival time to the gas sensor after a command for scent release. If the distance between a user and the device is 1 meter and the PWM duty cycle of the blower is 60%, then a scent can reach the user after about 2.34 s. In addition, the user is able to recognize smells after about 4.84 s (2.34 s + 2.5 s) since the total perception time of the average user is about 2.5 s. An additional reason for selecting a total perception time of 2.5 s is as follows: the odor perception time after sniff onset is about 1,000 to 1,200 ms, as we have said in the last chapter, but inhalations of breath are intermittent, not continuous. Assuming the average user has a respiration rate of 20 times per minute, we can then calculate that the average user is to have a respiration cycle of 3 s. So, it is reasonable to assume that the present breathing state of the average user is then the midpoint of the respiration cycle; that is, 1.5 s. Therefore, the total perception time of the average user is said to be fixed at 2.5 s (the perception time 1,000 ms + midpoint 1.5 s). Supposing that the distance between user and device is x meters and that the PWM duty cycle of the blower is d (%), the total perception time of a user can be defined by the following equation: (2) $$y=\frac{x}{v}+2.5,$$ where v is velocity by d (%). This equation is very important for accurately synchronizing scents and images. If you wanted to display a specific scent during a certain scene of a video, then you have to send a command to the scent display device about y seconds before the scene were to begin. The developed scent display device for broadcasting services can display 20 scents. Many people may think that more would be required to cover a greater range of scent effects. However, this is simply not the case in many real-world applications. Frequent displays of scents during a video do not augment emotion, as most of us are aware. Besides, time is needed for the dissipation of any smells produced by the scent display device. Table 7 shows the 20 scents of the developed scent display device. Information for the scent display comes in two data formats, as shown in Fig. 5 . One such data format is device capability information, for which a scent display device offers full functional capability of itself to the authoring tool, as shown in Table 8 . The other such data format is device command information, for which the authoring tool makes a request to the scent display device for the emission of a scent, as shown in Table 9 . The authoring tool can make requests for emission of scents in such a manner so as to be suited to the scent display device’s capability by using the specific information on the scent display device. The syntax and semantics of the metadata are based on the XML schema of MPEG-V (ISO/IEC 23005). ScentDeviceCapabilityType is a tool for describing the capability of a device to give out scents and control the scent parameters. ScentCommandType is a tool for describing the commands of the scent display device. In this paper, we designed and developed an olfactory-media authoring tool for an emotion-media broadcasting system using both the proposed data format and the implemented scent display device. The overall system architecture for the olfactory-media broadcasting service is shown in Fig. 6 . If you want to augment immersion and reality by using the olfactory display, then, first of all, you have to input a command for scent display at the editor GUI of the authoring tool at every desired scene. Figure 7 shows the developed authoring tool. The developed authoring tool can create scent effect metadata based on MPEG-V to synchronize the existing video with the proposed capability and command information in Tables 8 and 9 . After selecting the required representative images, you have only to input the necessary information into the “editor for scent display,” shown in Fig. 8 , to display scents suitable to the emotions of your selected scenes. The left part of Fig. 8 is used for scent editing of a selected section, and the left part shows all of the scent information that is applied to the database of the scent display device. The scent intensity, persistence, and velocity can be altered by adjusting the relevant percentage sliders and are to be done so on the basis of the capability information in Table 8 . We have developed a new scent display device and the corresponding emotion-media authoring tool for scent display to augment emotion at broadcasting services. The developed authoring tool allows olfactory display to be properly synchronized with the existing video service by using both the proposed data format and the developed scent display device. For optimal design of the authoring tool, we analyzed the combined modality and cross modality related to olfactory displays. From the results of our analysis, we derived a set of parameters for olfactory displays that are related to human emotion. In addition, we have designed a scent display device that can generate many kinds of scents and control the intensity, persistence, and velocity of a scent display. Corresponding Author  jdkim@hoseo.edu Jeong Do Kim received his BS, MS, and PhD degrees in electronic engineering from Sungkyunkwan University, Suwon Rep. of Korea, in 1987, 1990, and 1994, respectively. He is currently a professor of electronics engineering at Hoseo University, Asan, Rep. of Korea. His main research interests are e-nose and olfactory displays, and he has done research in the area of signal processing and pattern recognition applicable to multi-sensor fusion. cjh@etri.re.kr Ji Hoon Choi received his BS and MS degrees in electronic engineering from Kyunghee University, Suwon, Rep. of Korea, in 1999 and 2001, respectively. In 2001, he joined the Data Broadcasting Research Group, Electronics and Telecommunications Research Institute, Daejeon, Rep. of Korea, where he has been working on the interactive data broadcasting system. His research interests are data broadcasting and network QoS. mn04mn04@naver.com Seung Ju Lim received his BS and MS degrees in electronic engineering from Hoseo University, Asan, Rep. of Korea, in 2010 and 2012, respectively. Currently, he is pursuing his PhD degree in electronic engineering at Hoseo University, Asan, Rep. of Korea. His research interests include digital signal processing and pattern recognition. sdpark@control.hoseo.ac.kr Sung Dae Park received his BS, MS, and PhD degrees in electronic engineering from Hoseo University, Asan, Rep. of Korea, in 2007, 2009, and 2014, respectively. His research interests include smart sensors, healthcare, and embedded systems. ichromosome@nate.com Jung Ju Kim received his BS and MS degrees in electronic engineering from Hoseo University, Asan, Rep. of Korea, in 2006 and 2008, respectively. Currently, he is pursuing his PhD degree in electronics at Hoseo University. His research interests are ubiquitous, human–computer interaction and embedded systems. hyun@etri.re.kr Chung Hyun Ahn received his PhD degree in GIS/RS from Chiba University, Japan, in 1995 and worked as a member of the research and teaching staff at Chiba University. He has been working at ETRI since 1996, leading many projects in GIS/RS (1996–2000) and digital broadcasting areas (2001–present). He was the leader of the Next Broadcasting Service Planning Team. Currently, his major research focus is emotion-based broadcasting services and new broadcasting services for disabled people.