Adaptive Multimodal In-Vehicle Information System for Safe Driving
Adaptive Multimodal In-Vehicle Information System for Safe Driving
ETRI Journal. 2015. May, 37(3): 626-636
Copyright © 2015, Electronics and Telecommunications Research Institute (ETRI)
  • Received : September 17, 2014
  • Accepted : March 23, 2015
  • Published : May 01, 2015
Export by style
Cited by
About the Authors
Hye Sun, Park
Kyong-Ho, Kim

This paper proposes an adaptive multimodal in-vehicle information system for safe driving. The proposed system filters input information based on both the priority assigned to the information and the given driving situation, to effectively manage input information and intelligently provide information to the driver. It then interacts with the driver using an adaptive multimodal interface by considering both the driving workload and the driver’s cognitive reaction to the information it provides. It is shown experimentally that the proposed system can promote driver safety and enhance a driver’s understanding of the information it provides by filtering the input information. In addition, the system can reduce a driver’s workload by selecting an appropriate modality and corresponding level with which to communicate. An analysis of subjective questionnaires regarding the proposed system reveals that more than 85% of the respondents are satisfied with it. The proposed system is expected to provide prioritized information through an easily understood modality.
I. Introduction
- 1. Necessity for Adaptive Multimodal Interaction for Safe Driving
These days, cars are equipped with human–machine interface (HMI) devices that provide both information and entertainment, such as navigation devices and various AV players. In addition, cars of the future are to be mounted with interfaces that will require alternative methods of interaction, such as gesture recognition–based interfaces [1] [2] , or tracking technologies that can track the motion of the human body [3] [4] . Research conducted by [5] [6] indicates that HMI devices in a car are deeply related with driver safety. Inappropriately designed HMIs can threaten driver safety. To prevent car accidents that are a result of a driver’s interaction with an in-vehicle HMI device, it is essential to consider alternative designs of such devices that focus on reducing the required level of interaction between driver and device. A more effective approach to improving driver safety in such cases would be to try to develop ways to make drivers better understand various types of input information. For instance, in the case of elderly drivers, one could consider using different types of modalities or varying the level at which a modality operates so as to ensure that an elderly person is able to easily recognize a type of input information.
One emerging solution to the aforementioned problem is that of adaptive multimodal interaction . Firstly, a car’s surroundings are detected while its driver drives the car, and then secondly, the most suitable modalities for providing information to the driver are selected. Finally, if necessary, the levels of these modalities can be adjusted according to whether the information is very important or if there is a warning that should be urgently delivered to the driver.
In this context, the proposed adaptive multimodal in-vehicle information system (AMiVIS) determines the order of priority of the information to be provided to the driver having already considered the driving situation and driver’s cognitive state; selects the modality with the minimum driving workload; and then informs the driver of the necessary information. Next, according to the driver’s reaction, it adjusts the multimodalities or their corresponding levels as is necessary.
- 2. Necessity for Intelligent Information Management for Safe Driving
The modern age has been referred to as an information society [7] . An exponential increase in the amount of information in the world has occurred, and we are now able to access a variety of information systems through a network-based distributed computing environment. In light of this, the need for information technology for processing huge amounts of information has arisen.
T. Masue and others proposed a method for transferring information to control the usage of content more quickly and efficiently, and thereby prohibit any overhead incurred when transferring usage control information for several types of content [8] . In addition, J. Georgy and others presented a method for transferring vehicle navigation information faster and more effectively through a continuous and accurate system integrating low-cost MEMS-based inertial sensors, a vehicle odometer, a GPS, and map data from road networks [9] . These methods focus on how to combine different types of information and offer the resulting information more quickly to a system user.
However, to provide intelligent information, it is most important to consider user-centered interaction. One of the methods used to intelligently offer information is to provide the most easy-to-understand and prioritized information to the user. The authors in [10] introduced a method for filtering and sharing various types of information based on AI technology. Their method then provided users with only the information that was necessary from the resulting filtered information.
To manage all in-vehicle information automatically, D.R. Tufano and others [11] developed an in-vehicle information system (IVIS) that could manage and display all in-vehicle information, in the sense that in-vehicle information is first prioritized and then filtered or in some cases fused prior to being displayed to the driver. This system was developed for the US Federal Highway Administration. The system supports a more efficient uptake of the driver’s available information and minimizes the potential for driver distraction, thus enhancing driver safety.
Through the development of intelligent information systems that select and recommend information using both a preference estimation model and a collaborative filtering technique, various information filtering methods based on a priority ranking scheme have been proposed [12] [14] , as have methods applicable to an intelligent vehicle information system [15] [18] .
Owing to the rapid developments in automotive telematics devices and the intelligent automotive industry, vehicles have recently become more than a simple transport medium. Drivers are now able to conduct various activities while driving and receive various types of information provided from their vehicles. Using in-vehicle and out-of-vehicle network technologies, telematics can provide many more services to drivers, such as transmissions on both infrastructure and vehicle information. In addition, an intelligent transport system (ITS), such as an advanced traffic management system (ATIS), an advanced public transportation system (APTS), or an advanced vehicle and highway system (ATMS), can quickly provide greater amounts of road and traffic information to drivers [19] [21] . The development of these technologies has also brought about greater convenience and enjoyment. However, offering a greater variety of information while driving may endanger the safety of the driver. In particular, providing unnecessary information may distract a driver; furthermore, the operation of various in-vehicle devices may result in traffic accidents [22] [31] .
According to the authors of [29] and [30] , the number of car accidents occurring as a direct result of a driver (one that was involved in such accidents) using a hand-held mobile phone while driving has increased sharply since 2007. Such a cause now accounts for 17% of all car accidents. Nevertheless, it is increasingly likely that drivers will continue to be provided with ever-increasing amounts of information from their in-vehicle HMI devices. Therefore, the necessity to develop an intelligent IVIS to manage and control such information is only going to increase [31] . What exactly an intelligent IVIS is and how to manage and control such a system intelligently are important questions. Such a system provides necessary information to a driver at the right place and at the right time. Thus, this type of system can help increase driver safety and bring about driving convenience with a minimization of driver workload. How to determine what information is needed and how to provide this information to the driver in an appropriate way when required are the next questions that need to be answered.
- 1. Overview of Proposed AMiVIS
For the development of an intelligent IVIS, this paper first proposes an adaptive information interaction method that estimates and extracts information regarding a driving situation, the driver, and the driver’s driving workload; selects a proper modality and adjusts its level in accordance with the given driving situation and driver’s cognitive state; and offers the information to the driver. Second, this paper proposes a smart information management method for filtering indiscriminate input information that is collected owing to the recent rapid development of vehicle technology. This paper calls the system with these two intelligent information provision functions an AMiVIS.
The AMiVIS is a software system equipped inside of an intelligent in-vehicle navigation system. Figure 1 illustrates an overview of the intelligent in-vehicle navigation system included with the proposed AMiVIS. The intelligent in-vehicle navigation system was designed such that the left side of the screen shows the navigation information and the right side presents information regarding the driver, vehicle, and environment, including the driving workload values extracted from the sensors in real time. In addition, the intelligent in-vehicle navigation system provides urgent information with a higher priority than the route information to the driver through a warning voice or a display on the left side of the screen.
PPT Slide
Lager Image
Overview of intelligent in-vehicle navigation system.
The intelligent in-vehicle navigation system is implemented by connecting to four different software (S/W). The first S/W estimates the vehicle status, driver state, and ambient situation. First, vehicle state ( S V ) estimation infers the driving tasks, such as driving, stopping, and accelerating, using the measured vehicle information. The vehicle information is detected from an OBD-II device based on the vehicle’s speed, steering angle, braking, and other information. Next, a driver state ( S D ) estimation tracks the driver’s head and eye movements from an eye-tracker and camera mounted in the vehicle. In addition, it also obtains the driver’s physical state, such as their pulse, respiration, and temperature. From this information, the proposed system can estimate the driver’s level of distraction, drowsiness, and general health. Finally, the ambient situation ( S E ) estimation recognizes obstacles and lanes using a sensor fusion based on two radars and two cameras equipped in the vehicle. Thereafter, it can infer the driving situations and degree of danger in the surrounding environment. The second S/W, the proposed AMiVIS, integrates, manages, and controls the input information. The third is a navigation S/W. The last S/W, a call control S/W, is related to the driver’s smartphone connected with the vehicle, and through the use of an app (HVI), it can intelligently control the receiving and transmitting of messages. The app and AMiVIS are connected through Bluetooth. If a driver receives a call while driving, then navigation S/W automatically blocks the call and sends a text message to the caller depending on the driving situation. In addition, during an emergency, this S/W includes automatic connection to a caller through a voice transmission service.
A flowchart of the proposed AMiVIS is presented in Fig. 2 . Once three types of driving condition information are received; that is, the driver state, vehicle status, and ambient situation (indicated by “3S” in Fig. 2 ), the navigation information and phone information are input, and the system starts to filter the provided information. First, the system prioritizes the information using a predefined database (DB), and thereafter determines the best time to provide the information based on the driving situation awareness. Thus, the information to be provided and the time to provide it are selected. Next, the system then determines the proper modality and its level based on the results of a driving-workload measurement. Finally, the system checks whether the driver recognizes the provided information. If the driver is not fully aware of the output information, then the system readjusts the provided modality and its associated level.
PPT Slide
Lager Image
Flowchart of proposed AMiVIS.
The proposed system has the following advantages:
  • ■ The AMiVIS filters input information from various in-vehicle devices, and the system preferentially provides high-priority information. Thus, it improves driver safety by reducing driver distraction from providing multiple types of information indiscriminately.
  • ■ The AMiVIS measures the driver workload according to four channels in real time; selects the modality with the lowest workload among the visual, auditory, and tactile modalities; and provides information to the driver. It thus helps reduce the risk to the driver by reducing their driving workload and improving their understanding of the information provided.
  • ■ The AMiVIS uses, and adjusts the levels of, multiple modalities to allow the driver to operate the vehicle safely even when the driver overlooks important information or does not react to a given warning.
- 2. Proposed AMiVIS Configuration
The proposed system consists of the following four stages: information priority determination, driving situation awareness, optimized workload–based modality selection, and driver’s cognitive reaction–based modality-level control. First, the information priority determination stage formulates the order of priority of the input information type — each of which has two associated attribute types. Second, the driving situation awareness stage determines the information to be provided and the transit time, based on the road type, travel direction, speed, and so on. Next, the optimized workload–based modality selection stage offers the driver high-priority information through an interface through which the driver can most easily understand the information. To select the proper interface, the driving workload value and combined driving workload value for the four channels are calculated, and the optimal modality is extracted. Finally, the driver’s cognitive reaction–based modality-level control stage observes the driver’s reaction in real time. If the driver does not recognize the received information well, then at this stage the modality channel is reselected and the corresponding modality level is upgraded. More detailed explanations are given in the following sub-sections.
A. Information Priority Determination
The information priority determination stage, which consists of two steps (dispatcher and scheduler), prioritizes the input information and determines which information the AMiVIS will offer.
First, when the dispatcher receives information, it determines what group and type the information belongs to; discerns the attributes of the information according to the driving situation; and delivers to the scheduler the basic values that were pre-set according to the groups and types of information and the weights that were set according to the information attributes. These attributes are safety and timeliness and can be changed depending on the driving situation. Here, safety is information related with safety, and other information is given for convenience.
Next, the scheduler calculates the order value of the information ( Oi ) using (1) based on the two delivered factors (basic values and weights). Through the calculated order value, the priority of the input information is determined. At this time, the bigger the order value is, the higher the priority it is given.
O i = b i +( 2 w a i 1 +1.5 w a i 2 ).
In (1), O is the order value, bi a basic value,
a i 1
is the safety (the first attribute of information), and
a i 2
is the timeliness (the second information attribute). The basic value of information ( bi ) is one that was calculated in advance according to the groups and types of information based on the following four heuristic rules:
  • ■ The information provided from AMiVIS is divided into four groups —driveris composed of information related with the driver;vehicleis information related with the vehicle,deviceis information related with the devices within the vehicle, andobstacleis information related with obstacle warnings.
  • ■ Each group is made up of more than one type. In addition, each type is assigned with a different basic value (bi). The smaller the basic value that is assigned, the less important the information is. The smallest basic value that is assigned is 1, and the basic value increases as the number of types increases.
  • ■ For the basic values of all types in the driver group, 1.5 is added to each basic value assigned for each type. On the contrary, for navigation information for all types of thedevicegroup, 0.5 is subtracted from each basic value, and for phone information, 1.0 is subtracted from each value.
  • ■ The value added to the basic value or the value subtracted from the basic value for each group is, based on the order value calculated by (1), adjusted to consistently produce the same information priority under the same conditions.
In (1), the weight ( w ) of an information attribute is given to only information with the attribute of safety or timeliness. Thus, types with an attribute of safety or timeliness are assigned with a particular weight. Basically, since the attribute of safety
( a i 1 )
is more informant than the attribute of timeliness
( a i 2 )
in calculating the order value, (1) calculates the weight of each attribute reflecting this fact.
Table 1 shows an example of the generated information order results for all information that may be provided from the proposed AMiVIS.
Predefined DB of attributes and priority according to information type.
Group Type 1st attribute 2nd attribute Priority
Driver Distraction Safety Timeliness 2 2
Drowsiness Safety Timeliness 1 1
Health check Safety Non- 4 5
Vehicle Components problem Safety Non- 5 6
Device Navi Route guide Convenience Non- 7
Safety Non- 4
Phone Receiving call Convenience Non- 6 7
Obstacle Warning Safety Timeliness 3 3
B. Driving Situation Awareness
The driving situation awareness stage determines the information provided and the transit time based on the road type, steering wheel angle, speed, and other factors. This stage should judge whether the driver is in a situation in which they can recognize the information. If the driver is not in such a situation, then this stage should select the proper time to provide the information and then give it at that time. Among the vehicle information estimated from the vehicle status estimation S/W, this stage determines whether the driver is in a safe enough situation to receive the information using both the steering wheel angle and the vehicle speed. For example, if an urgent call comes while driving, then this stage can receive the call from the terminal. However, when the driver should focus on a driving situation such as a sharp curve, the system automatically cuts the call and sends the sender a text message letting them know that the recipient is not able to take their call right now. In addition, when the driver can safely answer the phone (for example, when there are no other vehicles nearby on a straight road), the system automatically receives the call.
C. Optimized Workload–Based Modality Selection
This stage offers the driver prioritized information through an interface through which the driver can most easily understand the information. To select the proper interface, AMiVIS calculates the driving workload value combined from the four channels and then extracts the optimum modality and its level. The driving workload value is determined using the total driving workload (DW T ) combined with each of four workload values (DW A , DW C , DW V , and DW P ). Here, DW refers to driver workload, and A, C, V, P and T indicate auditory, cognitive, visual, psychomotor, and total, respectively. The workload values for the four channels are extracted using the following method.
First, the driver completes a survey, which measures the workloads for the four channels, before being allowed to begin to drive. The survey consists of questions that were made based on driving behavior determinants, driving confidence levels, and subjective workload assessment tools (NASA-TLX, driving activity load index (DALI), and so on), and that can predict subjective driving workloads. Based on the results of this survey, the initial workloads for the channels are designated. In addition, the initial workload values, which are set according to the driver’s gaze; the operating condition of the navigation system and mobile phone; and the states of the surrounding vehicles and environment, are updated in real time through the three above-mentioned S/W programs, excluding the proposed AMiVIS S/W.
Next, the total driving workload is evaluated as below. When optimally based on the driving workload values calculated in real time as explained above, the information is given through the modality belonging to the channel whose workload is the smallest among the four channels. To optimize the workload, the total driving workload (DW T ) is updated by this stage using
DW T = ch{ A,C,V, P } DW ch ω ch ,
where DW T is defined as the weighted sum of DW A , DW C , DW V , and DW P at time t . In addition, DW A,t is the current auditory channel, which is updated using the weighted sum of DW A.t−1 , S A.t.D , S A.t.V , and S A.t.E ; DW A.t−1 is a previous auditory channel; and S A.t.D , S A.t.V , and S A.t.E are the current auditory states of the driver, vehicle, and the environment, respectively. These states, which are given as measured values from each sensor, are determined using a codebook; DW C.t , DW V.t , and DW P.t are also updated in the same way. The updated equation is as follows:
DW ch,t =ρ f{ D,V,E } S ch,t,f ω f +( 1ρ ) DW ch,t1 . ch{ A,C,V,P }.
Finally, this stage selects the modality, controls the level of modality, and determines the transfer time. To select the transferring modality, this stage selects the minimum number of channels required, as compared to using all four channels, based on the optimized driving workload. Thus, the transferring modality is determined through
DW T = argmin  ch{A,C,V, P} DW ch ω ch .
Next, if DW T is over a given threshold value, then the AMiVIS does not transfer information to the driver. Otherwise, the AMiVIS provides the information using the selected channel based on (4). There are three levels of modality — levels 1, 2, and 3. Level 1 is the default level. In this paper, the level indicates the transfer strength.
D. Driver’s Cognitive Reaction–Based Modality-Level Control
The driver’s cognitive reaction–based modality-level control stage observes the driver’s reactions in real time, and the modality and its associated level are updated according to the reaction results. The judgment of the driver’s awareness differs depending on the type of the provided information. For example, when an obstacle warning is issued, if the driver turns the steering wheel or reduces their speed to avoid the obstacle, then it is judged that the driver was made aware of the information provided. This judgment of the driver’s awareness uses the vehicle speed and steering angle, which are extracted from the vehicle status estimation S/W. If the driver does not recognize the received information well, then the system re-selects the channel using (4), and ω ch.t is then updated using
ω ch,t =k× ω ch,t1 .
In both (4) and (5), k and ω are determined experimentally. Finally, the modality level is upgraded when the selected modality is reselected.
III. Experiments
The experiments to verify the usefulness of the proposed AMiVIS prove that the two proposed functions (a function that filters input information using the driving situation detection and information priority, and a function that offers information with the optimal modality and level based on the detection of the driver’s cognitive reaction) improve driving safety and make the driver easily understand the information provided by minimizing the driver’s workload. In addition, the results of a survey questionnaire given to the users of the AMiVIS indicate that they were satisfied with the adaptive multi-interaction system. To prove the two functions of the system, two experiment scenarios were created.
- 1. Experiment Scenario 1
  • 1) The driver drives toward a destination.
  • 2) While driving, the phone rings while on a sharply curving road where the vehicle should turn to the left or make a U-turn.Figure 3shows the actual field test used in the experiment. The subjects were instructed to perform a 90-degree left turn (sharp curve) on the circular road section shown inFig. 3. The subjects turned left based on the numbering order (from 1 to 5), as shown inFig. 3.Vehicle with the AMiVIS. When the vehicle should turn to the left while driving, the system cuts off the information that the phone is ringing for driving safety. When the vehicle comes to a straight road and can drive at a regular speed, the system lets the driver know that the phone is ringing and provides an automatic call service with voice guidance.Vehicle with a general navigation device (without an AMiVIS). This vehicle has no information filtering or automatic call service function. It only provides simple information about the driving course (the phone continues to ring until the driver answers it).
  • 3) A memory test on words the driver hears while the phone is ringing.
PPT Slide
Lager Image
Field test and test vehicle.
- 2. Experiment Scenario 2
  • 1) The driver drives toward a destination.
  • 2) While driving, the driver conducts a second task (word recall tasks).
  • 3) While driving, the driver avoids an obstacle.Vehicle with the AMiVIS. The system detects an obstacle in advance and lets the driver know about it. When the system judges that the driver has failed to be aware of the warning (when they do not reduce their driving speed or turn the steering wheel to avoid the obstacle), the system continues to send the obstacle warning. At this time, the information is sent with the modality with the minimum driving workload.Vehicle with a general navigation device (without an AMiVIS). The navigation device has no function to detect an obstacle and provide a warning. It only offers simple information on the driving route.
Under these scenarios, a vehicle with an AMiVIS mounted and a vehicle without an AMiVIS each drove on the experiment road where various driving situations that can occur on a real road can be reproduced. The results were then compared and analyzed. Since the experiment has a within-subject design, the order of the experiments was counterbalanced to prevent a possible practice effect. A total of 30 subjects (21 men and 9 women) with an average age of 30.19 years and an average of 6.37 driving years were used. To prevent the experimental data from distortion owing to unfamiliarity of the driving route and subject tension, the participants were given plenty of practice in a test bed before conducting the real experiments. Each of the 30 participants drove the test vehicle from the start point to each destination for the three scenarios under each type of navigation conditions, the order of which was counter balanced. During the experiment, the subjects were encouraged to follow certain guidelines: observe the sign boards and follow the instructions as secondary tasks; maintain a speed limit of 60 to 70 km/h; avoid sudden obstacles; and refrain from passing the preceding vehicle.
IV. Results
- 1. Driving Performance
The results of the experiments for the two scenarios were analyzed as follows.
To determine how safely the subjects turned during scenario 1, their lateral accelerations (LA), which were analyzed under the geometric parameters of the road, are taken into account in the road safety criteria [32] [35] . Here, LA is used as one of the risk criteria related to vehicle dynamics, and is usually a variable that limits the vehicle speed on curves under good friction conditions [32] [37] . In [36] [38] , the experimental results prove that actual driving speeds on curves are lower than the theoretical speeds, and that this reduction depends on the drivers and their vehicles. Thus, Table 2 [32] [35] shows the criteria used to judge whether the driver made a safe turn based on the LA and according to the results of the proposed system.
This paper analyzed the driving stability performances based on the criteria in Table 2 . Figure 4 shows the results of the analysis. As shown in Fig. 4 , the average LA when the AMiVIS was mounted in the vehicle was 2.67 m/s 2 , and according to the criteria in Table 2 , this proves the driving stability of the AMiVIS-mounted vehicle. On the contrary, the average LA when the AMiVIS was not mounted in the vehicle was 3.36 m/s 2 . However, for seven subjects (23%), the average LA when the AMiVIS was not mounted in the vehicle was 4 m/s 2 or more. The experiments for driving stability were conducted through a field test, but when conducted under actual road conditions, more obstacles will appear, making the proposed AMiVIS more effective. The reason for this is that the drivers should always observe the driving road situations through the side mirror, rear-view mirror, and front windshield while driving. Thus, visual and cognitive loads occur while driving. To effectively convey this situation, we created an experimental environment that was similar to a real road by. applying secondary tasks. Nevertheless, none of the actual road situations can be portrayed using only a secondary task Because the driver should observe more visual elements such as an unexpected operation of another vehicle or a signal, in a real driving situation, the AMiVIS is expected to promote a larger effect on safe driving on an actual road.
Effects of lateral acceleration.
LA (m/s2) Effects
< 2 Every vehicle and every driver can deal with them.
2–4 Majority of drivers can deal with them.
4–6 Modern vehicles can maintain stability but drivers have problems dealing with them.
6–8 Vehicles and drivers cannot deal with them.
PPT Slide
Lager Image
Driving stability performance results using LA.
To analyze how safely the subjects avoided an obstacle for scenario 2, the steering angle, vehicle speed, and braking were analyzed. Table 3 shows the results. In Table 3 , SD, M , N , p , and t represent the standard deviation, mean, number of subject, values, which are calculated with T-test for continuous variables, and T-test score, respectively.
Analysis results of driving stability.
Condition N M (SD) t
Steering angle MAX AMiVIS 30 305.88 (156.79) −.61*
Non- 30 310.32 (172.62) 2.74**
Vehicle speed (km/h) Average AMiVIS 30 2.05 (1.79) 2.74**
Non- 30 1.12 (0.66)
MIN AMiVIS 30 0.71 (0.61) 5.28**
Non- 30 0.05 (0.12)
Brake Pressure MAX AMiVIS 30 8.75 (0.15) 2.66**
Non- 30 8.94 (0.40)
Frequency AMiVIS 30 0.35 (0.30) 3.72**
Non- 30 0.09 (0.18)
*p < .05, **p < .01,***p < .001
Comparison results from when the AMiVIS was mounted and not mounted in the vehicle show that the vehicle with the AMiVIS had a lower average and standard deviation of the MAX value of the SAS angle than those of the vehicle without the AMiVIS. This shows that when the subjects were warned of an oncoming obstacle from the AMiVIS they avoided the obstacle with a safer steering wheel angle. In addition, the results of analyzing the average vehicle speed show that while the vehicle with the AMiVIS maintained the average speed, the vehicle without the AMiVIS had a smaller MIN value and average speed because many of the drivers failed to avoid the obstacle in advance and had to suddenly stop. In the case of the braking, the vehicle without the AMiVIS had a high pressure value because the drivers hit the brakes strongly to quickly avoid a crash with an obstacle. Moreover, for the frequency of applying the brake, the vehicle with the AMiVIS had a higher frequency and a lower pressure value because the subjects engaged the brake several times with proper pressure to slowly reduce their speed to avoid an obstacle in advance. These results prove that a driver with an AMiVIS can detect an obstacle more quickly than a driver without an AMiVIS and thus the former driver has a higher possibility to avoid an obstacle safely.
- 2. Cognitive Performance
The results of the above two scenario experiments were analyzed to prove that the AMiVIS leads drivers to understand information more easily through a minimization of driving workload and helps them quickly recognize the driving-situation through a proper modality.
For scenario 1, the subjects took a test in which they recalled the words that they had heard while talking on the phone after arriving at their destination. They heard five words that did not have any particular association or connection with each other. The results show that, of the 30 subjects who drove the vehicle with the AMiVIS, eleven remembered all five words, 17 remembered three to four words, and two remembered less than two words. On the contrary, of the subjects who drove the vehicle without the AMiVIS, none remembered all five words, eleven remembered three to four words, and 19 remembered less than two words. Based on these results, it is thought that it is hard for the driver to recognize the information provided when they should focus on driving while at the same time taking care of their surroundings, such as when making a left turn, and that it is dangerous for the driver to react to the provided information owing to a large driving workload. It was, therefore, proved that under an unstable driving situation, the automatic call service promotes driving safety and leads the driver to understand the offered information easily; the automatic call service delivers the information through a speech recognition technology when it is judged that the driver is in a situation in which they can receive the information.
For scenario 2, we set up a sign post consisting of four words with different colors along the road, and instructed the driver to remember each word. After the experiment, we asked questions such as “What was the yellow word?” During the experiments, two word sets, A and B, were used randomly, as shown in Fig. 5 . The selected words have a lower frequency of occurrence, as determined in [39] . This is intended to prevent incorrect experimental results, according to a knowledge- or relevance-estimation-based recall.
PPT Slide
Lager Image
Real example of word recall tasks in scenario 2.
This experiment was conducted to determine the effects of the AMiVIS in detecting an obstacle and providing a service concurrently. If an obstacle is placed on the road in advance, then the drivers will avoid it regardless of whether a warning is given. Therefore, to reflect the intention of the experiment, the second task was conducted to prevent the subjects from seeing and avoiding the obstacle in advance. The results of the experiment show that when the driver is distracted while driving, they are less likely to recognize an obstacle in front of them, and there is therefore a high possibility to crash into the obstacle.
In such a case, if the auditory modality warns the driver about crashing into the front obstacle, then it can lead the driver to pay attention to what’s in front of them and prevent them from crashing with the obstacle. The proposed AMiVIS adjusts the level of the auditory modality using time to collision, which is based on the distance and the relative velocity from a front obstacle. It then provides the driver with the necessary information. This function shows that AMiVIS improves the ability of a driver to recognize information. The time that it took the subjects to receive a warning and reduce their driving speed was on average 1.7 s quicker for the vehicle with the AMiVIS mounted than for the vehicle without the AMiVIS. Thus, it was judged that the vehicle with the AMiVIS can detect the driving situation quickly and speedily cope with the given information.
- 3. Subjective Performance
To access the subjects’ levels of satisfaction with AMiVIS with the information filtering function and information provision function through an optimal modality and level, a survey questionnaire was given after the subjects finished both scenarios of the experiment.
Table 4 shows the results of analyzing their satisfaction for each function of the AMiVIS. In Table 4 , the information filtering function of the AMiVIS is called Function 1, and the information provision function through the optimal modality and level is called Function 2.
As shown in Table 4 , the total satisfaction of the subjects for the vehicle with the AMiVIS was 85.00%, which is 35.01% higher than that of the subjects for the vehicle without the AMiVIS. The main reason why they were unsatisfied with Function 1 is because they felt displeasure that the auto information filtering function of the AMiVIS blocked an incoming call by judging that it was unsafe for the drivers to receive it. That is, they were displeased with the fact that they were controlled by a machine. They were mainly unsatisfied with Function 2 because the sound of the auditory modality was not loud enough that they could recognize it. But, the experimental result of the cognitive reaction shows that the subjects reacted quickly based on the sound volume. Thus, it was concluded that after the experiments they did not recall the sound volume of the auditory modality.
Analysis results of usability performance.
Satisfaction level Condition Function 1 Function 2
Very satisfied AMiVIS 13.33 6.67
Non- 0.00 3.33
Satisfied AMiVIS 50.00 46.67
Non- 13.33 23.33
Normal AMiVIS 13.33 40.00
Non- 33.33 26.67
Dissatisfied AMiVIS 20.00 3.33
Non- 40.00 43.33
Very dissatisfied AMiVIS 3.33 3.33
Non- 13.33 3.33
V. Conclusion
This paper proposed an adaptive multimodal in-vehicle information system (AMiVIS) that filters the input information and selects an adaptive modality and its level. The experimental results show the following:
  • ■ Information filtering based on a discernment of the information priority is effective in reducing the cognitive workload of the driver caused by indiscretion in the input information.
  • ■ Information filtering based on driving situation awareness helps drivers drive safely by delivering only safety-related information while omitting other input information. For example, when the driver should be paying attention to where they are going and maintaining awareness of the situation around their vehicle, as when making a left turn, the system classifies the information to be delivered and omitted through a filtering process.
  • ■ The provisioning of information through an adaptive modality that considers the driver workload improves the driver’s understanding of the information provided.
  • ■ Finally, when it is judged that the driver has failed to recognize the safety-related information, it is effective in that the information is provided again, the level of modality is adjusted, or the information is provided through only a few modalities.
The analysis results of the driving stability performance, in which a vehicle mounted with the AMiVIS drove on an experiment road modeled after a real road, shows that the information-filtering function of the AMiVIS reduces driver distractions and promotes safe driving. In addition, the modality selection and adjustment of the modality level improve the driver’s understanding of the information and lead them to pay attention to the safety information. The analysis results of the usability performance show that user satisfaction of the AMiVIS is more than 85%.
This work was supported by the Industrial Strategic Technology Development Program and the Development of Driver Adaptive Intelligent HVI (Human-Vehicle Interface) Technology (10033346) funded by the Ministry of Knowledge Economy (MKE, Rep. of Korea).
Corresponding Author
Hye Sun Park has been a senior engineer with the Computer Graphics Center, Electronics and Telecommunications Research Institute, Daejeon, Rep. of Korea, since December 2009. She received her PhD in computer engineering from the School of Electrical Engineering and Computer Science, Kyungpook National University, Daegu, Rep. of Korea, in 2007, where she was a member of the Artificial Intelligence Lab. She was selected as a delegate from Korea for the 2nd China-Japan-Korea Young Researchers Workshop. She also received the Japan Society for the Promotion of Science during her studies as a postdoctoral fellow at Kyoto University, Japan. In addition, she was a cooperative overseas researcher at Kyoto University, during her master’s and doctoral studies with support from Brain Korea 21 and the National Research Foundation of Korea. Her research areas include computer vision and human–computer interaction.
Kyong-Ho Kim is a principal researcher and director of the Human–Vehicle Interaction Research Center, Electronics and Telecommunications Research Institute, Daejeon, Rep. of Korea. He received his BS and MS degrees in electronic engineering from Kyungpook National University, Daegu, Rep. of Korea, in 1993 and 1995, respectively, and his PhD degree in computer science from the Korea Advanced Institute of Science and Technology, Daejeon, Rep. of Korea, in 2010. Since 1994, he has been with ETRI. His current research topics include intelligent vehicles, human–computer interaction, head-up displays, and augmented reality applications in vehicles.
Kim H. , Song H. 2014 “Evaluation of the Safety and Usability of Touch Gestures in Operating in-Vehicle Information Systems with Visual Occlusion,” Appl. Ergonomics 45 (3) 789 - 798    DOI : 10.1016/j.apergo.2013.10.013
Ahmad B.I. 2015 “Intent Inference for Hand Pointing Gesture-Based Interactions in Vehicles,” IEEE Trans. Cybern. 1 -
Jo D. 2013 “Tracking and Interaction Based on Hybrid Sensing for Virtual Environments,” ETRI J. 35 (2) 356 - 359    DOI : 10.4218/etrij.13.0212.0170
Cheng S.Y. , Park S. , Trivedi M.M. 2007 “Multi-spectral and Multi-perspective Video Arrays for Driver Body Tracking and Activity Analysis,” Comput. Vis. Image Understanding 106 (2–3) 245 - 257    DOI : 10.1016/j.cviu.2006.08.010
2010 Visual-Manual NHTSA Driver Distraction Guidelines
2008 Measuring Distraction Potential of Operating In-Vehicle Devices
Webster F. 2006 “Theories of the Information Society,” Taylor & Francis e-Library New York, NY, USA 1 - 323
Masue T. , Hirai T. , Shikama T. 2011 “Transfer Acceleration of Content Usage Control Information by Using Base-Values Reference Method,” IEEE Trans. Consum. Electron. 57 (3) 1141 - 1147    DOI : 10.1109/TCE.2011.6018867
Georgy J. , Noureldin A. , Goodall C. 2012 “Vehicle Navigator Using a Mixture Particle Filter for Inertial Sensors/Odometer/Map Data/GPS Integration,” IEEE Trans. Consum. Electron. 58 (2) 544 - 552    DOI : 10.1109/TCE.2012.6227459
Malone T.W. , Grant K.R. , Turbak F.A. 1986 “The Information Lens: An Intelligent System for Information Sharing in Organizations,” ACM Int. Conf. SIGCHI Human Factors Comput. Syst. New York, NY, USA 17 (4) 1 - 8
Tufano D.R. , Spelt P.F. , Knee H.E. 1997 “In-Vehicle Information System Functions,” Annual Meeting Exposition Intell. Transp. Soc. America (Grant No. DE-AC05–96OR22464) 1 - 11
Kim J. 2000 “Dynamic Information Ranking Using Multiple Information Filtering,” M.S. thesis Inha University Incheon, Rep. of Korea
Ekstrand M.D. , Riedl J.T. , Konstan J.A. 2010 “Collaborative Filtering Recommender Systems,” Human–Comput. Interaction 4 (2) 81 - 173
Oku K. “Context-Aware SVM for Context-Dependent Information Recommendation,” Int. Conf. Mobile Data Manag. Nara, Japan May 10–12, 2006 109 - 113
Ye Z. , Wang Y. , Wen Y. 2013 “Effects of Information Filtering on Driving,” Procedia - Social Behavioral Sci. 96 (6) 1755 - 1763    DOI : 10.1016/j.sbspro.2013.08.200
Hancock P. , Lesch M. , Simmons L. 2003 “The Distraction Effects of Phone Use during a Crucial Driving Maneuver,” Accident Anal. Prevention 35 (4) 501 - 514    DOI : 10.1016/S0001-4575(02)00028-3
Kantowitz B.H. , Moyer M.J. 1999 “Integration of Driver In-Vehicle ITS Information,” Federal Highway Administration Conf. Proc. ITS America DC, USA 1 - 10
2002 “ITS In-Vehicle Message Priority,” Int. SAE, Automotive, SAE J. 2395 1 - 7
Dadashnialehi A. 2014 “Intelligent Sensorless ABS for In-Wheel Electric Vehicles,” IEEE Trans. Ind. Electron. 61 (4) 1957 - 1969    DOI : 10.1109/TIE.2013.2266085
Vashitz G. , Shinar D. , Blum Y. 2008 “In-Vehicle Information Systems to Improve Traffic Safety in Road Tunnels,” Transp. Res. Part F: Traffic Psychology Behavior 11 (1) 61 - 74    DOI : 10.1016/j.trf.2007.07.001
Morris A. “Studying the Effects of In-Vehicle Information Systems on Driver Visual Behavior – Implications for Design,” Int. Res. Council Biomechanics Injury Gothenburg, Sweden Sept. 11–13, 2013 641 - 651
Tijerina L. 1996 Final Report--Workload Assessment of In-Cab Text Message System and Cellular Phone Use by Heavy Vehicle Drivers on the Road US Department of Transportation, NHTSA Washington, DC
Tijerina L. 2000 Driver Distraction with Route Guidance Systems US Department of Transportation, NHTSA Washington, DC
Tijerina L. 2000 Issues in the Evaluation of Driver Distraction Associated with In-Vehicle Information and Telecommunication Systems Transportation Research Center Inc.
McKnight A.J. , McKnight A.S. 1993 “The Effect of Cellular Phone Use upon Driver Attention,” Accident Anal. Prevention 25 (3) 259 - 265    DOI : 10.1016/0001-4575(93)90020-W
Haigney D. , Westerman S.J. 2001 “Mobile (Cellular) Phone Use and Driving: A Critical Review of Research Methodology,” Ergonomics 44 (2) 132 - 143    DOI : 10.1080/00140130118417
Harbluk J.L. 2007 “An On-Road Assessment of Cognitive Distraction: Impacts on Drivers’ Visual Behavior and Braking Performance,” Accident Anal. Prevention 39 (2) 372 - 379    DOI : 10.1016/j.aap.2006.08.013
Burnett G.E. , Summerskill S.J. , Porter J.M. 2004 “On-the-Move Destination Entry for Vehicle Navigation Systems: Unsafe by Any Means?,” Behavior Inf. Technol. 23 (4) 265 - 272    DOI : 10.1080/01449290410001669950
2005 Statistical Analysis of Traffic Accident: RTSA Annual Report 61 - 70
Park S.K. 2008 “Road Accidents in Korea,” Int. Association Traffic Safety Sci. Res. 32 (2)
Green P. 2004 “Driver Distraction, Telematics Design, and Workload Managers: Safety Issues and Solutions,” SAE Int. Convergence Int. Congr. Exposition Transp. Electron. Pittsburgh, PA, USA Session: User Interface Challenges Amid the Increasing Complexity of Automotive Interiors, paper no: 2004-21-0022 165 - 180
Lamm R. 1998 “A Practical Safety Approach to Highway Geometric Design International Case Studies: Germany, Greece, Lebanon, and The United States,” Transp. Res. Circular (E-C003) 9:1 - 14
Lamm R. 2001 “Introduction of a New Approach to Geometric Design and Road Safety,” South African Transp. Conf. South Africa: Meeting the Transp. Challenges in Southern Africa Pretoria, South Africa 1 - 14
2003 State Highway Geometric Des. Manual NZ Transport Agency, Waka Kotahi Wellington, New Zealand “Section 2: Basic Design Criteria,” 1 - 32
Bertolazzi E. 2010 “Supporting Drivers in Keeping Safe Speed and Safe Distance: The SASPENCE Subproject within the European Framework Programme 6 Integrating Project PReVENT,” IEEE Trans. Trans. Intell. Transp. Syst. 11 (3) 525 - 538    DOI : 10.1109/TITS.2009.2035925
Jimenez F. , Aparicio F. , Paez J. 2008 “Evaluation of In-Vehicle Dynamic Speed Assistance in Spain: Algorithm and Driver Behavior,” IET Intell. Transp. Syst., Special Session 2 (2) 132 - 142    DOI : 10.1049/iet-its:20070019
Liu Y.C. , Ou Y.K. 2011 “Effects of Age and the Use of Hands-Free Cellular Phones on Driving Behavior and Task Performance,” Traffic Injury Prevention 12 (6) 550 - 558    DOI : 10.1080/15389588.2011.607197
Lee R.Y.W. , Carlisle A.J. 2011 “Detection of Falls Using Accelerometers and Mobile Phone Technology,” Age Ageing 40 (6) 690 - 696    DOI : 10.1093/ageing/afr050
Lee G.O. , Bae S.B. 2009 “Effects of Word Cognition on Frequency of the Morpheme and Syllable Notation,” Korean J. Cognitive Sci. 20 (3) 309 - 333