The survey showed video-clips that included illumination behaviour of two concepts, called ‘illumination concept’ and ‘base concept’. Both concepts convey attentional cues on two levels. The first level concerns ‘soft’ warnings that request attention to potentially dangerous road-traffic situation. The second level comprises ‘hard’ warnings requiring drivers to retake control because automation stops. Based on previous research (Van den Beukel et al., 2016), both concepts use the same sound signals to distinguish between soft and hard warning. Both concepts also use a blue-illuminated ring around the steering wheel as an indicator when automation is active. In addition to the base concept, the illumination concept highlights frame-sections at the sides of the windscreen to provide a spatial cue where attention is needed. For example: Figure 1b shows a light-cue at the left side of the windscreen to warn for a cutting-in car. In parallel to sound, the illumination concept conveys urgency with colour: a request for attention (moderate urgency, i.e. soft warning) is combined with yellow illumination. Required take-over (high urgency, i.e. hard warning) comes with red illumination. When using the illumination concept, the assumption is that the directional light-cues improve driver’s observation of the road-traffic situation which in turn improves understanding why attention is needed (Merat et al., 2014). Therefore, the configuration of concepts was chosen to identify if the additional directional light-cue of the illumination concept raises understanding why attention is needed compared to the base concept without such cue.

Table 1 lists the scenarios used in this study with visual examples. Except for scenario 3, the scenarios are selected from a previous study (Van den Beukel & Van der Voort, 2017) and distinguish: (a) ‘hazardous’ scenarios when attention from the driver is requested (i.e. ‘soft’ warning), but without immediate danger for an accident; (b) ‘critical’ scenarios, which hold a danger for collision. During the second category the concepts raise a ‘hard’ warning and require the driver to retake control. The video-clips provide a simulated motorway environment in line with the described scenarios. The video-clips showed other vehicles on close distances, as to simulate traffic congestion. Timing when a soft warning or hard warning was raised depended on the scenario’s specific traffic situation and is explained in Figure 2. Each video-clip lasted between 80 and 100 seconds.

The independent variables comprise Concept-type (illumination concept or base concept) and Scenario (shown with the video-clips). The dependent variables are Situation Awareness (SA) and Concept Acceptance. Measurement of SA covers cognitive capabilities with regard to driver’s perception of automation-mode and understanding of how a system reacts to different situations (Endsley, 1995; Jin & Kaber, 2009). Measurement of SA is therefore widely accepted in areas important to supervisory tasks (Salmon et al., 2009). In parallel to Endsley’s definition of SA (Endsley, 1995), we measured: (1) Operator’s perception of automation-mode (correct knowledge what mode the automation is in); (2) Understanding why the automation is in a particular mode; (3) Interpretation of driver’s role (i.e. request for attention or required intervention). Measurement of SA is based on freeze-probe technique (Salmon et al., 2009). Concept Acceptance was tested by asking respondents’ consent how suitable the signals were for the specific scenario, as well as how hazardous the situation was perceived.

The survey is set-up as a between-subjects comparison of concept. Therefore the survey is distributed in two samples, each representing a Concept-type. The video-clips included the signal-behaviour of a concept. Questions after each video were intended to find out participant’s understanding of the signals. The survey was designed to show participants two times a set of four videos, see Table 2. Each set included scenarios 1, 2, 4 and 5 in random order. After each video within the first set, the survey either asked for respondents’ perception of automation-mode (Q1) or interpretation of their role to control the automation (Q3). Thereafter, the survey always asked for respondents’ understanding of automation-mode (Q2) and their judgement of the signals’ suitability (Q4) for the scenario at hand. The questions on perception of automation-mode and driver’s role were multiple choice. Participants’ understanding of automation-mode was asked with an open question. The combination of multiple choice and open question was done to allow sufficiently easy response for respondents and at the same time acquire sufficiently detailed answers for qualitative assessment of respondents’ understanding. The order of questions was mixed to avoid prejudice about the aspects to be evaluated.

The second set showed again scenarios 1, 2, 4 and 5 in randomized order. Now, the survey asked for either respondents’ perception (Q1) or interpretation of the driver’s role (Q3), interchanging the aspects as asked with the first four videos. In addition, the survey asked how hazardously respondents perceived the situations in each video (Q5). Rehearsal of the videos avoided that answering all five questions with each video would rely too much on visual memory. However, to avoid that respondents become accustomed to the situations shown in the videos, the order in each set was randomized. Furthermore, the first and second set was separated with a scenario not shown before (i.e. scenario 3). When a participant finished watching a video, it could not be replayed. The survey could be interrupted and continued, but needed to be finished within 72 hours. On exceedance of this time-span data were excluded from analysis.

The survey was set-up and hosted with web-based software from Qualtrics (www.qualtrics.com) and distributed in the Netherlands, United Kingdom and Germany. Ethical approval of the survey was confirmed by the authors’ department board at University of Twente. To enable participation, respondents needed to confirm possession of a driving license and a minimum age of 20 years. Invitations included a link to the Qualtrics-website where the survey was posted. The invitation explained that participation is anonymous, at a voluntary basis and without remuneration. Invitations were sent by the authors’ employers support offices via e-mail and social media. In addition, invitations were sent within the authors’ professional and private networks. The authors’ direct group of colleagues was instructed not to take the questionnaire because they could be familiar with the concepts. Because participation was anonymous and other than the socio-demographic questions no detailed personal data were taken, exact numbers about participant’s occupation and education are not available.

Participants were instructed to consider themselves the responsible ‘driver’ of a vehicle that drives automatically. It was explained that automation was only possible within boundary conditions, like travelling on motorways, with low speeds and while a target-vehicle is recognised. Respondents were informed that situations could happen requiring extra attention or to retake control. Participants were explained that in the videos the automation will announce signals (i.e. stimuli provided by the concepts) to either ask for attention or urge for intervention. However, respondents were not familiarized with the signals beforehand. Table 3 lists the instruction which was spoken during the introduction video.

Due to the survey’s between-subject set-up, data-sets from two survey-samples were created. Per question, the number of responses varied between 49 and 55 for sample one, and between 39 and 45 for sample two. This variation is due to not all questions being mandatory to answer and posed a mistake in the surveys’ set-up discovered after the questionnaires were already being distributed. In total 100 people answered the questionnaire. Table 4 shows that respondents’ average age was 44.2 and driving experience was in average 23.0 years. Vehicle-usage was dominated by commuting and business trips. Gender was comparably divided over both survey-samples with a minimum of 25% female respondents. Both survey-samples show the same distribution of respondent’s characteristics and therefore allow comparison. With respect to experience with automation, 82% of participants had experience with cruise control (equally divided over both surveys), None of the participants’ vehicles was equipped with adaptive cruise control, but seven participants in survey 1 and one participant in survey 2 had vehicles equipped with automated parallel parking.

The question “What aspect of the automation changed shortly before the video stopped?” measured participant’s perception of automation-mode. The options were: The automation…: (a) stopped, (b) became unavailable, (c) did not change, (d) became available, (e) became active, or (f) I don’t know. During the two hazardous scenarios (1 and 2) the situation only required attention and option (c) was correct. At the end of the two critical scenarios (4 and 5) the automation stopped. Figure 3 shows that in all scenarios the illumination concept provides better (more correct answers) or equally good perception of automation-mode.

Figure 4 visualizes the distribution of participants’ selected options per scenario and per concept. Within scenarios 2 “Vehicle passing” (Figure 4b) correct perception of automation-mode was higher with the illumination concept (60%) compared to the base concept (41%): χ²(1, N = 99) = 3.64, p-value = 0.056 (two-tailed). The results also show that correct perception was difficult in this scenario: With the base concept almost half of the participants (44%) did “not know” the automation-mode compared to 29% with the illumination concept. In scenario 2, the alert was induced by traffic behaviour aside (i.e. illegally overtaking). In scenario 1 “Complex road” the alert was induced by a change in road situation ahead. When comparing results between both situations, analysis of regression revealed that Scenario significantly predicted perception scores, β = 0.232, t(185) = 3.320, p < 0.005 (R² = 0.056, F(1, 185) = 11.02). Concept showed less power to predict perception scores within the hazardous scenarios, β = -0.133, t(185) = -1.861, p = 0.064 (R² = 0.018, F(1, 185) = 3.46).

Within the two critical scenarios (4: “Merging-out target-vehicle” and 5: “Close cut-in”) most respondents answered with either “automation stopped” or “automation became unavailable”. In scenario 4, a target-vehicle disappears without the availability of a new target-vehicle. Therefore, both answers are considered correct: It makes the system become unavailable and it subsequently stopped. Results show slightly better perception with the illumination-concept (82% correct perception), compared to the base-concept (72% correct perception), see Figure 4c. These numbers do not denote statistical differences.

Although the answers of option (a) “The automation stopped” and option (b) “The automation became unavailable seem identical, there is in scenario 5 (“Close cut-in”) a difference in meaning between both answers: Here automation stopped because the cut-in vehicle violated minimum required follow-distances. Based on assumed system-design, deceleration-rates of the ego-vehicle would not suffice to bring the vehicle to a safe stand-still. This is why automation stopped. The technical preconditions for automation, i.e. line and target-recognition, plus driving on a motorway, did not change. Therefore, the answer “automation became unavailable” does strictly speaking not apply. However, based on general interpretation of the words “unavailable” and “stopped”, it is defendable to consider both options correct. In the first case correct perception is 44% with the base-concept and 41% with the illumination-concept, see Figure 4d. In the second case, correct perception is with each concept identical in this scenario, i.e. 64%.

When comparing results between the two critical scenarios (i.e. scenario 4 “Merge-out” and 5 “Close cut-in”), analysis of regression revealed again that Scenario significantly predicted perception scores β = 0.136, t(185) = 2.044, p = 0.042 (R² = 0.022, F(1, 185) = 4.18) – but Concept did not reveal power to improve perception.

Overall, we see that correct perception of automation-mode is better with the illumination-concept (70% correct perception) compared to the base-concept (60% correct perception), see the bottom item in Figure 3. For this overall result both answers “automation stopped” and “automation became unavailable” are considered correct within the two critical scenarios. These numbers denote a trend that the illumination-concept improves correct perception of automation-mode (χ²(1, N = 374) = 3.82, p-value = 0.051, two-tailed). Moreover, when we consider in scenario 5 (“Close cut-in”) only the option “automation stopped” correct (because strictly speaking the other option does not apply), the concepts show over all scenarios statistically significant differences in favour of the illumination concept: χ²(1, N = 374) = 4.01, p-value = 0.046, two-tailed.

Measurement of understanding was based on respondents’ answers to the question: “Was there with regard to the traffic situation a reason to give signals?” This was an open question and participants were asked to give their answer with a (short) explanation. The appendix summarizes the results and shows participants’ understanding why an alert was raised per scenario and per concept.

Results for understanding automation-mode in scenario 1 (“Complex road”) do not differ between concepts: 70% of respondents understood the likelihood that other traffic would enter the main road. Understanding of automation-mode within scenario 2 (“Vehicle passing”) does show differences between concepts. 53% Of respondents had correct understanding with the base-concept, compared to 72% with the illumination-concept (χ²(1, N = 88) = 3.215, p-value = 0.073 two-tailed).

Results during scenario 3 (“Approaching combined on/off-ramp) show that the illumination-concept caused misunderstanding. An alert was raised for entering a main road and only 21% of respondents understood this correctly with the illumination-concept, compared to 46% with the base-concept. A specific explanation why the illumination-concept scored worse is difficult to find. In general the situation was difficult to understand: While it was explained that the system would only work on motorways, the vehicle was in this scenario on an entrance-lane and respondents could have had doubts whether the entrance-lane was part of a motorway or not. The scores also reveal a general difficulty to distinguish whether the signal refers to a road-situation or to potentially dangerous behaviour of other road-users: Hence, a considerable part of respondents (almost 15%) gave in the category ‘something different’ reasons like: “vehicles coming too close”, or “car in blind spot”.

In scenario 4 (“Merge-out target-vehicle”) understanding was generally low and results show that the concepts did not influence understanding of mode-changes. Only 1/3 correctly understood that the target-vehicle left the lane and that there was no new target-vehicle. The road situation was the same as in scenario 3 and again confusion whether the exit-lane belonged to the motorway or not, could be an explanation for misunderstanding. Moreover, after the automation stopped, the uncontrolled ‘ego’-vehicle ran out of its lane. It is likely that it was not the absence of a target-vehicle, but the road-position and behaviour of the ‘ego’-vehicle that made respondents suspicious of the need for attention.

Although results for participants’ perception (see previous paragraph) within the close cut-in scenario (5) showed indifferent concept-scores, understanding of automation-changes show some results in favour of the illumination-concept. In this close cut-in scenario, not the cutting-in as such caused mode-changes, but the close distance caused violation of minimum required follow-distance. With the illumination-concept respondents showed a trend to distinguish better (40% correct understanding) that a minimum follow-distance is being erupted, compared to the base-concept (23% correct understanding): χ²(1, N = 95) = 3.514, p-value = 0.061 (two-tailed). However, if we broaden correct response to more generally considering that cutting in is dangerous and that this behaviour is often referred to in the answers, we don’t observe differences between the concepts (χ²(1, N = 95) = 0.375, p-value = 0.541, two-tailed).

To summarize: If changes were only induced by the behaviour of other road users (i.e., scenario 2 “Vehicle passing” and 5 “Close cut-in”), respondents showed better understanding of automation-mode with the illumination-concept. In both scenarios signals were raised because of hazards that originated from the driver’s side-views. Therefore, we consider these results a confirmation of the directional cue offered by the illumination-concept. If the alert was induced by the road situation ahead (i.e.; scenario 1 “Approaching complex road”), understanding was high but indifferent between concepts. If the situation was characterized by an interplay between road-situation and behaviour of other road users (i.e.; scenario 3 “Approaching an on/off ramp” and 4 “Merge-out target-vehicle”), understanding was low and we assume that confusion whether the alert (or warning) was induced by the road-situation or by the behaviour of other road users, tempered correct understanding.

Situation Awareness on level 3, i.e. projection of future state, requires correct interpretation of driver’s role. To measure SA-level 3 this survey therefore asked: “What do you think the signals at the end of the video wanted to tell you?” Possible answers were: “I should take over”; “I should watch traffic carefully” and “I don’t know”.

Table 5 shows only within scenario 3 “approaching on/off ramp” differences between concepts to help interpreting driver’s role. Then, the illumination-concept (78% correct interpretation) shows a trend for superior interpretation compared to the base-concept (62% correct interpretation): χ²(1, N = 93) = 2.91, p-value = 0.088 (two-tailed). In all other scenarios, scores between concepts are indifferent and show relatively high values of correct interpretation. Remarkably, the previous section showed that respondents had difficulties with understanding the situations in scenario 3 and 4. Although these difficulties in understanding were based on similar confusion due to the interplay between road-situation and behaviour of other road-users, a vast majority correctly recognised that their responsibility differed between both scenarios and correctly interpreted their role.

Figure 5 shows results for acceptance. Respondents were asked after each video to rate their consent with the statement “The signal was suitable for the situation” on a Likert-scale ranging from 1 (“totally disagree”) to 5 (“totally agree”). Differences in concept-scores are in favour of the illumination-concept. Within two scenarios, the illumination-concept receives significant better acceptance. These two scenarios (2 and 3) are both ‘hazardous scenarios’, requiring supervisory control. Based on cumulative scores (taken all scenarios together), illumination (M = 3,57; SD = 0,99) is perceived more suitable than no illumination (i.e. the base-concept) (M = 3,12; SD = 1,14). The difference between both concepts is extremely significant: t(478) = 4,597, p-value < 0.001.

Figure 6 shows that the illumination-concept has influenced respondents’ opinion how hazardous they perceived the situations to be. Within scenario 1 “Approaching complex road”, respondents perceived the same scenario with the illumination-concept significantly more hazardous than with the base-concept: t(86) = 2.962, p-value = 0.004. The relatively low scores in perceived ‘hazardousness’ are in line with the designed low level of urgency for this scenario.

A concern is that illumination influenced ‘perceived hazardousness’, i.e. made situations perceived more urgent for possible intervention. Although the additional level of ‘hazardousness’ is small, it is consistently observed in all scenarios. Besides, in scenario 1 – a scenario with a generally low ‘hazardousness’ – this effect is significant. A small increase would not necessarily be disadvantageous, as it might increase driver’s carefulness. However, a larger influence raises two concerns. Firstly, if in practice soft-warnings are perceived as high urgency this would raise false warnings and cause misunderstanding. Secondly, urgency is in general related to annoyance (Baldwin & Lewis, 2014) and if perceived urgency is considerably higher than intended, this would lower acceptance. The potential influence of directional illumination on annoyance needs therefore investigation during further development of the concept.

When interpreting the results, a concern is that the survey was filled out by participants themselves using their own equipment to watch the videos without the ability to control screen sizes. For participants’ convenience the survey could be interrupted and continued (with time restrictions) at a later stage. Also, this study was based on watching short video-clips without a need for actually retaking control and neither the temptation to monitor the road less, something that would be the case during prolonged driving with partially automation.

Providing visual cues within the windscreen or side windows might be less effective when drivers have taken their eyes completely off the road. Other means to provide directional feedback – for example via the steering wheel (Johns, Mok, Talamonti, Sibi, & Ju, 2017), are not included in this study but need consideration in future research.

Another limitation is that older people and female are underrepresented in our study. The average age was 44.2 years and 28% of respondents were female. An older group of participants and more equal division of gender may have influenced results. Furthermore, the questionnaire has not been controlled to represent a cross profile in societal groups. Hence, a majority of respondents originated from the authors’ social and professional networks. This may have biased the results. Therefore, future research should include trust and acceptance while acknowledging societal differences.

Another comment concerns the absence of competing signals and immersion. In practice, more input from competing functions is likely to cause a larger variety of signals which might worsen correct perception. Therefore, care should be taken with interpreting the results for real-world circumstances. With regard to perception of automation-mode this study revealed that the influence of Scenario on mode awareness was larger than the influence of Concept. Probably, the contextual situation defined situation awareness more than can be compensated for through interface-design. In real world circumstances, the driving situations and events are likely to be more diverse and immersive and might therefore have an even larger influence on mode awareness. At the same time, in real world circumstances the need for support is larger as drivers are expected to monitor the road less when using partially automated driving. Furthermore, the high scores with regard to correct interpretation of driver’s role might be a trade-off of our particular option-based questionnaire, rather than the result of particular interface-features. Finally, a limitation is that mode awareness has not been compared with a traditional interface designed to provide mode-feedback. Consequently, further research is needed within circumstances closer to real-world situations and with additional concept comparison in order to make more reliable predictions on the ability of directional illumination to raise mode awareness and to improve drivers’ understanding of the control task.