MenShen:Systematically Ensuring The Confidence of Real Time Home Automaton IoT Systems
Introduction Download Case Studies User Studies

User Studies


User Studies For MenShen

To support the motivation of this work, we have also performed some user studies based on the MS-3 scenario used in the benchmark. We have generated 3 different safety properties and have two group of volunteers. Each group has 45 participants. The study participants in the first group include researchers and interns from Microsoft Research Asia, and PH.D. and Master Students in software engineering discipline at Nanjing University. The participants in the second group include Non-CS major college students, high school students, and also some housewives.

We gave the participants the floor plan of the room with all the rules connecting the devices. Then, we ask them to decide whether the connected IoT system can violate any safety specifications. We also ask the participants to try to fix the error manually. The executable user study program is available form here. The data of first group we get can be downloaded here and second group here. We organized the data in the table below. From the data, we can see most of the participants cannot find the bug, and only few of them can fix the system successfully. On the other hand, MenShen is able to find the bug and fix them in seconds.

 

User Study on MS-4 Scenario from CS-Majored Students and Researchers

Problem Total Participants Average Total-Time(s) Average Check(s) Average Fix(s)* Able to Find Conflicts Able to Fix Menshen Check(s) Menshen Fix(s) User Acceptance
Q1
45
453
444
82
7/45
2/45
3.56
10.50
39/45
Q2
45
98
90
63
7/45
2/45
3.82
17.77
37/45
Q3
45
75
63
57
11/45
2/45
3.14
9.72
45/45


User Study on MS-4 Scenario from Non-CS-Majored Participants

Problem Total Participants Average Total-Time(s) Average Check(s) Average Fix(s)* Able to Find Conflicts Able to Fix Menshen Check(s) Menshen Fix(s) User Acceptance
Q1
45
326
443
61
14/45
5/45
3.56
10.50
30/45
Q2
45
85
70
26
6/45
2/45
3.82
17.77
33/45
Q3
45
48
47
32
13/45
10/45
3.14
9.72
31/45

*For the average fixing time, we only consider the results from the studies which give fix suggestions. As we can see from the table, most of the participants cannot conduct this task. Therefore, their fixing time is not took into consideration.

 

 


©2017 SEG, Nanjing University