Collaborative design of software applications: the role of users
Mirri et al. Hum. Cent. Comput. Inf. Sci.
Collaborative design of software applications: the role of users
Drawing on a 1-year application design, implementation and evaluation experience, this paper examines how engaging users in the early design phases of a software application is tightly bound to the success of that application in use. Through the comparison between two different approaches to collaborative application design (namely, user-centered vs participatory), we reveal how sensitivity to the role that users may play during that collaborative practice rebounds to a good level of user satisfaction during the evaluation process. Our paper also contributes to conversations and reflections on the differences between those two design approaches, while providing evidences that the participatory approach may better sensitize designers to issues of users' satisfaction. We finally offer our study as a resource and a methodology for recognizing and understanding the role of active users during a process of development of a software application.
Participatory design; User-centered design; Users' evaluation; Users' satisfaction; Co-creation
The role of users in design and creation processes of computational artifacts is at the
basis of many HCI methodologies. In the evolution of such methodologies, the role of
the user is becoming more active, being the main character in several steps of the most
commonly used approaches for the development and release of applications and tools.
We are witnessing at a wide diffusion of many initiatives of open innovation and
cocreation in different fields, where different kind of organizations are gradually moving
away from traditional design models, becoming more inclusive in their approaches to
design and development. Both co-creation and open innovation design techniques are
based on a strong involvement of users in the design process [
]. While open
innovation suggests active collaboration among different organizations and the sharing of
intellectual property, co-creation involves users of a product or service in exchanging their
knowledge and resources with the aim to deliver a personalized experience. Co-creation
relates more specifically to the relationship between an organization and a defined group
of users, who are engaged directly by involving them in the creation of a product or a
service, i.e. in its design and in its development processes. Definitions of open
innovation and co-creation can be found in [
© The Author(s) 2018. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License
(http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium,
provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and
indicate if changes were made.
Focusing on the co-creation of software applications, there are different HCI design
methodologies involving users that can be applied, such as user-centered design and
participatory design. The User-centered design approach was defined more than
30 years ago [
] and it consists in designing and developing applications or products
where a team of designers focuses on users’ needs in an iterative way. The team
members together plan, create, and develop a project, focusing mainly on design activities
and processes. While having the user at the center of the process, a user-centered design
can be done without any actual participation of real users, who can be virtualized during
the whole design/development/evaluation cycle (modelling stakeholders as personas, or
simulating them) [
]. In few words, the user-centered design process can be conducted
without assigning to users any active role, but designers are focused on the application
being designed, looking for ways to ensure that it meets the needs of the user.
A structured and official definition of participatory design came into the scene later, in
]: this design methodology consists in letting the users actively contribute to the
design and to the content development process. The participatory design involves users
in the process as co-designers, in a more in-depth way. In fact, they can propose and
generate design issues, functionalities, services of the application being designed. Hence,
participatory design supports different ways of planning and thinking, making
applications and products more responsive to human needs. It basically brings together
designers, developers, and users to think and define the contexts of use at the design phase [
Summarizing, in participatory design experiences, the roles of designers and developers
blur, while the role of users becomes a more prominent component of the process.
Differences and similarities of these two approaches are still debated, as witnessed by
discussions and topics in the most commonly used social networking sites for scientists
and researchers (such as Researchgate.net [
] and Academia.edu [
]). In particular,
intersections and borders of these approaches, together with their application, are still arising
interest in the academia. Some interesting examples are reported in [
]. Indeed, one
of the most interesting interpretation is that participatory design is user-centered, when
it focuses on the interest and on the satisfaction of the end-users .
In this context, our aim is to investigate if a more deeply engagement of users (reached
through a participatory design process) corresponds to a higher level of satisfaction in
users of the designed application, when compared with the satisfaction level of the users
who are exploiting an application resulting from a more traditional process, based on a
user-centered design approach.
In order to reach this goal, we have applied the user-centered design and participatory
design approaches to the development of two applications, where end-users have been
engaged. Then, we have involved users in the evaluation of the two resulting prototypes,
with the aim of measuring their satisfaction in using them, and of discussing emerged
differences. In particular, here we present two experiences conducted at the University
of Bologna, where students collaborated in the design of two applications thought to be
used by exploiting mobile devices:
1. An application designed to support foreign students (Erasmus, exchange, overseas,
International, mobility students, etc.) in their daily activities, getting in touch with
local language and culture, which enhances their integration and their interaction
with the country hosting them.
2. An application designed to support high school students while approaching their
undergraduate programme and become a freshman.
While the first application was designed with a user-centered approach (users were
not involved in defining goals, functionalities, and services of the application, they
were engaged just in a latter design phase, by means of questionnaires, focus groups,
and experience prototyping activities), the second one was based on a participatory
approach (users were asked to propose ideas in specific co-creation sessions). After the
development of two prototype applications, we have conducted some tests with
stakeholders, letting them evaluate the application. Then we asked them to fill a survey, with
the aim of comparing users’ satisfaction and of evaluating how the resulting prototypes
meet users’ expectations and needs. Moreover, we want to discuss about differences in
expectations and satisfaction when users were involved in both the design and in the
evaluation processes, and when they were involved only in the final evaluation phase.
In particular, we aim to observe if there are differences between the satisfaction levels
reported by these two groups of users. The results we have obtained show how the
prototype resulting from the participatory design corresponds to a higher level of users’
satisfaction and how it is more compliant with users’ expectations than the application
coming from the user-centered design approach.
The remainder of the paper is organized as follows. “Research questions and
methodology” section reports our research questions and the methodology we have applied.
“User-centered design: the ILocalApp experience” section presents the experience based
on the user centered design approach, while “Participatory design: the AlmaOrienta
App experience” section describes the process based on the participatory design. “On
comparing application functionalities and services” section illustrates some common
elements and functionalities emerged by the two design processes, highlighting main
differences. A final evaluation phase conducted with end users is reported in “On
evaluating the design processes and their results” section, detailing users’ satisfaction and
assessing how the resulting prototypes meet users’ expectation. Finally, “Conclusion”
section concludes the paper illustrating main findings and further work.
Research questions and methodology
In this section, we report the research questions (RQ) we have formed and the
methodology we have applied with the aim of prove them:
• RQ1: Users’ satisfaction is proportional to real users’ engagement in the design
process of an application.
• RQ2: Applications resulting from participatory design approach better meets users’
expectation than the ones resulting from user-centered design.
In order to answer to our research questions, we have taken the chance of applying
different methodologies in the process of designing two applications devoted to meet
different needs of users at the University of Bologna, with different levels of users’
engagement in such a process.
In the first case, we have applied a user-centered design approach while defining goals,
functionalities, services, user interface and interaction for an application devoted to
support foreign students hosted by a university while they are experiencing an exchange
programme. In particular, in this experience, a team of designers has defined the goals,
the services and the functionalities offered by the application, on the basis of a user
model, aiming to meet the needs of the user. In a second phase of the design process,
real stakeholders (i.e., International students, students of mobility programmes, past
and future Erasmus students) have been engaged by filling a questionnaire and by taking
part in focus groups and experience prototyping activities, as detailed in “User-centered
design: the ILocalApp experience” section.
In the second case, we have applied a participatory design approach to the
development of an application devoted to support high school students while they are choosing
their undergraduate programme and while they are becoming freshmen. In both cases,
we have involved target users in the design processes. In this context, the experience was
mainly based on participatory design sessions, where high school students have taken
part, by proposing ideas and by sharing requirements and desiderata, and on a
development contest, where undergraduate degree proposed their solutions and prototypes, as
illustrated in “Participatory design: the AlmaOrienta App experience” section.
After the design phase, we have developed the applications (thought to be used by
means of mobile devices), whose main features are presented in “On comparing
application functionalities and services” section. Then, we have involved target users in
testing our prototypes and we have collected feedbacks by letting them fill a questionnaire.
Some of the users involved in this last phase were originally involved in the design
process, while some others did not take part to these initial activities, letting us discuss
about how these latter users show a higher level of satisfaction (as described in “On
evaluating the design processes and their results” section).
User‑centered design: the ILocalApp experience
The ILocalApp project (incidentally learning other cultures and languages through an
APP) is a 3-year Erasmus+ KA2 project, carried out by a transnational consortium
(University of Bologna, Italy; the Adam Mickiewicz University, Poland; University of
Lapland, Finland; Centre for Social Studies, Portugal). Its main goal is to design and develop
a mobile application for incidental learning [
] of four cultures and languages:
Finnish, Italian, Polish, and Portuguese . ILocalApp aims to let its users acquire language
and culture, thanks to learners’ participation in an informal way [
]. Learners would
be able to wander around the city while using the application to enjoy language and
culture geolocalized contents in a context-aware way. Thus, mobility students (e.g. Erasmus,
exchange, overseas, International, mobility students, etc.) would be supported in their
daily activities, getting in touch with local language and culture in a simple and
innovative way, which enhances their integration and their interaction with the country hosting
The project has been based on a user-centered design approach, with the aim of
designing and developing a mobile app, which takes into account the user and his/her
needs, preferences, abilities, since the very first step (as described in [
]). In order to
reach this goal, the project has carried out many activities, such as focus groups [
and experience prototyping [
], by involving undergraduate and international students
at the four partners sides, after an initial phase during which designers have defined
goals, features, and services of the application. The results coming from such activities
have driven the development. In particular, students were asked to participate by filling
a questionnaire (as described in “Phase one: questionnaire” section), by attending focus
groups (as presented in “Phase two: focus groups” section), and by evaluating a
preliminary prototype through an experience prototyping activity (as reported in “Phase three:
experience prototyping” section).
Phase one: questionnaire
An online survey has been launched with the aim of finding out students’ views on
language learning apps and the role of apps and mobile devices in their everyday life. The
survey was launched from February 16th, 2016 to March 28th, 2016 and it had reached
altogether 2350 answers all over Europe.
The survey was structured in 20 questions, grouped in three parts, devoted to identify
and collect information about: (i) culture areas of interest, views on using language and
culture apps; (ii) usage of apps and mobile devices (e.g., operating system on the smart
phone, most commonly used apps, frequency of use, etc.); (iii) personal details (e.g., age,
gender, first language, etc.). The most relevant results coming from such a survey were
exploited to better identify the nine categories of contents, the issues motivating users
of language and culture apps (short and useful learning moments, interactive, tracking
progress, efficient), and the most commonly used language and culture apps (such as
Duolingo and Memrise). 53.5% of the users declared they usually install apps with offline
maps and/or city guides or similar when they are planning a trip. Taking into account
personal details, 42.9% are aged 19–22 years, while 37% are between 23 and 26 years,
confirming the targeted audience of the project, which was constituted by young and
emerging adults. Finally, the targeted groups of users have been also confirmed by how
the users have identified themselves: students who are planning to go on an exchange
(39.4%); users who currently are or have previously been exchange students (33.1%);
international degree students (12.9%).
Phase two: focus groups
In April 2016 focus groups were held in all the ILocalApp institutions with the aim of
getting feedback and comments from participants on the functionalities, the
interactions, and the flow of the application that we were developing (in terms of how and
when offering contents and activities). In particular, the goal of the focus groups was
progressing in the definition and the design of the application functionalities and
services. Table 1 summarizes the students involved from the 4 partners in the focus groups
(as well as in the experience prototyping activities, described in “Phase three: experience
prototyping” section). Participants were selected among the identified target groups
(mainly international students, past, present and future Erasmus students, exchange
students, overseas students, etc.).
The discussion of all the sessions was structured in five main topics, as agreed by all
the involved institutions as detailed in the following. Some mockups have been exploited
so as to facilitate the discussion among the participants. The discussion brought
concrete suggestions and hints from the participants who actively discussed the proposed
topics during the sessions. Reporting all the collected feedback and comments coming
from the 4 partners, we can summarize as follows the focus groups results:
1. Content organization The importance of having information related to the user’s
position emerged as a common element in all the conducted sessions. In
particular, it has been identified as strategic having an integration of mobile/web mapping
services (such as Google Maps) in the application. Information about public
transportation, events, and points of interest have been defined as interesting, as well as
information about useful services (e.g. pharmacies, hospitals, citizenship/immigrant
services) in the nearby. Particularly appreciated would be having language tips of
immediate use, context-related, “how-to” phrases, including survival dictionaries and
handy expressions. Cultural issues have been reported as crucial, in terms of places,
monuments, cultural heritage, but also as a concrete way of living.
2. Application usage The usage of the application before going to the place has been
identified as very useful to start to know the language and the place, to have
communication tips, and to get used to the application. Obviously, the emphasis is
recognized during the stay, when the application is mostly important.
3. Usage frequency Among all the institutions, participants marked that the application
should not force the users, even if some notifications or services could be exploited
as motivation (eventually by means of game-like elements [
]). It is important to
let the user free to decide and modify the level of notification. The rhythm of usage
may vary (ideally, it could be higher at the beginning of the stay), without forcing the
user to do a minimum of activities.
4. Interface, icons vs language of instruction, presentation video, progress check
Participants declared that they would prefer a user interface based on a combination of
icons and language of instruction, with priority to images. The user should have the
possibility to change the language (English in principle, but then they could switch to
the local language). Game-like elements have been recognized as useful to stimulate
and to monitor progress.
5. Interaction with other learners and with native speakers This last topic was the one
with most opposite and contrasting results. In particular, two complete opposite
positions emerged from all the focus group sessions. On the one side, participants
in Bologna and in Coimbra considered as relevant the interaction with other users
(both native speakers and other learners), eventually integrating or linking other
existing networks and social media in the application. On the other side, participants
in Poznan and Rovaniemi claimed that there is no need to add functionalities and
services to support interaction, since there are plenty of means, tools, channels, and
strategies with this aim that can be exploited.
Phase three: experience prototyping
In May 2016, experience prototyping sessions were held in the ILOCALAPP institutions,
with the aim of getting feedback and comments from participants on the functionalities,
the interactions, and the flow of the application that we were developing, by means of a
realistic simulation in a real world context, by exploiting the “Think aloud protocol” [
Table 1 summarizes the students involved in the experience prototyping activities. In
particular, the goals of the experience prototyping sessions were: (i) understanding users
and their experiences with the app prototype in a real world context; (ii) understanding
users’ needs and users’ point of view in a real user environment.
To reach these goals, we have identified three tasks the participants had to complete
during the experience prototyping session. The tasks are the same ones for all the
ILOCALAPP partners: (i) buy a bus ticket; (ii) visit a monument/museum/library; (iii) eat a
typical food. Starting from some “common” mockups, we have customized them with
specific multimedia content (in terms of pictures and texts) for each partner, as
prototype models (see Fig. 1 showing the prototype preparation phase in Bologna). Figure 2
shows a picture taken during experience prototyping sessions conducted in Bologna.
We have analyzed the notes and the reports provided by all the partners involved in
such an experience, and we have identified the main common remarks about the
• Navigation among the contents and the services It plays a strategic role. Navigation
tools should be adequately provided to come back to the previous content/menu/
interface, to come back to the home page of the application, and to move across
different services/content/functionalities related to the same content (i.e. language tips,
cultural and historical information, practical information, games, external links, etc.).
• Interface, layout and their configuration/personalization The participants have shown
opposite positions about some layout and interface elements, such as font size. This
means that interface personalization is needed [
]. The application should offer
a set of configurations to let the user choose his/her preferred combination of layout
and typographical aspects, meeting his/her needs, so as to improve the user’s
experience, also in terms of learning experience .
• External links and social network connections External links and connections to
social networks are considered important. In particular, participants expressed
interest in having: (i) external/official information related to the topic; (ii) a way to share
information/pictures/comments; (iii) exploiting already existing social networks.
• Practical and cultural content A good balance between historical/cultural content
and practical information should be found. Designers and developers have to pay
attention in providing not only practical information or not only historical/cultural
• Audio content Audios are needed, in particular for the language tips, with the aim of
effectively support users in their daily activities and in learning the language.
These results confirmed some issues already emerged during the focus groups and let
the designers and developers better understand the direction of the final design of the
application and better identify the direction of further steps in its development.
Participatory design: the AlmaOrienta App experience
To promote its programs, the University of Bologna organizes a spring college fair, called
AlmaOrienta. The fair is held in the Bologna Exhibition Center (Bologna Fiere) in 3
exhibition halls that covers more than 25,000 mq. Every year about 30,000 high school
students visit AlmaOrienta with the aim of finding support in deciding about their future
opportunities at the University of Bologna. The complexity of the programme catalogue
and the admission procedures together with the wideness of the AlmaOrienta event
drove the University to provide students with a mobile application to support them
while approaching their undergraduate programme and become a freshman. Thinking
to the digital native generation who is approaching the university in the next 2 years, we
decided to use co-creation in designing the new application, involving students in the
design process. This goal is generally challenging, but it is much more complex in the
specific situation, due to difficulties in identifying a community of users to be involved
in co-creation. On the one hand the target community, students who are attending the
last 2 years of high school have the need to be supported during and after the fair, while
deciding about their future studies, but they are not enough aware of the complexity of
the programmes catalogue to completely design a similar solution. On the other hand,
the community of university students have the appropriate level of expertise about how
the university works, but they are not necessarily aware of the real needs of the next
generation of students. To overcome this situation, we decided to involve both the
communities of students in the application design, using a combination of two co-design
• A participatory design phase involving the target group of high school students in
providing proposals. While this group of users have a clear understanding of their
own needs in terms of support in deciding and starting college studies, they have
a very partial view of the complexity of the programme catalogue offered by the
University. The participatory design phase took place during AlmaOrienta 2017, in
• A contest (with a call for ideas and a prototypes’ development track), involving
University of Bologna undergraduate students in structuring, improving and
implementing proposals arose during the participatory design phase. Students enrolled in the
University know difficulties they got in deciding and, at the same time, they have a
reasonable perception of the complexity of the programme catalogue offered by the
University. The contest took place in spring 2017 and it was completed during a final
contest session in May 2017.
The combination of the two phases produced a wide set of proposals of different on
line support services, thought to be used during the AlmaOrienta fair and after (while
high school students are deciding about their future university studies).
Phase one: participatory design
During the 2017 edition of AlmaOrienta, we conducted several participatory design
sessions, involving high school students who were attending the fair in design
activities under the moderation of colleagues from the Computer Science and Engineering
department. At the end of the 2 days of fair, we have globally collected 45 project ideas,
proposed by a community of more than 200 high school students. Figure 3 shows one
of the mockup proposed by one team composed by three of those students. Proposals
focus on 8 main topics:
a. Programmes catalogue Almost all the proposals asked for a structured presentation
of the catalogue and some support tool to better understand differences among
b. Identity The collective value of the University of Bologna is expressed though clear
and strong identity message, including a clear visual identity.
c. Tests a significant part of the University of Bologna programmes has an assessment
admissions test. Students’ designs underlined the need to be supported in studying
for each specific admission test and in managing the application for the test and the
following application for the selected programme.
d. Fair AlmaOrienta fair many design proposed by students include services to present,
explain and navigate the AlmaOrienta fair itself, together with open days and all the
events organized by the University and its departments, schools, and programmes.
e. Places The University of Bologna has a multi-campus structure, based on 5
campuses in 5 different cities. The Bologna Campus itself is structured in 9 different areas
inside and outside the city. Projects proposed to support students in physical
orienteering in the wide university space, including information about the city (e.g.,
transportation, dormitory and other residential facilities, lifestyle).
f. Job traditionally programme catalogues aim to support students in understanding
the structure of programmes, as they are expected to run for the given year. The
programme and unit catalogues are organised by school and level (first cycle, second
cycle, single cycle) without considering that different programmes can lead to similar
competences and consequently to similar jobs. Proposals arose during the
participatory design mainly focus on what jobs or professional activity users can do once
g. International although students involved in the participatory design come from cities
nearby Bologna, some proposal underlined the international dimension of the
University and proposed new services to international students or to students who may
apply exchange programmes (i.e. Erasmus, Overseas, etc.).
h. Communication style most of the participatory design projects underlined the need
of a clearer and direct communication style. They explicitly refer to chat, messaging,
forums and other social communication tools to contact counsellors, tutors,
professors, and also other students and alumni.
Phase two: mobile application development contest
The second design phase was organized as an on line (remote) application design
contest, named AlmaContest and open to the whole community of students of the
University of Bologna. Almacontest was structured in two tracks in order to exploit:
A. Ideas Track A was devoted to collect project ideas to be used in the development of
the university fair app.
B. Prototypes Track B asked students to develop a prototype to be used as a support for
high school students.
Students applied to AlmaContest in teams, ranging from 2 to 5 members, selecting one
of the two tracks. For each track, a monetary reward was available for the best 3 projects
of the track. A jury of experts, including ICT professors, design specialists and college
counselling managers, was nominated to select the best projects. Globally 59 students
applied for the AlmaContest, organized in 22 teams, 14 of them for the track A (ideas)
and 8 for the track B (prototypes). The best 5 projects of each track were involved in a
contest final session, where each selected group gave a speech (as a 5 min presentation
with slides) to the jury. The best 3 projects for each track were selected and announced
as winners immediately after the speech session. Figure 4 shows a picture taken during
I mestieri di Unibo (Unibo Jobs) focuses on future work opportunities of
graduated students, connecting professional perspectives to programmes
AlmaMatch suggests to use clustering and classification algorithms to find
similar programmes and provide potential users with a conceptual map
instead of a traditional programmes catalogue
Unibo International describes an app to support international students in
overcoming specific issues (such as obtaining VISAs, finding home, learn
AlmaOrienteering is an Android app with many features including: a guide to B
AlmaOrienta fair, a chat to talk with course tutors, an organized view of the
course catalogue, a set of metrics to compare similar courses on the basis of
Universitalk is an Android app devoted to support users in studying for the
different admission tests. It includes some gamification features in order to
keep the user engaged in exploiting some trial tests
Smartsearch is a webapp based on a crowdsourcing classification of
programmes and a semantic search engine
A, B, D, F
G, E B, H
A, B, D, E, H
1 of the 5 min presentation, in the speech session. Table 2 reassumes characteristics and
covered topics of the 6 winner projects.
Although the most part of AlmaContest projects, including the winner ones, refer to
topics underlined during the participatory design phase, some significant aspects arose
from this second design phase. Projects introduced both creative and implementation
novelties. Some very new services were proposed such as (i) using ebeacons to support
geo-localization inside AlmaOrienta fair and track users’ while they explore the fair [
(ii) using gamification so as to keep users engaged both during the fair and after (project
5, Universitalk); (iii) using a chatbot to answer students’ questions; (iv) computing
similarity among programmes by using a clusterization algorithm (project 2, AlmaMatch).
On comparing application functionalities and services
In order to compare the results emerged from the two design experiences in terms of
application functionalities and services, in this section we present the most interesting
issues emerged from applying these two design approaches, highlighting differences and
• Geolocalized information and services Both the applied design approaches let emerge
or reinforces the need of exploiting data and services on the basis of their
geolocations and of users’ position.
• Game-like elements and gamification The use of gaming factors emerged as
significant and strategic from both the applied design approaches. Mainly they could be
applied with the aim of involving users and supporting them in conducting activities
related to the application [
• Communications paradigms Different needs in terms of communication emerged.
On the one hand, from the participatory design experience it was strong the need
of a synchronous and direct mechanism of communications. Users involved in that
experience suggested the use of chats as well as chatbots. They expressed the need
of direct communications among people with different roles: from their peers, to
alumni, from professors, to administrative staff. On the other hand, a direct form of
communication with other users is not emerged as so crucial in the user-centered
• Notifications The use of push notifications is recognized as essential in both the
experiences. In user-centered design approach, it should be exploited with the aim of
letting the user be aware about the presence of points of Interest in the nearby, while
in the participatory design approach it should be used in order to report deadlines
and training activities.
• Personalization/preferences/settings The need of adapted information and services,
availability of setting preferences and personalized configurations is considered as
crucial by the users involved in both the experiences. The two adopted design
processes let this need emerge as one of the strongest requirements.
• Incidental learning mechanisms This emerged as a strong need in the user-centered
design experience, while it is not relevant in the participatory design obtained results.
• Interaction and social networks integrations The need of social networks integration
emerged from both the experiences. It has been reported as a more important need
in the participatory design side, than in the user-centered design experience. In
particular, users showed interested in the integration of communication systems similar
Some of these emerged functionalities and elements are common at the two
experiences, while some others differ. In particular, it is interesting the role and the need of
communication among users, which emerged in a totally different way in the two design
On evaluating the design processes and their results
After the development of the two prototypes, before launching the applications, we have
involved the users in testing them, collecting their feedbacks and comments by means
of a survey. In particular, we involved users who have previously taken part in the design
activities (groups A) and users who did not, who have neither bias nor specific
expectations about the applications and their purposes (groups B).
The survey was structured in three parts:
1. Personal details (e.g., age, gender, current position, etc.);
3. Additional open comments.
An additional fourth part was provided to users of groups A, who were involved in
the design activities, with the aim of evaluating their appreciation about their
involvement in the design process. We use the Likert scale [
] approach in ranking the second
part questions based on a 5 values symmetric agree disagree scale, in order of strength.
In particular, the adopted scale was as follows: strongly agree, agree, neutral, disagree,
strongly disagree. Moreover, we have included an open question, to let the users add
some details about their answers. Questions of this survey part are reported in Table 3.
Moreover, we gave participants the possibility to provide personal comments in the
third part of the survey. We have provided the questionnaires to 2 different groups of
users for each design experience, as follows:
1.a Erasmus, exchange, International, mobility students, who took part at the focus
groups and experience prototyping activities in Bologna, during the user-centered
design phase (10 students).
1.b Erasmus, exchange, International, mobility students, who were not involved in the
user-centered design phase (11 students).
2.a High School students, who took part at the participatory design activities (13
2.b High School students, who were not involved in the participatory design phase (12
Obtained data were analyzed from both a quantitative viewpoint and from a
qualitative perspective, obtaining interesting results, as here reported.
On a quantitative perspective, taking into account users’ satisfaction about the
application, a first element of interest is the fact that all the users enjoyed the app
(Question 1), providing positive feedbacks (“strongly agree” and “agree”), claiming they would
use it (Question 4) in choosing their undergraduate programme or in getting confident
with his/her hosting university during an exchanging programme. Moreover, only 6
users (13.04%) claim that the app does not meet their expectations (in particular, users
belonging to group 1.a, 1.b, and 2.a) and only 2 users (4.34%) declare that the app is
boring (both were involved in the participatory design experience, 1 belonging to group 2.a
and 1 belonging to group 2.b). Aggregated data reporting users’ answers about questions
related to users’ satisfaction are shown in Fig. 5.
Figure 6 depicts answers users have given about issues related to the ease of use of the
applications. In particular, only 5 users (10.87%) claim that using the application has not
been simple (Question 5) and that they have faced some troubles while accomplishing
some tasks (Question 6), while 7 users (15.22%) do not appreciate the apps layout and
interface (Question 8). Finally, only 5 users (10.87%) answered that they need a
consistent amount of time to learn how to use the app, stating its ease of use.
On the open comments in part 3, a user (group 2.b) wrote: “I will surely use this
application to choose my undergraduate program. It is better than the official web site, clearer
and more focused.” While another one (group 1.b) commented: “The application could
be useful not only for foreign students but also for offsite students and for freshmen who
are not used to live and study in Bologna.”
We have observed general better response from users involved in the participatory
design experience, with a more in-depth involvement of the users, who were asked to
provide their own ideas. Table 4 compares the percentage of positive feedbacks for the
two prototypes, focusing on the users’ satisfaction. In particular, we have taken into
account “strongly agree” and “agree” answers for all the questions, except for Question
3, which is a control test question regarding users’ satisfaction (hence for this item we
report the percentage related to “disagree” and “strongly disagree” answers). Data shown
in Table 4 confirms that the more target users are involved in the design process, and the
more end users appreciate using the resulting prototypes.
Concluding, the results here reported confirm the RQ1 research question we have
formed in “Research questions and methodology” section: “Users’ satisfaction is
proportional to real users’ engagement in the design process of an application”.
In order to verify the RQ2 research question (“Applications resulting from
participatory design approach better meets users’ expectation than the ones resulting from
usercentered design”), we reported users’ answered divided in A and B groups to Question
2 (“The application meets my expectation”). In this sense, we have observed a better
response from users belonging to the B groups, in other words from the users who were
not involved in the design processes, as shown in Table 5, which reports details about
the answers of the 4 groups of users. These data show that groups B responds with a
higher percentage of positive feedback (“strongly agree” and “agree” answers) than
groups A of users. In particular, taking into account the user-centered design experience,
group 1.b gave 63% of positive feedback, while group 1.a gave 50%. Participatory design
users group A gave 61% of positive feedback, while group B 83%. Similar results can be
observed in the answers of the most part of the other questions. This can be due to a
bias: groups A of users were involved in the design phase, and, during these activities,
they have built specific expectations of the mobile app, while the final prototype could
have met only parts of these expectations. Again, it is evident that the participatory
design approach have obtained more positive feedbacks than the user-centered design
one, both from users previously involved in the design phases and from users who did
not take part in those activities.
To assess the significance of the experimental results from a statistical point of view,
we carried out an additional statistical analysis. In particular, for three items in the
questionnaires (provided to the four groups of the users involved) we defined a null
hypothesis and calculated the corresponding p-value using the standard two-tailed formula for
the one sample t-test. As each item in the questionnaire can be responded with a value
in the interval [− 2,2], we assumed 0 as the expected average value (accordingly with the
related null hypothesis) and considered a confidence level as great as the 95%. Hence, the
level of significance was set to 0.05. From an analysis of Table 6, it is easy to verify that
each of these p-values calculated were smaller than the 0.05, thus providing a statistical
significance to our experiments and allowing us to reject each null hypothesis.
Participatory design is based on a more deeply engagement of target users in the design
phases if compared with the user-centered design approach. This paper aims to prove
that a stronger involvement of users in the design phases corresponds to a higher level of
users’ satisfaction while using the resulting application. In order to demonstrate it, here
we present two experiences of design process, where user-centered and participatory
design approaches were applied. In particular, target users were involved in different
ways in the design phases of two applications, one devoted to support foreign students
during their exchange programme and one devoted to support high school students
while they are choosing their undergraduate programme. The paper describes how we
have conducted the design phases and how we have involved their target users in the
process. After the development of the prototypes, we went back to target users,
involving them in an evaluation phase and asking them to provide feedbacks by means of
questionnaire. In particular, at this stage, we have involved users who took part at the design
phase and users who were not involved at that time, hence users who have no
expectation and no idea about the applications purposes and goals. Results we have obtained
confirm the two research questions we have formed:
• RQ1: Users’ satisfaction is proportional to real users’ engagement in the design
process of an application.
• RQ2: Applications resulting from participatory design approach better meets users’
expectation than the ones resulting from user-centered design.
Users of the application resulting from the participation design obtained a better
evaluation in terms of meeting users’ expectation, compared with the user-centered
designed one, confirming RQ2 research question. Moreover, users claimed they are
satisfied by the two prototypes, but the application resulting from the participatory design
approach got higher scores from users in terms of their satisfaction, confirming our RQ1
research question, hence letting emerge a relationship, a proportion, between target
users’ involvement in the design process and users’ satisfaction in using the application
RQ: research question; ILocalApp: incidentally learning other cultures and languages through an APP.
PS carried out the participatory design phases, drafted the manuscript, and participated in reviewing and editing. SM
carried out the user centered design phases, drafted the manuscript, and participated in reviewing and editing. MR
carried out the qualitative and quantitative analysis, drafted the manuscript, and participated in reviewing and editing. All
authors read and approved the final manuscript.
Silvia Mirri is assistant professor at the Department of Computer Science and Engineering at the University of Bologna
(Italy). She was also a Visiting Researcher at the University of Toronto (Ontario, Canada). Her research interests focus on
Human Computer Interaction, Accessibility, Multimedia and Web applications and technologies.
Marco Roccetti is full professor of Computer Science at the University of Bologna (Italy). He was also a Visiting
Scholar at the University of California Los Angeles and a Visiting Scientist at the International Computer Science Institute
in Berkeley. His research interests focus on Human Computer Interaction, digital multimedia applications and allied
Paola Salomoni is associate professor at the Department of Computer Science and Engineering at the University of
Bologna (Italy). She is currently serving as Vice Rector to Digital Technologies at the University of Bologna. Her research
interests focus on Multimedia and Web applications and technologies, Human Computer Interaction, and accessibility.
The authors thank all the colleagues involved in the design process of the two apps, with particular regards to Nelda
Parisini for the participatory design process (the AlmaOrienta project), and to Antonella Valva for the user centered design
process (the ILocalApp project). A special thank goes to all the users involved in the design and in the evaluation phases
of both projects.
The authors declare that they have no competing interests.
Availability of data and materials
Ethics approval and consent to participate
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
1. Bano M , Zowghi D ( 2013 ). User involvement in software development and system success: a systematic literature review . In: Proceedings of the 17th international conference on evaluation and assessment in software engineering. ACM , New York, pp 125 - 130
2. Chesbrough H , Di Minin A ( 2014 ) Open social innovation . New Front Open Innovation 16 : 301 - 315
3. Sanders EBN , Stappers PJ ( 2008 ) Co-creation and the new landscapes of design. Co-design 4(1 ): 5 - 18
4. Norman DA , Draper SW ( 1986 ) User centered system design . Hillsdale , NJ, pp 1 - 2
5. Fischer G ( 2003 ) Meta-design: beyond user-centered and participatory design . In: Proceedings of HCI international , vol 4 . pp 88 - 92
6. Schuler D , Namioka A ( 1993 ) Participatory design: principles and practices . CRC Press, Boca Raton
7. Sanders EBN ( 2002 ) From user-centered to participatory design approaches . Design Soc Sci 1 : 8
8. ResearchGate.net ( 2015 ) What are the key differences between user-centered design and participatory design? https://www .researchgate.net/post/What_ are_the_key_differences_between_user-centered_design_and_participatory_design . Accessed Dec 2017
9. Academia.edu, Co-design/Co-creation/participatory design. https://www.academia.edu/People/Co-design_ Cocreation_participatory_design . Accessed Dec 2017
10. Carpentier N ( 2009 ) Participation is not enough: the conditions of possibility of mediated participatory practices . Eur J Commun 24 ( 4 ): 407 - 420
11. Frow P , Nenonen S , Payne A , Storbacka K ( 2015 ) Managing co-creation design: a strategic approach to innovation . Br J Manag 26 ( 3 ): 463 - 483
12. Vroom RW , Horváth I , Rusák Z , de Smit B , Opiyo EZ ( 2012 ) An exploration of the user role in product ideas containing ubiquitous technologies to elaborate on design support tools . In: Proceedings of TMCE 2012 symposium . pp 1265 - 1276
13. Gaved M , Kukulska-Hulme A , Jones A , Scanlon E , Dunwell I , Lameras P , Akiki O ( 2013 ) Creating coherent incidental learning journeys on mobile devices through feedback and progress indicators . In: QScience proceedings 12th world conference on mobile and contextual learning [mLearn 2013 ]. p 13
14. Kukulska-Hulme A , Gaved M , Paletta L , Scanlon E , Jones A , Brasher A ( 2015 ) Mobile incidental learning to support the inclusion of recent immigrants . Ubiquitous Learn 7 ( 2 ): 9 - 21
15. The ILOCALAPP Project . http://www.ilocalapp.eu/. Accessed Dec 2017
16. Ceccherelli A , Cervini C , Magni E , Mirri S , Roccetti M , Salomoni P , Valva A ( 2016 ) The ILOCALAPP project: a smart approach to language and culture acquisition . In: The future of education conference proceedings . pp 270 - 275
17. Cervini C , Solovova O , Jakkula A , Ruta K ( 2016 ) Mobile assisted language learning of less commonly taught languages: learning in an incidental and situated way through an app . In: CALL communities and culture-short papers from EUROCALL 2016 , Research-publishing. net . pp 81 - 86
18. Mirri S , Roccetti M , Salomoni P , Mambelli G , Valva A ( 2017 ) On the design of an app for foreign languages incidental learning . In: 2017 IEEE symposium on computers and communications (ISCC) , pp 111 - 116
19. Morgan D ( 1997 ) The focus group guidebook , vol 1 . Sage publications , Thousand Oaks
20. Buchenau M , Suri JF ( 2000 ) Experience prototyping . In: Proceedings of the 3rd conference on designing interactive systems: processes, practices, methods, and techniques. ACM , New York, pp 424 - 433
21. Furini M ( 2016 ) On gamifying the transcription of digital video lectures . Entertain Comput 14 : 23 - 31
22. Bujari A , Ciman M , Gaggi O , Palazzi CE ( 2017 ) Using gamification to discover cultural heritage locations from geotagged photos . Pers Ubiquit Comput 21 ( 2 ): 235 - 252
23. Lewis C ( 1982 ) Using the “thinking-aloud” method in cognitive interface design . Research report RC9265 , IBM TJ Watson Research Center, New York
24. Ferretti S , Mirri S , Prandi C , Salomoni P ( 2016 ) Automatic web content personalization through reinforcement learning . J Syst Softw 121 : 157 - 169
25. Salomoni P , Mirri S , Ferretti S , Roccetti M ( 2008 ) A multimedia broker to support accessible and mobile learning through learning objects adaptation . ACM Trans Internet Technol 8 ( 2 ): 4
26. Salomoni P , Mirri S , Ferretti S , Roccetti M ( 2007 ) Profiling learners with special needs for custom e-learning experiences, a closed case? In: Proceedings of the 2007 international cross-disciplinary conference on web accessibility (W4A) . ACM, New York, pp 84 - 92
27. Kwak J , Sung Y ( 2017 ) Beacon-based indoor location measurement method to enhanced common chord-based trilateration . J Inf Proces Syst 13 ( 6 ): 1640
28. Prandi C , Roccetti M , Salomoni P , Nisi V , Nunes NJ ( 2017 ) Fighting exclusion: a multimedia mobile app with zombies and maps as a medium for civic engagement and design . Multimedia Tools Appl 76 ( 4 ): 4951 - 4979
29. Bertram D ( 2013 ) Likert scale is the meaning of life . University of Belgrade, Yugoslavia. http://poincare.matf.bg.ac. rs/~kristina/topic-dane-likert.pdf