Evaluation
Assessing the quality and effectiveness of our system
Focal Evaluation Task:
The main purpose of our application is to provide the user with a platform that promotes increased involvement in UMD events and culture. That is why we chose to use the click-through Marvel prototype from Project Assignment 4 part one for our focal evaluation task. This task requires the user to walk through a series of different landing pages, which will navigate them to the calendar page which contains details of various events.
When it came to selecting recruits for our test, our team decided to divide the task amongst all five of our members. Each member of our team presented one or more friends/colleagues with the Marvel application so that they could walk through a simulation of our applications features. It was important to our team that we capture a range of demographics of potential users. We wanted data from people with different levels of involvement.
Our focal evaluation task was premised with an overview of the application, that way our users felt more comfortable with the application’s interface before beginning their task. After each user received the breakdown of the functionality of TerpRoots, we asked our peers to navigate to the page labeled “Upcoming Events.” During their navigation, our team used a hands-off approach while our peers used the application; each of the users were required to find and select the name of an event without the aid of our team members.
The flow of the task was straightforward. Initially, the users were directed to the page labeled “Sign In.” From the sign in page, the users entered their information and logged in, which then took them to the homepage. From the homepage, the user clicked on the icon located on the top right of the page, which lead them to the pages that list upcoming events. On the page of upcoming events, the user clicked on the first event that was listed, where they then received more information on that particular event. After they received this information, a back button appeared on the top left corner to return to the upcoming events page.
Data Collection Procedures:
Although user experience and usability are not directly measurable elements, we can still collect data by receiving qualitative and quantitative feedback from potential users. To do so, our team used a combination of the Think-Aloud and questionnaire techniques to collect our data. Once we told our participants an overview and purpose of the system, we gave them the task of locating the events calendar in the prototype that features a list of specific events. While our participants were carrying out our desired task, we asked them to also describe their thoughts about the system as they went through it page by page until the task was completed. We felt that by having them speaking their immediate thoughts on their assigned task would better allow us to understand the hesitations and issues they are having with the system. Our team found this direct feedback from various participants to be extremely vital because it allows us to understand issues that other potential users may run into and look for ways to improve the use of the system.
As for quantitative data, we gathered it in two ways. The first method of gathering was done by observing the number of clicks on the screen it took our user to complete our given task. We associated a higher amount of clicks with more difficulty of navigating the system. The second method was through a verbal questionnaire where we asked for Likert-scale responses, with each feeling associated with a number. The Likert-scale response options were as follows: (1) Very Dissatisfied (2) Slightly Dissatisfied (3) Neutral (4) Slightly Satisfied (5) Very Satisfied. Once receiving the participants responses, we summed up the numbers associated with each of their responses to in order to compose a composite score. Similar to the qualitative data collected, our numerical data will be used by our team to improve the design and functionality of our TerpRoots platform.
Evaluation Results:
Through the Think-Aloud and questionnaire results, we were able to analyze the data in order to evaluate our design for possible improvements. We each gathered input and responses from various individuals for the Think-Aloud prototype click through evaluation and verbal questionnaire. The Think-Aloud technique proved to be very beneficial as our team received a variety of positive and negative responses from our participants. Some of the positive responses were as followed:
“Performing the assigned task to locate the events was very straight-forward”
“The navigation from the start of the app to the calendar was simple as possible”
“The setup of the calendar was well thought out to include all the necessary information but also not bogged down with too much information”
“I think this setup is pretty good; you can find everything on here that is contained need in one place quickly”
“Getting to this calendar was pretty quick; I like that I could quickly find the events, so I can get up and go”
Along with the good, we also received some responses from participants expressing some of the issues they had during and after the evaluation task. Here are a few of the responses:
“It’s not responding to my touch like I think it should; I’m tapping where I want to go but it’s not working”
“I clicked on the wrong section and now I’m trying to go back so that I can get to the calendar, but it’s not working right; I can’t find the back button”
“It did the job for sure, but I think could look better at the same time”
“This seems low quality; I knew what to do and how to get there, but it’s not appealing to look at”
“There’s no back button so I had to start all over to get where I needed to go”
When conducting our field test, the metrics we were collecting were the number of clicks it took the user to go through the prototype and a questionnaire measuring the prototype’s usability. Our results are as follows:
Our first user clicked on the screen a total of 12 times in order to complete the assigned task. They scored the prototype’s usability as a 4.5 on a 5-point scale. For the second user, it took them only 10 touches on the screen to finish the task and gave a 5 out of 5 score for the usability. User number three took the most amount of touches at 15 to complete the task. They did not have as good of an experience as they gave a 3 out of 5 usability score. The next user took only 11 clicks to complete the task and gave the usability experience a 4.7 out of 5. Our final user also took only 10 touches to complete the assigned task and also had a great experience as they gave 5 out of 5 score for the usability.
Alternative Design Steps:
In the future, if we went through a second design iteration we would implement a transportation aspect of the application. This transportation application would show bus routes and current bus locations. The transportation component would also use location services to show the user how far they are from their end destination. Additionally, if we went through a second design iteration, we would make the prototype more functional by highlighting more of the features we hope to include in our prototype.