Following up on the post reviewing the paper by Lam et al, we were asked to design our own evaluation. As you might have read, there were three evaluation scenarios that were relevant to us. One of them was of the “process” group, the other two were part of the “visualization” group. Prof. Duval asked us to ignore the latter, so that leaves us with one scenario: the VDAR (visual data analysis and reasoning) scenario.
However, this doesn’t mean that you can no longer give feedback/comments on usability of our visualization. We intent to keep posting new versions online, hoping to get this kind of feedback. In this way, we will keep the “Informal Evaluation” thing going, as it is called in the paper by Lam et al.
We also intent to perform a decent evaluation of the VDAR type. This scenario focuses on knowledge and hypothesis generation. The goal of our evaluation would be to discover in what way our visualization helps people drawing conclusions on the last few years of belgian football. Given the fact that our visualization is intended for casual users, the methods described as ‘case studies’ wouldn’t make much sense for us, as they mostly aim on (long-term) specialist behavior. A controlled experiment, probably combined with a questionnaire, is probably the best in our case.
Since we don’t really have experience with designing such an experiment, we are not quite sure yet what tasks and questions the experiment will consist of. Since the deadline for this week is still a couple of days away, we still have time to ask for your suggestions. If anyone has good ideas of questions to put in such a VDAR questionnaire, please leave’m in the comments 🙂
Update: After discussing the issue together, we came up with the following: we’d make the test person fill out a questionnaire before performing the test (thx to Niels for that one). This questionnaire would consist of football-oriented questions, like ‘where would you rank team x for their overall performance over the last 15 seasons?’ , ‘who do you think is the best team of the past 15 years?’ or ‘How many years did team x play in first division?’. After filling out the complete questionnaire, we would ask them to fill the same thing out a second time, but with our visualization to help them. Obviously, we would hope that they’d perform much better. We would also try to observe how long the test subject needs for every question. This way, we would be able to see how our visualization affects the generation of hypotheses, and what kind of knowledge can be easily extracted from it.
Pj
#MUME13 
I think it would be interesting to present a questionnaire before and after they have used your visualization. This way you can start to ask with what they expect of such a visualization and afterwards you can ask how well your visualization fulfilled their initial ideas.
Interesting approach! Maybe we will use this for our own visualization too!
I especially think the timing aspect of the experiment will be very relevant. I suppose you could also do this for the questionnaire before they use the visualisation. That way you can compare how long a user takes to generate/evaluate hypotheses with and without the visualisation.
I think this comparison would be more valuable than comparing the correctness of their answers: people who know little about soccer will almost certainly perform bad on the first one, while anyone with some proficiency for using digital tools will be able to complete the second questionnaire flawlessly. Therefore an improvement of the results (as you indicate yourself) is to be expected: this makes for an (in my opinion) more or less insignificant observation.
The comparison of the answers could be interesting for people that do know about football (or believe they know about it). It would tell us whether people change their views after/during our visualization. As you said comparing a correct answer to “I don’t know” wouldn’t be interesting, but comparing the first answer (“I think team x is the best”) to the second (“ok no, after using your visualization, I think team y is the best”), couldn’t that be useful? Most information about football is subjective to some degree, and people tend to have very different opinions.
This implies that we need to include an “I don’t know” answer when asking multiple choice questions 😉