Dialog Design DialogDesign ved Rolf Molich English English Dansk Dansk
DialogDesign, Skovkrogen 3, 3660 Stenløse, Denmark, dialogdesign@dialogdesign.dk, +45 4717 1731
Home Free Resources CUE Products Events Books About DialogDesign
CUE Papers CUE-1 CUE-2 CUE-3 CUE-4 CUE-5 CUE-6 CUE-7 CUE-8 CUE-9

CUE-8 - Task Measurement

You can’t manage what you don’t measure - but can you manage measurements to generate reproducible and useful results?

CUE-8 compared practical approaches to usability task measurement. Fifteen participating teams carried out independent, simultaneous measurements of the Budget.com car rental website and compared results.

CUE-8 results were presented and discussed at a workshop at the UPA 2009 conference in Portland, OR, USA, on Tuesday 9 June 2009.

Available Downloads

·       CUE-8 instructions to participating teams (PDF, 40 KB).

·       All 15 reports and appendices plus workshop agenda plus results overview (Zip, 17 MB, password cueeight)

·       Our UPA 2010 conference paper "Rent a car in just 60, 120, 240 seconds - Comparative Usability Measurement", 8 pages. The paper was presented by Rolf Molich, Jurek Kirakowski and Tomer Sharon at  a full 90-minute session at the UPA 2010 conference in Munich, May 2010

·       A preliminary version of our paper "Rent a Car in Just 0, 60, 240 or 1,217 Seconds? – Comparative Usability Measurement, CUE-8", 17 pages. The paper has been accepted for publication in the November 2010 issue of JUS, the Journal of Usability Studies.

 

Advice from CUE-8 about Task Measurement

·      Adhere strictly to precisely defined measurement procedures for quantitative tests.

·      Report time-on-task, success/failure rate and satisfaction for quantitative tests.

·      Exclude failed times from average task completion times.

·      Understand the inherent variability from samples. Use strict participant screening criteria. Provide confidence intervals around your results if this is possible. Keep in mind that time-on-task is not normally distributed and therefore confidence intervals as commonly computed on raw scores may be misleading.

·      Combine qualitative and quantitative findings in your report. Present what happened (quantitative data) and support it with why it happened (qualitative data). Qualitative data provide considerable insight regarding the serious obstacles that users faced and it is counterproductive not to report this insight.

·      Justify the composition and size of your participant samples. This is the only way you have to allow your client to judge how much confidence they should place in your results.

·      When using unmoderated methodologies for quantitative tests ensure that you can distinguish between extreme and incorrect results. Although unmoderated testing can exhibit a remarkable productivity in terms of user tasks measured with a limited effort, quantity of data is no substitute for clean data.



Home    Free Resources    CUE    Products    Events    Books    About DialogDesign