Sketchnoting research

Studying illustrative notetaking with Microsoft Research

Sketchnoting is a form of notetaking with a mixture of drawings, handwriting, typography, and other visual elements. Despite its known cognitive benefits, the practice is not common. I studied and analyzed the live sketchnoting process, particularly focusing on the role of time on decision-making. With the rise in interest in Microsoft’s digital pens and tablets, sketchnoting brings increased possibilities and challenges for their hardware.

Research proposed and conducted by Ann Tran

Mentored by Nicolai Marquardt @ Microsoft Research

1 — Literature review

We studied benefits of sketchnoting, challenges in adopting sketchnoting, and research provided to make video recording research sessions useful.

Benefits of sketchnoting

Encourages cognitive patternmaking to remember topics

Enhances attentional engagement for memorable learning

Deepens people’s understanding of complex topics by creating their own conventions of representation

More successfully communicates and explains information to peers

Sketchnoting adoption

Why it’s uncommon:

  • Despite that drawing abilities should not be important, sketchnote novices believed their drawing skills limited their success

  • Demands higher levels of patience and practice than conventional notetaking

Ways to make it easier:

  • Situate oneself in an environment conducive to focusing

  • Write in available relevant information before the notetaking session

  • Make key information layout decisions in advance to the session

  • Practice simplifying ideas into basic icons that are quick to draw

Videotaping sessions

Capturing behavioral studies using video data is useful to:

  • Distinguish moments close to one another

  • Study human behavior with extreme attention to detail

  • Replay, review, and reproduce interactions

There are limitations, however, in choosing video capture for observational studies. These include:

  • Lack of a setting’s contextual data

  • Analyzing such rich data is time- and resource-consuming

  • Participants’ behaviors may be affected under known surveillance

2 — Behavioral study

The study had a within-subjects design.

We recorded ten participants use analog tools (e.g., pens, paper, pencil) as they took visual notes for two video presentations, respectively four minutes and 19 minutes long.

Participants had at least one year of experience sketchnoting; were over 18 years old; and comfortable using analog tools.

We focused our observations on participants’ real-time sketchnoting activities, especially related to time constraints, visual-spatial processing, and tool applications.

Sketchnotes from participants of the 4-minute video are displayed below.

Sketchnotes from participants of the 18-minute video are displayed below.

3 — Follow-up interviews

With five participants post-session, we conducted 45-minute semi-structured interviews on their sketchnoting behaviors.

Applying thematic analysis, I highlighted quotations in every interview across and extracted 10 themes. I then returned to the interviews to mark which participants mentioned the respective themes.

Questions included:

What are ways that you prepare for sketchnoting? Conclude for?

How does your layout structure get planned, changed, and adapted during your live sketchnoting process? 

How are shortcuts and placeholders employed during your live sketchnoting process? (icons, etc.) 

What are ways that you try to stay focused while sketchnoting?

What are ways you manage running out of space? Too much space?

Walk me through the tools you use and why. Why that many? What does each offer that’s different from the other?

What common sketchnoting practices and strategies that remain consistent for most of your sketchnotes?

What decisions do you make when you’re sketching out something with limited time? What do you do to jog your memory when you lose out on something?

4 — Codify video recordings

First, I annotated any obvious patterns across participants.

Next, I translated the sketchnotes into a new visual language to summarize each participant’s activity over times. Using symbols, colors, and color gradients, I simplified participant activity to visualize overarching trends.

Finally, for each video recording, I color coded participant activity minute-by-minute to understand participant behavior before, during, and after the lectures they were taking notes on. As you can see, embellishment mostly happened before or after the lectures.

There are several ways that we pulled patterns from participant sketchnotes. Across the research, we focused on the following themes to study:

Typography for information hierarchy

Strategies to deal with sketching under time constraints

Planning layouts

Strategies to deal with sketching with limited excessive space

Switching tools

Note taking pre- and post- lecture

Visual embellishment

Changing and adjusting sketching styles over time

5 — Analysis and findings

First, we noticed that each participant had preferences structuring page layout. Below is an example of P3’s sketchnote before, during, and after the video session. Before, participants might write the header or section the page based on talk points. During, they would typically populate information intuitively and top-down. After, they would often link information or add colors.

Below is an example of P1’s sketchnote during the live presentation and post-presentation where empty space has been filled with connectors.

Second, from having too little space to having too much, participants had various ways of balancing spatial density on their note pages. Below are two examples of how they did so: creating a container with their note page’s extra room, or appending another page to their notes when their page was saturated.

Third, the tools participants used empowered them to take better notes. That being said, participants limited their tools to improve focus and reduce cognitive load. Below are calculations on the number of tools used per video and number of interchanged tools used per video.

Fourth, participants used a number of techniques to draw out important details. Below on the left you can see P4’s use of iconography and color. On the right, you can see how all ten participants approached emphasizing ‘trust and respect’ differently, through color, type treatment, bolded lines, and containers.

Finally, we noticed the personal trends among participants to continuously self-improve their note-taking techniques; desire praise and understandability from the note-taking community; and relax into their creative impulses.

Outcomes and learnings

There were several takeaways I concluded for Microsoft’s hardware development. First, participants responded well to the feeling of analog tools. They remarked on the tactile feedback and its personalness. Simulating different weights and haptic experiences could retain analog tool benefits with more versatility. Second, participants constrained their own toolset to limit distractions while notetaking. Digital tablets could lock access to colors, tools, and erasers until post-presentation. Thirdly, the sketchnote community thrives on social media, and digital tablets make file and image sharing, easy, presenting a strong case in favor of using tablets. Finally, a benefit of digital sketchnoting can be the ease of collaging and duplicating information on a page. With pen and paper, users can’t do that so easily.

My dissertation also had a profound impact on my understanding of user research. First, I learned that recruitment takes more time than expected. The majority of the project’s timeline involved reaching out to participants via X (formerly Twitter), sending them materials, scheduling meetings, and being flexible for meeting changes. Second, participants don’t always behave the way we expect. For example, although I expected participants to use a big palette of pen shapes, colors and sizes, in reality, they limited their kit to limit cognitive load, which in turn supports the thesis that sketchnoting as an activity and skill heightens focus. Finally, this work established a foundation for me to enjoy informing future design decisions with research. I loved seeing research in conversation with design and it grew my love for getting in the research weeds for all future design projects!