PHASE 2: Usability Test Findings

Results of our usability tests

What’s good:

• The table of contents was easily navigated by everyone. Most users complimented how easy the navigation was with the granular drawer. Much easier than the PDF or other modes they use now.
• Everyone found the right sidebar “write a comment” links. They were able to go through the flow of creating a comment and submitting a comment.
• The confirmation screen shows the information the user expects, and more.
• Understanding that the search would cover both the Preamble and CFR tabs.
• The behavior change is more work for these users, but if it’s going to be worth it for the agency then it is worth it.

Looking at the CFR diffs:

“Want to reach through the internet and kiss you for this!”

At the end of the test:

“Actually, this is much better than anything I’ve ever seen.”

What needs work:

• Bugs:

  • The attachment upload is too wide (upload a thing with a short file name to see). [FIXED]
  • Scroll logic may be broken on comment write mode pages. [FIXED]
  • CFR tab and preamble tab active at the same time. [FIXED]

• Switching between pending comments

  • Expected the whole box in write mode to be clickable, not just the edit icon to switch between draft comments.
  • Some users found the edit icon.

• Review was initially confusing

  • After saving comment initially thought that it was submitted by saving it. [Already fixed - Button only said “review” at the time - we have adjusted it to say “review and submit”]

• Tabs are still a little confusing:

  • Write mode tab vs links:
    • “Write a comment was a little confusing, maybe view comments?”
  • CFR vs Preamble tabs:
    • The ALT text, “CFR Changes” helped her understand. She said more explanation would be helpful there.

Needs not met:

• Overall comments:

  • “We always include stock language in our comment that explains who we are. I could fit that into something but wouldn’t want it to be a comment on a part of the preamble.”
  • Always submit on letterhead with CEOs signature. “The inability to brand our comment is one of the bigger hurdles for our organization.”
  • Harder to tell a story and harder to advocate in the context where you have to comment section by section - One thing that would be helpful if you are pushing this is providing the opportunity to submit a 1-pager summary along with the comments on each section.

• External links:

  • Wants to refer to other CFR citations or USC citations. Helps users have all the context they need to write a good, legal comment. [Lawyers are already trained to find these documents in other ways, but making it easier would enhance their experience and trust in the tool.]

---

General background

• Phase 1: Generative research
  • Original research plan
  • Findings
  • Workshop with EPA results and user stories
• Phase 2: Testing the tool
  • Findings < You are here!
• Phase 3: Success?
  • Comment period research plan
  • Findings