by Joe Miller
I’m weirdly conflicted about AI. Earlier this year, I wrote a whole series that revolved around the question of whether the current crop of AI (Chat GPT, Claude, Gemini, Meta, Grok and the like) is really intelligent. I think there are good reasons to believe that it’s not intelligence in any meaningful way.
And yet, a lot of the journalists and technologists whom I’ve come to trust over the years keep writing long essays about how incredibly useful AI has gotten.
Since I’m an optimist about technology – which has, since the Industrial Revolution, lifted much of the world out of the desperate poverty that was the norm for most of human existence – I figured that maybe I should try using AI.
I’ve already used AI for simple tasks, like transcribing Zoom calls to text. I wanted to give it something more complex. But it also needed to be material that I know well so that I could fully evaluate its performance.
As it happened, I had just finished a round of usability testing as part of my work for the National Weather Service’s Weather Prediction Center (WPC).
Those tests involved me asking regular WPC website users to complete a series of tasks while sharing their screen with me. The test takers then narrated their actions while I recorded everything. I then reviewed all the tests to figure out what is working and what is not.
The goal is to uncover patterns and then craft designs that incorporate those patterns.
It’s a little bit like detective work. I do usability testing because I know that something is wrong, but I won’t really understand why it’s going wrong until I watch people trying to do stuff. Watch enough of them and the patterns reveal themselves.
My latest round of testing resulted in 60 hours of video to review. I spent a few days re-watching all of the tests, putting Post-It notes all over my chalkboard and drawing arrows between them. My office sometimes resembles that conspiracy theory meme.
After about two weeks of work, I’d identified patterns, revised some designs and drafted a report with findings and recommendations.
With that report fresh in my mind, I asked Claude to analyze the same raw data and produce recommendations and compile all of it into a report. I then went downstairs to get a cup of coffee.
When I got back, the report was finished. Claude identified all the same patterns I found and made more-or-less the same suggestions.
I was honestly surprised at how good it was.
A few years ago, I hired a junior colleague who was brand new to the world of information architecture. She turned out to be a rock star. Another agency hired her into a senior job less than two years after I hired her.
Claude was as good as Adriana in her first year.
The work wasn’t perfect. Claude identified some things that, while true, weren’t relevant to the project. The writing in the report was mediocre. It wasn’t ready to go to a client, but it also wasn’t hard to envision a path to getting it there.
It also took five minutes to complete what took me two weeks. By human standards, I’m fast at this job.
I wouldn’t trust it to run a complex project like the one for WPC without some serious oversight.
The issue is that most projects aren’t as complex as WPC’s. I just spent 140 hours on a single round of tests. The vast majority of websites we built at my previous agency had only around 400 hours for the entire project – that’s everything from kickoff, through discovery, design, development, migration, testing and launch.
We worked pretty much exclusively with nonprofits, and they simply couldn’t afford to pay anyone to spend two weeks doing detective work.
And the thing is, I work with nonprofits and government agencies because I believe in their mission. I give smaller organizations big discounts because I believe they are doing world-class work and need world-class websites on which to publish that work.
I have always made it a rule to give my clients the best advice I can, even when that advice costs me money.
