I’m curious if the Blender workflow has any process in place for measuring and testing a user’s experience within the program. As the user base grows, are there ways to measure the effectiveness of changes, or even use data to inform future changes? If so—great! One thing that I’ve seen is that what users say and what they do are both important and yet can be at odds.
Collecting verbal feedback to a screenshot or video can be helpful for general opinions but can also be misleading when basing large feature decisions. I’ve seen a lot of value in tests where you see what users do based on a provided goal. Insight from data might paint truly different pictures than verbal responses that are offered up from online forums or polls (though those are still helpful). There could be testing like:
How many users truly use right vs left-click for select (twitter polls can only tell so much )
Given a task of adding a new primitive, how many users are using Shift-A vs the Create > Add Primitive panel
If proposing a new panel layout or visual UI change, how many users are able to find the new change on their own? Is it faster? Slower? Same? Are those complaining of slower workflow actually correct for everyone?
These are just crude examples, but knowing how the program is used on a large scale could offer some really useful insight when planning features. And it doesn’t mean you have to follow data—but when adding disrupting features, you can plan ahead by knowing how to ease users into a new workflow to follow a larger goal.
Even proposed ideas like Andrew Peel’s draw primitive or particle system painting proposals (thread here), where common tasks that users might often repeat over and over (maybe data and tests would confirm) could eventually be combined into convenience features without replacing original functionality. I think his proposals are really interesting and insights like these could be easier to identify with user testing.
I’m not sure of the Blender dev process right now, maybe they already do this. I’m just curious if anyone knows or if they are open to integrating this type of work into the process eventually. It can certainly slow down a development cycle so maybe a slow integration is the ideal, but with the right tasks and tests it could be pretty informative.
What do you think? Or does anyone know if this happens already?
I know both Unity Tech. and Microsoft use something known as telemetry to determine just how users have set up their workflows and what they do in the application, but such a thing is not going to fly in the world of Open Source where data collection is seen as wildly unpopular,
To have a method for collecting user data among those who use Blender would have to be a completely optional thing that can be turned off at any time at best (and even then, it might attract controversy as some here have pretty extreme views on privacy topics).
Totally! Wide data collection should only be used with a clear opt-in and with correct privacy policies. But there are even other ways to conduct user testing that’s somewhere between user-initiated input and a Telemetry-style analytics tool. I just have no idea if it’s part of the process so far, or how features are chosen or tested currently.
Bad ideas are bad ideas!
I once trusted for example Firefox and participated in the Shield Study to help improve the program and reporting and sending of “anonymized” data.
Breaches of trust which put massive dents in the image of Firefox?
at least twice; CliqZ advertisement7usability “tests” and an unsolicited installation of a freakishly stupid and bad
advertisement for a TV-show.
mouse movements and keystrokes are no ones business, just go away with stupid nefarious ideas.
I think this could only work as a separate blender build. I don’t think most open source enthusiasts would trust a version of blender with telemetry even if you could opt-out.
Yeah telemetry isn’t really the point I’m after, but rather focused user testing as a whole process which can be done many ways. I’m just curious how features are tested for users before being implemented. I’ve heard back from the team and there’s not really any formal process set in place, though they have plenty of feedback coming their way. So it’s good for now.
The Ideas here are regressions of the current workflow since right now the Blender Institute among many other professional developer teams are always seeking and preferring communication with short paths with clear input and valuable data. Quality over quantity.
Seasoned Artists capable of producing higher quality output with Blender are in teams and giving direct feedback to developers while being in the same building.
THIS is one of the most valuable ways to optimize and gather quality input + you get promotion material for Blender. its a true win.
One thing that I’ve seen is that what users say and what they do are both important and yet can be at odds.
)
