It is amusing that even years after surveys were invented and used for data collection purposes, we still need to talk about how to design surveys well and to keep them relevant to the task at hand.
Nevertheless, it is the need of the hour.
Problem with surveys today
The first problem I see in most surveys is a complete lack of respect for the survey taker’s time.
I remember getting a survey from Asana to give them feedback on the product, and boy was it a long survey. Seriously, what makes you think that I am going to take half of my life answering your goddamn feedback survey? I am a Product Manager by profession, and whenever I see a survey from a product team, I really empathize with them and genuinely want to help by answering honestly, but their survey, I couldn’t go past a few minutes. Questions after questions, they just kept coming.
People today do not have the patience to do even one thing for a few minutes continuously, and your expectation is for them to fill out a long survey honestly. Even if a few fill it out, you can be sure that the data coming would be more noise than signal. We are not living in the 2000s anymore; there is more distraction and competition, and a survey is always going to lose to them.
The second problem I see is a lack of “relevancy“.
On a feedback survey, you ask “What is your gender?”, seriously?
The problem here is that people do not start with a “learning goal”. What do I want to know from these people that will increase my understanding of them to a degree that will assist me in making a reasonably good decision?
Instead of generically asking for product feedback (which also has its importance), ask a more specific one which is pointed towards probably a specific problem or a feature. Without a learning goal, surveys tend to go all over the place. While it may give someone the impression that we collected a lot of data, unfortunately, you cannot do anything with it. How many times has it happened that we have tons of data but ultimately we still decide based on gut as the data in front of us just doesn’t tell us anything.
The third problem I see is asking too many questions (like Asana did).
It is extremely hard to fight the instinct to ask too many questions on the survey. “Hey, maybe we can ask this and maybe even this might help…” The problem with this is that it just ends up making the survey bloated with unnecessary questions. This is also a strong case where every survey should start with a “Learning goal”. A good product decision is never solely based on data. It requires more than just collecting data on relevant topics, it also requires skill to see something that the data doesn’t tell you to dig deeper into your previous learnings. If everything needed to decide came just from a survey the world would probably see a lot more successful products, but that isn’t the case is it?
The fourth problem I see is asking users hypothetical questions.
It is simple: you get a hypothesis for a hypothesis.
Would you be willing to use this if …?
There is no way you are getting any good data out of this question because it’s just too hypothetical. A better way to ask this is to find out what their actual behavior is right now and if they are using something similar.
A better question would be: What was your experience like using x or in this situation?
The problem with hypothetical questions is that it introduces biases, and it’s a bad idea to base decisions on biased responses.
The point of conducting surveys and collecting data is to make something better, a product or a service or a training, etc. If this objective is not being fulfilled, why collect data at all?
No surveys are perfect, but that’s not an excuse for creating a half-assed, half-baked, long survey that neither helps the survey taker and nor helps us.
