Welcome to Carl’s Newsletter.

In today’s issue:

• An incredible thread on networking in Tech

• The secret path into product management

• Google’s new Gemini vs ChatGPT for product work

• A complete guide to writing great user surveys

Forwarded this email? Sign up here.

Today’s post is sponsored by Alex Rechevskiy. He’s an ex-Google Group PM turned top-tier product content creator and career coach.

Last month, he launched the Product Career Accelerator. It’s an all-inclusive monthly program designed to help you land the right next product role, with at least three live sessions every week, including:

• Resume review

• Interview & negotiation prep

• Application & Follow-up Methodology

Alex’s clients have received offers from Google, Facebook, and Amazon. Others have negotiated $50k - $100k+ salary increases during the offer process.

The January and February cohorts sold out almost immediately, but the March waitlist is open now.

From Around the Product-Verse

1. Your Network is Your Net Worth: Legendary a16z partner Andrew Chen published an absolute banger about networking this week. He talks about networking in the Bay Area specifically, but these lessons would work in most cities. It’s a fantastic read.

2. The Secret Path to Product Management: I’ve recently started following a few aspiring PMs on LinkedIn who do incredible product case studies about companies they want to work for. In this example, Jahnavi Shah managed to get on a call with LinkedIn’s CPO by cold emailing members of the LinkedIn product team her work. Lots of people ask me about breaking into product management – if you’re not having any luck, learn from Jahnavi’s example.

3. The Gemini Wars: Last week, Google upgraded and rebranded Bard to Gemini. I guess that’s an improvement? Here’s my thread comparing it to ChatGPT for product work. Gemini Advanced has a 2-month free trial, which I highly recommend trying it out.

   

A PM’s Guide to Writing Insightful User Surveys

I’ve written over 100 user surveys. They’re an underestimated user research tool for product managers. Good surveys let you gather insights from lots of users quickly, cheaply, and easily.

The operative word in that sentence is “good.” It’s amazingly easy to create a survey with bad questions and useless results.

In this guide, I'll walk you through when and how to create good surveys that help you learn what you really need to know.

Why Should A PM Know How to Write a Survey?

Even if you’re blessed enough to work with a UX Researcher, there are still lots of good reasons to know how to write a good survey.

This skill will help you:

1. Recognize research opportunities - Understanding best practices gives you intuition for scenarios where a survey would be helpful, and lets you proactively ‌propose them at the right times.

2. Improve research relevance - Since you likely have the most context on your product, you can help ensure surveys tie directly to relevant decisions and priorities.

3. Accelerate insights - By owning survey creation and analysis, you can get the answers you need when you need them instead of competing for often overloaded researcher bandwidth.

What You Should, and Shouldn’t, Use Surveys For

With interviews or usability studies, you're limited to feedback from 5-10 people at a time. But surveys can collect responses from hundreds of users in days.

For example, you could observe 5 users interacting with a new feature, or you could survey 500. This a larger sample size helps eliminate any bias from only talking to a select group of users.

Specifically, surveys are good for:

• Evaluating new features or concepts (e.g. ask users to rate potential new features on a 1-5 scale)

• Benchmarking metrics like satisfaction (e.g. use rating scale questions to track churn risk over time)

• Quantifying perceptions of quality (e.g. "On a scale of 1-7, how would you rate the search feature's relevance?")

• Identify major pain points (e.g. "What is your biggest frustration with [X feature]?")

The simplest reason for a survey being bad is trying to use them for things surveys are bad for, like:

• Deep diagnostic insights (e.g. understanding all the factors influencing a user behavior)

• Innovative ideas or discoveries (e.g. imagining completely new features or products)

• Hypothesis testing (e.g. proving or disproving assumptions about cause and effect)

• Open-ended feedback requiring extensive qualitative analysis (e.g. "What do you think about our onboarding flow?" which requires coding responses)

For these, you’ll want to rely on things like ethnography, prototyping, and A/B testing.

Preparing the Survey

Before writing questions, you need to define:

Overall Objectives: Clearly outline what you aim to learn or achieve with the survey. This guides the entire process and helps you make sure the questions are relevant and purposeful.

Questions for the Survey to Answer: Make a bulleted list of specific things you want to learn to achieve your objective. Don’t worry about specific questions here, just want you want to learn. This is an extremely important step!

Target Audience: Defining the target audience helps tailor language and questions.

Doing these things upfront focuses the survey on outcomes vs just curiosity. Research should always tie to product strategy.

Crafting Effective Questions

Okay! We’re finally ready to start writing our questions. In general, there are 6 types of questions you can ask:

1) Multiple Choice Questions

• Allow respondents to select one answer from a predetermined set of options.

• Best for closed-ended questions with a limited number of possible answers.

• Provide an "Other" option to capture responses outside the predetermined choices.

• Limit to 4-5 answer choices. Too many can overwhelm respondents.

2) Ranking Questions

• Ask respondents to rank options in order of preference.

• Useful for prioritizing features or evaluating preferences.

• Limit the number of items to rank to avoid fatigue. 5-7 items is ideal.

3) Rating Scale / Likert Questions

• Ask respondents to indicate level of agreement, frequency, importance, etc.

• Use 5 or 7 point rating scales. (Odd numbers give respondents a neutral middle option.)

• Include descriptive labels for scale points (e.g. Strongly Disagree to Strongly Agree).

4) Open-Ended Questions

• Allow respondents to answer in their own words.

• Gather more qualitative insights and detailed feedback.

• Best for the end of sections to capture additional thoughts.

5) Demographic Questions

• Gather data on attributes like age, gender, location, job role.

• Can help analyze and segment survey results.

• Keep these basic and at the beginning or end of the survey.

6) Use pick/group card sorting questions

• Allow respondents to categorize items to understand mental models

• Avoid priming users with predefined categories

• Useful for developing taxonomies, navigation, IA

Refining Your Survey

Now that you’ve got your questions, here is an unprioritized list of things to consider to improve your survey:

• Prioritize key goals: Front-load questions tied to your most important objectives in case respondents drop off.

• Funnel from general to specific: Start with broad, easy questions to warm up respondents before diving into detailed product feedback.

• Group thematically: Organize related questions into sections or pages around specific topic areas.

• Keep it simple: Avoid long, complex, double-barreled questions. Only ask one thing at a time.

• Balance closed and open-ended: Closed-ended questions provide quantitative data that's easy to analyze. Open-ended questions reveal deeper qualitative insights.

• Test double-sided questions: Ask both positive and negatively-framed questions to check for response bias.

• Randomize choices: Shuffle multiple choice options so order doesn't influence selection.

• Add “attention checks”: Ask users to do things like “select number 3 from this list” to make sure they’re really paying attention.

Building, Testing, and Sending Your Survey

Everything in this article can be done for free on Google Forms, but your company may already have a survey platform.

The links below will help you with any other aspects of writing a user survey

• 7 Simple ways to recruit research participants

• How much research participants want to be paid

That’s All For Today

Don’t forget that as a subscriber, you get free access to:

1. My massive database of resources

2. A huge list of ChatGPT megaprompts to help you with product work

3. A 60-day guided tour of the best product content ever written

Last things:

1. Upcoming AI course: I am getting close to finishing my first-ever paid course “AI for Product Managers” which goes in-depth with tactical ways PMs can leverage AI tools to save hundreds of hours at work. If that sounds interesting to you, drop your email in this 1-question form and I’ll send you updates, sneak previews, and a discount code when the course is launched.

2. Sponsors: Carl’s Newsletter has its first official sponsor starting in two weeks! But there is still room beyond February, so if you or your company is interested in sponsoring this newsletter, I’ve posted sponsorship information on this snazzy new sponsorship page.

Here’s where else you can find me:

1. Follow me on X, where I post the most content, including lots of memes.

2. Follow me on LinkedIn, where I post longer-form content.

3. Follow me on Instagram if you just want the memes.