Episode 283 · 12 March 2026 · 53:46
How Google and Netflix design engagement surveys that actually work, with Bill Yost
Featuring Bill Yost
How Google and Netflix design engagement surveys that actually work, with Bill Yost
0:000:00
Tip: click any timestamp below to jump there and keep playing.
Enjoying the show? Subscribe for free
Chapters & timestamps
Click any timestamp to jump the player to that moment.
Intro: Bill Yost, Googlegeist and the 90% response rate
The 'snake eating itself' — how performative surveys destroy trust
interview#survey-fatigue#trust#engagement
Bill's love-hate relationship with dashboards
Why engagement surveys exist in the first place
interview#history#engagement-surveys#origins
How Googlegeist actually worked
The 'Goodwill Account' — asking without acting is a withdrawal
Don't ask questions you don't intend to act on
interview#survey-design#actionable-insights#employee-feedback
The Yost coefficient — how long should a survey be?
Pop-up surveys — are they worth it?
What Google does with the people who speed-run surveys
Confidential vs. anonymous — the gold standard
The return-to-office bias — leaders see what they want to see
interview#return-to-office#bias#leadership
The manager who only takes what they want from the data
Vendor assessments — what to look for
Where to find Bill
Top 3 takeaways
Show notes
Why Your Engagement Survey Is Failing — and How Google and Netflix Fix It — with Bill Yost
Guest: Bill Yost — People Analytics Lead at Netflix, former architect of Googlegeist, Google's legendary employee engagement survey that consistently achieved a 90% response rate.
Employee engagement surveys. You send them. People fill them in. Nothing changes. Sound familiar?
Bill Yost has run surveys at the very highest level — four years launching and running Googlegeist at Google, and now leading people analytics at Netflix. He joins Al and Leanne to explain why most engagement surveys are essentially performative, how the "snake eating itself" destroys employee trust, and what tech giants do differently to turn data into genuine organizational change.
They dig into the "Goodwill Account" — why asking for feedback without intending to act is a withdrawal you can't afford — why confidential surveys beat anonymous ones, how Google built an entire department from a single survey finding, and why leaders so often see only what they want to see in the data.
Top 3 Takeaways
- Only ask questions you're willing to act on. If you can't give everyone a pay rise, don't ask about pay. Every question you ask sets an expectation — and every expectation you ignore is a withdrawal from the goodwill account.
- Close the feedback loop loudly and clearly. When Google's survey flagged a structural problem, they built an entire department in response — and told everyone about it. If you act quietly, it might as well not have happened.
- Confidential beats anonymous. Nearly every well-run survey is confidential, not truly anonymous. Confidentiality lets you identify specific team issues without compromising trust — anonymity just gives you vague data you can't act on.
More From Bill Yost
- LinkedIn: linkedin.com/in/billyost (look for the pirate flag 🏴☠️)
Special offer: US listeners — try incredible cookies at cookiesworthsharon.com and use the code TLW for 15% off (valid for two months from release).
Connect With Us
Get in touch, book a call, or find Al and Leanne on LinkedIn — all in one place: truthliesandwork.com/connect
Mental health support: findahelpline.com — UK: Samaritans 116 123 · Mind 0300 123 3393 — US: 988 — Australia: Lifeline 13 11 14
Truth, Lies & Work is part of the HubSpot Podcast Network.
Full transcript
Expand full transcript
Leanne ElliottOur guest today, Bill Yost, spent almost 4 years running Google's flagship employee engagement survey, the Googlegeist, which hit a 90% response rate year after year. Now he does the same at Netflix, and he says what most companies are doing isn't the right way.
Bill YostI was on the employee engagement team at Google for almost 4 years. I, in that time, launched, I think, 7 of our annual employee engagement surveys called Googlegeist. We had a really high response rate. It was always like 89%, 90%. There was always a big push to have Googlers fill this survey out. And it was like a cultural moment for the company when the survey went out. And then again, when the results came out, because there was a very good readback of the results to employees and executives did a great job of, you know, standing up and saying, here's what we learned from the survey. Here's what we're going to change. We appreciate you taking it. And it's that feedback loop that's really important.
Al ElliottBill calls it the snake eating itself, where employees stop trusting Surveys stop filling them in, and leaders stop sending them because nobody actually responds. And he's got a very specific framework for breaking that cycle.
Leanne ElliottToday, we're getting the inside track on how Google and Netflix design surveys that actually change things, and why yours probably isn't working yet. Hello, and welcome to Truth, Lies and Work, the award-winning podcast where behavioural science meets workplace culture, brought to you by the HubSpot Podcast Network. The audio destination for business professionals. My name is Leanne. I'm a chartered occupational psychologist.
Al ElliottMy name is Al. I'm a business owner, and we are here to help you simplify the science of work. So today we're asking, what separates an engagement survey that actually moves the needle from one that just burns your employees' goodwill and lands in the bin? Bill Yost spent nearly a decade in people analytics, 4 of those running Googlegeist. This is Google's annual company-wide survey that consistently got 90% of 100,000 employees to respond. He's now doing the same work at Netflix.
Leanne ElliottAnd he's also one of LinkedIn's most entertaining follows. But more importantly, Bill has seen every survey mistake there is, made a few of them himself, and has very strong opinions on what actually works.
Al ElliottAfter this really quick break, we'll find out why most engagement surveys are essentially performative, what Google did when the survey results were uncomfortable, and Bill's framework for getting results that leaders can't ignore. Do not go anywhere. The dirty little secret is that most businesses think they know their customers. They have the data, the records, the history, but it's scattered across 3 teams and 4 systems. And in our case, it's about 216,000 spreadsheets, which means nobody can actually use it.
Leanne ElliottWhat you should be doing is using HubSpot, because why? HubSpot connects it all— every interaction, every support ticket, every conversation— into one platform. Every team can work from. So when sales talks to a customer, marketing already knows the full story. And when you know more, you grow more.
Bill YostNice.
Al ElliottCheck out hopspot.com, the agentic customer platform for growing businesses.
Bill YostMy name is Bill Yost. It's a pleasure to be here today. Thanks for having me, Al. I work in the people analytics field, which is data analytics for human resources. I've Been in this field for almost a decade now. I broke into it working at Google, where I was there for 8 years, bounced across multiple different people analytics teams, spent a lot of time in the employee engagement survey space, which we're going to talk a lot about today, but also worked a lot on scaled data products vis-à-vis dashboards. Data products is a much sexier way to put it, in my opinion. I left Google after almost 8 years. As I mentioned, I started at Netflix last year. I'm just coming up on my My 1-year anniversary here in 8 days. So time flies. I'm in that reflective moment of, wow, things have escalated quickly. And something I'm known for is people seem to like me on LinkedIn. I've started posting humorous stuff about dashboards and spreadsheets and kind of the, the shared misery that a lot of data analysts go through around this time last year. And it's, it's worked quite well. So if you're into spreadsheet humor, bad dashboard jokes, or really just me making fun of myself for being a data nerd, you can check out my LinkedIn feed, like you mentioned, I've got the pirate flag in my name.
Al ElliottSo what is it with you and dashboards?
Bill YostI have a love-hate, love-hate, love-hate, love-hate dashboard relationship, I would say. I think every data analyst starts their career seeing building dashboards as like the mecca, the thing that they want to do to deliver metrics and analytics to business stakeholders. But the reality is the barrier to entry for creating a dashboard these days is quite low, and the resulting impact of that is there is so many bad ones out there. I feel like everybody, like you mentioned, has their little story of I have this dashboard, I hate it. And so for me, the evolution of my analytic journey started from make dashboards everywhere I possibly can. Literally before I was at Google, I was working for a company called Protivity. I was working kind of breaking into HR analytics there without even really realizing it. And it was just every spreadsheet I could get my hands on. I was going to put it into Tableau, which was my weapon of choice at the time.
Leanne ElliottYeah.
Bill YostMake a dashboard, blow people's minds. Tableau people's minds was a joke that was made all the time when I was working in this field. And I learned that you don't always, you know, rattle the earth when you put these dashboards out into existence. You have people who are looking at them, they're feeling overwhelmed, they're feeling confused, and eventually they're thinking, I have 60 dashboards, which ones actually move the needle for me? And the answer is a very, very small percent of them. So as I matured in this space, I started focusing on making better dashboards or data products, ones that actually solve the business problem. And the reality is, when you put a spotlight on all of the corpus of data and metrics that you could possibly put a dashboard over top of, many of them are not worthy of it. Sometimes it could be a spreadsheet, sometimes it could be an email, sometimes you could put it into a parody song and deliver it that way musically. I've actually done that a handful of times, but Creating a dashboard takes a lot of work, takes a lot of maintenance, takes a lot of training, and then at the end of it, if people are never clicking on it, you're just going to feel very, very defeated. So at this point in my career, I'm focusing on making much fewer but higher impact dashboards, which also gives me a lot of mechanisms to make humor out of it when I'm on LinkedIn. Because again, everybody has a dashboard that they hate and they want to punch at, and they're so abundant that it makes it easy to discuss. So Again, yeah, I love dashboards when they're good and they get the job done. But if you've flashed up any given dashboard at any point in time, I could probably roast it pretty effectively just because I've seen a number of ways not to do it. Made those mistakes myself. It is part of the learning journey though. So I will say if you're a data analyst starting out, definitely fool around making dashboards, but always like apply a critical lens to it and think, how could this be better? Is this actually achieving the outcome I'm looking towards, or is this just fancy flashy metrics and charts in a centralized location that's going to overwhelm somebody.
Al ElliottCorpus of data. You threw that in fairly early doors. I like that. Written that down. I'm going to be using that at some point on LinkedIn this week to make myself look clever.
Bill YostI read books, you know, I got, I got some, I got some words, vocabulary.
Al ElliottI got words.
Bill YostWhat are you going to do about it? Might be the fanciest word I say in this entire pod, but The bar is very low.
Al ElliottSo we're going to talk about engagement surveys. Before we do, I mean, just, this is such an expansive question you may not be able to answer in like 90 seconds, but is there a rule, a Yost rule for dashboards?
Bill YostYes, it goes back to Greek and Roman mythology. I think it might have been Apollo who said, know thyself. I think that's the most important thing out of the gate. Dashboards can be a lot of things to— I'll try not to get too technically jargony with you, but there's really this spectrum of what a dashboard can be wherein you have curation, which is like we're bringing like a very concrete picture to a stakeholder to help them understand the data really, really quickly. This means there's not a lot of levers to pull. There's not a lot of filters, which are little dropdowns that you click to change the view that you're seeing related to the data. And it doesn't you know, completely overwhelm them when they log in. Those are really, really useful for like your executive stakeholders, people who need the data really, really quickly in a pinch in a predictable way. The other side of this is what we call exploration or exploratory dashboards, which is kind of putting an analyst in your pocket and letting you literally explore the data to try to uncover any kind of trends or patterns or breakdowns that you're looking for within the data. The tradeoff then, of course, is there are lots of buttons to press, right? You might have 10 or 12 filters on the left side that let somebody tweak it exactly how they want to, you're going to lose a C-suite person immediately as soon as you put that in front of them, right? Or any VP business executive who does not have time, they're going to get lost in the weeds of that. That's not to say that those aren't useful, right? There are lots of exploration dashboards that are available out there that serve a lot of use cases. But the problem is if you try to do both, you're going to have a really, really bad time. So the first thing I look at when we start to design a dashboard is, is this curated or is this exploratory? And sometimes people's eyes will roll back in their heads and be like, what are you talking about? But it is the number one key to figuring that out, out of the gate, because if you do everything, by definition, you're doing nothing.
Al ElliottSo before we go off and short Chubb Tableau's stock, we need to say that there is a place for that kind of, like, lots of knobs and twiddles and buttons and stuff. But there's also a place for, is the graph going up and to the right, or is it going down and to the right?
Bill YostAbsolutely. And to be honest, we just made this decision at work a couple months ago, because I'm working on a highly curated data product for business leaders, right, that we're trying to make be as intuitive and simple as possible, but that sacrifices flexibility. So if we have users who are like, well, I need this very specific metric and breakdown that doesn't exist in your dashboard, we don't want to leave them high and dry. So we're also developing what we call the exploratory companion to that curated dashboard, and we're working to make them really tightly linked to each other. So that if you need to get down into the weeds and get more information, you can get there. But it's not going to sacrifice our priority zero, if you will, journey of trying to make these— this a useful data product for our core audience of business leaders who often don't want to be in the weeds mucking around with these extended metrics and breakdowns. So there's a place for both. And I think just doing both well is, is critically important if you're trying to create a useful data ecosystem. for your users. Ecosystem, another big word I'll use. Corpus, ecosystem. We'll keep them, keep them written down there.
Al ElliottI'm writing, I'm writing this down. An ecosystem of corpae. Is it Latin or Greek? How do we, how do we pluralize it? I don't know. Probably a question for another day.
Bill YostYeah, you need another podcast guest to figure that one out.
Al ElliottOkay, let's talk about engagement surveys because that's what we're really here to talk about. I think there's a plethora, there's another great word, there's a plethora of platforms out there that say we'll do an engagement survey and we'll give you a beautiful dashboard. Not looking at anyone in particular. Hi, Bob. Not anyone in particular, but most of them are just frankly bullshit. So let's talk about your thoughts on a great engagement survey. You ran them at Google, is that right?
Bill YostYeah, I was on the employee engagement team at Google for almost 4 years. I, in that time, launched, I think, 7 of our annual employee engagement surveys called Googlegeist. Which is, uh, it's a fun name on like the voice of the Googler, uh, another foreign language. I don't even actually know what the root of that one is, but it does mean the voice of the user, um, in this case for Googlers. And you can search the word Googlegeist. It's actually pretty, it was pretty well publicized, uh, in like the 2010s, which sounds really wild to say this far into 2020, but, uh, as one of the like flagship employee sentiment surveys, because We had a really high response rate. It was always like 89, 90%. There was always a big, big push to have Googlers fill this survey out. And it was like a cultural moment for the company when the survey went out. And then again, when the results came out, because there was a very good readback of the results to employees and executives did a great job of, you know, standing up and saying, here's what we learned from the survey. Here's what we're going to change. We appreciate you taking it. And it's that feedback. So, there's a feedback loop that's really important, and I'm sure we'll get into more about how we can make a survey successful once we ultimately launch it.
Al ElliottOne of the biggest mistakes people do when they do an engagement survey is they take the results and they go, hmm, interesting. And then nobody ever hears anything, anything ever again. So, what other mistakes do you see, or do you think other businesses make when they're running these engagement surveys?
Bill YostI will say, you know, for every Googlegeist survey that I launched at Google, I also launched one or two that probably didn't get to the outcome that we were ultimately looking for. And there's A number of reasons for that, but I think it also, it ultimately goes back to respecting the user's time. The user in this case for us are the employees who are filling out the survey. But if you're doing this in a product marketing sort of space, this might be the users of your product or what have you. It takes time to fill out a survey, right? You have to open the email, you have to click all the buttons, you have to click submit, you have a myriad list of other things that you need to do to get through your day. And at the ultimately, you need to make it worth it for the user to take that survey and take that time. And so I think the biggest mistake that I'm ever going to see in terms of sending out a survey is, are you planning to do anything with it? Are you just doing this performatively? Are you trying to validate an opinion that's already solidified in your mind? I have a lot of great examples of where that manifests, particularly coming out of the pandemic, when there's lots of different new space that people are trying to get a read on, but also lots of opportunities for people to have their own biases and kind of be dug into their own ideas. So the biggest mistake succinctly is sending a survey and then immediately sending the results to the shredder, right? It's not just that you've wasted the respondent's time. It also takes a lot of effort from a survey operations team, like where I was sitting for many years, to actually get that survey out the door, get it QA'd, get it approved by legal and privacy, have the results roll in accurately, aggregate the results into a format that we can present back to stakeholders.
Leanne ElliottYeah.
Bill YostAnd then from there, for it to go absolutely nowhere, you're also gonna, you're also gonna piss off your survey team, if I'm being candid, right? Because they're gonna be like, why do I need to send this survey if you're not gonna do anything with it? But I think that's, that's a lower priority. The higher priority is you don't wanna burn people out, right? Because this becomes an endless cycle of, well, I'm not gonna take this survey because they're not gonna do anything with it. The response rate goes down with each subsequent survey because people are realizing that. And then you have stakeholders on their side going, I don't want to send a survey because no one's going to fill it out, right? It becomes this, this very vicious cycle, like the snake eating itself is how I refer to it at this point. So succinctly, send a survey, do something with it. Like that's how you can ultimately be successful if you're trying to capture this really unique information that you get, because there's really no other— you don't get employee sentiment through Workday or any of your standard HR systems that are like capturing employee lifecycle stuff. Like you're truly capturing somebody's voice, their feelings, how they're perceiving the company. And that's really important if you're going to actually use it for something useful.
Al ElliottSo we're going to come on to performative surveys in a second. But the way you were describing that, I'm imagining a bank of goodwill. And what we're doing is if we're sending a survey, we're withdrawing from the account. And then when we actually give the results and send the results back, we're kind of like almost depositing But is there anything else we can do, like any other things that would withdraw from it or any of the things that would deposit into that account, if you'll forgive my flaky analogy?
Bill YostI love that analogy. I wouldn't, I wouldn't call it flaky at all. So Googlegeist was also very popular because there were tangible changes in the company that could point right to the surveys being inputs for why they did such a thing. There was an entire function at Google called Core. That existed because of Googlegeist feedback that did not come in as positively as I think a lot of the stakeholders would like. And it pointed to a need for like a pretty big restructuring and a new team to fill some of these gaps that were being identified through the survey. So they like did it. They staffed a VP, put hundreds, maybe thousands of heads behind this function, and it became one of the like institutional benchmarks of what we did well with this survey. It was like, we heard your feedback. We made a whole new organization to try to address this specific feedback, right? That's the, that's the top-level thing you can do. I don't think every company is going to like staff a 1,000-person organization just based on survey feedback. But you can imagine there's lots of trickle-down smaller things that could manifest, right? If you're seeing in survey feedback that parental leave is something that leaves a lot to be desired, you know, you could work with your benefits team to actually get a more compelling or competitive parental leave package in place. And then when you launch it, make sure you say, we did this because we've heard your feedback. You know, based on the results of this, we know we can do better. So we sought out to do so. And now here's this new program. You know, we hope that it meets your needs and we'll ask that same question next cycle. So we can actually check this with data as well. And we can have the pre-post sort of observations. of what happened after we launched such a program.
Al ElliottI think a lot of people who are reticent about sending surveys are exactly what you described there. Everyone says, we want some kind of parental leave, and they're like, we can't do that. We're a company of 15 people. So what would you say to those people who don't want to do the survey because they're worried that people are going to ask for something they can't give?
Bill YostThat's a great question. So I would say first, Don't waste time asking questions that you don't intend on doing anything with, right? The textbook example of this is, hey, do you feel like you're paid enough? Because no one in their right mind is going to say, yes, I do. I mean, okay, I will say I've seen this result. Some people do say that, but you would be silly to not say, no, I want you to pay me more, right? But if you don't actually have budget to do so, the company's not in the financial position to give everybody raises, you shouldn't go ask everybody if they want raises.
Leanne ElliottRight.
Bill YostBecause you know where this is going to land exactly. So focus the survey, laser focus the survey on things that are actually in your control and you're willing to change. Because there's another trade-off that you need to consider every time you send a survey, which is called survey fatigue. Means exactly what it sounds like, right? If the survey is too long, people are going to be like, this is too long, I don't want to do it. And they're just not going to do it because ultimately it's optional.
Leanne ElliottYeah.
Bill YostOr if they get too many surveys, even if those surveys are of an appropriate length, they're still going to be like, I'm not going to do a survey. I did a survey 2 days ago and I'm going to get another survey next week. All I have is surveys coming out my, coming out my ears here. So I don't actually want to do any of this. That eventually leads to a place where you're not going to have response rates for anything. And, and unfortunately, it was not a favorite part of my job when I was working at this at Google. But we did have to do a lot of consultations in the same way that I tell people, you don't really need a dashboard here. I had to say, you don't really need a survey here because you're not actually trying to get to new information. You're probably trying to validate something you already know. And the meta trade-off here is yet another survey that's going to burn everybody out, and it's going to lead to them probably not taking my survey, the big annual one, which we want them to take.
Leanne ElliottDid you know that the UK's number one management podcast, That's us, by the way.
Al ElliottYeah, yeah.
Leanne ElliottAnd the UK's number one marketing podcast are both on the same podcast network.
Al ElliottWe actually have a lot in common. We both use behavioural science to help people do better at work.
Leanne ElliottAnd we both had to wrangle Rory Sutherland on an episode.
Bill YostAnd 2 out of 3 of us are devilishly good looking.
Al ElliottAnd you're not gonna say which. Fel, host of Nudge, UK's number one marketing podcast. It's brought to you— we need to do this in 3s—
Leanne Elliottthe Hobbs The audio destination for business professionals.
Al ElliottSeamless. Seamless. Tell us about your latest episode, Phil.
Bill YostUh, we've just done an episode on fake fandom and how New York indie bands are paying agencies to create fake TikTok videos about how much they like their work. And we talk about the behavioral science behind fandom and how that encourages people to enjoy the music and all of that good stuff and do some big things about how this affects the world. It is.
Leanne ElliottIt's such, it's such a good show. Of course, you'll hear all the stuff you want to hear about how to grow your business by using behavioural science in your marketing. But there's also just some really interesting episodes that'll be right up your street. Personally, I enjoyed Can Balsamic Vinegar Make Beer Taste Better?
Bill YostIt can.
Leanne ElliottAnd Are We All Just Status-Seeking Monkeys?
Al ElliottI am. Go and listen to Lunch wherever you get your podcasts.
Leanne ElliottBut come back.
Al ElliottYeah, come back. We'll be back next week.
Bill YostDefinitely come back because Nudge isn't as good as this.
Leanne ElliottSo come back.
Al ElliottI'll cut that out.
Leanne ElliottKeep that in.
Al ElliottSo let's invent something called the Yost coefficient. And this is a formula which is derived, which tells me how long a survey should be and how many times I should send it a year. Obviously, it depends. We both know it depends. But let's just say that I'm a small business, 25 employees. How many surveys should I be doing a year and what sort, how many questions should it have?
Bill YostIt does really depend. I will say the big survey that I sent, depending on who you were, because some functions or roles could have more questions than others, could creep into like 120, 130 questions, which is a lot. However, it didn't really impact response rates significantly when the survey was that long because those questions were all perceived as valuable or light weight enough that people could move through it easily and still get to the end of that submit response button, which they ultimately wanted to do so that their responses could be heard and their management chains could see the results and everything. That said, I wouldn't recommend a company of 25 sending a 130-person or 130-question survey out the door because there's— how much information are you really going to be able to make use of at that scale? So one of the variables in this coefficient is definitely size of the company. Another variable here is going to be your willingness to change or like the number of levers you have to pull to actually make meaningful change based on the results. And another variable is how often is that survey or all surveys going out all every year? Because if you think about when you start a new company at a new company, you might get a new hire onboarding survey. You might go to orientation and get a survey for how that orientation went right after. You might get a survey from the IT department being like, how is your laptop set up? Right? You can just be inundated by surveys really, really quickly out of the gate. And then it comes time for the twice-yearly or 4-times-yearly engagement survey that's also just being like, do you— are you proud to work with the company? If you were offered a competitive role elsewhere, would you take it? Like those sorts of questions. And so it's hard for me to say how many is too many for the individual because I have personally burned out on surveys just from being in the space for so long. Like, if Panera Bread sends me a, like, how was your meal survey? I'm like, nope, not doing that. There's just no way I can at this point. But I just, if I had to throw a rock at it, you know, 2 or 3 per quarter, you know, is probably on the upper max of what I would be looking at. It often will hinge on, you know, most companies will have their annual or semiannual employee engagement survey. At Google, like I mentioned, it was, it was once a year for the big one that I worked on. They've since changed that significantly. I'm not sure if they're still sending a big annual one, but they started shifting to a much lighter weight weekly survey to try to get less latent data because the big complaint was, well, that survey was in July and now it's December. So who gives a shit? You know, the things have changed and there is some merit to that, actually, to be totally honest with you. Uh, at Netflix, it's going— the sentiment survey is going out twice a year. So you get a little, a little more regular data, but you still get into this period of like, well, the survey went out in, let's say, February. The results came out in March. The next one is only going to come around in August. So there's quite a bit of time still between those two. So it's a very delicate balance. But I would say, yeah, maybe 2 to 3 per quarter across all the surveys that people are hitting is probably sufficient because more than that, it's just going to make me raise my eyebrow a little bit and be like, are they that valuable though? And they very well could be, like I said. So to summarize, it's number of questions, size of your company, how often you're sending surveys, throw that into the magic pot. That's the coefficient. I don't actually know what the ideal answer is. Your mileage may definitely vary.
Al ElliottWhat about these little pop-up surveys that come out like, like once a week and it goes, how are you feeling today on a scale of 1 to 5? Are you, do you want to throw rocks at your manager today on a scale of 1 to 5? Like sort of 5 little questions that pop up at the corner of your screen. Is that, does that work?
Bill YostIf the question was, do you want to throw rocks at your managers today? The legal implications of that aside, because I'm certain corporate lawyers would have something to say about that, you would hook me back in. I would take surveys again if you were asking me those sorts of questions. Unfortunately, the ones I've that I've programmed and wrote over the years are much more boring than that particular one. But what you're describing is called a pulse survey, right? Most regularly coined as just regular, you know, literally heartbeat pulse is what you're trying to catch when you're sending those questions out. I have heard mixed success of those across organizations. I don't know if this is still true, but famously Amazon would have a survey question when you would log into your laptop like it was a requirement. Be like, do you want to throw rocks at your manager today? Yes or no? You click yes, and then you can go to your inbox and like start your day or whatever. Not sure what response rates are for that or if that's successful for them. I know when such an idea was brought up as a, should we do it? It was always kind of like a, eh. From a research perspective, you don't really want to compel people to take surveys that way. You know, you want them— You want them to come to us. Not necessarily force them to us, because then you can potentially implement bias in the results because you're going to have people who are going to be like, strongly disagree to whatever the question is because they just want to log into their laptop and they hate the idea of this. Now, taking the forcing them to ask the question to log into their laptop part aside, like maybe you get a survey every week in your inbox that's like, hey, please take it. We want you to. We're not forcing you to. What I do see in practice is you launch those surveys, you'll start with a response rate, you know, that maybe you're happy with, let's say 40 to 50% response rate, simply meaning the number of people who take the survey divided by the number of people you invited to take the survey. And so you get maybe a 40 or 50, then the next week you see it tick down a percent, and the next week you see it tick down a half a percent, the next week another percent. And over this arc of time, you're going to drop, drop, drop, drop, drop because it does feel very operational at that point. And if you're asking a question every single week, to go back to your example, you know, it's a bank account, you're depositing and you're withdrawing, you're taking a lot of withdrawals from people. Like, it's still okay, 1, 2, 3 questions. It might be very, very quick, but it's still a withdrawal nonetheless. And like the same way that I go buy avocado toast every day and my paycheck only comes twice a month, You know, eventually the avocado toast is going to drag me out and I'm going to be like, I need more paycheck here. But that's a crude example. But what I'm saying is, is specifically like you're not doing the readback, you're not looping back with those results to your respondent base at that same cadence. So there's a real imbalance there. And so if you're going to try to explore this pulse survey type of stuff, there can certainly be merits to it. And it doesn't always have to be once a week. You could say a pulse survey might be every month or something like that. You just have to make sure that you're ultimately fulfilling the other side of that transaction, or eventually you're going to just make the survey be completely useless by way of not getting enough responses for it to be significant for anything.
Al ElliottI've heard a rumor that you guys are clever and you have a way to select out people like me who will just go, yeah, click, click, click, click, click, random, get on with my day. I don't care. Is that true?
Bill YostMathematically, we can do anything. I'll say that, right? We've got, we've got the data. We have unlimited time to figure it out, semi-limited time to figure it out. If we wanted to actually say like, Al is, is completely jaded and is not giving us quote unquote honest responses, maybe we could assign some sort of percent likelihood that that's the case. And then we just only look at the responses for people who are under that percentage threshold. In practice, you don't want to, you don't want to do that because you don't know for sure, right? I can never be 100% certain. To spin this into a different but related field is predicting when someone leaves the company, right, is going to quit is one of these hot-button issues in my field because we have AI and machine learning and we know a bunch of stuff about employees. So just like make it clear when somebody is about to leave. Even if I can create a model with near absolute certainty that says Al is about to leave the company, what are you practically going to do in that situation? You're going to sit you down and be like, hey, Al, computer told me you're going to quit. You know, we want to talk about that. In the same way, I can't be certain that you put strongly disagree. I can't be 100% certain that you put strongly disagree the whole way down the chain because you're jaded versus because you're really, really unhappy.
Al ElliottYeah.
Bill YostAnd if you're really, really unhappy, we don't want to dismiss your results, right? You're the reason we're sending this survey. We don't send surveys to pat ourselves on the back and be like, hell yeah, everybody's happy if you remove all the unhappy people. You know that now the snake is eating itself from our side of the house. And we don't, as professional survey practitioners, we don't, we'd never want to do that. I will say there is this methodology called the heartbeat analysis. Though, which tries to tackle this exact problem. I'm aware of this because a good friend of mine, Chris Patton, and another analyst named Justin Pearl, they created this at Google and then it won some fancy award through maybe the SIOP organization. It was like the Wiley Award for badass survey stuff. I don't actually remember what the title was, but it actually creates a baseline of responses and then looks at where for each person and then looks at where the questions where someone's baseline went above or below that. So if you went through and put strongly disagree for everything, your baseline is going to be strongly disagree. But if you picked 1 or 2 questions of agree, strongly agree, you know, you went the other direction, that's going to pop that as being like, hey, he went out of his way to change his, his responses here because he must feel really passionate about it. It's where the kind of the heartbeat element comes to it. So there are definitely ways to get to that answer without completely throwing your responses away. And also, we work really hard to get everybody to take the survey. So like, I don't want to throw responses away. I never want to do that. In fact, part of the survey administration process for us operationally was people would get cold feet after they submitted the survey. So maybe they, they said yes to the throw rocks at the manager question. They're like, oh my God, I just I was having a bad day. I don't want to throw rocks at my manager. Like, delete the whole response. And we would do that. Of course we would honor it. But that makes us sad because we want all the responses that we can get. High response rates validates our work. It makes it worthwhile. It actually, it also makes our responses statistically significant where it matters. So yeah, we love all survey responses one and the same, even if you're really, really jaded and hit no to everything else. You—
Al Elliottsomething you just said there, it kind of in passing, I think, would terrify a couple of people listening because you went, oh, I noticed Al said X. Now, I know it was just an example, but how anonymous are these surveys and how anonymous should they be?
Bill YostThis is a fantastic question. Okay. Sorry, listeners, we're gonna get really nerdy for a second. Most surveys are not anonymous. In fact, it's pretty much a— it's a pretty big taboo to say that you're giving an anonymous survey because anonymous means, I mean, technically speaking, we can do it, but it means we can't connect any information about your responses back to any metadata that matters. So one of the, one of the most popular ways to review survey results is through what we call a heat map, right? You look at the top level by each question. Let's assume we're using 5-point favorability questions here, like what percentage of people at the company level said they agree or strongly agree with something. We call those a favorability That's one set of one column in the heat map. And the other columns are variables that we care about, metadata related to the person. This could be the business function. This could be their VP. This could be their tenure at the company. This could be their location, region, country, site, you name it. And this is how you're able to get a lot more value from these survey responses because you can see pockets of the organization where sentiment differs.
Al ElliottYeah.
Bill YostYou can be like, whoa, new hires really freaking hate our benefits package, right? Like, that's okay. Why is that? You can dig into that further and maybe you can try to decide if this is, this is an onboarding problem or, you know, maybe the, the compensation packages for new hires is not as compelling as the ones for tenured employees. Like, it gives you a lot of, a lot of red meat to chew on, if you will, in terms of analytical value.
Al ElliottYeah.
Bill YostAnd the only way to do that is by what we call a confidential survey. And so confidential means a very, very limited number of people can actually tie Al's responses to Al's identity. But what we do is we collect it with your identity and then tag it to enough metadata that we publish out for, for end users to use, but not so much that we can— Right. Make it very, very simple for Al's manager to be like, Al's unhappy. Let's figure this out. Let's get him out of here or whatever it is, because privacy is still really, really important. But it becomes this trade-off where we do need to know some information about you to make those results as useful as they can be. And this is where I can nerd out about this all day is at Google, we used to do both. So there was a confidential version of the survey. And there was a fully anonymous version. Now, I mentioned earlier that some of your roles or locations or functions might get slightly different or expanded questions based on who they are. You couldn't get that, obviously, on the anonymous side of the survey because we fundamentally do not know who you are. So the anonymous survey was much shorter. It was really whittled down to what we called the core questions of, let's say, 20 to 25 general-purpose questions about Google. And then the confidential side had way more than that because we could customize stuff. You know, if you, if you as a manager, Al, had 7 people on your team, those 7 people in the survey, if they're electing to take the confidential version, could say, hey, how do we feel about Al's vision, about Al's commitment to our career development? You know, whatever. And we could literally inject your name in there so that it becomes a more personalized experience for them. But we did both because we conceded, you know, there might be people who are like, we find that creepy because we don't want ultimately people to be able to know who said what. But when it shook out in practice, we would have, let's say, 90% response rate of the company. 89 of that 90 went to the confidential survey. It was always 1% or less of the respondents who are going to the fully anonymous version because people wanted their responses to flow up to their management chain. They wanted their functional leads to know how they felt about things. And they ultimately had enough trust in our survey team to administer it fairly and know that we're not leaking their data or exposing their identities to people. And so this is where that trust, you know, that deposit withdrawal really, really takes hold because we do need to abide by that. And I was at one point one of 3 people in the company who could, you know, reach in and look at your responses, Al. And I'll be honest, not that interesting. You know, I'm not in this business to try to dox people or be like, Al hates his manager, get him. You know, I'm just, I'm a professional analytic person who is trying to do the best thing. And I also do have, you know, as much as I joke and laugh, have some ethics that I try to abide by, which is I want to continue to relate, to have trust from the survey respondents, but also stay gainfully employed. Because if I started leaking people's responses, I would face my own kind of disciplinary measures for sure alongside that. So eventually, I was very proud we got to turn off the anonymous survey. We're just like, we're not actually— the juice is not worth the squeeze for us because we're getting very limited information from a response size basis. And also the information we get, we can't really do a lot with it because it's just like we don't know how we can focus that information towards a specific set of people, know how pervasive or you know, narrow the problem might actually be. And it's a ton of work to have 2 surveys running in parallel. Like, it was a real headache for me. So I was very proud when we finally turned that off. So I'll say most surveys across industry are of the confidential variety. Netflix's is as well, right? We can ultimately tie it back to the user, but we keep that under lock and key. Really just the analytics team, a few survey admins who are able to do that. And And when we do need to get to someone's identity specifically, it's for reasons that are typically unfortunate but good-intentioned. So, meaning like, unfortunately, you'll have instances where you'll have comments where people are like, hey, I was harassed at a company function, or something like really awful happens, right? In that case, we do want to go in, figure out who that person is so we can get it to employee relations or legal and figure out how we can remedy that situation because we never want to find a situation where somebody said something really terrible in a survey. We have no idea how to fix that. And then we're just kind of saddled with this, this really disturbing problem that could be addressed. So yeah, that's my soapbox. Confidential versus anonymous. Most surveys are confidential. Sorry if you've turned the podcast off at this point, listeners.
Al ElliottWell, if they have, they've missed gold and also they won't hear you say sorry. So they'll just be blaming you for the rest of the day, weren't they? Honestly, that is genuinely, that's exactly the kind of answer I was looking for. Now, Bill, I don't know whether I've read this or whether we chatted about it when we did our little pre-call, but there's something I've got a note down here, something about how leaders, you can tell them the results of the survey and they just almost don't hear it and hear what they want to hear. Does that make any sense to you?
Bill YostYeah, I've seen that happen. I think, you know, have you ever watched a movie or a TV show with maybe a partner and something ambiguous is presented, and then like the episode ends and you're both like, well, I think X happened. And the other says, well, I think Y happened. That's because even though you've both been presented the exact same information, you're still viewing it through your own lens, right? This, this personal lens that's really distinct to you and quite hard to replicate across the board. We run into that a lot in the survey world when Data are messy. People are messy. The survey might be messy. You know, the question might not be as well written as you wanted it to be after you looked at it on the other side and you saw the results. As an analyst, it's, it's my job, it's our team's job to frame that information in a way that removes the noise and presents it to a stakeholder in a way that hopefully can be universally resonated with. But that's not always going to be the case.
Al ElliottYeah.
Bill YostA specific example of this that comes to mind is return to office, right? Hot button issue across every company that has to reconcile with, if they made pivots during the pandemic, some of their, or maybe most of their workforce went remote. And now the question is, do we bring them back to office? If we have a survey that, let's say, asks 9 questions about the benefits of working from home, And one question about the productivity gains that you have while working in the office, and the survey is screaming loud and clear that everybody likes the flexibility of being able to work remote or at home or whatever. And we present this to a room of I don't know let's say seven executives. At least one of those executives is going to have this lens of we're more productive in the office, right? And so doesn't matter that nine of those questions are saying like screaming blaring red we want to work. From home or we want to work flexibility, they're going to look at that one that goes, well, look, we're more productive in the office and I'm actually optimizing for productivity and not employee engagement or experience. Therefore, we need to get back to the office. That's going to happen, right? And anytime you think you have a concrete solution or answer to a specific problem, you're probably wrong because you're going to encounter numerous different lenses across these stakeholders, these executives, whomever you're presenting the answers to, and they're going to inject their bias, whether it's conscious or not, into these answers. And sometimes you can try to push back and say, well, that's not quite what I was saying here. But sometimes you just have to accept that this is the nature of the beast when we do this kind of work. And I've heard plenty of stories from other analysts in other spaces who went into a presentation with their concrete set of recommendations and then walked out with the executives being like, well, we're going to do the exact opposite set of recommendations. And it's just like, well, all right, I'm not going to die on this hill today. So I did what I can do in terms of gathering and presenting the information, but the business is going to move as it does. And this takes us back to, you know, were they planning on doing this to begin with? Was this all a performative survey? These are the problems you need to solve far, far upstream. By the time you encounter them where we are downstream, when we're presenting the results, it's definitely far too late at that point to try to make corrective action. So the best you can do to try to avoid these situations is have very, very strong survey design, well-written questions that removes ambiguity wherever possible and gets you to pretty conclusive results across the board. But also know you can do everything right. You know, you can drive the speed limit, you can buckle up, you still might get into a car crash. Like, these things just happen, unfortunately.
Al ElliottYeah. I wanna talk about someone who sort of falls between the, I only take what I wanna take from the survey, don't care what the results are, and someone who goes, I will accept every result as like, okay, that's what we need to do. Let's talk about the human person, the real person who's in the middle. They get results, they're disappointed with them because maybe they're a founder and there's 30 people. They've built every single one of these up from the ground up. They're disappointed with the results. How, how should I feel when I get results that that don't tally with the way I thought things were around here.
Bill YostI saw this most often at Google because managers would get reports tied to their name, their organization. And any number of those questions could have a poor result, and it can be personally hurtful, right? Nobody wants to hear that they're doing a bad job. Nobody wants to hear from 12 people on their team that they're doing a bad job in a very particular area. I would often say to them, one, you're not going to be perfect in every single area. There's just no way. There are no rock stars among us who are going to have 100% favorable across the board. If you do, it's because you have a team of 3 people and they're all really, really happy people, right? Eventually, when you get into the 7, 8, 9, 10 number of people who are filling out the survey, those varying lenses that we talked about earlier for the stakeholders and the executives, Also applies to the employees, and people are going to view the world differently from that angle. Also, some people are just quite harsh graders, right? A neutral to them, a 3 on the 1 to 5 point scale, might actually be pretty good. But to you as the manager, it's gonna be like, that sucks. That's a shitty score, that 3. I want a 4 or a 5. I want that favorable point. And so what we would tell them is, Focus on the people who did respond neutral. You know, not literally focus on who they are, but look at that as opportunity area, right? Neutral means you're fence-sitters. They can probably be swayed into becoming favorable next time around with very little change. Unfavorable, meaning they said strongly disagree or disagree to the question, that's going to be a little harder to bridge the gap on. But if you look, try to put the optimist lens over top of it, If your percent favorable is, let's say, 33%, and you're not thrilled with that, your percent neutral is 33%, and your percent unfavorable is 33%, you could shift just a small number of people, and next cycle you have a 66%, right? Because you moved that whole neutral bucket over to favorable. Now 2/3 of people are happy. Like, that's significant. Better, a significantly better result than where you started. So look at it as an opportunity to learn, grow, and do better. And then also on the survey administration side, a well-written survey is going to have resources for all of these questions. Like, we should never give a question that leaves somebody completely high and dry, right? If the question is, I receive valuable feedback from my manager, And that's, that's low, right? There should ideally be some sort of, from a curated guide from HR, that's like how to facilitate meaningful feedback conversations, the cadence at which you should do it. Like, there should be a roadmap in front of you to be able to try to drive that score higher. Hopefully a research-driven roadmap that says, like, we've seen when managers do X, Y, and Z, this score will raise by some percentage. And then that gives somebody a tangible path forward to try to address those, those issues. Because ultimately, if a manager is coming and saying, oh my God, these scores suck, I'm heartbroken, it means deep down they do want to do better, right? The willingness to change is absolutely there. So a well-designed survey will enable that for them, right? We're not here to just get bad scores and then say, oh, you're out of here, right? Continuous improvement is is the strategy. That's the mandate here. So yeah, I would say don't take it too personally. Look at your neutrals as an area where you can improve and lean on what the company is giving you to try to improve those particular scores. And if the company is not giving you anything, I think you as the manager should be empowered to say, I need more help here. You know, you can't just tell me this is bad, fix it. That's not productive for anybody. And I think most survey teams would look at you and go, you're Right. That's exactly, that's totally true. We do need better guidance here. Action plans are what we would typically call those in the survey jargon world.
Al ElliottThere are so many platforms out there that in terms of HR that say they do engagement surveys, some are clearly just a play for ARR, MRR, ARR. Some actually seem decent. Is there one out there that you think is pretty good?
Bill YostI would say Vendor assessments are very important if you're a large company and you need to, let's say, onboard a new survey provider. There are, as you say, lots of names in this space. Qualtrics, Culture Amp, Perceptics, Medallia, to name a few, right, are all dabbling in the employee engagement world. The reality is you could facilitate an entire employee engagement survey through a Google Form. If you really saw fit, so I would say you need to meet your company's needs and objectives with the platform at hand and try to find the right fit. I definitely have my favorite, and I have my least favorite. I will not drop them on this pod because I don't want to throw anybody under the bus. But it's really, it's really again, like your mileage may vary. What are you trying to solve for? I know if you look at market share, I think. Qualtrics has quite a bit of it, and it's because they've been around for a really long time. They have a very robust solution offering. But that's not to say that any of the, you know, 20 or 30 other vendors in the space might not be the right fit for you because there's the cost element to this. There's the number of people you're trying to administer the survey to. There's the analytical value out of the box that you need from the survey vendor. Because what I often— what we often did at Google and what we do somewhat at Netflix now is we take the results back from the vendor, bring them into our internal data warehouse, and do our analysis and our reporting inside the box of, of what we own, because there are lots of good reasons for us to do that. But if you don't have a mature business intelligence function that's ready to take on that data, you might want to look at something that's more sophisticated because the cost-benefit there is going to be a different value proposition.
Al ElliottBill, people are going to want to hear more from you. Out of all of our guests, you're be in the top 5 people who want to follow you. I'm guessing LinkedIn is the best place, is it?
Bill YostLinkedIn is definitely the best place. You can search Bill Yost and it's the only one with a pirate flag in there. And also, if you're in the United States and you want some cookies, you can go to www.cookiesworthsharon.com. And what's the, what's our acronym here? I was thinking Truth, Lies and Work. Let's say code TLW will get you 15% off cookies for the next, let's say, 2 months after this show release.
Leanne ElliottThat was Bill Yost, People Analytics Lead at Netflix, former Google Geist Architect, and quite possibly the only person who can make survey design genuinely riveting, apart from me, of course.
Al ElliottOf course.
Leanne Elliott3 takeaways for leaders. First, only ask questions you're willing to act on. If you can't give everyone a pay rise, don't ask if they think they're paid enough. Laser focus your survey on things actually within your control. Otherwise, you're making withdrawals from a goodwill account you can't afford to drain.
Al ElliottSecond, close the feedback loop loudly and clearly. When Google's survey flagged a structural problem, they built an entire new function to fix it and told everyone that was why. It doesn't matter if you hire 25 or 25,000 people. You do need to say, we heard you, and here's what's changed. That's what gets you the 90% response rate.
Leanne ElliottAnd third, confidential beats anonymous. Nearly every well-run survey is confidential, not truly anonymous. And that's a feature, not a flaw. It's the only way to find out that, say, your new hires hate the benefits package, or that one team is quietly miserable while the rest of the company is fine.
Al ElliottYou can find Bill on LinkedIn. Search for Bill Yost. Bill Yost, that's Y-O-S-T. Look for the pirate flag. And if you're in the US and want 15% off some delicious cookies, head over to cookiesworthsharon.com and use the code TLW for a discount.
Leanne ElliottHow much of a discount, Al?
Al ElliottThat we will find out when we order our own.
Leanne ElliottMy notes say 15%.
Al ElliottBill, sorry if it's less than 15, but now it's 15%.
Leanne ElliottThis is Truth, Lies and Work. We will see you next week with with cookies.
Al ElliottThank you.