Your Survey is Lying to You

How bad data leads to bad decisions — and the analytics that can solve it all

An employee engagement survey is a smart first step toward shaping an intentional culture that mitigates risk, creates value, and delivers bottom-line results. After all, how do you know if your organization is headed in the right direction if you don’t know where you stand?

But a survey is only useful if it provides an accurate picture of what’s really going on in your organization. They help to ensure your employees feel heard and show you’re willing to listen to their input. Done right, surveys can build the trust and confidence of your workforce, and research shows this has a direct impact on engagement.

Unfortunately, not all surveys are reliable — and sometimes the consequences can be significant. In one example, the Harvard Business Review reported that United Parcel Service suffered a “spectacular” loss after its annual survey failed to uncover issues around the increase of part-time jobs. Employees went on strike as a result, costing the company hundreds of millions of dollars.

So, let’s dive in and discuss the different ways your survey can do more harm than good — and what you can do about it.

First things first, administering a survey is the easy part

Knowing what to do next is where it gets tricky. All too often, momentum is lost in the analysis, understanding, and sharing of the employee survey results. Efforts stall and the data just sits there, collecting dust. And there goes the meaningful insights, quick action, and culture improvement.

In fact, doing nothing with survey results is worse than not surveying at all. Neglecting to close the loop with your employees soon after the survey closes provokes disengagement and erodes trust. Not to mention, you lose out on the opportunity to get valuable feedback from those who know your organization best and are left without a compass to guide your decision making.

Six ways your survey can steer you wrong

1. The process takes way too long. Some survey providers take months to process and present your data. When it comes to surveying your employees, the faster you can act, the better the result. Otherwise, you’re acting on old data and damaging trust with your employees.
2. Your survey isn’t backed by research. With all of the technology available these days, it’s tempting to craft the survey yourself. And with 100+ providers out there, it’s entirely possible you’re working with someone who lacks the right qualifications to handle your sensitive data.
3. You’re comparing your organization only to an industry benchmark. Your true culture story can be easily hidden — or even misinterpreted — if your data isn’t properly sliced and diced, taking into consideration nuances and segments such as department, tenure, and location.
4. Your survey asks the wrong questions or bad questions. What your survey asks employees — and even how the statements are worded — has a direct impact on the quality of data it yields. Examples include leading or biased statements, double-barreled questions, ambiguity, and more.
5. The survey isn’t anonymous. It’s human nature to want to resolve workplace issues at the source. But this can lead to blaming or berating people. And oftentimes, managers aren’t prepared to handle negative feedback, especially if it’s aimed at them.
6. You can’t make sense of the survey data. Numbers, figures, and overly complicated charts can make it difficult to truly understand what’s going on inside your culture. And that makes it almost impossible for you to move from data to insights and action.

Or worse, your survey may be lying to you

Yep, we said it: lying. How’s that possible? When a survey produces bad data, you’re not getting an accurate picture of what’s really happening within your culture. Bad data opens the likelihood you’ll make bad decisions and take the wrong actions. And that can have significant consequences for your organization.

Here’s what can happen and why

Meet Jack. He’s 20 and started working for your company eight months ago. It’s his first full-time job. His pay is low but fair enough. He rents an apartment in the dingy student housing area of the city. Things don’t work all the time, but it’s what he can afford. He drives an old hand-me-down car he doubts will pass the next inspection.

Now meet Jill. She’s 43. She works for your company too, but this is not her first job. As a seasoned executive, she has a wealth of professional experience. She receives a competitive salary, and over the years her role — and compensation — have increased substantially. She owns a new home that requires little maintenance. When repairs are needed, she gets things fixed right away because she can. She leases a new car that’s under warranty.

And today, your organization launched its annual survey. Both Jack and Jill were given the opportunity to respond to the same series of survey statements.

Here’s an example:

Survey statement:

“At this company, we do things efficiently and well.”

Let’s say both Jack and Jill respond to this statement with “Slightly Agree.” Looking at the employee survey results, we see the same response.

But this is where things can go wrong

There’s more to the story than meets the eye, because similar survey responses don’t always tell the same story. Jack and Jill are two people at very different stages of their careers. They have different salaries and live in different environments.

Outside of work, Jack is surrounded by things in need of repair. But, when he shows up at the office each morning, everything is in working order. Jill, on the other hand, expects things to be in working order. And when they’re not, she has the means to get them fixed.

They responded to the same statement. The survey responses are the same. Yet, we see two different stories. Why?

Employees have different expectations, experiences, and priorities

When you treat the survey responses from Jack and Jill as equal, you’re ignoring the context each brings with them to work. Ignoring benchmarking at the individual level can lead to misguided decision making.

The survey statement, “At this company, we do things efficiently and well” is a great example. Our research and experience show that highly paid employees tend to respond more negatively to the “Efficiency” statement than those who are paid less.

Let’s look at another example:

Survey statement:

“I feel genuinely appreciated at this company.”

Meet Adam. Adam is an account executive at your company who makes approximately $100,000 each year. He’s new to the team, joining your organization only six months ago.

Now meet Cole. Cole is also an account executive. In fact, he and Adam are on the same team and his annual salary is similar. The biggest difference is that he’s been with your organization for 15 years.

When survey time rolls around, Adam and Cole respond to the “Appreciation” statement with “Agree.” That’s good, right? Not so fast. Once again, there’s more to the story, and it’s important to interpret these responses correctly.

While both answered “Agree,” Cole’s response is a more impressive score than Adam’s. This is because new employees tend to respond more positively to the “Appreciation” statement. Long-tenured employees are less likely to “Agree” with the “Appreciation” statement, making Cole’s response cause for celebration.

These are just two fairly simple examples of how you can be misled without the proper tools in place. Let’s dig even deeper.

Surveys you can trust consider the context

To get to your true culture story, you need to go deeper than the company level. Deeper than department, tenure, or location. You need to get to the individual level — and also ensure your survey uses the right math.

Energage Organization Insights does just that. Tapping into our database of 20 million employee surveys from well over 58,000 organizations across the globe, our AI-driven analytics gives you instant access to what’s truly meaningful and what’s just useless noise. That means you’ll move from insights to smart decision making and real change — with precision and confidence. It’s what separates our survey from the rest. Turns out, the U.S. Patent and Trademark Office agrees with us — so much so that we received a patent for it.

Here are some of the other things that separate your average survey from insights that get to the real truth:

Smart benchmarking

• Individualized benchmarking: Intelligence backed benchmarking understands employee responses differ based on their relationship with your company.
• Mixed sector benchmarking: Some organizations don’t fit neatly into a single sector. The right solution applies the right benchmarks to each area of your business.
• Cultural overlay benchmarking: Geographic differences affect how employees respond on surveys. These variations need to be taken into account.

Patented analytics

• Accurate response scoring: A probability-based approach adapted from the high-stakes testing industry produces the most precise results.
• Smart focus areas: A core blend of benchmarked score, change, and importance as well as hierarchy-awareness and “spidering,” distribution awareness, actionability, overlap, comprehensiveness, and demographic adjacency.
• Intelligent comment analysis: Purpose-built for employee survey comments, benchmarked theming, and benchmarked true-sentiment.

And comparative analytics gives you even more power

Advanced well beyond simple benchmarking, our comparative analytics applies reference data to individual data points prior to aggregation rather than after the data points have been aggregated.

Here’s what it looks like:

Comparative analytics leads to actionable insights

1. The green slices are where meaningful insights are found.
2. Different slices are made up of employees who have different inherent relationships with your organization.
3. Different inherent relationships drive them to respond to employee surveys in different ways.
4. These “proclivities” are adjusted to get to the real story behind your data.
5. Green slices are the output of complex math that’s been simplified for you.
6. Now with the right insights, you can take the right actions.

Comparative analytics gives you actionable intelligence. So, if your survey doesn’t utilize this science, we’re willing to bet your investment in employee feedback will have no impact, wasting valuable time and resources. It might even cause you to make decisions that do damage to your organization.

Make sure the survey you choose can be trusted

Employee surveys are as important as ever — we know we don’t need to tell you that. But with all of the options available, how can you know for sure you’ve chosen one that produces actionable intelligence?

Here’s your checklist:

• Start to finish, the survey is anonymous, the process is quick, and the statements are scientifically sound.
• Survey insights clearly and accurately identify your culture strengths and focus areas.
• Data visualization makes it easy to interpret results and see what’s really happening within your organization and its culture.
• Smart benchmarking considers employee responses at the individual level, and where needed, mixed sectors and cultural overlay.
• Comparative analytics applies reference data to individual data points prior to aggregation.

But don’t stop there: there’s more than surveying

An employee engagement survey is the right first step toward shaping an intentional culture that mitigates risk, creates value, and delivers bottom-line results. But remember, a survey alone won’t fix anything — even if you’ve chosen the best survey available.

Failing to close the loop and act on the feedback you receive can do more harm than not surveying at all. Without proper communication and a plan for next steps, you’ll fail to realize the full potential of your organization. Here’s what we recommend:

Listen and connect

Once you understand your culture story, you’re ready to dive deeper to get to the root causes. First, listen to your employees — then explore their responses and find opportunities to expand on their feedback. Second, connect on your strategy and ensure alignment.

The right strategy involves:

• Thanking employees and creating a custom follow-up strategy.
• Making sure every manager and team lead gets results for their team.
• Sharing local results with every employee to inspire further insight.
• Connecting successful teams and managers to people who can learn from them.
• Using survey insights to help consider culture in every leadership conversation.

Engage and grow

Now, it’s time to act on your survey insights or — even more effectively — let employees act on it. It’s their feedback that started your culture journey, so empower them to move everyone forward. This is the fun part, really. Employees get to solve their own problems, stretch their skills, and unleash their potential. They’ll be glad for the opportunity to improve the organization. You’ll see better results — and more engaged employees!

To improve your culture and performance, make sure to:

• Start all action at the front line, only escalating topics that can’t be handled at that level.
• Seek and empower passionate employees, no matter their role.
• Make it easy to visibly track action and celebrate success.
• Keep doing what works, but make a plan to re-evaluate later.
• Enforce win-win solutions across the company; anything that isn’t working for one group soon won’t work for others.

These steps are essential to building an intentional culture strategy that unlocks potential and unleashes the boundless energy of working better together.