ASU Lodestar Center Blog

Current nonprofit sector research and recommendations for effective day-to-day practice from ASU faculty, staff, students, and the nonprofit and philanthropic community.

Friday, September 16, 2011 - 10:08am

posted by
Robert F. Ashcraft, Ph.D.

Executive Director
ASU Lodestar Center

Welcome to Research Friday! As part of a continuing weekly series, each Friday we invite a nonprofit expert to highlight a research report or study and discuss how it can inform and improve day-to-day nonprofit practice.

A popular television and print marketing promotion in the 1970’s showcased the “Shell Answer Man,” an all-knowing expert on automotive issues. There seemingly was no end to what the “Man” could answer, based upon research and reason, and it was quite beside the point that Shell is an oil company seeking customers.

In guiding our ASU Lodestar Center, I reflect on this type of marketing because of the inquiries we receive each week for research-based answers to every sort of question imaginable. Like the Shell Answer Man, we respond to dozens of inquiries each month on wide-ranging questions.

Though we do our best to respond to requests, unlike the Shell Answer Man, we are not fueled by the research and marketing budget of a global oil company. Unfortunately, there is little interest on the part of most funders to invest in high quality, methodologically sound research on leadership and philanthropy topics that are so very important to those who lead, manage, and support nonprofits.

Do you know how effective you are?
Find out with our new web-based self-assessment.
Click here to learn more about the Principles of Effectiveness.

As a knowledge enterprise, our Center finds the wherewithal to regularly produce a few core research products in our Giving and Volunteering Study, our Arizona Scope of the Nonprofit Sector, and our Nonprofit Compensation and Benefits Study, along with our work on Models of Collaboration. We are committed to these research products because of their high demand and utility toward an understanding of nonprofits and philanthropy in Arizona and beyond.

The challenge arises when stakeholders come to us for customized research. These requests require a high level of expertise, time, and output, and they often come with unreasonable expectations about costs, complexity of design, and other essential ingredients to successful inquiry and deliverables.

Our Center’s work with the AIM Alliance, funded by the W. K. Kellogg Foundation, was designed around the fusion of three outstanding nonprofit academic centers. Of the several goals for the Alliance, involving ASU’s Lodestar Center for Philanthropy and Nonprofit Innovation, the Indiana University Center on Philanthropy, and the Johnson Center for Philanthropy at Grand Valley State University (Michigan), was the matter of using joint research protocols in studying selected topics in philanthropy.

During an AIM Alliance meeting, I expressed frustration about those who want great research results with accompanying unreasonable expectations regarding time, cost, and quality. The executive director of the IU Center and a colleague in the world of nonprofit and philanthropy research, Dr. Patrick Rooney nailed the issue by describing his experience with what many stakeholders want from university research centers. He notes four parameters: Bigger! Better! Faster! Cheaper! Given these, Dr. Rooney's response that follows is that he can give them three of the four variables, and they can choose which of the factors they want! If they want the first three (bigger, better, faster), then they cannot have cheaper. Conversely, if they want cheaper, they cannot have bigger, better, and faster.

What is the downside to lack of adequate funding for high quality research in our field? There are many. Just one is that small-scale studies are funded, often to advance a particular point of view, conducted by organizations who produce beautiful reports from data based upon flawed methodologies. We are seeing more research reports based on convenience samples (i.e., a survey monkey sent out to e-mail contacts), which is then touted as portraying some universal understanding of the nonprofit sector. Such sampling may not be representative of the sector and may under-represent, over-represent or not represent at all the reality of the issue under study. Yet, these “quick and dirty” studies are blasted out through social media channels, often attract media attention, and occasionally become highly touted. These studies are certainly not bigger or better, but they are cheaper and faster!

A famous case of flawed methodology leading to faulty conclusions involves the United States presidential election of 1936 pitting Franklin D. Roosevelt against Alf Landon. A study conducted by The Literary Digest predicted that Landon would be the winner with 370 electoral votes. In fact, Roosevelt won by a landslide, carrying all but eight electoral votes and every state but Maine and Vermont. Roosevelt also won the popular vote by a large margin. What was the reason for such an error by a publication that had successfully predicted the prior five elections? They relied on voluntary responses to a survey — from a convenience sample of their own subscribers, the results of which were grossly erroneous. The magazine’s credibility suffered, and it became extinct within months after the election.

I wonder what the Shell Answer Man would say when asked to produce research that is BIGGER! BETTER! CHEAPER! FASTER!?

Dr. Ashcraft is the executive director of the ASU Lodestar Center for Philanthropy & Nonprofit Innovation at Arizona State University, and he is associate professor of nonprofit studies in ASU's School of Community Resources and Development, located at the downtown Phoenix campus.

Like this article? Get another!

Click here to read "Research Friday: What Affects Hispanic Volunteering - Comparing Three Surveys in the United States" — by Lily Wang.


Well done! As one who has one foot in the non-academic world ( I won't say "real world") and at least a toe or two in the academic world, I can understand the tension. Funders do want quick action and in research it can lead to fluffy and incorrect results. And yes they do get picked up on social media and by thoughtless politicians. Researchers also have their responsibilities to be responsive to demands for information without "cooking the books". Lodestar Center keep doing what is right and relevant and continue to refresh your product.

Excellent points! Many organizations are conducting and producing survey and research reports. They often use nonprobability convenience samples and then generalize the results to a whole population or sector-- a methodological error. Some education and awareness is needed on this topic so readers and users can discern when survey findings can be generalized to a wider population and when they really only apply to the responding organizations or individuals.

Stephanie and/or Robert, It would be great to have a follow-up blog on some fundamental or essential things to look for when analyzing research reports or surveys done in the nonprofit sector. What makes for good methodology? Or are there some common methodological errors to watch out for? How could the research / surveying being done better so that it yields valid conclusions? Do you think people are purposely misleading readers or just under-prepared for doing the analysis /surveying and then reaching conclusions based on flawed methods?

You make great points! i wish that instead of spending so much time on research trying to make people understand how important non-profits are that people would stop asking why and just help non-profit programs and the professionals can start using all their time for what they are really trying to accomplish.
-sharae jennings

Very true. I think this "Bigger, Better, Faster, Cheaper" goes beyond university research studies as well. It just seems to be the type of society we live in today. People expect results fast and accurate for next to nothing, but when it comes to research (good, quality research) that is simply impossible. I think good results that take time to aquire are more important than misleading results. Funding for these projects needs to grow because they are very important to the non-profit sector (which is still struggling to pronounce itself as a legit profession) and other organizations as well. -Ariana Ontko

You have some great points here. If more people just donated their time, we wouldn't have to spend so much time researching this topic. Or at least have enough resources to conduct proper research. It takes more time and effort to do something, "Bigger, Better, Faster, Cheaper!" and get it all wrong, than to just do it properly the first time.

This is a very familliar problem these days. It seems that mankind has been overwhelmed with greed to the point where bigger, better, faster, and cheaper and now mandatory to our economy. Perhaps this is why companies rely planned and perceived obsolescence.
Shasta Neal

Great point. Our society is obsessed and getting swallowed up by the notion of "Bigger, Better, Faster, Stronger". It's simple, people want MORE for LESS, and that is only going to ruin our society in the end. If funders allowed for more time for research and were not so impatient, information would be more precise leading to BETTER results. You can not have faster and better in the same bundle. We need to find a happy medium and balance between these qualities. Priorities need to be straightened out in order to advance.

-Kendall Klingaman

This brings up wonderful points. In the society we live in today we want everything to be instantaneous or as close to that as we can get. But not only do we want and want it now we want to spend next to nothing to get it. How can we believe that good thought out research is going to come from anything fast and cheap? In my mind if you want something that will help you in the future you are inevitably going to have to invest some time and some money to get it right the first time rather than wasting valuable time, money and resources in the long run.

-Valerie Randolph

Blog Archive