Measure Twice, Tweet Once

February 1, 2011

I confess I’ve been the lagging indicator on this blog for any number of reasons. My goal was to post a multipart examination of how to measure outcomes in Social Media (SM). In my genius moment, I wanted to interview a number of experts in outcomes measurement, market research, and users who would share their knowledge and expertise. Two trusted colleagues took the time to respond to my questions and thoughts on outcomes and SM. This is a starting point for this discussion.

I do not view SM in and of itself as a major driver of change in healthcare (HC). It is a tactic, a tool, and one element of any number of strategies (e.g. marketing, sales, awareness, listening, etc.). In my estimation, SM is a powerful tactic within a learning strategy. Measuring outcomes in SM will aid in understanding what works, what doesn’t, and why. With that knowledge we will be able to create strategies that include targeted SM in HC to drive change through learning and application of knowledge. And further analysis of hard clinical outcomes can follow.

The best definition I could find on outcomes is from a statistical glossary:
“An outcome is the result of an experiment or other situation involving uncertainty.

We are uncertain what works in HCSM, the degree to which outcomes are a function of other variables, and why.

The two colleagues I asked to help me with outcomes are:

Derek Dietze is the founder and owner of Improve CME ( During the past 10 years Derek has focused on measuring outcomes in CME and finding unique ways to measure learning.

Scott Fishman is a well-regarded market researcher and principal of Ethos Lifescience Advisors (, which supports market development in the life sciences, and provides guidance on the clinical and commercial value of new technologies.

Derek and Scott each looked at the topic in a slightly different fashion but overlapped in some critical areas. I will share their comments and thoughts before my own.

Derek offered the following regarding HCSM:

“For many years, we in the CME community have agonized about how to measure the effects of learning interventions on performance change in healthcare professionals and patient outcomes in practical, objective ways within reasonable budget limitations.

While methods and means to do this have come some distance, I’ve discovered that you really have to walk before you run with respect to measurement, both in an individual organizational sense, and in a community of practice sense. I’ve found it most productive to build on a strong foundation of simple measurement, easily understood by the consumers of the results, than advance to higher forms of measurement based on experiential learning. So initially, we focused on measuring changes in knowledge, confidence, and intention to change behavior (called “competence” by many in the CME community). These are still our staple forms of measurement, but now we have advanced to self-reported performance change and self-reported observations of patient outcomes based on CME activity participation. We are also piloting studies using national scope medical claims data to objectively measure performance change for national scope CME initiatives.

While ultimately the goal might be to measure how the use of SM results in better patient and community health, I would recommend starting at lower measurement levels, such as changes in knowledge, attitudes, confidence, and intention to change behavior among those interacting through SM. An article by Moore et al 2009 provides a good frame of reference for measuring changes in physicians and other healthcare professionals, and the Transtheoretical Model may prove to be of some help with respect to changes in patients.

Initially, I thought of SM as an “intervention” (like an educational intervention or activity in CME) when it’s focus is bringing about increases in knowledge, reshaped attitudes, and intention to change behavior. While I suppose it could be considered an intervention or “treatment” for the purposes of measurement, I realize it has many other functions that might be measured. SM seems to be a facilitator, and also has empathic and emotive characteristics, given its relative immediacy and potential frequency of use.”
A reasonable starting point for measuring the outcomes of social media (SM) in health care would be to clearly define:
• The different characteristics/uses of SM in healthcare
• What reasonably might be expected to change through its use in each of those circumstances
• In whom those changes are expected to occur, and
• What’s important about outcomes for the people/organizations invested in the results.

Scott Fishman responded to HCSM and some specific questions with this response:

I asked Scott:
Should our outcomes be qualitative or quantitative? Would you use combinations? Would we want to measure just clinical outcomes? His answer:

“Definitively both. Clinical outcomes measures are obvious and you’ll find them in any clinical paper or meta-analysis of a given condition. But there will also be references to “softer” metrics such as QOL scales, and I would suggest incorporating such measures. Of course, you will have to deal with the issue of generally accepted (e.g. “faces” pain scale) vs. novel metrics. The latter are also useful, but will need to be correlated with improvement in quantitative clinical and accepted QOL measures, which is another whole task in itself, and has the potential for either an “aha” moment or to entirely
undercut your hypotheses.”

Scott also addressed the health care provider (HCP). What would we measure there to determine outcomes in SM?

“Fewer phone calls from unsatisfied or frustrated patients. Shorter visits. Fewer changes in medication or dosages, less agita, fewer slaps from, and more incentives provided by insurers. Then, of course, there are all the “I’m a good doctor I have healthy patients” social acceptability measures.”

I asked him to address the question of what measures are most appropriate for patients?

Happiness. Does the patient feel better holistically? Do we see improvements on ADL scales? Of course we can measure condition-dependent clinical improvement (e.g. Hemoglobin A1c, BP, memory performance, and health). Are patients satisfied with their health care provider? Change in the number of days of lost work. Hospital admissions.
I might also include the quality and content of communications between HCP and patient as a potentially valuable measure for the effect of SM on outcomes.”

Both Scott and Derek have started the discussion that I would like to continue with you.

In healthcare SM’s most powerful application is as part of a learning strategy. HC is driven by knowledge and application of knowledge to improve an outcome. Knowledge is the locus for medicine. The physician is an intermediary in knowledge and treatment. Add to that these evidence-based conclusions about learning:

• Adults want to learn solutions to problems they already have (and, conversely, they seldom learn solutions to problems they do not have),
• Adults want to participate in their own learning, and
• Instruction for adults must respect the multiple demands in their lives.

Derek added the following comment:
“Mark, these measure a patient’s PERCEPTION of their own knowledge level and how it might have changed, which I admit may be valuable. However, when measuring physicians, I don’t trust them to self-evaluation whether they are knowledgeable. Instead I ask knowledge questions to test their knowledge. Anyone of use may be confidently misinformed or have incorrect perceptions regarding our illness/disease. A really simple way to validate correct knowledge (without making the patient feel like they are being tested), is to use “clinical assertion” agreement scale questions. Make a correct statement about a disease or condition, then ask for their level of agreement with the statement using a Likert scale or semantic differential.”

If we accept the fact HCSM is first and foremost a learning tool I believe Derek has presented a way to begin to understand HCSM strength and its potential. We can observe what happens to those patients engaged in SM with a HCP or others. Has the patient (caregiver)/learner: improved/increased knowledge, gained confidence, and intention to change behavior.
Patients get ill, get a diagnosis, and want to avoid an illness*. All of these are triggers to begin a process of learning. They come into the moment with a specific set of information. Patients will surf, call, read, tweet, FB and more. At some point they either stop adding to their personal compendium of knowledge because they have changed or are satisfied they have the knowledge they need. They have closed a personal knowledge gap. If we can measure changes in that gap comparing those who use social media against those who don’t in HC we can begin to understand SM as a learning tool and its value. We can compare it to non SM learning.

Areas of inquiry we may ask patients, caregiver, and HCP using SM in HC are:
• What is their current level of knowledge regarding the illness/dx/treatment you are seeking information on?
• Has their knowledge or understanding changed?
• Which formal and informal information of the following sources provided the greatest knowledge to them? (e.g. web sites, physicians, friends, others with similar conditions, networks on FB, tweeting, chat rooms)
• Based on what they have learned, how are they changing their behavior and/or approach to your care and management? What actually happens when we measure longitudinally from intent to behavior?

This is not an exhaustive analysis of HCSM and outcomes. It is a starting point to an open source discussion. What are your thoughts on how to measure outcomes in SM? Can we determine its relative value within HC? Are strict clinical outcomes the only measure we should make? This is one hypothesis on SMHC and outcomes. How can we dive into this area and improve it?

*I am fascinated by the idea that those patients who are heavy users of HCSM represent a small subset of all patients and are patients who are pre-disposed to learning etc.

%d bloggers like this: