Are you seriously still using email and excel? https://t.co/n1KUBKoRTm #HITsm
— Chris Richie (@ChrisMRichie) February 25, 2016
This tweet highlighted an interesting observation I had after experiencing so many healthcare analytics pitches going into and at HIMSS. I’ll set aside the email comment for now (email is still very powerful if done right) and instead focus on Excel. Here’s what I discovered about healthcare analytics:
Excel is a healthcare analytics company’s biggest competitor.
It’s crazy to think about, but it’s true. When a healthcare organization is evaluating healthcare analytics platform the “legacy system” that they’re usually trying to replace is Excel. I can’t tell you how many times I heard analytics vendors say that “Hospital A was doing all of this previously on a bunch of Excel spreadsheets.” If you work at a hospital, you know that you have your own garden of Excel spreadsheets that are used to run your healthcare organization as well.
When you think about the features of Excel, it’s no wonder why it’s so popular in healthcare and why it’s a challenging competitor for most healthcare organizations. First, it’s free. Ok, it’s not technically free, but every healthcare organization has to buy it for a lot of reasons so that cost is already in their standard budget. Second, every computer in the organization has a copy of Excel on it. Third, the majority of people in healthcare are familiar with how to use Excel. Since we love to talk about healthcare IT usability, Excel is extremely usable. Fourth, Excel is surprisingly powerful. I know many healthcare analytics organizations could argue its limitations, but Excel is more powerful than most people realize.
That’s not to say that Excel doesn’t have its weaknesses. I’m sure that most organizations have experienced time wasted trying to figure out which Excel file has the accurate data or is the most up to date. No doubt you’ve experienced the multiple copy problem where 2 people are editing the same file and now you have 2 versions of the same file that need to be merged. Document management software has helped with this situation in many regards as it locks the file when someone starts to edit it and things like that. However, it’s still often a problem.
Another problem with Excel as compared with a true analytics platform is when you want to go in and slice and dice the data. What’s possible with a true analytics platform is so much more powerful when you want to really dive in and chop up the data in unique ways.
While possible in Excel, most uses of Excel are backwards facing data analysis and tracking. You can do some near real-time data analysis in Excel, but newer analytics platforms do a much better job of real time analytics using the latest data.
Of course, the biggest problem long term with Excel is that it can’t scale. Once you reach a certain amount of data points or a certain amount of complexity in the data, Excel falls on its face. However, most healthcare organizations are still working on small data, so Excel’s worked fine.
I’m sure there are many more issues. Hopefully some analytics vendors will chime in with more examples in the comments or on their own blogs. However, it’s worth acknowledging that for many organizations it’s really hard for them to find a healthcare analytics solutions that’s so much better than Excel. Plus, many of these expensive analytics solutions fail when it comes to some of the things that makes Excel great (ie. Free, Usable, Ubiquitous).
John, you post raises good points to which I respond below. My response explains important yet generally unrecognized Excel capabilities and offers simple solutions to Excel’s apparent weaknesses.
Having been developing Excel-based applications for over three decades, including numerous healthcare programs, I’ve become astutely aware of its real potential and how to overcome the weakness you point out. The same goes for e-mail. The following addresses your key points and I appreciate this opportunity to discuss.
1. Free, Usable, Ubiquitous. Yes. There is a modest cost, however, for certain advanced versions of Excel.
2. Trouble maintaining accurate, up to date data.
One way to avoid this problem is to use a database to store shared data and have Excel as a front-end that inputs and outputs the shared data. We also use Excel in a node-to-node network architecture. In this architecture, data from a central store are sent by the database manager, or by an automated (macro-driven) Excel program connected to the database, in encrypted data files (xlsx or csv) to remote Excel users for local storage, processing, and rendering. This extract and transport process can be done upon request or on a timed schedule. Likewise, data files can be sent from the remote user nodes to the central node for manual or automated input into the database. This node-to-node network architecture can be done using e-mail, such as DIRECT transport (including through HISPs or through the Simple SMTP method).
Note that the distribution of data between nodes using Excel and e-mail provides vast scalability at very low cost and without infrastructure build-out (no additional hardware or software required).
3. Excel is imitated as a true analytics platform for slicing and dicing data.
Excel’s pivot tables have powerful slice & dice capabilities, as well as drill down, sort, filter, etc. Excel also has considerable statistical packages built-in (including its Analysis Tool Pack), that can do what-if scenarios and have a host of third-party statistical add-ins.
In addition, Excel has the free Power Pivot add-in that “enables you to import millions of rows of data from multiple data sources into a single Excel workbook, create relationships between heterogeneous data, create calculated columns and measures using formulas, build PivotTables and PivotCharts, and then further analyze the data so that you can make timely business decisions without requiring IT assistance” (reference).
Furthermore, the free Power Query add-in “has an intuitive and consistent experience for discovering, combining, and refining data across a wide variety of sources including relational, structured and semi-structured, OData, Web, Hadoop, Azure Marketplace, and more. Power Query also provides you with the ability to search for public data from sources such as Wikipedia” (reference).
Finally, we’ve also created Excel templates that consume and build complex XML files, create and implement sophisticate electronic forms, automate intricate workflows, interface Outlook, and interoperate with other software tools.
In conclusion, Excel out-of-the-box is a powerful tool that is free, easy to use and almost everywhere. Its add-ins enhance its native functions with additional capabilities that greatly extend its power and usefulness, while eliminating many of its shortcomings. And, as part of the MS Office suite, it integrates seamlessly with Outlook to provide automated e-mail functions (as well as with other Office tools).
Stephen,
Thanks for the added analysis. That was similar to the response I got from an analytics vendor. “Export the data from my platform and view it in Excel.” Although, you also added some more insights. I agree that Excel is much more powerful than most people realize.
Another problem with a true analytics platform as compared with Excel is that the user needs to be a true analyst, or data scientist; and there are so few of them compared to Excel users. But, if you are handling large volumes or performing difficult operations you need to hire the professional and give them the engines for the job, whether that refers to goods and truck drivers, or data and analysts.
Good points, John and Patrick.
No one tool can do everything for everyone. In some cases, a professional analytics platform used by a data scientist is needed. In those situations, Excel could be used to feed the analytics platform and the results of the analyses can be returned to Excel for visualization and presented along with data from other sources.
But one of the most important distinctions often overlooked is that Excel not only has a built-in computation engine, but also has a macro language (VBA) for automation, electronic forms (ActiveX) for data inputs and views, charting, query wizards, and worksheets for rendering the data and storing them in tables, as well as many add-ins. All of these features and functions are fully integrated so, e.g., macros can automatically grab data from local and remote stores, transform the data into any required formats, perform calculations using both cells formulas and macro functions, manipulate alphanumeric strings, cleanse and validate data, as well as transport data and the spreadsheet template models that consume those data among networks of collaborators.
Taking advantage of all these capabilities for particular use cases requires technical knowledge, of course. There are many good online forums, educational materials, consultants and Excel-based products available worldwide that can provide needed solutions.
Stephen,
Indeed. I knew a couple people who built entire EHR software on Excel. Crazy to consider, but the options you mentioned are why it was possible. Although, meaningful use kind of wiped them out. Still illustrates how powerful Excel can be in the right hands.
John,
For the past five years, I’ve been involved in several ONC technical workgroups. SDC about creating and electronic forms, DAF is about querying remote data stores, 360X is about closed-loop referrals, and DIRECT is about using e-mail transport. We’ve been making the case that Excel is a low-cost yet powerful software development platform whose programs are able to satisfy the ONC’s goals and comply with emerging standards.
This meant enabling Excel to consume and create complex XML files (including C-CDA templates and forms designs), handle complex clinical workflows, use FHIR resource, and transport data securely via SMTP and HTTP.
There has been considerable pushback against our efforts and we have not received any gov’t funding. Nevertheless, we have received strong support from others.
A key requirement has been for us to support MU, which meant we had to develop processes in Excel that are not typically recognized as spreadsheet functions. Having done so, we are now in a position to demonstrate the potential Excel and e-mail in support of ONC goals through three upcoming pilots.
Anyone interested in discussing this can contact me at sbeller@nhds.com.
One more thing, I posted the question “Is Excel a true analytics program” to members of an Excel developers’ forum to which I belong. One person posted: “No matter how much Excel is put down and trampled upon it is still the most prolific. Analytics is a process. A process of dissecting data to reveal trends, relationships, forecasts etc. If the knowledge set is available anything that can be done with another tool can be done in Excel.” To which another replied: “And I was going to write ‘unless of course, you are NASA’ – til I stumbed on this by Googling. Not sure Google helping me make any cases nowadays. Does NASA use Excel – You bet they do — https://goo.gl/5TyMpp”
I responded: “This thread makes the case that Excel analytics, coupled with its strong data integration and presentation capabilities, as well as its transport capabilities (when combined with Outlook or RESTful services), make it a very powerful development platform for robust applications that help solve complex problems, build knowledge, and promote collaboration.
Others pointed out that the ease with which Excel can be programmed could have a downside, e.g.: “Excel makes is easy to write programs (but quite a few users don’t think they are writing programs when setting up spreadsheets). And since it easy to write programs in Excel, it is also easy to write bad programs. It is harder to write programs with other tools, but programmers can write bad code with any programming tool.”
I responded: “Like just about every software program, Excel’s capabilities depend on the technical requirements, as well as the level of skill and knowledge of the user. I don’t think anyone can, however, refute that Excel as the POTENTIAL to be incredibly competent and useful, even in the most challenging situations, thanks to its flexibility + analytic power + presentation abilities + decades of evolution + ubiquity + low cost + interoperability with other tools.”
Thanks for your detailed response Stephen. That’s interesting that you’re working on something with ONC and they’re giving you the cold shoulder. Not surprising though.
As far as the discussion of Excel, I agree that it’s extremely powerful. More powerful than most realize. However, I also think there is something to say about using the right tool for the job. Just because something could be done with Excel doesn’t mean it should. I could build a house and cut all the wood with a handsaw, but that doesn’t mean I should do it when a power saw would be much faster. I think we have to be thoughtful about when Excel is the best tool for the job and realize that sometimes other options could be better, easier, faster, etc.
I agree with your power saw analogy, which reflect Mark Twain’s quote: “To a man with a hammer, everything looks like a nail.” To avoid this narrow view, I’ve focused on the point that people are unaware of Excel’s extensive capabilities and how Excel can be used in ways never before been imagined to address many of ONC’s daunting challenges. I’m not, however, rejecting any analytic tools that are the best/better fit for a particular use case, be it SSAS, MATLAB, SAS, or Excel. But I am rejecting the watered-down narrow-minded view of Excel espoused by the unenlightened.
Furthermore, like you, I’m not surprised by ONC’s cold shoulder. After all, I’m describing a disruptive innovation and it’s well-known that such innovations are not immediately accepted by the mainstream. Instead, it starts with early adaptors, such as technology enthusiasts and visionaries.
I’m glad that ONC recently appears to be more willing to examine such innovations through its current pilot projects and I applaud its apparent readiness to establish health IT standards based on what the pilots demonstrate to work (rather than being overly prescriptive and inflexible).
And please note that the disruptive innovation I’ve been promoting not only focuses on Excel, but also e-mail transport. As such, I was blown away, John, by your initial post since you mentioned both Excel and e-mail. You are a technical enthusiast with a visionary spirit and an open, objective, critical thinker.
I won’t go into detail about e-mail, but would like to make a couple of brief points. There’s been an effort to constrain the use of DIRECT e-mail in ONC’s initiatives (e.g., limited to closed-loop referrals). What detractors don’t realize is that Excel’s application development platform + e-mail based transport architecture provide a cost-effective way to support “distributed analytics” to grow and share knowledge. That is, these tools enable collaborative networks of researchers and clinicians to exchange analytic models and the data they analyze without the constraints of organizational and regional boundaries.
I assume you know I set aside email because I like to keep my posts short and focused and not because I underestimate emails power. Just a topic for another post.
In closing my comments on Excel, let me just list a sampling of its capabilities out-of-the-box that people may not realize:
1. Its spreadsheet data grids can: (a) store data in tables, lists, and linked-lists; (b) do numeric calculations (including statistics), logic functions, and string manipulations; (c) reference/retrieve data from other Excel spreadsheets and from remote data sources; (d) be populated through database queries, built-in electronic (Active-X) forms, macros procedures, and manual input; (e) organize the data into computational models; (f) sort and filter data; (g) validate data; (h) format data and display the data in charts and dashboards; (i) provide what-if analysis, and (j) provide a convenient way to share data and information.
2. Its query functions work on many different types of data stores and simplified with wizards.
3. Its chart sheets and pivot tables provide dynamic data views and interactive slice, dice, drill-down scenario.
4. Its VBA macros can: (a) perform iterative and looping procedures numbers and strings (including building and consuming XML); (b) automate procedures that interact with its spreadsheet grids, tables, and charts, as well as enabling automated queries; (c) interact with third-party products; (c) launch hyperlinks to websites and local files; (d) do many directory file functions; (e) provide transport capabilities; (f) improve the user experience by accommodating sophisticated workflow requirements; (g) interact seamlessly with other MS Office tools; and more.
And I didn’t even list the wealth of analytical and presentation add-ins.
Bottom line: I know of no other analytic platform with such a rich set of capabilities and inherent flexibility. Excel has great untapped potential in healthcare and beyond. This potential can be realized today to solve many of ONC’s daunting challenges. Doing so would be spurred by collaboration between developers and users of all kinds.
The goal of my involvement with the ONC workgroups is to use my 35 years’ experience with spreadsheet technology to demonstrate what is possible and to share my knowledge with others in a collaborative environment. Although it’s been very frustrating, there appears to a light at the tunnels end … We shall see!