A healthcare IT foundation built on gooey clay

Today, there was a report from the Solicitor General of Singapore about the data breach of the SingHealth systems that happened in July.

These systems have been in place for many years. They are almost exclusively running Microsoft Windows along with a mix of other proprietary software including Citrix and Allscript.  The article referred to above failed to highlight that the compromised “end-user workstation” was a Windows machine. That is the very crucial information that always gets left out in all of these reports of breaches.

I have had the privilege of being part of an IT advisory committee for a local hospital since about 2004 (that committee has disbanded a couple of years ago, btw).

Every year, budgetary proposals for updates, new versions etc., of the software that the advisory committee gets for consideration and possible approval. Almost always, I would be the exception in the committee in questioning the continued use of expensive proprietary software for these healthcare systems (a contributory factor to increasing health care costs). But because I am the lone contrarian voice, inevitably, the vote will be made to approve and hence continue, IMHO, the wasteful path of spending enormous amounts of monies in these proprietary systems.

I did try, many times, to propose using open source solutions like, for example, virtualization from KVM. This is already built in into the Linux kernel that you can get full commercial support from Red Hat (disclosure: I work for Red Hat). You pay a subscription and we make sure that the systems are running securely (via SELinux for a start) and that enterprise can focus on their core business. But no, they continued with VMware.

I did propose open source solutions like OpenEMR and many other very viable solutions for the National Electronic Medical Records system – but none of them were accepted. (It has been brought to my attention that there are plans to mandate private sector healthcare providers to use the NEHR. There is considerable opposition to it both from the hassle (from their point of view) and added costs since the solution is proprietary and expensive).

There were some glimpses of hope in the early years of being on the committee, but it was quickly snuffed out because the “powers that be” did not think open source solutions would be “good enough”. And open source solutions are still not accepted as part of the healthcare IT architecture.

Why would that be the case?

Part of the reason is because decision makers (then and now) only have experience in dealing with proprietary vendor solutions. Some of it might be the only ones available and the open source world has not created equivalent or better offerings. But where there are possibly good enough or even superior open source offerings, they would never be considered – “Rather go with the devil I know, than the devil I don’t know. After all, this is only a job. When I leave, it is someone else’s problem.” (Yeah, I am paraphrasing many conversations and not only from the healthcare sector).

I recall a project that I was involved with – before being a Red Hatter – to create a solution to create a “computer on wheels” solution to help with blood collection. As part of that solution, there was a need to check the particulars of the patient who the nurse was taking samples from. That patient info was stored on some admission system that did not provide a means for remote, API-based query. The vendor of that system wanted tens of thousands of dollars to just allow the query to happen. Daylight robbery. I worked around it – did screen scrapping to extract the relevant information.

Healthcare IT providers look at healthcare systems as a cashcow and want to milk it to the fullest extent possible (the end consumer bears the cost in the end).

Add that to the dearth of technical IT skills supporting the healthcare providers, you quickly fall into that vendor lock-in scenario where the healthcare systems are at the total mercy of the proprietary vendors.

Singapore is not unique at all. This is a global problem.

Singapore, however, has the potential to break out of this dismal state if only there is both technical, management and political leadership in the healthcare system. The type of leadership that would want to actively pursue by all means possible to make healthcare IT as low cost and yet supportable, reliable and more importantly, be able to create a domestic ecosystem to support (not via Government-linked companies).

I did propose many times to create skunkworks projects and/or run hackathons to create solutions using open source tools to seed the next generation of local solutions providers. As I write this, it has not happened.

To compound the lack of thought leadership, the push in the 2000s to “outsource IT” meant that what remaining technically skilled people there were, got shortchanged as the work went to these contract providers (some of these skilled people were transferred to these outsourcee firms but left shortly after, because it was just BS).

This also meant that over time, the various entities who outsourced IT were just relationship managers with the outsourcee companies.It is not in the interest of the outsourcee companies to propose solutions that could lower the cost overall as it could affect the outsourcee’s revenue model. So, you have a catch-22: no in-house IT/architecture skills and no interest at all on the part of the outsourcees to propose a lower cost and perhaps better solutions.

I would be happy, if asked, to put together a set of solutions that will steadily address all of the healthcare IT requirements/solutions. I want this to then trigger the creation of a local ecosystem of companies that can drive these solutions not only for Singapore’s own consumption as well as to export it globally.

We have the smarts to do this. The technical community of open source developers are, I am very confident, able to rise to the challenge. We need political thought leadership to make this so.

Give me the new hospital in Woodlands to make the solutions work. I want to be able to do as much of it using commercially supported open source products (see this for a discussion of open source projects vs open source products), and build a whole suite of supportable open source solutions that are open to the whole world to innovate upon. It would be wonderful to see https://health.gov.sg/opensource/ (no it does not exist yet).

There are plenty of ancient, leaky, and crufty systems in the current healthcare IT systems locally. We need to make a clean break from it for the future.

The Smart Nation begs for it. 

Dr Vivian Balakrishnan said the following at GeekCamp SG 2015 (video):

I believe in a #SmartNation with an open source society and immense opportunities; where everyone can share, participate and co-create solutions to enrich and improve quality of life for ourselves and our fellow Singaporeans.

And for completeness, the actual post is here (it is a public page; i.e., no account needed):

I am ready.  Please join me.


  1. Your logic is: 90% of NTUC Comfort taxis are Hyundais, Accidents so far involve only Hyundais. Therefore anyone who drives a Hyundai will get into an accident.
    Your post contains a jumble of the badness of the limitations of proprietary software, and that open-source is automatically more secure than proprietary software (a point you have pushed more strongly in the past) which is blatantly false.
    The root cause of the issue is that about 50% of all IT managers in Singapore are not qualified, as in like a person who has never spent more than 5 hours at the wheels of a motor vehicle being made a bus driver. The percentage is higher at the higher level, like 90% at the CIO/CTO level. (These are my estmates, you don’t have to believe them.) However, large systems are complex and the problem is not always technical. There are pulls in many directions by interested parties facing life or death. Red Hat would want the customer to go one way. IBM will want the company to go another way. Everyone has to look after his own survival.
    Another major cause is the political structure of companies. When the CEO of DBS sat in a press conference to explain how an IBM technician used the wrong disk cable and caused a nationwide ATM outage, the underlying subtext was “hey, we have bought the services of what the best money can buy, you can’t blame us.”, never mind that the statement can never be proven. Imagine if DBS had done the maintenance themselves and saved shareholders $300m every year doing so, that exact same technical fault would have caused the CEO to lose his job. Shareholders will ask, “Why are you in the computer maintenance business?”

    • Thanks for your comment. There are lots of things we both agree on. Open source is *not* the panacea. It is an important big first step. Ask NYSE. Ask CERN, Ask almost any of the fortune 1000 companies. Whether they are Red Hat customers or not, they have benefited from the security and high degree of flexibility that open source products offers them that they could *never* have gotten from proprietary products.

      We are on the same page as far as the IT managers are concerned. For years, I have always said that there are perhaps a small handful of CIOs worth their weight in gold. The bulk of them are mere pushing some vendors agenda. Even in the IT healthcare sector. There are some very strong, technically capable and worthy individuals who do a brilliant job. But they are largely thwarted by “it’s only a job” types.

    • Thanks Ivan for the comment. Security is indeed the cornerstone of open source software solutions regardless what that CSOline article says. If this was an issue, Singapore own Ministry of Defence would not use it. They are huge consumers/contributors to open source solutions. If it was a problem, the New York Stock Exchange would not use it (NYSE defaults to open source solutions from Red Hat). What is needed is supported open source solutions. That is what businesses like Red Hat does. Red Hat provides accountability and peace of mind to the use and deployment of open source products from mundane use to mission critical systems.

  2. [it is better to post comments here and not in a walled garden, so I am cutting and pasting it here – I can’t seem to get a public URL. So, to see the original post, you have to have an account in the walled garden and the link is this: https://m.facebook.com/story.php?story_fbid=10160850858295627&id=812715626&comment_id=10160875937695627&_rdr%5D%5D

    “From Jesse Sng:
    Harish Pillay, I don’t think the issue of organizations overlooking Open Source solutions is strictly a tech issue. We may need to look deeper into the organizational culture.

    I highly doubt that the typical corporate IT manager would choose Open Source because they are unable to pad their resumes with important, branded names. Coders, hackers and Start-up types will think differently, but not the corporate types and they are not going to stake their careers on it.

    When one chooses to go with a big name, even if the project fails, top management is forgiving and will find a way for you to justify more expenditure to make this work. If you go with Open Source, they will be waiting for you to tender your resignation letter if there’s so much as a hiccup.

    My contention is that inherent risks aside, the choices are decided based on whatever is less risky as for as the IT manager’s career is concerned.”

    I believe this comment is really about “it is a job, just go with what is easy, and a few years later, move on. it will be someone else’s problem and you can pad your resume with ‘brand names'”. I’ve nothing to add here except to point out the catastrophic failure of an Avanade(Accenture)+Microsoft project that switched the London Stock Exchange to a .net environment in 2008. The CEO of LSE was fired eventually for the switch and failure.

    1. https://www.zdnet.com/article/about-that-london-stock-exchange-it-failure/
    2. https://www.computerworld.com/article/2480424/data-center/london-stock-exchange-suffers–net-crash.html
    3. https://www.reuters.com/article/us-lse/london-stock-exchange-crippled-by-system-outage-idUSL01084620080908
    4. https://www.zdnet.com/article/the-london-stock-exchange-moves-to-novell-linux/

    So, should I hire the person who made the choice of going with the “brand name” because “no one gets fired for going with a brandname”?

Leave a Reply