On the use of Artificial Intelligence

While gaining some benefits, people and organisations all over the world are quickly leaning into the use of AI and often losing their balance. Many are unaware of the tradeoffs they are making. The lure of the hype can lead to poor decision making, and we think it’s important to think ahead about the downstream effects of change.

This article aims to illustrate why we intentionally tread with care and caution when considering the use of AI, or any new technology for that matter.

We’d love to encourage debate and discussion, so please read ahead, however for those who are short on time, here are the key points:

• Privacy and security of your data is paramount. Using new tools and platforms that leverage AI can inadvertently lead to data leakage and potentially a loss of control. Be informed before you proceed, so you can maintain sovereignty and safety.
• While potentially boosting productivity, AI usage can produce unreliable and low quality results. Review by an expert is always a good idea.
• There are many ways to solve problems. Sometimes a simpler solution yields a better, more reliable and consistent outcome than introducing non-deterministic solutions like AI. The term “non-deterministic” is further explained below.
• Don’t avoid and don’t go all in. This is about optimisation and your decisions need to take many factors into account. The optimal choice today will need to change, so re-evaluation and adaptation is key.
• Our professional commitment and principles demand that we develop definitive knowledge in this new field, while charting a safe course for our customers and those whom they serve.

Privacy & Security

Organisations in any sector should, and in healthcare must, prioritise privacy, control and security of data. They should collect the bare minimum of data, and it should only be accessible by those with the appropriate role and authority. A good software product should prioritise these concerns on behalf of their customers from day one. The inappropriate use of AI within software infrastructure can inadvertently lead to data loss and leakage.

Without adequate controls, data which might seem to be private can leak via AI tools and agents which are ultimately connected to data centres and platforms in various countries. Adjacent to concerns over loss of data privacy and control, My Health Record data specifically must remain in Australia by law, and state laws such as NSW and Victorian health privacy laws make onshore hosting the practical default for health information. Your data should remain yours, without a nasty, belated discovery that your data now resides in offshore data centres controlled by third party organisations in other legal jurisdictions.

Reliability & Quality

In a simplified view, technology products are essentially containers for data with numerous mechanisms to present people with useful ways to achieve their objectives. Without data, a technology product is an empty vessel with no utility. Products can use AI with said data to great effect. We are exploring the use of “natural language” interaction in Carespective, as a part of human centric design, to run reports and gain insights that enable continuous improvement of care delivery. However, this needs to be done very carefully, so as to avoid another key risk: ambiguity and hallucination.

While improving rapidly, AI is famous for producing unreliable and incoherent information. For high quality and reliable interactions, AI needs to be trained and contained in a specific, structured context. The temptation to “just add AI” might lead to unreliable and low quality outcomes.

Consistency & Safety

Is AI the best choice to solve every problem? From a technical perspective, not necessarily. Ask an AI agent or tool the same question numerous times and you’re unlikely to get a consistent answer. Do this in parallel with a few of your colleagues and compare your experience. This is known as non-determinism, or in other words, given the same input, the software produces different outputs and traverses different pathways to reach a conclusion.

In many ways, AI can provide an excellent productivity boost, but what remains clear is that we still need to verify outputs and check their integrity. So if consistency and safety are a fundamental concern, it pays to stick to deterministic software which behaves predictably by design.

Sovereignty, Continuity & Control

We cannot control everything in our lives. That’s an unrealistic expectation. However in the context of building resilient and long lasting organisations, we can think deeply and carefully about the concentration of risk we create when we depend too heavily on one entity. Imagine spending a day at work without an internet search engine, or access to your favourite AI tool. Or the day when your AI tool shifts to a new model, changing it’s behaviour significantly or subtly. This was aptly demonstrated when Anthropic were restricted in supplying their most powerful consumer model to their non-US national customers [reference].

Another major risk is the imminent and predictable rise in costs of using AI. It costs a lot to produce, and is heavily under priced today. A sharp correction on this front will have significant impact for all users. The most resilient systems are designed to maintain continuity under duress.

Things in the realm of AI are moving rapidly (massive understatement right?!). Sometimes it pays to calmly watch others ride a rollercoaster from a stable stance.

How we use AI today

Organisations can rely on Carespective to achieve healthcare workforce compliance, while focusing on those who are in their care and those who are delivering this care.

We operate with safety, quality and high performance in mind, all the time.

All of these attributes are synonymous with compliance. In addition, systems and technology in healthcare need to be highly reliable, consistent, efficient and effective.

Gaby, John and I use Claude Design to create prototypes which we test in conversation with our customers and partners. John and I use Claude Code for our work to build Carespective. We have developed what we call the “Claude Kit”. This software development kit is an expression of our collective experience in building high quality technology. It enables us to develop product features with clarity of purpose, implementation, software design, security, performance and ultimately yields an outcome which is peer reviewed by the other person. With efficiency and efficacy in mind, our backlog of work is integrated with our use of Claude Code.

All the data which Claude can see is fictitious, “dummy” data that we use to simulate product usage and behaviour.

Our philosophy

Our professional commitment and principles demand that we develop definitive knowledge in this new field, while charting a safe course for our customers and those whom they serve.

We are comfortable with rapid rate of change and ambiguity; that’s what we’ve been practicing for decades. We are neither sceptics nor zealots. We neither love nor hate AI. We are motivated to build the most impactful and reliable product possible, and are intent on calmly doing so with the right underlying technologies and techniques.

Two examples to illustrate our approach

Amongst numerous other ideas, and as mentioned above, we’re working on the concept of a “natural language” interaction with Carespective. For instance, you can ask a question in every day language, and receive a conversational response supported with a structured, queryable set of data. Non technical staff can interact at ease to see patterns that need to be considered and addressed. If executed carefully, considering all the concerns above, this is a great candidate for AI.

Underpinning our effort to be the compliance and quality partner that never sleeps, Carespective provides workflows that can be instigated by specific triggers. If some mandatory training is expiring, or a background check needs to be refreshed, Carespective can illustrate the gaps and pending your review, then take action on what needs to be done. This in contrast, is a great candidate for a simple, deterministic approach. No AI necessary.

We objectively choose an approach that best achieves the outcome.

In conclusion

Thanks for reading, we hope you found this to be useful and if you got this far, please know that this article was hand crafted and peer reviewed without AI support, with care and consideration for you.

Please get in touch with us if you wish to discuss this and see what Carespective can do for you.