Skip to content
olizillaPublic

About

Problems with the Federated Data Platform

Resources

Stars

2 stars

Watchers

0 watching

Forks

Latest commit

 

History

2 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 

Repository files navigation

Problems with the Federated Data Platform

A place to collect the problems with the FDP. Open issues to flag others.

Primer: How the FDP works

Every hospital installs Palantir’s data collection agent. They give it credentials to access the many different databases for the existing hospital IT systems.

The agent copies the data up to a database in Palantir Foundry, a cloud service.

As those copies come from systems built by many different companies, the data is in a wild array of different shapes. The hospital uses Palantir Foundry tools to write code to transform those copies into a common shape: the “NHS Canonical Data Model” (CDM).

Each hospital then uses Palantir Foundry tools to create dashboards to help them run the hospital. Those dashboards pull data from the transformed copy of the data in the Palantir Foundry database, rather than directly from the source systems in the hospital. Those dashboards can be shared with other hospitals’ Foundry accounts, once they also do the work to install the Palantir data collection agent, grant it permission to copy data from their systems, and write the code to transform their data into the Canonical Data Model.

Hospitals can also build 'integrated apps' inside Palantir Foundry. Like the dashboards, these use the transformed data rather than the source data. Apps can be shared with other Foundry accounts. Apps may create new clinical workflows like the Cancer 360 app.

Foundry provides an API so that systems built by other companies could build apps on top of NHS data in Foundry. There are no examples of this yet.

NHS England has a Foundry account. Some of the data from each participating hospital’s Foundry account is copied over to NHS England’s Foundry account so they can see what is going on in hospitals across England.

The diagram below shows the dependencies that the FDP architecture creates. Grey boxes represent digital infrastructure that would have to be replaced or rebuilt in order to switch provider.

Flow diagram of data and dependency between NHS trusts and Palantir Foundry

Figure 1: The dependency architecture of the FDP. Derived from Palantir Foundry developer documentation.

The Problems

Vendor lock-in: The data collection agent does not belong to the NHS

  • The Palantir data collection agent will only send data to Palantir Foundry.
  • The agent is granted access to many other NHS systems. For Foundry apps to be useful, this must include permission to read and write back to those systems. That hands a lot of power and trust to Palantir.
  • It has taken over 2 years to get 60% of hospitals to install and configure the agent. That work has to be dismantled and redone if we want to switch providers. This is a sunk cost that weakens our bargaining position when renegotiating the contract.

If we owned the data collection agent, we could send the data to multiple data processors to compare their services, and to other destinations like the upcoming Health Data Research (HDRS) project. Right now the only way we can use that data is after it is stored in Foundry.

Vendor lock-in: The data transforms are all in Foundry

  • The data transforms represent useful, bespoke work, done by each hospital to translate their local data into a shared national format. The code for those transforms is held in Foundry, and includes Foundry specific annotations to load the source data.
  • We could demand that all hospitals store the transforms they write in a national repository, but this creates additional work, and those transforms all need editing to work outside of Foundry. This increases the switching cost.

Vendor lock-in: The app platform does not belong to the NHS

  • Integrated apps built inside Foundry increase lock-in. We can’t take them with us if we switch provider.
  • An external app ecosystem built on an API provided by Foundry will radically increase our lock-in. Moving that API to a new platform provider may be possible in theory, but would not be seamless without first tackling the hardcoded reference to Palantir in the API, and then solving all the other lock-in challenges first. Moving an ecosystem is hard, it represents a huge switching cost.
  • The only public expression of this API is hardcoded to use Foundry; see: nhsengland/fdp-canonical-data-model#2.
  • The root domain for that API is https://ptukhealth.palantirfoundry.co.uk.
  • All the API paths have the same prefix /api/v2/ontologies/palantirukhealth-ontology that includes a trademarked name. This makes it unlikely that any other vendor would implement this API as is.
  • The stated commitment to monthly public API updates on GitHub stalled in Feb 2026. There have only been 3 rounds of updates since Oct 2024. See: https://github.com/nhsengland/fdp-canonical-data-model/commits/main/.

A common data layer is needed. An app platform that splits the services from the data, so we can switch between competing implementations would be a huge boost to the ecosystem and the procurement process. But the NHS needs to own the foundations of that ecosystem.

Vendor lock-in: Foundry becomes essential for running our hospitals

  • With the clean version of the data only existing in Foundry, along with all operational dashboards, the day to day running of each hospital will depend on everyone logging in to Foundry. This will be felt as an improvement for the staff as some of the existing systems are painfully slow and unintuitive to use. The impact here is each hospital's local process will be replaced with a Foundry-based workflow.
  • This complete operational dependency creates a huge switching cost.

Spin: The FDP is not “Federated”

  • A federated architecture is desirable for this problem, and the FDP's name implies one has been adopted.
  • In the current setup, each hospital simply copies its data to its own partitioned user account on a centralized, single-vendor cloud service.
  • This fails the technical definition of a federated system, though proponents argue it is "federated" at the data governance level because each hospital is currently the data controller for its own account.
  • The upcoming NHS Modernisation Bill will invalidate that argument too. The UK Government is set to become the central data controller for both clinical data (via the Single Patient Record mandate) and operational data (via the dissolution of NHS England).

Spin: The FDP stores health data, not just operational data

  • There is a lot of dissonance in the communication around the scope of the FDP. Initially it was going to be for operational data only, just for running hospitals. This reduced the amount of push back it received. People care more about their health records, than how we report hospital admin.
  • Apps like Cancer360 are clearly using health data. The scope of the FDP has increased to include clinical data.

Spin: Efficiency gains are driven by NHS effort, not Foundry

  • Any significant investment in improving a workflow would yield benefits. An example hospital saw a 10% efficiency improvement when it switched from using an excel spreadsheet to using a Foundry app for managing its operating theatre.
  • Benefits are incorrectly attributed to the FDP rather than the effort made by each hospital to improve their process, and then used to justify the national tech choice. The stats given do not account for any other changes made as part of that effort.

Apps that encode workflows can reduce errors that are common with processes based on spreadsheets. Those apps could live anywhere. The real gains are the attention each site gives to how it works.

Spin: The contract does not protect against lock-in

  • We are on the meter. The quoted £330mn figure is just a usage estimate. The more we use Foundry the more we pay.
  • Being locked in to a single vendor has similar consequences to a monopoly market; the price is likely to go up without any improvement to the service. That price only needs to be slightly lower than the total cost of walking away. The multiple points of vendor lock-in make the switching cost for the NHS much higher.
  • The NHSE FDP FAQ frames the fixed 7 year contract term with Palantir as a key protection from lock-in. It seems more likely that this is the earliest point at which they will experience rent-seeking and the usual shifts in power that occur when you lack a credible exit at the negotiating table.

Governance: CDM improvement process is unclear

  • The CDM defines the shape of the public API that external apps could build on. It is vital that we have clear and open governance for the CDM to encourage a healthy, vendor-neutral ecosystem.
  • A former NHSE staff member suggests the CDM is intended to become "the foundation for the Single Patient Record. EPRs should align to it. All applications live in the NHS should map to it."
  • It is a centrally designed “one schema to rule them all”. We have seen this pattern fail previously (NPfIT).
  • It’s not unreasonable for the centre to define a common schema for operational data (how full is this hospital, etc). NHSE should not and cannot hope to tackle the full complexity of solving the diversity of clinical data needs. Better, open tools exist for that.
  • Public updates to the CDM spec on NHS Futures stalled in September 2025.
  • Schema design appears to be happening in an Excel spreadsheet by a single core team rather than in an open forum and a fit for purpose tool and governance process.

What we actually need

FDP does things the NHS needs, but in a way that maximally locks us in to one vendor, creating a monopoly over our data.

  • Open Protocols not proprietary platforms.
  • Unbundle the key steps (collect, transform, visualise, workflows).
  • Open process and forum for CDM development.
  • Nationally owned data collection agent.
    • Pipe to any ETL / data insights platform.
    • Pipe to medical research - link to HDRS stream.
  • Extremely clear boundary between operational and clinical data.
  • openEHR CDR per hospital for portable health records.
  • FHIR for linking systems at the edge.
  • OMOP for long range analytics.

Note

There are many wise people who love the NHS and have good ideas for what better looks like here. Let's flesh this proposal out. Issues and PRs welcome.

Sources and Resources

In most cases, you will want to create an integrated application. An integrated Slate application can make use of the Foundry ecosystem to create widgets, use functions, configure Events and Actions, and more. Integrated applications are published to Foundry users within your Organization and can be viewed or edited based on user permissions. - https://www.palantir.com/docs/foundry/slate/applications-types/#integrated-applications

About

Problems with the Federated Data Platform

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors