cGxP Directory · Data Platform
How a record gets into the directory
Two real companies, ten real feeds, and every table the data touches on its way from an HTTP response to a profile page. Press Generate data and the pipeline narrates itself — including the two records it refuses to trust.
How to read the values on this page
Live means the value was pulled from the public API during preparation of this page and can be re-fetched with the curl shown beside it.
Curated means a hand-entered demo value standing in for what a curator or a commercial feed would supply. Nothing on this page is invented and presented as regulatory fact.
The sample
Two companies, chosen to break in different places
One small-molecule drug and one biologic. They exercise opposite halves of the schema: an NDA routes through Drugs@FDA and the Orange Book with patents and exclusivity; a BLA routes through the Purple Book and never appears in Drugs@FDA at all. Their names also fail to match in three different ways, which is the more interesting half of the demo.
Pfizer Inc.
companies.company_slug = pfizer-inc
- Product
- XELJANZ (tofacitinib) Live
- Route
- NDA → Orange Book
- Application
- NDA203214 Live
- NDC
- 0069-1002 Live
- UNII
- O1FF4DIV0D Live
- Labeler
- 0069
- Trial
- NCT02092467 Live
- Paper
- PMID 35081280 Live
- Exchange
- NYSE : PFE Curated
Moderna, Inc.
companies.company_slug = moderna-inc
- Product
- mRESVIA (mRNA-1345) Live
- Route
- BLA → Purple Book
- Application
- BLA125796 Live
- NDC
- 80777-345 Live
- Substance
- RNA-100-AR02 Live
ingredient name — the feed carries no UNII - Labeler
- 80777
- Trial
- NCT05127434 Live
- Paper
- PMID 38091530 Live
- Exchange
- NASDAQ : MRNA Curated
Neither company's name arrives from a feed the way it is stored. openFDA calls Pfizer Pfizer Laboratories Div Pfizer Inc; Drugs@FDA lists the NDA203214 sponsor as PF PRISM CV; ClinicalTrials.gov says just Pfizer; the shortage feed says Pfizer Inc.. Moderna arrives as Moderna US, INC. and ModernaTX, Inc.. Five spellings, one company — and one of them scores so low that no fuzzy matcher will ever find it.
Step one
Where the data comes from
Every feed is a row in data_sources. The row is not a label — trust_rank is the number the survivorship function sorts on when two sources disagree, and is_licensed is what keeps MedDRA terms out of a public export.
| Source | Endpoint | Tier | Trust | Access | Cadence | Feeds tables |
|---|
Trust rank is deliberately an integer rather than an ordering derived from tier: two feeds in the same tier still need a tiebreak, and Drugs@FDA must beat an EMA scrape.
The demo
Run the pipeline
Three presses, in order. The first lands the data and holds what it cannot verify. The second is a human clearing the hold. The third is the next morning's scheduled run, which is where the design earns its keep.
Rows written, by layer
Tables written by the run will appear here, newest first.
In run 1: one reject (a malformed record refused rather than coerced to NULL), four rows held at Review with their foreign key left NULL, and one row that matches exactly and auto-confirms. In run 3: the freshness gate lets the run through because the feed did publish, content_hash then short-circuits the records that did not change, and the alias a human taught it in step 2 matches at 1.00 without asking again.
Re-running is meant to be cheap. Gate one asks, once per run, whether the feed has published anything since the last success — if not, nothing is fetched at all. Gate two asks, per record, whether this payload is byte-identical to the last one landed — if so, only last_seen_at is stamped.
They answer different questions, which is why there are two commands people reach for wrongly. --force lifts gate one only, so after a mapper change it re-downloads the whole corpus and then skips every record at gate two. The job that actually re-derives staging from stored payloads is npm run ingest:retransform: it reads raw_source_payload and never touches the network. It is also the only thing in the system that reads the payload column — which is what the landing layer exists for.
Reference
The seven layers, and what each one refuses to do
| Layer | Tables | Written by | The rule it enforces |
|---|---|---|---|
| CTL Control plane | data_sources, ingest_runs, source_watermarks, ingest_rejects | Every loader | Which run wrote this row, where the feed got to, and what it refused. A bad run is revertible because every staging row carries its ingest_run_id. source_watermarks.last_success_at is what the freshness gate reads — and a bounded pilot run deliberately does not advance it. |
| L0 Landing | raw_source_payload | Loader, before anything is derived | Append-only. Never updated, never read by the API. A mapping bug becomes a re-transform instead of a re-download — which matters because shortages and CT.gov publish current state only. ingest:retransform is the job that cashes that in. |
| L1 Staging | ndc_products, fda_applications, clinical_trials, drug_labels, drug_shortages, adverse_event_reports, publications, orange_book_products, purple_book_products | *-mapper.ts | Typed, keyed on the source's own record id. Uniqueness is per source, not global, so two feeds publishing the same NDC do not overwrite each other. |
| L2 Resolution | entity_xrefs, company_aliases | *-matcher.service.ts | A staging foreign key is written only from an xref at Confirmed, and from the entity that row names — never from the candidate the current run computed. Anything less confident leaves the FK NULL; a Rejected ruling keeps it NULL permanently. An unattributed record is recoverable; a wrongly attributed one quietly poisons every derived trend. |
| CORE Canonical | companies, products, people, facilities, deals, and their satellites | Curators & the write-back job | One row per real-world thing. This is what the API serves and what a slug points at. |
| L3 Provenance | fact_claims, field_overrides | Loaders & the governance workflow | Source URL, as-of date and confidence on every contested fact. Losing claims are never deleted — "three sources disagree with this value" is a feature of the profile. |
| REF Vocabulary | taxonomy_terms, taxonomy_codes, taxonomy_crosswalks, ingredients, routes | Reference loaders | One vocabulary, many code systems. MeSH, ICD-10, SNOMED CT, ATC and MedDRA all hang off the same term, so "trials for this product's indication" is a join. |
Survivorship: which value the profile page actually shows
Defined once, in SQL, by resolve_entity() — not in application code, because the write-back job needs the same ordering without going through Node.
field_overrides— a human decided. Nothing outranks it, permanently.data_sources.trust_rankdescending — regulatory authority over secondary source.fact_claims.observed_atdescending — the source's own as-of date, not when we fetched it.fact_claims.confidencedescending — primary over inferred over crowdsourced.
Reference
Industrial identifier for every table
All 95 tables. For each: the primary key the database uses, the natural key that makes a re-ingest an upsert rather than an append, and the identifier the outside industry would use to name the same thing. Where there is no external identifier the cell says so — a blank cell invites someone to invent one.
Four classes, and only one of them is a column
“Does this table have an industry identifier?” has four different answers, and conflating them is how a schema ends up with a lei column that cannot hold the truth.
A registrar issues exactly one identifier that names this row, unique worldwide. Store it as a column, constrain it, trust it.
14 tables · NDC, NDA/BLA, NCT, PMID, DOI, FEIRegistrar identifiers exist, but several apply to one row. A single column cannot hold them. Needs a child identifier table.
6 tables · companies, products, ingredients…The identifier only means something underneath its parent — product number 001 is meaningless without its application. Composite key.
19 tablesNo external identifier exists. Junctions, the control plane, governance and app tables. The natural key is correct and inventing one is wrong.
56 tablescompanies: no, and it cannot be one column
There is no single identifier that names a pharma company across the industry, because the registrars name legal entities and a directory entry is a group. Four separate LEIs, all live in GLEIF's register today, all belonging to what your directory calls one company:
| LEI (ISO 17442) | Legal name in the register | Jurisdiction | Status | Where it shows up in your feeds |
|---|---|---|---|---|
765LHXWGK1KXCLTFYQ30 | PFIZER INC. | US-NY | Active | the directory entry itself |
5493007F3JGT3G124417 | PFIZER PRODUCTS INC. | US-CT | Active | manufacturing / labeling records |
549300PXUU34P6IQ3A11 | PF PRISM C.V. | NL | Active | the NDA203214 sponsor in Drugs@FDA |
549300NPEN3DIEAYWM33 | Pfizer Canada Inc. | CA-QC | Inactive | historical CA filings |
This also settles the matcher failure from run 1. PF PRISM CV scored 0.10 against “Pfizer Inc.” and no fuzzy threshold would ever have caught it — correctly, because it is a genuinely different legal person, registered in the Netherlands. The link between them is corporate hierarchy, not name similarity, and no amount of tuning the trigram threshold recovers it.
The other identifiers behave the same way. SEC CIK covers only US registrants (0000078003 for PFE, 0001682852 for MRNA) and stops at the filing entity. DUNS is assigned per site and per legal entity, so a group has dozens. FDA labeler codes are issued per labeler, and Pfizer holds many beyond 0069.
products has the same problem — UNII, RxCUI, ATC and GTIN all apply at once — and it is already solved by product_identifiers. taxonomy_terms has it too, solved by taxonomy_codes. companies is the one GLOBAL×N table with no equivalent.
Recommendation: add company_identifiers, mirroring product_identifiers exactly — (company_id, identifier_type, identifier_value, issuing_authority, confidence) unique on the triple, carrying LEI, DUNS, CIK, labeler code and any national registration number. Until then entity_xrefs can carry them keyed by source system, which works but mixes “this feed record maps to this company” with “this company is this registered entity” — two different statements in one table.
A GLEIF name search for Pfizer Inc. returns 5 records. The same search for Moderna, Inc. returns 76,537 — because moderna is an everyday adjective in Italian, Spanish and Portuguese, so the register is full of Belgian printers, Madrid clothing shops and Italian holding companies. Pfizer is a rare token; Moderna is not.
That asymmetry is exactly why the two companies produced such different trigram scores in run 1, and it is not fixable by tuning. A name is a lookup hint. A registrar identifier is an answer.
One limit, stated plainly: GLEIF returns HTTP 404 for both direct-parent and ultimate-parent on PF PRISM C.V. and on PFIZER PRODUCTS INC. — no hierarchy is published for either. So an LEI tells you which legal entity a record belongs to, but not whose subsidiary it is. Rolling those four entities up to one directory company remains a curated decision, which is what company_aliases and Company.parentCompanyId are for.
Reading the Example column. Values in accent colour were fetched from the live API while this page was written and can be looked up as printed. Values in grey with a dotted underline are format illustrations showing the shape of the identifier — they were not fetched and should not be quoted as records. Where a value is part-verified the note says which half.
| Table | Layer | Class | Primary key | Natural key in this DB | Industrial identifier | Issuing authority | Example |
|---|
ndc_products.product_ndc, fda_applications.application_number, clinical_trials.nct_id and drug_labels.(spl_set_id, spl_version) were all demoted from @unique. The per-source partial index on (data_source_id, source_record_id) says one row per source per record; a global unique on the business key says one row full stop. With twenty-plus feeds those two statements contradict each other the moment a second source publishes the same key, and the loser is an ON CONFLICT that silently overwrites another source's row. Uniqueness of the real-world thing lives on the core entity and is reached through entity_xrefs. This is a design decision, not a missing constraint.
Verified values for the first three are listed above. LEI, DUNS and SEC CIK for companies; ORCID for people; FDA recall, Warning Letter and Form 483 reference numbers for regulatory_events; GS1 GLN for company_locations. Each can be carried today as an entity_xrefs row keyed by source system, which is the intended route — but none of them has a first-class column, and no constraint currently enforces their formats. Flagged here rather than left blank.
Honesty
The three empty states
Some tables stay empty for these two companies, and the reason differs each time. One NULL cannot encode three different reasons, so field_coverage carries the distinction at section grain. A clean compliance panel is only a finding if the reader knows the platform actually looked.
We looked; the source published nothing. A genuine finding. Example on this run: company_awards — no awards feed returned a row for either company.
Nobody has looked yet. Absence of evidence, not evidence of absence. Example: facilities — the FDA establishment registration feed is not wired, so both companies show zero sites and no FEI numbers.
Structurally irrelevant to this company's vertical. Example: vendors and vendor_services — both companies are manufacturers, not service vendors, so the Vendors panel is not rendered at all.