Approve fields
The existing ITS contract says Client A may access HIFZ and certification, but not address.
High-level design · review draft
Publish once. Keep every approved system current.
Replace repeated SOAP and REST polling with permission-aware Kafka streams carrying the latest authoritative ITS changes.
- People
- 500,000
- Client systems
- 200–400
- Peak changes
- 2,000/day
- Replay window
- 7 days
Who owns what
ITS owns the source, broker, schemas, and permissions. Clients own only their consumer and their local update logic.
ITS OWNSCloud platform
SQL Serverauthoritative state
→Change capturedetect ITS ID + field
→Kafkapublish once
hifz.changed
certification.changed
address.changed
CLIENT OWNSAny technology
Kafka consumerITS-issued credentials
↓Client databasestore latest state
One shared topic per permission-aligned field or domain. No queue or topic per client.
Follow one HIFZ update
1. WriteStaff or service updates SQL Server.
2. CaptureITS identifies HIFZ and the ITS ID.
3. PublishLatest value enters hifz.changed once.
4. Fan outEvery approved consumer group reads it.
5. CommitClient stores data, then commits offset.
Ready: ITS ID 30305110 · HIFZ status changes from 5 to 6
How subscription works
“Subscribe” is not an open signup. ITS approves access first, then Kafka enforces exactly those topic permissions.
Provision access
ITS creates unique credentials, an allowed consumer-group identity, and READ ACLs for approved topics.
Connect and consume
The client subscribes with those credentials. Kafka rejects every topic that its ACL does not allow.
Contract and failure rules
The database remains authoritative. Messaging can lag or retry, but it must never block an ITS write.
Latest state only
Events contain the latest authorized value, not every intermediate value and never the previous value.
At least once
Clients deduplicate by eventId and reject stale domainVersion values.
Ordering is local
Kafka keys by ITS ID. Order is preserved within a topic, not across domains.
Large data stays out
Photograph events carry metadata only. Approved clients fetch image bytes through the API.
Deletion is explicit
person.deleted is a separate event. A cleared field is explicit null, not an omitted field.
Finite responsibility
Kafka retains seven days. Longer-disconnected clients reconcile through existing APIs.
99.9%available within five minutes
1 houroutage backlog recovery
7 daystopic retention
1 regionmulti-zone availability
Open before the LLD
The HLD is stable without these answers. The low-level design is not. Expand each area during technical discovery with ITS.
- Can SQL Server CDC be enabled on every relevant database?
- How are direct staff edits and multi-database joins captured?
- Can on-premises environments connect outbound to the cloud securely?
- Where is the atomic domain version generated?
- What are the final 15–20 fields and permission groupings?
- Which fields may safely share a topic?
- Which schema registry and compatibility policy applies?
- Should rapid updates consolidate over 0, 5, or 10 seconds?
- Where are current client field permissions stored?
- How will permission changes synchronize to Kafka ACLs?
- mTLS, SASL/SCRAM, OAuth, or cloud IAM?
- How are credentials issued, rotated, revoked, and audited?
- AWS or Azure, and which managed Kafka service?
- Partition counts, message limits, and seven-day storage sizing?
- Alerts for capture lag, publish lag, and consumer lag?
- Disaster recovery, replay, schema rollback, and incident runbooks?
- Which reference consumers or SDKs will ITS publish?
- How will clients test before production access?
- How should poison events be handled without blocking a partition?
- What schema version deprecation window is supported?
- Which API retrieves photographs and other oversized fields?
- How does a client reconcile after seven days offline?
- Does permission revocation require a downstream deletion event?
- What evidence proves no source change was lost?
The design in one sentence
ITS publishes each latest change once to a permission-aligned Kafka topic; every approved client reads it independently through its own secured consumer group.