Architecture and geo-redundancy

ExpTech real-time earthquake data uses geo-redundancy. Two core sites, Tainan (Taiwan) and Tokyo (Japan), both receive data from every station. Each site runs its own real-time processing, and the two sites sync and de-duplicate. If one core site stops, the other still produces real-time seismic intensity and waveforms. The metadata database replicates between the two sites and fails over automatically under a quorum of Tainan, Tokyo, and Taipei. Real-time streams are served by two load-balancer entries, in Taipei and Kaohsiung.

StationsDual uplink50 Hz 3-axis acceleration; intensity computed on the station; sent every 500 ms

Primary linkTainanUDP · WebSocket fallbackBackup linkTokyoUDP · WebSocket fallback

TainanTaiwan · TNN-1Core site

Station intakeSignature check, replay protection

Real-time intensity RTS

Real-time waveform

Time-series storeWaveforms and intensity, kept 7 daysClickHouse

Metadata databaseWrites go to the current primary onlyPostgreSQLHAProxy

TokyoJapan · TYO-1Core site

Station intakeSignature check, replay protection

Real-time intensity RTS

Real-time waveform

Time-series storeWaveforms and intensity, kept 7 daysClickHouse

Metadata databaseWrites go to the current primary onlyPostgreSQLHAProxy

Message clusterCross-site sync · de-duplication

Synchronous replicationStandby promoted automatically if the primary failsPatroni

Three-site quorumetcdTainanTokyoTaipeiElects the primary; still a majority if one site fails

Served by either site

Push serviceReal-time data to apps

DevelopersLoad balancers · Taipei / KaohsiungSSE streams, time syncTPE-1 · KHH-1Core sites · Tainan / TokyoData APITNN-1 · TYO-1

Primary link
Backup link
Cross-site sync and de-duplication
Database replication and quorum
Distribution

Both uplinks send at the same time; the backup link is not a standby. Each link uses UDP and switches to WebSocket after repeated timeouts.

How data gets from a station to you

Station measurement: The sensor samples three-axis acceleration at 50 Hz, converts it to JMA seismic intensity on the station, and sends 2 packets every 500 ms (seismic intensity and sensor data).
Dual uplink: Each station stays connected to Tainan (primary link) and Tokyo (backup link) at the same time. Both sites receive every packet. Both links carry traffic whenever the station is up.
Transport and checks: Each link uses UDP. If no acknowledgment arrives within 250 ms, the station retransmits, at most once. Every packet and every acknowledgment is signed with HMAC-SHA256. The server rejects replayed packets.
Cross-site sync: The two core sites are connected by a message cluster. They sync waveforms and seismic intensity with each other and de-duplicate, so the same record is not processed twice.
Each site computes on its own: Each site runs a real-time seismic intensity (RTS) service and a real-time waveform service. Both subscribe to the same real-time data and each produces its own results. Waveforms and seismic intensity are also written to that node’s time-series database and kept for 7 days.
Distribution: Real-time data is broadcast through the message cluster, and the push service delivers it to apps. Developers reach SSE real-time streams and time sync through the Taipei and Kaohsiung load-balancer entries. Data APIs such as earthquake reports are served by the Tainan and Tokyo nodes. URLs for each service are on Servers and Base URL.

Every server syncs its clock once a minute and aligns sample timestamps, so time matches across nodes. When your program compares data times, use the time sync endpoint to correct the local clock.

How geo-redundancy works

Node	Location	Role	Serves
`TNN-1`	Tainan	Core node (primary station uplink), metadata database, consensus node	Data APIs; weather and map tiles are served only here
`TYO-1`	Tokyo	Core node (backup station uplink), metadata database, consensus node	Data APIs
`TPE-1`	Taipei	Load-balancing entry point, consensus node	SSE real-time streams, time sync
`KHH-1`	Kaohsiung	Load-balancing entry point	SSE real-time streams, time sync

See Servers and Base URL for each node’s address.

Dual uplink

Each station keeps two uplink links. The primary link goes to Tainan and the backup link goes to Tokyo, and both send at the same time. If one site disconnects or the network is congested, the other site already has the same data. There is no wait for a switch.

Each link chooses its own transport:

Normally it uses UDP. Latency is low, and the number of retransmits is bounded.
After 4 timeouts in a row, that link switches to WebSocket (TCP).
While it is on WebSocket, it probes with UDP every 5 seconds. After 3 successes in a row, it switches back to UDP.

The server tracks each link’s quality with smoothed RTT and the loss rate over 5 minutes.

Each site computes on its own

Tainan and Tokyo each run a full real-time pipeline: station intake, real-time seismic intensity (RTS), real-time waveforms, and a time-series database. The two sites process the same data and do not depend on each other. If one site stops, the other still produces results.

Cross-site sync and de-duplication

The two core sites are connected by a message cluster and sync the waveforms and seismic intensity each of them received. The same packet arrives at both sites over the two links. Sync de-duplicates it, so each record is processed once.

Automatic database failover

The metadata database (PostgreSQL) has one instance in Tainan and one in Tokyo. Patroni manages synchronous streaming replication:

When the primary fails, the standby is promoted to the new primary.
Which instance is primary is decided by a quorum (etcd) of Tainan, Tokyo, and Taipei. If any one of the three fails, the other two are still a majority, and the election continues. Taipei is the third quorum node, so a majority still exists when one of the two core sites fails.
Each node’s routing layer (HAProxy) sends writes only to the current primary.

Public entry points

The entry points developers connect to are also in different places:

SSE real-time streams and time sync: two load-balancer entries, Taipei (TPE-1) and Kaohsiung (KHH-1), serving the same routes.
Data API: the two core sites, Tainan (TNN-1) and Tokyo (TYO-1), serve it directly. An Auto name is also assigned to a node by DNS.

The entry point does not switch nodes for your program. Auto is assigned by DNS, and DNS may not drop a failed node immediately. To avoid depending on a single node, prepare backup hosts in the client as described in Switch the Base URL yourself.

What happens when a node fails

Situation	What happens	What you should do
One core site stops (Tainan or Tokyo)	Station data still reaches the other site over the other link. Real-time seismic intensity and waveforms are still produced. Requests sent straight to the stopped node fail. Requests through Auto can also fail until DNS removes that node.	For data APIs, switch to the other node (TNN-1 ↔ TYO-1) as described in Switch the Base URL yourself. Switch only on a connection error, a timeout, or a 5xx. If an SSE stream drops, reconnect using the rules in Reconnect.
The Tainan node stops	Same as above. Weather and map tiles are served only in Tainan, so they are unavailable during this time.	These endpoints have no backup node. Another host does not serve them. Show that the data is temporarily unavailable, and try again later.
An uplink gets worse or drops	That link switches to WebSocket on its own, and switches back to UDP after it recovers. The other link keeps sending, so both sites still have data.	Nothing to handle. The public API and stream formats stay the same.
The database primary fails	The standby is promoted to primary. Writes on each node go to the new primary.	Usually nothing to handle. If you get a 5xx during the switch, retry using the rules in Switch the Base URL yourself.
One quorum node fails	The other two sites are still a majority. Primary election continues as usual.	Nothing to handle.
A load-balancer entry stops (Taipei or Kaohsiung)	The other entry serves the same routes. Connections through Auto can fail until DNS removes the failed entry.	After an SSE disconnect, wait 3 seconds per `retry: 3000`, then reconnect. After several failures in a row (for example 3), switch to the other entry (TPE-1 ↔ KHH-1). Do the same for time sync.

Current limits

Geo-redundancy covers real-time earthquake data: stations, seismic intensity, waveforms, and earthquake reports.
Weather, and map tiles such as radar, satellite, and wind, are served only on the Tainan node today. They have no geo-redundancy. When Tainan is down, these endpoints are unavailable. See Routes served only by TNN-1.
Auto names are assigned to a node by DNS. A failed node is not guaranteed to be removed immediately. To avoid depending on a single node, the client has to switch hosts itself.
This page describes the architecture. It is not a commitment about availability or failover time.

Servers and Base URLhost groups, how the nodes differ, and how to switch hosts in the client.

SSE connectionconnecting, heartbeats, and reconnecting for real-time streams.

Time synccorrect the local clock with the server time.