Skip to content
Console

Moving the CWA EEW relay stream from Lua to Go

The CWA Earthquake Early Warning (CWA EEW) that DPIP and TREM-Lite show is issued by the Central Weather Administration (CWA), and our servers relay it to the apps. Until Nov 16, the stream that does this relaying (the EEW stream from here on) was a piece of Lua inside OpenResty on our load balancers. The real-time seismic intensity (RTS) stream, on the other hand, runs on sse-server-go, our SSE server written in Go. TREM-Lite lost its connection to the EEW stream more often than its connection to RTS. From 04:00 to 09:20 on Nov 16 (Taiwan time, UTC+8, as are all times below), TREM-Lite connections to the EEW stream lasted only 23 minutes on average, against 36 minutes on RTS.

Between 02:30 and 03:01 on Nov 17, we moved the EEW stream into sse-server-go and removed all the Lua that served it from both load balancers.

Our two load balancers (LBs), lb-tpe1 and lb-khh1, run OpenResty, and the EEW stream lived entirely inside it. One worker held the subscription to upstream warnings on our message cluster. When a message arrived, a cross-worker event broker, lua-resty-events, woke the other workers. Each connection waited on its own semaphore, sent new content when there was some, and otherwise sent a heartbeat every 60 seconds. The same data also fed a polling GET with an ETag, which answered 304 Not Modified when nothing had changed.

That Lua came to 529 lines per LB: 185 for the hub, 318 for the message cluster client and 26 for compression, plus Lua blocks in two server configs. To get the core module that lua-resty-events needs, we built nginx from source, which is why the OpenResty Dockerfile ran to 112 lines.

The polling GET was busy. From 04:00 to 09:20 on Nov 16, lb-tpe1 averaged 79.7 requests per second, and 76.4% of them got a 304.

Everything else had already left Lua. Since September 2025 we have been rewriting our services in Go (see From JavaScript to Golang), and the RTS, intensity report and earthquake report streams all run on sse-server-go. The EEW stream was the only stream on the LBs still in Lua.

The gap in disconnects showed up only in TREM-Lite. DPIP connections averaged under 3 minutes on both streams, 173 seconds on the EEW stream and 169 on RTS, with no real difference between them.

The cause was the heartbeat. For each connection, the Lua version waited 60 seconds, then checked whether 60 seconds had passed since it last sent anything, and sent a heartbeat only if they had. On some connections the check came out a hair short of 60 seconds, so that round sent nothing and the first heartbeat went out only at 120 seconds. TREM-Lite treats a connection that has sent no bytes for too long as dead and reconnects; the current version waits 90 seconds. DPIP waits 132 seconds and was not affected. From 04:00 to 09:20 on Nov 16, 255 TREM-Lite connections to the EEW stream ended at exactly 120 seconds, and none of them carried a single heartbeat after the opening data.

Two behaviors in the Lua version were also not what we wanted. When a warning expired, the Lua version cleared it only once the next upstream message arrived, so while the upstream stayed quiet, the expired warning could stay in the snapshot. And when two updates arrived in the same millisecond, the Lua version dropped the second one.

While getting ready for the switch on the evening of Nov 16, we found one more problem. The message cluster client reconnected in a loop inside a timer and never checked whether its worker was shutting down. Each config reload closed the old worker’s connection, the loop connected again, and so the old worker never exited. On lb-tpe1, 24 of these old workers had piled up, and the oldest had been there for 23 days. Together they used about 1.8 GB of memory, and each one still held its own duplicate subscription. Another Lua module written the same way had the same problem. That night we fixed the loop and cleared out the old workers.

RTS had been stable on sse-server-go, so we moved the EEW stream there too.

The new EEW module lives in sse-server-go. It takes warnings from three upstream sources and merges them into one snapshot, the list of warnings currently in force:

  • For each earthquake event, only the message with the highest serial is kept.
  • A warning leaves the snapshot 240 seconds after its origin time. Expiry is checked every second, without waiting for the next upstream message.
  • If the content has not changed, nothing is sent again.
  • Snapshot versions strictly increase, so two updates in the same millisecond both go out.

Each snapshot is compressed once, and every connection shares the result. A new connection gets the current snapshot right away. A client that can’t keep up holds only the newest snapshot instead of a backlog of old ones.

Older app versions don’t need an update. The old EEW stream and the polling GET are still there, now produced by sse-server-go from the same snapshot in the old format, byte for byte. Both are deprecated, with no end date set.

The module is about 736 lines of Go, with another 880 lines of tests that run with the race detector on.

With the stream in Go, every EEW connection now also passes through OpenResty to sse-server-go. That hop used to be TCP. Each SSE connection proxied to the backend took one source port on OpenResty, with a ceiling of about 55,000 per destination, and two connection-tracking (conntrack) entries: one from client to LB and one from LB to backend. Over the past 7 days, concurrent SSE and WebSocket connections on lb-tpe1 peaked at 22,583, and we are planning capacity for 100,000.

OpenResty now reaches sse-server-go over a unix socket, with TCP kept as a backup. SSE connections no longer use source ports, and each takes one conntrack entry. After the switch at 03:01, TCP connections from OpenResty to the backend on lb-tpe1 went from 496 to 0.

With the Lua gone, we had no reason left to build nginx ourselves. OpenResty now runs the official image with only sqlite added.

Item Before After
OpenResty Dockerfile 112 lines 12 lines
OpenResty image, lb-khh1 311 MB 236 MB
OpenResty image, lb-tpe1 213 MB 162 MB
Conntrack entries per SSE connection 2 1

We switched one LB at a time, starting with lb-khh1.

Time Event
02:30 lb-khh1 gets the new backend; /api/v1/trem/sse/report is retired on this LB
About 02:30 The status page probes of the earthquake report stream start failing
02:31 OpenResty on lb-khh1 reloads its config
02:31 Internal monitoring of the EEW stream starts failing
02:35 lb-tpe1 gets the new backend and reloads; /api/v1/trem/sse/report is retired on this LB
02:43 A fix goes live that reports the actual state of the upstream link
02:43 Internal monitoring recovers
02:45 The earthquake report stream probes now request /api/v1/cwa/sse/report and recover
02:49 The status page’s own config is updated to the new URL
02:58 lb-khh1 restarts OpenResty with the unix socket and the new image
03:01 lb-tpe1 restarts OpenResty with the unix socket and the new image

The config reloads produced no 5xx at all, and the old workers all exited within their 30-second limit. Restarting OpenResty itself was different: each of the two restarts returned 502 or 503 for about 10 seconds while it started. Every backend restart also cuts every connection once, and it is up to each client to reconnect.

The new code only subscribes to warnings and has no permission to publish them. Before each restart, we checked the logs to confirm that no warning was in force. We verified the switch with logs and metrics only. We sent no test warnings and opened no test connections to the live EEW stream.

During the switch, we broke two of our own monitors.

The first was the Service status page’s probe of the earthquake report stream. /api/v1/trem/sse/report was retired between 02:30 and 02:35 as each LB got the new backend, but the probes were still requesting that path. They started failing at about 02:30 and recovered only after we pointed them at /api/v1/cwa/sse/report at 02:45. In between, the status page sent outage and recovery notices. Its own config was updated only at 02:49, so those notices still showed the old URL. We apologize to everyone who received them.

The second was our internal monitoring of the EEW stream. The old EEW stream’s response indicates whether the server’s link to the upstream is up, and our internal monitors check exactly that. The new server reported “not started yet” until the first warning arrived, and with no warnings that night, it stayed that way. The internal monitors failed from 02:31 until 02:43, when a fix made the server report the actual state of the upstream link.

Both had the same cause. The probes and monitors are clients too, and we left them to be changed after the server was swapped. The paths and the status they depend on should have changed in the same step as the server. As a result, the earthquake report stream probes were wrong for about 15 minutes and the internal monitors for 12.

The status page has since gained a row for the CWA EEW stream covering TPE-1, KHH-1 and Auto, with the probed URL hidden.

The table below compares two mornings on lb-tpe1, both from 04:00 to 09:20: Nov 16 in the Lua era and Nov 17 in the Go era, both weekdays. Connection lengths come from the access logs, with the apps told apart by User-Agent.

Item Lua (Nov 16) Go (Nov 17)
Average TREM-Lite connection, EEW stream 23 min 32 min
Average TREM-Lite connection, RTS (for comparison) 36 min 32 min
Average DPIP connection, EEW stream 173 s 177 s
Average DPIP connection, RTS (for comparison) 169 s 176 s
TREM-Lite EEW stream connections that ended at exactly 120 s 255 0
Polling GET, requests per second 79.7 83.7
Polling GET, share answered with 304 76.4% 77.2%
Polling GET response time Under 1 ms Median 1 ms, p99 3 ms
5xx from the EEW stream and polling GET 0 0

The average TREM-Lite connection on the EEW stream grew from 23 minutes to 32. Over the same period, RTS went from 36 minutes to 32, so the gap between the two streams is gone. DPIP looked about the same in both eras.

Connections ending at exactly 120 seconds are gone. In the Go version one timer sends the heartbeat to every connection, so none is skipped.

The polling GET now takes one more hop, with OpenResty passing the request to sse-server-go. The Lua version answered inside OpenResty, and every logged time was under 1 ms. Now the median is 1 ms and p99 is 3 ms.

Host-level numbers on the LB dropped too, but not all of that is down to the move. The old workers mentioned above were cleared on the evening of Nov 16, right between the two windows, so the table below shows both changes together and we can’t tell how much each one contributed. Load was similar in both windows: on average 6,627 and 6,584 concurrent SSE connections, and 3.57 million and 3.74 million requests in total.

lb-tpe1 host Lua (Nov 16) Go (Nov 17)
CPU usage, average 21.6% 16.5%
Conntrack entries, average / peak 13,328 / 37,670 11,152 / 24,430
Memory usage, average 5.4 GB 3.9 GB

This is one morning on one LB. The TREM-Lite sample is small, with only about 1,000 to 2,800 connections per stream in each window.

The Go era still had one kind of drop. From 04:00 to 09:20 on Nov 17, 144 TREM-Lite connections broke off at about 116.5 seconds: 97 on the EEW stream and 47 on RTS in sleep mode. All of them came from 6 IPs, and every drop happened in the second the server sent its heartbeat. Live-mode RTS, which carries data every second, did not drop once. RTS had the same drops in the Lua era.

These clients’ networks appear to cut a connection that sits idle for about 60 seconds, which races a heartbeat sent every 60 seconds. The connection is cut at a heartbeat and the client reconnects about 3.5 seconds later. The next heartbeat arrives 56.5 seconds after that, in time; the one after it comes a full 60 seconds later, too late.

At 10:13 on Nov 17, we moved the heartbeat to every 30 seconds. That is one more heartbeat per connection per minute, about 20 bytes. With about 7,000 connections on lb-tpe1, it adds about 2 KB per second, under 1% of all SSE traffic. The advice for clients is unchanged: reconnect when no bytes arrive for 90 seconds. See Heartbeat.

From the change to 12:45, no TREM-Lite connection dropped at about 116.5 seconds; the day before, the same window had 28. Many TREM-Lite users moved to the new version during this time, so connection counts don’t compare directly with the day before. We also followed the 7 IPs that had been cut 3 or more times between 04:00 and 10:13 on Nov 17. Three of them left 10 connections after the change, and none dropped. One of these, still on the old version, kept a connection for 61 minutes; before the change, 19 of its 88 connections had been cut. The other 4 IPs left no finished connection in the logs during this window.

The change needed a restart of sse-server-go, and this one was in the daytime. In the minute of 10:13, lb-tpe1 returned 502 4,222 times, and SSE connections fell from about 9,200 to about 6,200, back to only about 8,500 by 12:30. Only about one in ten of the DPIP connections that were cut retried during the few seconds of downtime, and those were back within a minute. The rest reappeared over the following minutes to hours.

The apps are changing too. TREM-Lite 26w47a, a preview released at 09:33 on Nov 17, opens a single connection in place of the two it used before. For earthquake reports, it now fetches the list once at startup and then follows the earthquake report stream for updates. The same change for DPIP is still in testing. The 04:00 to 09:20 data in “After the move” all predates this release, so it reflects the older app versions.

If your code still connects to /api/v1/trem/sse/report, that path was retired on Nov 17. Connect to /api/v1/cwa/sse/report instead, or use /api/v1/sse to receive several topics on one connection. The three multiplexed entries are compared in SSE connection.