Yesterday Japan experienced the Internet equivalent of rolling blackouts. From around 12:22 local time into the late-afternoon, dozens of prominent websites, apps, and services were unavailable. 接続障害 (connection issues) trended on Twitter, and DownDetector.jp had a field day.

Timing of the outage

The outage occurred during the middle of the business day on Friday, August 25, 2017.

• NTT Communications reported that there were intermittent connection issues on their OCN backbone due to foreign-source routing instability from 12:22 to 12:45 JST.
• KDDI reported instability due to large number of route changes from 12:24 to 12:39 JST.
• Many impacted sites remained unavailable into the afternoon, presumably due to second-order effects. For example, Sakura Internet reported that outages on multiple upstream carriers caused connectivity issues from 12:25 to 17:25 JST. In my case, the Mobile Suica app was still down at 15:50, which seems consistent with reports on Twitter.
• Update (2017/8/28): On Saturday a Google spokesperson confirmed to the Asahi newspaper that the outage was caused by a ‘misconfiguration’ (誤設定) on their part and apologized for the incident. According to their statement, the issue was resolved in less than 8 minutes.

Cause

Reports here and here suggest that (1) NTT Communications/OCN (AS4713) peers with Google (AS15169), (2) Google advertised a large number of bad BGP routes for destinations that should have been handled by OCN, and additionally (3) the sheer volume of route changes may have overloaded edge routers for customers connected to OCN’s network. According to JPNIC, the BGP routes affected approximately 25,000 IP addresses in Japan and 100,000 IP addresses worldwide.

Update (2017/8/28): Andree Toonk has published a more complete technical analysis of exactly what happened on the BGPmon Blog.

Notable affected services

Note: While some services on this list were completely down for all users, others are included due to user reports of connection issues on Twitter.

• Merukari: The wildly popular online market that is a rising star of Japan’s startup scene had to briefly shut down their service from around 13:00 to 14:40 JST. (Source) (Source) (Source) (Source)
• Mobile Suica: The major mobile payment card and train pass with Apple Pay integration had both its app and its website go down. While the cards themselves remained functional, for several hours you couldn’t top up your balance with the app.
• Rakuten Securities: This major online brokerage was (at least partially) taken offline. (Outage Report)
• SBI Securities: This major online brokerage was also (at least partially) taken offline. (Source), (Source)
• Jibun Bank: A bank funded by both KDDI and The Bank of Tokyo-Mitsubishi UFJ was (at least partially) offline from 12:24 to 15:28 JST. (Outage Report)
• Update (2017-08-28): According to this report in the Japan Times, Resona Bank, Saitama Resona Bank, and Kinki Osaka Bank were also affected.
• Nintendo: Online play was affected from 12:25 to 17:30 JST. (Outage Report)
• Sony PSN: Although their website shows an outage from 12:00 to 16:00 JST, they pointedly note that any connection problems were caused by user’s ISPs. (Source)
• Sakura Internet: One of Japan’s more prominent cloud providers experienced connectivity issues. (Outage Report)
• Netflix: Users reported connection issues on Twitter. (Source) (Source)
• Slack: Remained online for me, but other users reported connection issues on Twitter. (Source) (Source)
• GitHub: Remained online for me, but other users reported connection issues on Twitter. (Source) (Source)

Thoughts

So what do you do when the Internet goes down?

In practice, the answer is ‘not much.’

So you wait for it to come back up.

Updates

1. 2017/8/26 Initial post published
2. 2017/8/28 Updated with links to new articles with additional information about the outage.