How a hardware component knocked out the stock market of $ 6,000 billion?

At 8:00 a.m. last Thursday October 1, an autumn morning as usual for stock traders as they prepare for the new business day. But something unusual is happening. Price indexes and other data on the world’s third-largest commodity market did not appear as usual. Other people find the data constantly appearing and then disappearing.

Everyone was confused, not understanding what happened. Whether their company’s systems are crashing or the Tokyo Stock Exchange (TSE) – the world’s third largest commodity exchange worth $ 6,000 billion – is in trouble.

A minute later, the stock exchange issued the first notice to the system administrators of securities companies that TSE’s trading system was having problems. However, at some brokers, this information is not passed down to each employee immediately.

Electronic board with empty tickers, absolutely no transactions on Thursday morning of last week

At 8:5 a.m., on Twitter – the communication channel commonly used by brokers outside of the official communications system – began spreading rumors of a problem with TSE’s trading system. The feeling of confusion is increasing when there is no official announcement on the exchange.

At 8:36 a.m., TSE finally issued an official announcement to the securities companies that trading operations would be completely stopped.

This is the first time in nearly 15 years that this exchange has been completely stopped. The Tokyo stock market has a no-closing policy, even in the event of a natural disaster, so for many people preparing for that day, this is the first time they have encountered this situation. .

Only one hardware component fails

This policy of the Tokyo exchange is made possible by its ArrowHead (Arrowhead) electronic trading system. This is a completely electronic transaction system, designed based on the cooperation with Fujitsu.

Since its inception in early 2010 until now, it has become the leading solution for modern trading systems with fast and reliable transaction speed. A system with 350 servers that processes buy and sell orders at levels in just a few milliseconds and is capable of doubling capacity when needed. Despite a few hiccups since its launch, it hasn’t had any major problems so far.

However, that changed last Thursday morning, when a hardware component named a shared drive device number 1 – one of the two square data storage boxes – detected a memory error. This device is used to store administrative data for a variety of other servers to use and distribute information, such as commands with ID combined with password for transaction monitoring devices.

Usually, when the No. 1 drive fails, the system automatically switches to the backup device – the shared drive number 2. But for some reason, the TSE exchange’s operators don’t understand. This process does not happen. That directly affects the servers acting as information distribution gateways, making it impossible for them to send market trading information to stock traders.

And finally, the chain effect happened.

Historical decision

Coincidentally, that day is the day when many important economic data will be released and they could affect as many as 2,500 stocks traded on various stock exchanges, including the Nikkei 225 and Topix. .

That adds to the impatience of many people and the hope that the trading system will be up and running again soon – as it did in 2012 when the system also went down, but soon returned to work. But this incident is completely different.

From pausing trading, TSE finally decided to completely stop trading for the whole day.

After the system did not automatically switch to a backup storage device, the TSE exchange was forced to manually switch to the number 2 shared drive. But at this point, administrators were faced with a Another option: they can restart same-day trading, but this will force a full system reset – power off and restart.

This also means that the data for buy and sell orders received by the securities companies is lost – instead of being canceled. This means that the amount and the goods that the traders have placed in the order may not be refunded and, according to the securities firms, this will lead to chaos. Finally, after talking to the market participants, TSE decided to come to an historic decision: to stop trading for the whole day.

For the first time since 1999 until now, this exchange has stopped working for so long.

TSE executives bowed to apologize during a press conference that afternoon

At 4:30 that afternoon, 4 TSE executives held a press conference to explain this incident. If in the morning, the exchange was criticized for the information about the incident, then in the afternoon, they would be praised for how the press conference was handled.

Instead of blaming Fujitsu, the provider of this transactional system, TSE executives bluntly accepted responsibility for the incident. Inquiries from the media were answered with relative ease, as well as discussed areas such as system architecture in high-tech terms.

The next day, the problem was fixed, and trading was back with. While the incident has passed without major issues, many questions have yet to be answered. One of them is whether a similar hardware failure happens to other stock markets.

Consult Bloomberg