The incident occurred after a software update on the university"s supercomputer crashed. The backup software update accidentally deleted 34 million important files over two days.
The cause was a faulty script, whose original purpose was to delete old data, including unimportant log files on the Cray/HPE supercomputer, which were part of a software update.
Instead of deleting old files that are no longer in use, the update removes important research results that reach 77TB in size from the /LARGE0 backup disc, which occurred on December 14-16, 2021.
But this number is smaller than the initial estimate, which was originally estimated at 100TB. The reason is that the problematic software update deletes almost all files that are older than 10 days.
Still, the loss of that 77TB of research data would be huge. Because 34 million deleted files impacted 14 research groups.
Although Kyoto University did not reveal what data was deleted, they confirmed that there were 4 groups of researchers whose research data could not be recovered at all, as quoted by detikINET from Techspot, Thursday (6/1/2022).
The supplier of the supercomputer used by the university is Hewlett Packard (HPE) Japan, which has claimed 100% responsibility for the incident, which then sent an apology letter to the university.
According to HPE, the software update contains a script that has been modified to improve visibility and readability. However, HPE was not aware of the side effects of the modified script, which resulted in the deletion of millions of files from the supercomputer"s backup storage.
After this incident, Kyoto University temporarily stopped the backup process on their supercomputer. The goal is to make improvements and prevent similar things from happening in the future. One way is to add a backup mirror, and also do small backups periodically.
