Today, June 2, 2016 year, there was a technical failure, which I will remember for a long time. Several factors led to a serious malfunction of the Polygon system.
The fact that in the middle of May Polygon was transferred to servers located in Mail.Ru data centers. To transfer with almost no downtime (Polygon has a lot of data, about a terabyte) lsyncd utility has been used, which quickly pushed the file changes from the old server to the new one.
Today there was a reboot of the old server, which forced the unplanned start of lsyncd daemon. As a result it did the synchronization of data from the old server to the new again. So, data stored in the file system have been rolled back to the transfer time (May 13). The most unlucky fast that, unlike the old server where the backups system was set up well and reliable, on the new server, I quickly managed to set up only replication using the same lsyncd, which does not help in the incident of files deletion (as it forces the removals on the replica). Obviously, RAID-1 disks didn't save the data.
As a result, all file changes in the Polygon system made in three-week period were lost. Some information (statements, etc) is stored in the database, so accessible.
This night I urgently implemented the special link "Scraps" to download the archive with the available data. If a problem has been used in the Codeforces infrastructure, the packages remain in its cache of the judging system, they are available in archive.
I feel lousy realizing that my lack of attention to detail has led to such consequences. I distinctly remember the black day of Codeforces — and as a result many things have been much revised (everywhere mirrored disk arrays, replicas and backups). Unfortunately, this time after moving to new hardware I did not return all scripts and here is the result. Problem writers please accept my deepest apologies for my mistake.
