MikeMirzayanov's blog

Black Day of Polygon: 02.06.2016
By MikeMirzayanov, 8 years ago, translation, In English

Today, June 2, 2016 year, there was a technical failure, which I will remember for a long time. Several factors led to a serious malfunction of the Polygon system.

The fact that in the middle of May Polygon was transferred to servers located in Mail.Ru data centers. To transfer with almost no downtime (Polygon has a lot of data, about a terabyte) lsyncd utility has been used, which quickly pushed the file changes from the old server to the new one.

Today there was a reboot of the old server, which forced the unplanned start of lsyncd daemon. As a result it did the synchronization of data from the old server to the new again. So, data stored in the file system have been rolled back to the transfer time (May 13). The most unlucky fast that, unlike the old server where the backups system was set up well and reliable, on the new server, I quickly managed to set up only replication using the same lsyncd, which does not help in the incident of files deletion (as it forces the removals on the replica). Obviously, RAID-1 disks didn't save the data.

As a result, all file changes in the Polygon system made in three-week period were lost. Some information (statements, etc) is stored in the database, so accessible.

This night I urgently implemented the special link "Scraps" to download the archive with the available data. If a problem has been used in the Codeforces infrastructure, the packages remain in its cache of the judging system, they are available in archive.

I feel lousy realizing that my lack of attention to detail has led to such consequences. I distinctly remember the black day of Codeforces — and as a result many things have been much revised (everywhere mirrored disk arrays, replicas and backups). Unfortunately, this time after moving to new hardware I did not return all scripts and here is the result. Problem writers please accept my deepest apologies for my mistake.

  • Vote: I like it
  • +589
  • Vote: I do not like it

Comments Comments (17)
| Write comment?
»
8 years ago, # |
  Vote: I like it +135 Vote: I do not like it

That's why you shouldn't trust polygon and make your own backups.

→ Reply
»
8 years ago, # |
  Vote: I like it +32 Vote: I do not like it

Maybe we can utilize copy-on-write snapshots to eliminate this kind of mistake?

→ Reply
»
8 years ago, # |
  Vote: I like it +58 Vote: I do not like it

Thank you so much for "Scraps". It contains statements (even one editorial I've written recently) and scripts. Files with codes (solutions, generators, validators) I store on my computer, so I think I lost literally only tags marked in some problems.

But yeah, back-ups are crucial for Polygon. I hope that nobody lost a lot of work, so the main effect of this event is that Polygon will be safer and thus better.

Right now it's impossible to hit "Start" or "Continue" next to any problem in Polygon (hitting it leads to an empty page). When will it be available?

→ Reply
  • »
    »
    8 years ago, # ^ |
      Vote: I like it +5 Vote: I do not like it

    Currently it is available to edit any problems that were created before May, 13.

    → Reply
»
8 years ago, # |
← Rev. 2   Vote: I like it +33 Vote: I do not like it

Do having something like git local repository for polygon can help us , speed up our works and prevent such events ?

behind its speed for users , in such bad events we may lost Invocations,Packages,... but we are always sure that last version of important codes are available in our computer and everything can be recreated.

→ Reply
  • »
    »
    8 years ago, # ^ |
      Vote: I like it +56 Vote: I do not like it

    Using my client you can automatically synchronize most of files between polygon and local computer. It's not as good solution, as local git, but anyway.

    → Reply
  • »
    »
    8 years ago, # ^ |
    ← Rev. 2   Vote: I like it +26 Vote: I do not like it

    Why downvotes?

    Integrating polygon with git server would be great.

    • It will support conflicts resolution (finally).

    • It will be possible to work on problems without network.

    • Events like this will be impossible.

    → Reply
»
8 years ago, # |
← Rev. 2   Vote: I like it +91 Vote: I do not like it

You can take all changes from e-mail if you don't have it in Scraps.

→ Reply
»
8 years ago, # |
  Vote: I like it +24 Vote: I do not like it

Should we create new problems and copy the statements, etc. from the "Scraps" or should we wait to be able to edit the problems again? Because right now clicking Start in a corrupted problem leads to a blank page.

→ Reply
»
8 years ago, # |
  Vote: I like it +88 Vote: I do not like it

Don't get sad :(

We love you <3

→ Reply
»
8 years ago, # |
  Vote: I like it +24 Vote: I do not like it

Hey don't worry we all make mistakes :)

→ Reply
»
8 years ago, # |
← Rev. 2   Vote: I like it -67 Vote: I do not like it

+428 upvotes!! really? I thought people can get sad or even angry with such loss. Is there any secret VoteForces working on some blogs?

Never mind Mike, I accept your apology, so won't give you downvote this time.

→ Reply
  • »
    »
    8 years ago, # ^ |
    ← Rev. 2   Vote: I like it +39 Vote: I do not like it

    Do you really think that Mike would cheat to get upvotes? He isn't even being shown in table of top contributors, although he has largest contribution. All community of codeforces really empathize with him. And less that everyone can do is to upvote this post or write a positive comment to cheer him up. That's why this post got so many upvotes, I think.

    → Reply
    • »
      »
      »
      8 years ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      you mean largest contribution right? Cause I can't see his rating on his profile.

      → Reply
      • »
        »
        »
        »
        8 years ago, # ^ |
          Vote: I like it 0 Vote: I do not like it

        Yes, I mean largest contribution. My mistake

        → Reply
  • »
    »
    8 years ago, # ^ |
      Vote: I like it +3 Vote: I do not like it

    out of curiosity why are you unrated for 6 months and what is your purpose?

    → Reply