MikeMirzayanov's blog

By MikeMirzayanov, history, 13 months ago, In English

This is a response to this post.

Originally, I intended to leave it as a comment, but the text turned out to be quite long. So I decided to publish it as a separate post.

Hi!

Let me start with the main idea: I don't think I ignore too many issues (though I do ignore some). In most cases, I do work on them, but these days I just don't feel like writing about it. That kind of communication takes a lot of energy — which I currently don't have.

Oh, it looks like you edited the beginning of the blog, and now it triggers less of an internal protest in me. Honestly, lately I've been in a difficult emotional state, and I just don't have the energy or desire to interact much. It's not just about Codeforces — it's generally harder for me to engage than it used to be. I won't go into details (health, relocation, family issues, changes in the nature of my work due to bots) — all of that affects my mental state and willingness to communicate.

I understand now why companies often have a spokesperson or a similar role. Responding properly often requires deep focus, careful wording, emotional energy, and switching context away from other tasks. At the current scale of the audience, reacting to everything directly is very resource-intensive. Some of the blog posts you mentioned I had completely missed and never saw.

Thank you for this post. Of course, this kind of silence on my part isn't right. Maybe I'll try to overcome myself and be more responsive in some cases. I understand how frustrating these issues can feel from the user side, especially when there's no visible explanation right away.

I've written it before, and nothing has changed: almost every day and night I'm doing a huge amount of work around the infrastructure. Unfortunately, I'm not very good at delegating (though I manage to do some of it, and I keep trying). The Codeforces infrastructure is quite extensive now, and things are constantly happening: equipment fails or needs upgrades, there are attacks, load patterns change and the code or settings need to be adapted, we do new things (like the recent blitz round, which didn't run itself), rounds require attention and time, internet disruptions or blocks lead to infrastructure changes, investigations and fixes are needed for reports of unusual behavior or bugs, and then there's work with sponsors, clients, and special projects. I'm working on strengthening the team, but it's hard — many of my plans were derailed in 2022. Also, I don't really take vacations — I'm always on call and handling issues. It's been like that for more than 15 years. That's why I sometimes get into a state where I try to conserve energy and avoid doing that drain it. For example, I've started writing less on the site.

Now, some context. Over the past year (or even longer), I've had to put in significantly more effort just to keep the site running and to avoid total data scraping by AI bots. They ignore both robots.txt and site-specific HTTP headers. It seems that more than half of the website downtime incidents are caused by the unnatural and irregular load they generate. Something happens every single day. For example, yesterday I had to ban several thousand bots that registered for a round. That took several hours. There have been many similar incidents before (you usually don't notice them because I react quickly), and their absence during rounds is thanks to the much-hated Cloudflare human verification checks. These bots aren't simple scripts — they're part of large-scale, organized operations. Some even mimic human behavior well enough to get past CAPTCHAs.

I recently saw this article from Wikipedia, and I can assure you that something very similar is happening on Codeforces at a noticeable scale.

I wish I could live in a world where I didn't have to wrap everything in protections and CAPTCHAs, but the world doesn't work that way.

I've often heard suggestions like “let's add SMS verification” or even more extreme ideas like video verification. I'm not optimistic about those. I like the idea of basic privacy, and SMS verification significantly breaks that. Also, international SMS delivery isn't reliable. And even that wouldn't solve the problem — there are large pools of phone numbers available for rent that attackers can use. Practice shows that bot authors work hard, invest effort, and spend considerable resources. It's common for attacks to involve thousands of IP addresses and registered accounts — and we have to deal with that somehow.

Seems I've started rambling a bit, so let me briefly respond to the specific points from your post.

1. Unreasonable delay for viewing submissions

It's true — I didn't respond directly to some of these concerns. I do not allow bots to scrape the site to train their models on user submissions. Fighting bots is hard, and we have to rely on heuristics that evolve over time. But I've been working hard to make these problems less visible to regular users. For example, even the title “Unreasonable delay for viewing submissions” no longer applies. It came from this post, which no longer reflects the current reality. Why? Because I changed the behavior months ago, partly in response to that complaint. I continue testing new approaches and adjusting implementations.

2. Cloudflare misconfiguration

I'm confident that things have greatly improved since this post. Unfortunately, we can't hold rounds without Cloudflare or a similar service. It protects us from bots and from DDoS attacks — some of which have lasted for weeks and were quite powerful, though you likely didn't even notice. Thanks to Cloudflare.

On the other hand, yes, it does sometimes get in the way. But if you remember how things used to be, you'll see we've made huge progress. Nowadays we rarely enable aggressive protections during rounds, which used to be standard practice. I don't see many “I'm tired of Cloudflare during the round” comments anymore. That's because a lot has been reconfigured — and we continue to improve.

By the way, in the context of that discussion, I responded with this post, and the low number of complaints (which I do read and use to adjust settings) suggests that many issues have been resolved.

3. IP ban + blocked by the administrator

Due to changing bot patterns, I had to temporarily restrict some IPs and accounts. But within a day or two I improved the strategy, and such reports mostly disappeared. If you look at this post, almost all messages are from two weeks ago — that incident has already been resolved.

If you expect me to comment in detail on every such case, I'd end up spending the entire day writing replies instead of implementing fixes and improvements.

The second post (link) didn't get much traction either, because I changed the strategy and started showing a message explaining the reason — maybe not always perfectly accurate, but usually right.

There are also those GIFs you shared about authentication. They aren't directly related to this issue. I'm honestly happy that AtCoder can fend off bots with a simple CAPTCHA. But Codeforces faces much stronger attacks. I've had to manually undo actions from bots that got past such CAPTCHAs dozens of times.

4. Recently, your account was used to crawl

This is essentially a continuation of the previous point. Now, whenever possible, I avoid fully restricting IPs or account access — only certain pages are blocked, and I try to show a warning message explaining why. The warnings are usually accurate, and even when they're not, you can generally understand the cause. As I said earlier, I'm constantly adapting detection methods, and I hope these restrictions will become less noticeable over time.

5. Hacking issues

If you had to dig up a 1.5-year-old post to find an example of this, maybe it's not so bad after all.

I don't recall that post clearly. Maybe the author didn't leave a comment under the round announcement or tag me in the post. Also, I vaguely remember banning this person for repeated behavior violations — maybe that's why somehow I skipped it. The round in question sounds like it had major problems — but note that we did pay attention to it, and there haven't been widespread similar issues since.

As for this comment, I flagged the bug to the coordinator and KAN. I think that's the appropriate level of involvement for me in such a situation. Looks like the team investigated and added updates to the announcement. Of course, mistakes in problems are unfortunate, but they happen — and always will (hopefully less and less). In this case, reasonable effort was made to understand the impact, rating was rolled back where needed, and everything was communicated transparently. What's the problem?

6. API issues

You might have noticed that it doesn't work for a long period of time. This is because Mike enabled cloudflare even for apis during contest, which got a workaround in meooow25/carrot#67

Does /api really get blocked by Cloudflare? I'm pretty sure that was fixed a long time ago.

It worked for a while again until he started to take down the user.ratedList api during contest too

Maybe I'm mistaken, but I believe everything is functioning as intended. I tested availability during today's contest, and it all worked fine. Could you point me to where exactly this issue was raised, if it still exists?

MikeMirzayanov did make a comment about this api 3 years ago. However, it seems like there was no further communication.

That's true — I forgot about that. Were there any other reminders I might have missed?

This post ended up much longer than I planned, but I just wanted to share what's been going on behind the scenes and why some things are the way they are. I really do appreciate the feedback and the care many of you show for Codeforces — it matters. I know I'm not always quick to respond or explain things, but I do read what's being said and try to make things better, bit by bit. Thanks for sticking around and being part of all this.

  • Vote: I like it
  • +1425
  • Vote: I do not like it

| Write comment?
»
13 months ago, hide # |
 
Vote: I like it +74 Vote: I do not like it

Thank you so much for taking the time to update us and address the recent issues with Codeforces. I personally appreciate you taking the time to address the common queries despite dealing with some health and personal challenges. Your dedication and commitment, even in tough times, is both inspiring and deeply valued. Wishing you all the best and looking forward to seeing Codeforces back on track!

Finally huge shoutout to Mike!

»
13 months ago, hide # |
 
Vote: I like it +16 Vote: I do not like it

I agree with all your points. In the recent blog about the red coder cheating, when I saw one of his recent codes that was skipped I was pleasantly surprised at the anti plag/LLM system your team had put together. Many times we don't give you enough credit that you deserve, while only remembering about things that didn't go our way.

Saying this, I wanted to add to your point regarding SMS verification. I think phone numbers are pretty 'public' and you can experiment with the idea, maybe make it optional for accounts or something (I know a lot of thinking has to go into it and there would be better people than me thinking about the issue).

Thank you mike for the wonderful site <3 and the community may criticize you at times but we all love you!!

»
13 months ago, hide # |
Rev. 2  
Vote: I like it +58 Vote: I do not like it

Thank you for your calm and detailed response. Although by looking at latest blogs it may seem that overall sentiment is on the negative side, but a lot of people, me included, appreciate your hard work even if we are not saying it out loud that often. That's why there are still tens of thousands of participants each round and that's why Codeforces is de-facto one of the state of the art benchmarks for the frontier AI. It's truly marvelous, thank you for that.

As for something that can help have you considered proof of work solutions like this https://github.com/TecharoHQ/anubis ? It may be a useful alternative against crawlers at least.

»
13 months ago, hide # |
 
Vote: I like it -241 Vote: I do not like it

This post is rewritten in chatGPT, no one uses '—'.

»
13 months ago, hide # |
 
Vote: I like it +115 Vote: I do not like it

Thank you for the post! I genuinely appreciate that you tried to explain all of this at once. This clarifies a lot of things. We didn't even have a clue about why all of these have been happening, and I'm glad now we got an official clarification on the reason.

I've noticed a lot things have changed over time, but they didn't seem to truly "solved" the issues because we have kept getting various restrictions one after another, and we can't really see what is actually improved. There is no way we can notice them without your announcement.

As mere users, we can only get feared on all the restrictions because we don't know how the system will further take action on us and how much we are internally flagged as malicious users. This is why it is needed to have a visible rule about them. We need to be able to know these to take actions within the rule, and be assured we won't get punished for using the website as long as we follow the rules.

About the restriction policies, maybe we can have a credit system for that? I think a similar system is already applied to plagiarism checks, where a more trusted participant only gets out of competition while others with less credit get a 0-solve rated participation and eventually get banned with multiple of them. Users who already normally participated in multiple contests are not likely to be bots, so it doesn't make much sense to keep doubting in them just for a small amount of submission page views. Also, your recently added message implies that our account was hacked so we should change our password, then it would also make sense to immediately remove the flag once we change our password (which is not the case so far, I tried).

It doesn't always have to be this long. Just a few words whenever something major has changed would suffice to make us rest assured. I hope we can keep getting updates from you in the future!

»
13 months ago, hide # |
 
Vote: I like it +37 Vote: I do not like it

About API issues: blogEntry.view doesn't return post text for about 6 years now or even more report

On submission view: could you please disable any checks when viewing your own submissions? Though it's pretty fast now it still requires some time and disturbs a lot

»
13 months ago, hide # |
 
Vote: I like it +7 Vote: I do not like it

thanks for replying, we appreciate it! i get how exhausting it must be to manage all the communication and fixes on your own, must be a lot. not to mention that codeforces isn’t what it used to be, it’s grown a lot with the user base booming like crazy, and new administrative + technical challenges, as you've mentioned yourself.

so, wouldn’t it be cool if the administrative and technical teams were also expanded a bit? delegating some tasks could lighten your load, Mike, and i think it’d do wonders for the community too, letting more folks pitch in to keep things rolling smoothly.

i’m no big shot either (just a random blue who enjoys competitive programming lol), but i feel like it’s less about the platform malfunctioning and more about how we look at it, and i think expansion, if possible, would help.

you’re doing incredible stuff! thanks for keeping this platform alive and kicking!

»
13 months ago, hide # |
 
Vote: I like it +19 Vote: I do not like it

Have you thought about hiring some guys for future for taking your place?

»
13 months ago, hide # |
 
Vote: I like it 0 Vote: I do not like it

Thank you so much for maintaining and keeping this amazing site alive. Without your efforts, Codeforces wouldn't be half of what it is today. Please take care of your mental health and don't let such comments affect you. We know how much effort you put in even though you don't owe anything to anyone and such comments just feel so ungrateful. Take care!!

»
13 months ago, hide # |
 
Vote: I like it -23 Vote: I do not like it

Thanks for everything, Sir. If you feel like community is not cooperating with you. Remember, you always have an option of deleting this site (lol).

»
13 months ago, hide # |
Rev. 3  
Vote: I like it +90 Vote: I do not like it

Thank you so much for the response Mike! I will go straight to the points:

1. Unreasonable delay for viewing submissions

Although the behavior certainly improved. There's still an edge case to consider:

If you open submissions at the same time codeforces decides to throw Please wait. Your browser is being checked, it can goes like this:

  • 1st tab: 3 loads to open
  • 2nd tab: 5 loads to open
  • 3rd tab: 3 loads to open
  • 4th tab: Hangs forever
  • 5th tab: 403 forbidden

After that I would have to turn on vpn to be able to view codeforces

2. Cloudflare misconfiguration

Nowadays we rarely enable aggressive protections during rounds, which used to be standard practice.

That's great to hear! I will talk more on this in the section 6.

3. IP ban + blocked by the administrator

Thank you for fixing the issue within a day or two but I think a small update on telegram like "Login issues are being fixed" wouldn't hurt.

You might also want to consider adopting a status page, like https://status.openai.com/. That way, if there's ever an issue, users can simply check status.codeforces.com for updates.

4. Recently, your account was used to crawl

I agree with djm03178's comment, a credit system sounds reasonable here.

5. Hacking issues

In this case, reasonable effort was made to understand the impact, rating was rolled back where needed, and everything was communicated transparently. What's the problem?

The problem was that dori hacking attempt should have succeeded but failed due to a bug in the interactor. He should've earned +100 points, but ended up with -50 points instead.

6. API issues

Does /api really get blocked by Cloudflare? I'm pretty sure that was fixed a long time ago.

Yes, I brought it up as one of the points about lack of communication. Although it might be fixed a long time ago, I would never know if you will decide to put cloudflare on /api again.

Maybe I'm mistaken, but I believe everything is functioning as intended. I tested availability during today's contest, and it all worked fine. Could you point me to where exactly this issue was raised, if it still exists?

The API user.ratedList is pretty unpredictable, sometimes it works sometimes it just throws 404 not found. The issue was raised here: meooow25/carrot#74 on 18 Feb and in my DM with meooow on 20 March, 11 March, 18 Feb, 13 Feb, ...

That's true — I forgot about that. Were there any other reminders I might have missed?

Yes, please take a look at GIFs in Problem Statements are Broken. Thank you so much!!

»
13 months ago, hide # |
Rev. 2  
Vote: I like it +1 Vote: I do not like it

Reading this felt like a Dad saying "Son, let's have a talk" with the utmost empathy

»
13 months ago, hide # |
 
Vote: I like it +41 Vote: I do not like it

Thank you MikeMirzayanov and the whole CodeForces team for putting in the work to maintain the website! And thank you for telling us some of this behind-the-scenes stuff. This blog post explained a lot to me.

»
9 months ago, hide # |
Rev. 2  
Vote: I like it 0 Vote: I do not like it

It's okay sir we understand..you are a human too.