My Reply on "Mike, please stop ignoring your userbase"

Revision en3, by MikeMirzayanov, 2025-04-06 13:09:02

This is a response to this post.

Originally, I intended to leave it as a comment, but the text turned out to be quite long. So I decided to publish it as a separate post.

Hi!

Let me start with the main idea: I don't think I ignore too many issues (though I do ignore some). In most cases, I do work on them, but these days I just don't feel like writing about it. That kind of communication takes a lot of energy — which I currently don't have.

Oh, it looks like you edited the beginning of the blog, and now it triggers less of an internal protest in me. Honestly, lately I've been in a difficult emotional state, and I just don't have the energy or desire to interact much. It's not just about Codeforces — it's generally harder for me to engage than it used to be. I won't go into details (health, relocation, family issues, changes in the nature of my work due to bots) — all of that affects my mental state and willingness to communicate.

I understand now why companies often have a spokesperson or a similar role. Responding properly often requires deep focus, careful wording, emotional energy, and switching context away from other tasks. At the current scale of the audience, reacting to everything directly is very resource-intensive. Some of the blog posts you mentioned I had completely missed and never saw.

Thank you for this post. Of course, this kind of silence on my part isn't right. Maybe I'll try to overcome myself and be more responsive in some cases. I understand how frustrating these issues can feel from the user side, especially when there's no visible explanation right away.

I've written it before, and nothing has changed: almost every day and night I'm doing a huge amount of work around the infrastructure. Unfortunately, I'm not very good at delegating (though I manage to do some of it, and I keep trying). The Codeforces infrastructure is quite extensive now, and things are constantly happening: equipment fails or needs upgrades, there are attacks, load patterns change and the code or settings need to be adapted, we do new things (like the recent blitz round, which didn't run itself), rounds require attention and time, internet disruptions or blocks lead to infrastructure changes, and then there's work with sponsors, clients, and special projects. I'm working on strengthening the team, but it's hard — many of my plans were derailed in 2022. Also, I don't really take vacations — I'm always on call and handling issues. It's been like that for more than 15 years. That's why I sometimes get into a state where I try to conserve energy and avoid doing that drain it. For example, I've started writing less on the site.

Now, some context. Over the past year (or even longer), I've had to put in significantly more effort just to keep the site running and to avoid total data scraping by AI bots. They ignore both robots.txt and site-specific HTTP headers. It seems that more than half of the website downtime incidents are caused by the unnatural and irregular load they generate. Something happens every single day. For example, today I had to ban several thousand bots that registered for a round. That took several hours. There have been many similar incidents before (you usually don't notice them because I react quickly), and their absence during rounds is thanks to the much-hated Cloudflare human verification checks. These bots aren't simple scripts — they're part of large-scale, organized scraping operations. Some even mimic human behavior well enough to get past CAPTCHAs.

I recently saw this article from Wikipedia, and I can assure you that something very similar is happening on Codeforces at a noticeable scale.

I wish I could live in a world where I didn't have to wrap everything in protections and CAPTCHAs, but the world doesn't work that way.

I've often heard suggestions like “let's add SMS verification” or even more extreme ideas like video verification. I'm not optimistic about those. I like the idea of basic privacy, and SMS verification significantly breaks that. Also, international SMS delivery isn't reliable. And even that wouldn't solve the problem — there are large pools of phone numbers available for rent that attackers can use. Practice shows that bot authors work hard, invest effort, and spend considerable resources. It's common for attacks to involve thousands of IP addresses and registered accounts — and we have to deal with that somehow.

Seems I've started rambling a bit, so let me briefly respond to the specific points from your post.

1. Unreasonable delay for viewing submissions

It's true — I didn't respond directly to some of these concerns. I do not allow bots to scrape the site to train their models on user submissions. Fighting bots is hard, and we have to rely on heuristics that evolve over time. But I've been working hard to make these problems less visible to regular users. For example, even the title “Unreasonable delay for viewing submissions” no longer applies. It came from this post, which no longer reflects the current reality. Why? Because I changed the behavior months ago, partly in response to that complaint. I continue testing new approaches and adjusting implementations.

2. Cloudflare misconfiguration

I'm confident that things have greatly improved since this post. Unfortunately, we can't hold rounds without Cloudflare or a similar service. It protects us from bots and from DDoS attacks — some of which have lasted for weeks and were quite powerful, though you likely didn't even notice. Thanks to Cloudflare.

On the other hand, yes, it does sometimes get in the way. But if you remember how things used to be, you'll see we've made huge progress. Nowadays we rarely enable aggressive protections during rounds, which used to be standard practice. I don't see many “I'm tired of Cloudflare during the round” comments anymore. That's because a lot has been reconfigured — and we continue to improve.

By the way, in the context of that discussion, I responded with this post, and the low number of complaints (which I do read and use to adjust settings) suggests that many issues have been resolved.

3. IP ban + blocked by the administrator

Due to changing bot patterns, I had to temporarily restrict some IPs and accounts. But within a day or two I improved the strategy, and such reports mostly disappeared. If you look at this post, almost all messages are from two weeks ago — that incident has already been resolved.

If you expect me to comment in detail on every such case, I'd end up spending the entire day writing replies instead of implementing fixes and improvements.

The second post (link) didn't get much traction either, because I changed the strategy and started showing a message explaining the reason — maybe not always perfectly accurate, but usually right.

There are also those GIFs you shared about authentication. They aren't directly related to this issue. I'm honestly happy that AtCoder can fend off bots with a simple CAPTCHA. But Codeforces faces much stronger attacks. I've had to manually undo actions from bots that got past such CAPTCHAs dozens of times.

4. Recently, your account was used to crawl

This is essentially a continuation of the previous point. Now, whenever possible, I avoid fully restricting IPs or account access — only certain pages are blocked, and I try to show a warning message explaining why. The warnings are usually accurate, and even when they're not, you can generally understand the cause. As I said earlier, I'm constantly adapting detection methods, and I hope these restrictions will become less noticeable over time.

5. Hacking issues

If you had to dig up a 1.5-year-old post to find an example of this, maybe it's not so bad after all.

I don't recall that post clearly. Maybe the author didn't leave a comment under the round announcement or tag me in the post. Also, I vaguely remember banning this person for repeated behavior violations — maybe that's why somehow I skipped it. The round in question sounds like it had major problems — but note that we did pay attention to it, and there haven't been widespread similar issues since.

As for this comment, I flagged the bug to the coordinator and KAN. I think that's the appropriate level of involvement for me in such a situation. Looks like the team investigated and added updates to the announcement. Of course, mistakes in problems are unfortunate, but they happen — and always will (hopefully less and less). In this case, reasonable effort was made to understand the impact, rating was rolled back where needed, and everything was communicated transparently. What's the problem?

6. API issues

You might have noticed that it doesn't work for a long period of time. This is because Mike enabled cloudflare even for apis during contest, which got a workaround in meooow25/carrot#67

Does /api really get blocked by Cloudflare? I'm pretty sure that was fixed a long time ago.

It worked for a while again until he started to take down the user.ratedList api during contest too

Maybe I'm mistaken, but I believe everything is functioning as intended. I tested availability during today's contest, and it all worked fine. Could you point me to where exactly this issue was raised, if it still exists?

MikeMirzayanov did make a comment about this api 3 years ago. However, it seems like there was no further communication.

That's true — I forgot about that. Were there any other reminders I might have missed?

This post ended up much longer than I planned, but I just wanted to share what's been going on behind the scenes and why some things are the way they are. I really do appreciate the feedback and the care many of you show for Codeforces — it matters. I know I'm not always quick to respond or explain things, but I do read what's being said and try to make things better, bit by bit. Thanks for sticking around and being part of all this.

History

 
 
 
 
Revisions
 
 
  Rev. Lang. By When Δ Comment
en4 English MikeMirzayanov 2025-04-06 13:14:39 130 (published)
en3 English MikeMirzayanov 2025-04-06 13:09:02 46
en2 English MikeMirzayanov 2025-04-06 01:22:40 297
en1 English MikeMirzayanov 2025-04-06 01:19:37 10450 Initial revision (saved to drafts)