rui_er's blog

More and More Suspicious Participants in AtCoder Beginner Contests!
By rui_er, history, 6 months ago, In English

Hello, CodeForces.

As I can't find any discussion thread in AtCoder, and the announcements of AtCoder contests are also posted here, I decided to post this blog in CodeForces.

Last time (ABC355), I reported a suspicious participant toyuzuko who solved problem ABCD in 51 seconds. They also did the similar thing in ABC354.

This time (ABC356), there are much more suspicious participants.

The average first-AC time (in seconds) of ABC347~ABC353 is in the following table:

Contest A B C D E F G
ABC347 21 45 93 244 177 300 824
ABC348 15 37 72 302 236 495 430
ABC349 23 67 54 116 598 139 634
ABC350 26 47 102 74 240 489 335
ABC351 22 23 106 352 199 122 902
ABC352 27 43 49 90 196 989 188
ABC353 24 79 172 88 206 1161 313
(minimum) 15 23 49 74 177 122 188
(prefix sum) 15 38 87 161 338 460 648

Note that to get first-AC in a problem doesn't require you to solve all the problems before it. You can just skip to the problem you want to get first-AC and manage to solve it at once. So, the least reasonable time to solve a subset of problems is at least the sum of the first-ACs.

To make sure there are fewer exceptions, I will call all the participants who solved any subset of problems faster than the minimum expected time "suspicious".

Now, let's take a look at (some of) the suspicious participants who solved at least 4 problems at last:

Some other participants seem to be suspicious, too. They solved a subset of problems just 10 seconds slower than the sum of the minimum first-AC time in the above table. I think it's impossible to have all the easy problems in a single round, but I didn't list them here, just to make sure there are fewer exceptions.

I guess they probably used some AI to generate the code for them after seeing what happened in ABC355. If my guess is true, I'm impressed with the ability of AI. I'm also worried that more and more people will use AI to generate the code and solve easy problems in the contest. If everyone can solve ABCD by simply copy-pasting AI's code, what's the point of having AtCoder Beginner Contests? Why not just delete ABCD and add some new problems to make an AtCoder Regular Contest?

You can leave your opinion in the comments.

  • Vote: I like it
  • +317
  • Vote: I do not like it

»
6 months ago, # |
  Vote: I like it +41 Vote: I do not like it

I feel scared that AI will be stronger than many people.

→ Reply
»
6 months ago, # |
  Vote: I like it +70 Vote: I do not like it

Easy problems in abc are designed to require no algorithm but programming language.

AI masters programming languages very well and also know a little about algorithms.

It's normal for AI to solve these problems, or AI would be too weak.

→ Reply
  • »
    »
    6 months ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    You're right.

    → Reply
  • »
    »
    6 months ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    But in Span for 5 Seconds he submitted ABC in today's contest like even copy-pasting and making the solution will take at least some more time he/she submitted A in the 16th sec, B in the 19th sec, and C in the 21st second which is suspicious.

    → Reply
    • »
      »
      »
      6 months ago, # ^ |
        Vote: I like it +5 Vote: I do not like it

      maybe created some extension

      → Reply
    • »
      »
      »
      2 months ago, # ^ |
        Vote: I like it +3 Vote: I do not like it

      Maybe multiple people are using the same account.

      → Reply
»
6 months ago, # |
  Vote: I like it 0 Vote: I do not like it

If anyone has access to gpt4, he can give the tasks to the model and submit the code on Atcoder to check if it gives AC or not.

→ Reply
»
6 months ago, # |
  Vote: I like it +12 Vote: I do not like it

It's actually confirmed that they used AI, because if you see their submissions such as https://atcoder.jp/contests/abc355/submissions/54073783 and https://atcoder.jp/contests/abc355/submissions/54073778, it says "Generated by gpt4-o" at the top.

→ Reply
  • »
    »
    6 months ago, # ^ |
    ← Rev. 2   Vote: I like it 0 Vote: I do not like it

    I guess they don't have time to remove that, because they're in a hurry to submit as it is, even without counting the time to remove it.

    → Reply
    • »
      »
      »
      6 months ago, # ^ |
        Vote: I like it +58 Vote: I do not like it

      i dont see why they should remove it either, its not like its disallowed

      → Reply
  • »
    »
    6 months ago, # ^ |
    ← Rev. 2   Vote: I like it 0 Vote: I do not like it

    This validates my comment above. Its not a good news tbh. It may be the start of the end of ABC contests. I wonder how gpt4 performs on codeforces div2s and div1s.

    → Reply
    • »
      »
      »
      6 months ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      Well, ABC are mostly for beginners anyway, and it only solves the very standard problems, which are usually not accepted on Div 2s (the problems on Div 2 may be standard for GM+ but they are still novel from the perspective of an AI which can only solve them if some exact same problem has appeared before).

      → Reply
      • »
        »
        »
        »
        6 months ago, # ^ |
          Vote: I like it 0 Vote: I do not like it

        But yes, this may be the start of the end of ABC contests, at least for $$$r \in [0, 800]$$$.

        → Reply
        • »
          »
          »
          »
          »
          6 months ago, # ^ |
            Vote: I like it +1 Vote: I do not like it

          Honestly I think we're probably pretty close to having a freely available AI that is able to solve most 1600-1800 problems, since even those aren't usually too involved.

          I think there might be classes of problems that are simple for humans but unusually hard for AI, like ones which require some kind of intuition or visualization that AI is not great at.

          → Reply
          • »
            »
            »
            »
            »
            »
            2 months ago, # ^ |
              Vote: I like it 0 Vote: I do not like it

            AI achieves 1800 CF rating

            AI gets Gold at IOI 2024 but performs poorly on relatively easier Day 2B

            Nice prediction : )

            → Reply
            • »
              »
              »
              »
              »
              »
              »
              2 months ago, # ^ |
                Vote: I like it 0 Vote: I do not like it

              AI gets gold in IOI is not accurate. It only got gold when allowed 10k submissions.

              → Reply
              • »
                »
                »
                »
                »
                »
                »
                »
                2 months ago, # ^ |
                  Vote: I like it 0 Vote: I do not like it

                True, it still flops at Day 2B with 10k submits...

                → Reply
              • »
                »
                »
                »
                »
                »
                »
                »
                2 months ago, # ^ |
                  Vote: I like it -9 Vote: I do not like it

                This 10k submissions can be easily avoided by using local stress testing. The AI needs to be able to generate the brute force solutions, which seems reasonable. There could be some strategy like submitting the brute force and making sure you get TLE instead of WA to guarantee correctness, etc. Also at IOI you do not have limits of submission I think, so AI actually played by the rules.

                → Reply
                • »
                  »
                  »
                  »
                  »
                  »
                  »
                  »
                  »
                  2 months ago, # ^ |
                    Vote: I like it 0 Vote: I do not like it

                  At IOI you're limited to 50 submissions per task actually. They mention the (lower) score of the model when respecting this constraint.

                  They mention the model used testcases it created itself in order to develop the solutions. I'm not sure if it was allowed to run actual code against a brute force or if it just checks its logic "by hand".

                  But yeah, I do think it might be a matter of time before we're absolutely destroyed by bots lol. If they didn't test the model allowing it to run code, I'm afraid it already is much better than we think.

                  → Reply
                • »
                  »
                  »
                  »
                  »
                  »
                  »
                  »
                  »
                  2 months ago, # ^ |
                    Vote: I like it 0 Vote: I do not like it

                  I think there might be limitations that AI cannot overcome while humans can. A big part of CP problems have a low level of originality, but I expect AI to struggle with original ad hoc problems. Maybe this is wishful thinking and AI will destroy us in every aspect, but I dont think so.

                  → Reply
                • »
                  »
                  »
                  »
                  »
                  »
                  »
                  »
                  »
                  2 months ago, # ^ |
                    Vote: I like it 0 Vote: I do not like it

                  I just learned here that the AI generated thousands of submissions, but it only submitted 50. Which is according to the rules. It did use a stress test strategy.

                  → Reply
»
6 months ago, # |
  Vote: I like it 0 Vote: I do not like it

Actually, some of they are use ChatGPT to get AC for some easiest problems.
Currently, there are no punishment rules for using AI, so skipping the first 2 or 3 problems in a few seconds are kind of "good" strategy. (background: AI has good compatibility to resolve ABC problems because some of them are very simple settings and solutions)
Recently (especially after launching gpt-4o) there have been many pros and cons about allowing or banning to use of ChatGPT in the Japanese community, and the rules regarding AI may be changed through the result of this discussion. Personally, I want to know the opinion of CF community members about this issue.

→ Reply
  • »
    »
    6 months ago, # ^ |
      Vote: I like it +8 Vote: I do not like it

    I think allowing AI is not a problem. Firstly, anyone can plug a given problem into ChatGPT, so its not giving an unfair advantage to particular users. Secondly, AI can't solve any problems with even somewhat challenging algorithmic thinking, so it really doesn't impact any serious competitors.

    → Reply
    • »
      »
      »
      6 months ago, # ^ |
      ← Rev. 3   Vote: I like it +11 Vote: I do not like it

      Thanks for reply!
      Exactly because of it's almost not a problem for ARC or above, serious competitors are almost not care this situation in the difficult round (and I also don't hope to be do hard banning of AI).
      But in ABC, the situation is different. In ABC356, fastest ABCD gets about 1400perf so every green or below coder "shold" use this method. From Japanese community, it seems this strategy problem and "lose to ChatGPT" itself demotivated them and some of community members think this isn't healthy.
      In addition, the thing that ABC focuses on education and guideline rather than competition makes this problem more complex, then switch ABC to more competitive (for example, make them more ad-hoc) is not a solution...

      → Reply
      • »
        »
        »
        »
        6 months ago, # ^ |
          Vote: I like it 0 Vote: I do not like it

        I think that anybody can very quickly become better at problem solving than AI in its current state, so I'm not too worried. It should not take more than a month or two of serious practice.

        → Reply
  • »
    »
    6 months ago, # ^ |
      Vote: I like it +3 Vote: I do not like it

    I has seen someone with the same id as yours took part in some Luogu contests. However, I think you should pay more attention to Luogu's punishment for AI, because it's really useful and powerful. Using AI is the most unfair behavior for every participant. Generally speaking, I support that AtCoder should inprove the power of punishment for AI (such as block the account).

    → Reply
»
6 months ago, # |
  Vote: I like it 0 Vote: I do not like it

if there was prize money, there would be wayyy more cheaters

→ Reply
  • »
    »
    6 months ago, # ^ |
      Vote: I like it +2 Vote: I do not like it

    Actually some of ABC has some prizes for Japanese, so <1min ABCD have an advantage to earn them(though, to earn prizes, in most case contestants must solve F or G and in such difficulty ChatGPT isn't strong enough).
    But please note, with current ruleset, they are not cheaters.

    → Reply
    • »
      »
      »
      6 months ago, # ^ |
        Vote: I like it -10 Vote: I do not like it

      should we be scared of AI ?

      → Reply
»
6 months ago, # |
  Vote: I like it +27 Vote: I do not like it

The rule of ABC has been changed. Please see this post.

→ Reply
  • »
    »
    6 months ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    I think this is something great. I hope codeforces does this too, many people from Div3 and Div2 are passing solutions using GPT models too thesedays.

    → Reply
  • »
    »
    6 months ago, # ^ |
    ← Rev. 2   Vote: I like it 0 Vote: I do not like it

    It is prohibited to directly input the text or images of problem statements provided by AtCoder into generative AI or other programs.

    How are they even gonna check if a participant does it ?

    → Reply
»
6 months ago, # |
  Vote: I like it 0 Vote: I do not like it

Actually, it works only for too easy A and B type problems after that it fails, so we need not to fear I know cheating is a concern by cheater can never learn and grow if they keep cheating.

→ Reply
»
5 months ago, # |
← Rev. 2   Vote: I like it +6 Vote: I do not like it

Finally, this problem has been solved. You're our hero.

But there is another problem, which is "cooperation among multiple people" in the annoucement. The announcement states that it's "impossible" to detect it, but we can do our best to prevent it, such as by improving the anti-cheating system. In fact, there're more cases where multiple participants participate in a contest and share code simultaneously than when multiple individuals participate with one account! I've seen many incidents of cheating on social media (QQ, private message of Luogu or something else), like this one (fortunately he didn't succeed):

pktJHpD.png

Translation: "I want the solution of problem D! I cannot debug it, I'm crazy!!! And I also want solutions for problem E and F!!!"

Some people's tests indicate that AtCoder's anti-cheating system isn't perfect; some extremely similar codes won't be discovered (at least this kind of thing is highly unlikely to happen on CF; I can't disclose more details because I'm concerned that some people may imitate these cheating behaviors). Maybe it's necessary for AtCoder to solve this problem.

→ Reply
»
5 months ago, # |
← Rev. 2   Vote: I like it -25 Vote: I do not like it

Deleted

→ Reply