BOOBA's blog

Codeforces rating deflation?

By BOOBA, history, 5 years ago, In English

Today I tried to analyze past rounds on codeforces.
I wrote a primitive parser for the rating changes table and collected some statistics from it.
From the results I realized that rating deflation takes place.
For me it was not obvious, so I decided to analyze more and the tendency was confirmed.


Here are some statistics:
Round #635 (Div. 1): number of participants=1406, av. rating=2170, total change=-11777, av. change=-8.4
Round #635 (Div. 2): number of participants=14830, av. rating=1377.4, total change=-155190, av. change=-10.5

Div2 rounds are difficult to analyze due to the large number of participants, so later I checked recent div1 rounds.

Round #626 (Div. 1): number of participants=788, av. rating=2195, total change=-5689, av. change=-7.2
Round #631 (Div. 1): number of participants=1273, av. rating=2175, total change=-1058, av. change=-0.8
Round #633 (Div. 1): number of participants=1392, av. rating=2161, total change=-14764, av. change=-10.6

From the above statistics, it is noticeable that the number of participants is growing, and the average rating is gradually decreasing.
I think this is due to the ever-increasing popularity of the site (and programming in general). The counting system maintains balance in this way.
P.S. This is not a reason not to write rounds :)

  • Vote: I like it
  • +55
  • Vote: I do not like it

»
5 years ago, # |
  Vote: I like it +5 Vote: I do not like it

It would be very interesting to look up the same info for the Div. 2 rounds. I suppose that the average change would be higher than Div. 1 (it's just my assumption).

→ Reply
»
5 years ago, # |
← Rev. 2   Vote: I like it +8 Vote: I do not like it

I thought the system was designed such that the total rating change is always zero. Why isn't this the case?

→ Reply
  • »
    »
    5 years ago, # ^ |
    ← Rev. 2   Vote: I like it +14 Vote: I do not like it

    Remember that average rating of active users is much less than 1500.

    I do not know how that came to be. Whatever the reasons are, this creates a problem: introducing new users with a starting rating of 1500 all but guarantees that the average rating is inflating by ~100 rating per new user. Therefore, to evade rating inflation, ratings are artificially lowered.

    In normal times, Codeforces experiences regular growth in the number of active users. And these deflationary measures were probably fit towards keeping the average rating constant.

    But in normal times we are not, and Codeforces is experiencing a rapid onslaught of

    a) previously active users whose skills have eroded, with their high old ratings;

    b) new users who are less skilled than a typical ~1500 user.

    This means the old parameters may even not prove enough to prevent inflation in the short term.

    Lowering the starting rating, and increasing the K-factor (or equivalent) for new users, could have been a possible workaround. I'm not sure what helps now, when growth has already taken place, and the converging will inflate the ratings over the next several months.

    → Reply
    • »
      »
      »
      5 years ago, # ^ |
        Vote: I like it +14 Vote: I do not like it

      (Also, there is a nontrivial amount of accounts who register, take part in a small amount of contests, and then never come back.

      I guess those also contribute in the inflationary direction for real active users, but I wouldn't bet on it being the dominant term. Is there evidence that I am mistaken?)

      → Reply
    • »
      »
      »
      5 years ago, # ^ |
        Vote: I like it +18 Vote: I do not like it

      I completely agree.

      In fact, I think that the average performance of a first time user in a contest is actually substantially lower than the average ~1400. Thus, with the influx of new users, it is natural to assume that old users have their ratings slightly pushed up.

      From a different perspective, rating is simply the relative skill of a user compared to other users on codeforces. As a result, when a new user joins that is less skilled than you, your rating will go up. On the flip side, if no more new users joined, everyone would just keep improving, and it would be much harder to gain rating as you have to increase your relative position. Therefore, there needs to be some kind of balance between the number of new users that join and the rate at which everyone is improving in order to reach an equilibrium. In the past, codeforces has actually done quite a good job of maintaining this, but now, with so many new users, it seems like it is leading to rating inflation.

      Perhaps the fact that the net rating gain in div1 is negative as shown in this post is an attempt to offset this rating inflation?

      → Reply
»
5 years ago, # |
  Vote: I like it +5 Vote: I do not like it

Small sample size. 4 rounds and a couple of weeks between them is not much. Also note that your data shows the average rating increased from 633 to 635.

In fact, the rating is inflating, which is seen when you observe ratings of top users from a couple years ago and now.

→ Reply