Hi! Two weeks ago I made a survey link here: about making Codeforces better. In this blog I am sharing the results. Consider reading the previous blog first, but this one should be self-contained as well.
TLDR
The quality of Codeforces problems has increased over time, and people are mostly happy. Participants would probably be even happier with a greater variety of problem topics. It's important for setters to consider the contest as a whole, including the target audience of a given task. Some users wrote essays longer than the entire survey.
How to access the results
Before we start, I would like to invite everyone to take a look at the raw data. If you are keen on showing off your data analysis skills, please do so! I hope that some of us will manage to find more interesting insights taking into account preferences of users from various skill groups.
General info
I got 329 responses from users of various skill levels and experience. Most of the questions have more interesting results than others, so apologies if I have omitted something uninteresting. Plots and charts are presented in spoilers to increase readability.
There was one non-question which I purposely left blank to see some creative answers. I was moderately disappointed, but a honorable mention goes to Markadiusz for a comment "Kocham prezesa". And to an anonymous high-rated user who advised me to learn how to use Google forms. Okay, enough shitposting, let's get back to the point. This is probably the most important of all charts:
I honestly don't know whether this is an indicator of good rounds or not. Perhaps the statement is too opinionated and some people are leaning towards lower grades? I guess everyone has their own perception about this, but the later answers clearly show there is still a room for improvement.
Problem topics
Over $$$50\%$$$ of respondents agree with me that there are too many adhoc problems in recent rounds, and only $$$28\%$$$ disagree. Other problem topics are somewhat neglected, and geometry is who would have thought mostly undesired. Two questions about standard techniques in contests show nearly contradictory results, which probably is my fault for stating them too ambiguously, as some respondents pointed out, apologies.
I also put some more controversial statements to see how people react. While there is no consensus on "IMO with a keyboard", an overwhelming majority expresses their belief about existence of potential in algorithmic problems. I am excited to see what new algorithmic problems are featured in future contests. Oh and by the way, "Standard problems have been used to oblivion" is just a clickbaity title, not my actual opinion. In fact, my personal opinion is "strongly disagree" on this :)
Codeforces Contests as a whole
Sometimes a bad detail can ruin a work of art. And so is the case with competitive programming contests. In this paragraph, answers to questions concerning the contests are featured.
In my opinion, the most important outcome arises from the "Problem target" question. If you are a problemsetter, please consider considering this.
Some people did not see that the following two questions were intended as opposites of each other (keyword: primarily). I mean, how can you both primarily care about two different things?
Surprisingly many are not afraid of implementation challenges and would like to see some in contests. I expected most people to disagree with me on this, but was pleasantly surprised.
Stories in problems are generally unappreciated. Some people have pointed out in the comments that stories are good if they ease understanding of the problem, but short formal statements are strongly preferred over completely unrelated details that just waste your time.
Several people expressed their concerns over combined rounds. However, I believe not much can actually be done here. Sponsored rounds are preferred to be combined because of prizes, and there are many sponsored rounds (or thematic events like Goodbye 2021) recently.
My concerns about Educational Rounds have been validated. This can be a lost case, too (as they are rated for users under 2100 so in theory shouldn't contain terrible standardness). Maybe awoo and BledDest can comment on this?
And finally, there aren't many, but still exist people who would like to see hacks on system tests. if you are one of those, please let me know about your round so that I can skip it
Long comments from respondents
Some people have written essays longer than my survey. I decided to not list them here. The authors, if they wish to share their opinion, should be rewarded contribution for their work. If you are author of a meaningful long long comment that made a point, please post it under this blog. Or message me if you would like to do it anonymously.
Instead, I will cherry-pick some short sentences of particular wisdom. Possibly slightly edited.
- Problems where it's cousin timmy's toy encryption on a tree but with cute bounds or whatever are beyond too synthetic to care.
- Real programming looks a lot closer to implementing a to-brainfuck compiler, than counting paths in graphs by modulo 998whatever.
- You can't expect to solve global warming and cure cancer with some subset of the exact same classic tricks.
- I honestly want there to be more diversity, which includes "obscure topics" — I'm here to learn, not for rating, anyway. HOWEVER, accordingly, the editorials should be more clearly written.
- The harder the problem the more important it is that it hasn't appeared in some old contest, it makes no difference for d2E/d1C but very important for d1E/d1F
- I don't want codeforces to be atcoder 2.0.
I promise I did not write that answer - We often "prefer" topics we're good at and bash on everything else. For topics/problems we don't practice (we're bad at) we lose the opportunity to like them; so I think having a variety of concepts ("adhoc", math, implementation, geo, etc.) forces us to be more open-minded and potentially like them.
- Combined rounds are too lethal if you brick on one of the early problems.
- As a writer of AtCoder Beginner Contests, I have huge respect for CF round setters(Especially, ECR and Div3 writers) because CF's "interesting problem" is unique to other contests, and I cannot solve this kind of problems in other contests, and mass-produce such problems. Keep it up, CF!
- The real gigantic issue I have with recent Div2A, B, C problems is that they feel... somewhat arbitrary and unmotivated. Define some weird arbitrary operation/sequence/whatever and prove some random thing about it. Okay, I guess?
Takeaway
Problemsetters cannot make everyone happy. No round will be perfect for everyone, and there will always be complaints. That doesn't mean, we shouldn't pursue better rounds. I hope the survey (despite some flaws) has created a field for a data-driven discussion about how to make Codeforces better. The floor is yours!
Concerning the issue with the ERs, I would like to revert them back to being unrated, this could allow us to use more classical problems which are meant to introduce some tricks/techniques in a simple way.
Unfortunately, the main reason why they were made rated is that most people care much more about rated rounds, and unrated ERs didn't get as much attention. So, it's kinda impossible to make them unrated again without losing a large part of their audience.
Perhaps one of the ways to remedy the problem is to revert their max rating bound back to 1900, so we could try using more "old ER-style" problems. I'll discuss it with the other authors and with Mike.
What are some problems with using classical problems in rated rounds? Do they still reflect skills? If people dislike classical problems in rated rounds, how does it differ in unrated ones?
Just DM me the classical problems instead. :)
52.6% started CP in 2020-2021, and 66.8% have rating >= 1600.
Maybe I don't understand something, but the ratings chart seems a bit strange.
What is strange here ?
I started CP in september 2020 (and codeforces in march 2021) and my rating is >= 1600, I don't think that this is strange.
If that's true then this is a great result! Honestly, I didn't know anyone who was able to get expert in such short period of time. Of course, maybe, LGMs were such people, but not so many people become LGM.
And from these numbers we can see that ~25% of people reached >=1600 in 1-2 years. I can't believe this.
if someone isnt completely new to coding / CP then it just takes a few contests to get to 1600
I don't think it is true for everyone
Looks like I have a lot of unpopular opinions, so I guess it's a good thing I don't do coordination work :)
Row 98 in the raw data if you are interested.I'm also not a problemsetter, so all my answers are from a participant's point of view.
I voted 4 on the edu question, and I don't think anything is wrong with that. I personally stopped treating edus differently from common div2s, and maybe you should too. Raw algorithm problems don't teach you problemsolving, because usually a large part of a solution is a reduction or even just guessing the topic. Raw algorithm problems don't belong to a rated contest, because they can be searched for on the internet. So I like edus as they are now.
I absolutely love stories in problems. Reading fun statements is soothing, and I take everything to feel less stressed out during the contest.
I found hacks really fun 1000 rating points ago. Would probably find them fun now as well if I wasn't forced to solve 4 div1/6 combined problems in a contest. Also, if all easy problems didn't have multitest.
Kinda sad that coordinators tend to not approve implementation challenges into contests. I like thinking about how to implement a problem in the neatest way possible.
I'd also love to support this comment:
However, not about div2ABC, but about one certain type of problem I hate with passion (as a participant). You are given a binary (sometimes, even ternary!) string/matrix and a set of operations you can perform on it. Count how many different strings/matrices can be produced. Or determine if one string can be transformed into another one.
I see a binary string, and my blood starts boiling. Honestly, the most artificial type of all problems. Look at me, I came up with operations and stared at strings to notice an obscure property. Now you find this property and forget about it forever because it's absolutely useless anywhere but this problem. Wow, so cool.
I quit atcoder because of these problems every contest, and I'm happy about this decision.
No particular strong reaction to other questions. I mostly don't care what the problems in the contest are. I have my share of fun from competing against other people. Thus, as long as a contest has problems and participants, I'm satisfied.
I dislike those operation problems in atcoder too, but somehow the mainstream opinion is that "atcoder requires more thinking".
Seconding the comment about random string operation problems. More generally, the reason I find Codeforces problems rewarding is that when I solve them, it feels like the conclusion of a process of logical deduction, where each step is motivated from the last by some feature of the problem. Along the same lines, when I fail to solve a problem, I can typically read the editorial and figure out how I could have thought of that solution in the future. In contrast, the sort of problems you describe rarely feel motivated, and the process of solving them looks more like writing a brute force and staring at an output table until I find an invariant. As a result, ultimately, when I solve these problems, I feel like I've stumbled upon the solutions by coincidence rather than as a consequence of some skill I've developed. This is generally why I tend to enjoy Codeforces problems more than AtCoder problems (though the real issue may be that I'm just not as good at solving AtCoder problems :p).
I think your initiative is great and I hope it turns out to be as productive as it promises to be. Thank you very much for your effort and time, as well as everyone who contributed.
Some of my opinions are:
Of course Codeforces gives me happiness, in fact, I use it as a drug.
I like ad-hocs problems a lot (I call them magic), I love the analyzes I have to do to get to the answer. I would like to see more of the algorithmic problems (but, that said algorithm can't be the problem itself, but to reduce it to it)
Implementation challenges are very important and I would like to see them more.
About things of primary importance: I like both options, I think it should be like that (I don't see them as opposites :'v)
I LOVE stories in the problems, maybe they should be mostly in italics. I like more to get involved in the story than to receive specific orders. (I know I'm not going to enjoy army)
I hate hacks.
Happy New Year :)
Oh, I said that! As other commenters have pointed out, I realize that this is more of a problem that I have with AtCoder problems than Codeforces (though I do like AtCoder problems in general; it's only this specific kind that I don't like). But I do still feel it more often than I'd like, from Codeforces rounds.
Anyway, I feel some amount of artificial-ness will always be inherent to problems in a 2-hour contest format. I don't expect to see completely realistic real-world programming problems in CF contests, because tbh that would kinda suck. But I hope to at least get a feeling of "Wow, this is interesting!" or even "Yes, this seems like a natural question to ask." I don't want to be thinking, "Who cares?"
Here's a recent problem I really liked: https://mirror.codeforces.com/contest/1617/problem/D2 This is certainly not a "realistic" problem, but the set-up does not feel contrived. In fact, I think it's quite natural! Bonus points for the fact that it uses neat insights to lead to an elegant solution.
You can also check out a recent long contest I wrote for my opinion on what constitutes a "well-motivated" problem: https://mirror.codeforces.com/gym/103496
I expect that many people will dislike that set for mostly consisting of "implementation problems". But to be honest, I don't mind implementation problems! "Just implement it" can be nontrivial for beginners, and I think they will find it more engaging than proving some arbitrary fact about an arbitrary definition on some arbitrary sequence (mod $$$998244353$$$)
I could do without stories.
Especially the lamest copypasta of all time: <Name> has a problem and needs your help. Can you help him? It really feels like the author couldn't think of any story, but thought it was mandatory to have one.
But I'd like to praise Deltix for having their stories accompanied by very nice pictures. Especially I'd like to point out 1523C - Compression and Expansion. There were quite a few complaining that they had trouble understanding the problem (and frankly I understand where they're coming from). I was able to understand it rather quickly because of the picture.
Wow, that picture was geniosity... I didn't notice until now, and had a very hard time understanding what is going on in the task during the contest
I calculated the correlation coefficient between each pair of columns. The survey shows that a high ranking positively correlates with having a lot of experience in competitive programming and liking the last rounds, and negatively with:
1) the opinion that there should be more questions on data structures / graphs / dynamic programming
2) the opinion that there should be fewer ad-hoc questions
3) the opinion that easy tasks should target low-ranked people
4) the opinion that the questions in the competition should be balanced in terms of problem topics.