lightning_mcqueen's blog

Weak test cases in meta hacker cup round 2 problem B
By lightning_mcqueen, history, 6 months ago, In English

Problem B has weak test cases.

Spoiler

My code got ac verdict but fails on test case 

1
6 6
1 2 3 4 5 6 
5 5 5 5 1 1

Output should be 5 but it gives 6.

PS-From around ranks 1200 to 2100, people solved only problems A and B most of them have wrong sol for B.

  • Vote: I like it
  • +169
  • Vote: I do not like it

»
6 months ago, hide # |
← Rev. 2  
Vote: I like it +11 Vote: I do not like it

Another test cases:

1
4 4
1 2 3 4
4 4 1 1

I got confused.

→ Reply
»
6 months ago, hide # |
← Rev. 2  
Vote: I like it +23 Vote: I do not like it

yes i realized my code fails this case immediately after submission but somehow got AC

1

5 5

5 4 3 2 1

10 10 10 10 0

→ Reply
»
6 months ago, hide # |
 
Vote: I like it 0 Vote: I do not like it

Auto comment: topic has been updated by lightning_mcqueen (previous revision, new revision, compare).

→ Reply
»
6 months ago, hide # |
← Rev. 3  
Vote: I like it 0 Vote: I do not like it

This too

1
5 4
0 0 1 1 2
0 0 0 4
→ Reply
»
6 months ago, hide # |
← Rev. 2  
Vote: I like it +16 Vote: I do not like it

Another simple test case:
4 3
3 3 2 1
1 4 4

A correct solution is outputting 4 for this case, but 4 is not the correct answer.

→ Reply
»
6 months ago, hide # |
 
Vote: I like it +14 Vote: I do not like it

CC: SecondThread

→ Reply
»
6 months ago, hide # |
 
Vote: I like it +26 Vote: I do not like it

please look into this and try to rejudge SecondThread

→ Reply
»
6 months ago, hide # |
← Rev. 2  
Vote: I like it 0 Vote: I do not like it

Same goes for this

8 2

3 2 1 1 1 1 1 1

6 6

Should be 2, but solutions with 8 is accepted.

I couldn't submit, almost drove me crazy.

→ Reply
»
6 months ago, hide # |
← Rev. 2  
Vote: I like it -73 Vote: I do not like it

There will always be a weak test case for every single problem created. let it go

→ Reply
»
6 months ago, hide # |
 
Vote: I like it +5 Vote: I do not like it

Another test case 

1
6 2
1 1 1 1 2 2
1 7

Expected Output: 2, Some solutions with output: 6 are Accepted

→ Reply
»
6 months ago, hide # |
 
Vote: I like it +1 Vote: I do not like it

I tried after contest and following code get AC:

    private static int SolveB(int n, int m, int[] a, int[] b)
    {
        Array.Sort(a);
        var map = new List<int>();
        for (int i = 0; i < n; i++)
        {
            if (i == 0)
                map.Add(1);
            else if (a[i] == a[i - 1])
                map[map.Count - 1]++;
            else map.Add(1);
        }

        var bSum = 0L;
        for (int i = 0; i < b.Length; i++) bSum += b[i];
        var sum = 0L;
        var cnt = 0;
        var end = Math.Max(0, map.Count - b.Length);
        for (int i = map.Count - 1; i >= end; i--)
        {
            cnt += map[i];

            sum += cnt;
            if (sum > bSum)
                return cnt - map[i];
        }

        return cnt;
    }
→ Reply
»
6 months ago, hide # |
 
Vote: I like it +5 Vote: I do not like it

We hope submissions will be rejuged

→ Reply
»
6 months ago, hide # |
← Rev. 3  
Vote: I like it +16 Vote: I do not like it

SecondThread , Please rejudge the solutions as correctness should be the top most priority -- because it is not passing pretests or something , it is abt coming up the best you can and results will be revealed after end of contest .

→ Reply
  • »
    »
    6 months ago, hide # ^ |
     
    Vote: I like it +5 Vote: I do not like it

    It is important for ppl who compete for T-shirts (Top 2000) like me .

    → Reply
    • »
      »
      »
      6 months ago, hide # ^ |
       
      Vote: I like it +4 Vote: I do not like it

      Getting t-shirt solving only A seems to be of no use actually

      → Reply
      • »
        »
        »
        »
        6 months ago, hide # ^ |
         
        Vote: I like it +10 Vote: I do not like it

        Agree. But there are a bunch of people who solved B right in the end of the contest (and some of them did it correctly in opposite to many above ranked contestants) and still finished out of top 2k.

        → Reply
    • »
      »
      »
      6 months ago, hide # ^ |
       
      Vote: I like it +1 Vote: I do not like it

      Also important for people who could qualify for R3, but couldn't due to the weak TCs. Even in top-500, most of the passed solutions I have seen so far are incorrect.SecondThread Please do something about it. If it cannot be rejudged, then atleast do some relaxation to the qualification criteria for Round-3 (as you guys did for R2).

      → Reply
  • »
    »
    6 months ago, hide # ^ |
     
    Vote: I like it -20 Vote: I do not like it

    I agree with you upto some extent. But as someone who's B will fail after rejudge, I am also disappointed with very weak validation tests. If validation tests are stronger than i would have rectified my mistake.

    → Reply
    • »
      »
      »
      6 months ago, hide # ^ |
       
      Vote: I like it +3 Vote: I do not like it

      If your code is wrong, then it should fail on the test cases provided after the validation input test cases, and you will only find out about it after the contest ends. Even I got a wrong answer on Question A — so should I blame the validation input or my own code?

      As for the scoreboard, someone who wrote the correct solution for Question B and received an AC should be ranked above those who wrote an incorrect solution but still got accepted.

      If the submissions for Question B are not rejudged, it would be a huge shame, as it breaks the very spirit of Hacker Cup.

      → Reply
      • »
        »
        »
        »
        6 months ago, hide # ^ |
         
        Vote: I like it +2 Vote: I do not like it

        "you will only find out about it after the contest ends"

        Is it perfectly possible to realize that the code is wrong during the 6-minute time window (for example, if it gets runtime error on the main tests), and you have 6 minutes to debug. If the main tests are weak and then the submissions are rejudged, you don't have this opportunity. So even the rejudge is unfair in some sense.

        → Reply
        • »
          »
          »
          »
          »
          6 months ago, hide # ^ |
           
          Vote: I like it +12 Vote: I do not like it

          Ok, but who really deserves to be placed higher — someone with the correct code and an AC, or someone whose incorrect code also got an AC?

          And your claim is only valid for a runtime error. If the code passes the validation input but gets a wrong answer after the contest ends, we will only come to know about it once the contest is over.

          → Reply
  • »
    »
    6 months ago, hide # ^ |
     
    Vote: I like it +9 Vote: I do not like it

    It’s not possible to 'rejudge' the solutions because during the contest we were expected to handle the large input tests that were provided at the time. You can’t add new tests like in codeforces. I am sure they don't have any mechanism to run new tests on the participants' codes

    → Reply
    • »
      »
      »
      6 months ago, hide # ^ |
      ← Rev. 4  
      Vote: I like it +9 Vote: I do not like it

      Let them decide . What can we do ? It is upto metappl what they want to do ? It is pretty common that ppl with crrt solution goes up after the contenst , that does not seem to happen in B this time that is all .

      → Reply
  • »
    »
    6 months ago, hide # ^ |
     
    Vote: I like it +28 Vote: I do not like it

    it is impossible to rejudge

    the code files are only expected to be there as proof of work, not to be run. You cannot assume how each code file runs internally, for example some submissions might only work on Windows/Mac, and may take input/output from files instead of standard IO

    → Reply
    • »
      »
      »
      6 months ago, hide # ^ |
      ← Rev. 2  
      Vote: I like it +3 Vote: I do not like it

      could be . but compiler(the purpose it is there) + good env conditions has the potential to do this .Also , there are several other alternatives (recontest , some relaxations etc) .

      → Reply
»
6 months ago, hide # |
 
Vote: I like it +6 Vote: I do not like it

how are you sure that most of the people from ranks 1200 to 2100 have wrong sol?

→ Reply
  • »
    »
    6 months ago, hide # ^ |
     
    Vote: I like it +5 Vote: I do not like it

    Just randomly download anybody's solution, and run it on the few TCs provided in this blog, in 90%+ cases, thier solutions are failing.

    This is true even for people ranked in top-500. Most of the solutions that I've seen so far are incorrect. This not only affect those getting T-shirts, but also for those who could have qualified for Round-3 (Top-500), had the test cases been strong enough.

    → Reply
    • »
      »
      »
      5 months ago, hide # ^ |
       
      Vote: I like it +7 Vote: I do not like it

      Wow, its like 75% of submissions that are incorrect on very simple edge cases.

      I would understand them keeping the rankings as is if it was something like a TLE edge case where the wrong time complexity can pass, but these are just straight up incorrect solutions. I think I actually have a chance of getting T500 now if they clean up these incorrect solutions.

      → Reply
»
5 months ago, hide # |
← Rev. 2  
Vote: I like it +8 Vote: I do not like it

I know Hacker cup typically doesn't allow for solutions to be hacked after the competition (such as solutions that would fail time complexity with a well crafted input), but wow, these test cases that are failing are WEAK.

I really feel for the organizers on this. Either they have to retroactively change the ruleset after the competition has completed, OR they have to accept many flat out incorrect solutions. Not a good look either way, but I think this kinda highlights the need for a rules change.

→ Reply
»
5 months ago, hide # |
← Rev. 2  
Vote: I like it 0 Vote: I do not like it

Another test case:

1

4 2

10 10 20 20

1 2000

Expected output : 2, many solutions give 4 as output

→ Reply