Mohamed.Bassem's blog

dTests — distributively run GCJ testcases on several machines
By Mohamed.Bassem, 10 years ago, In English

Hi,

I developed a python tool to distributively run GCJ, FB hacker cup — or any contest in which you submit the output file — testcases on several machines. This should theoretically divide the program running time by the number of machines running plus a small network overhead. It's not yet ready for real contest usage but I hope it'll be ready before the next GCJ or FB hacker cup round.

Github Repo : https://github.com/MohamedBassem/dTests

Or you can download it from : https://pypi.python.org/pypi?name=dTests&:action=display

Check the repo for more info, installation and usage guide.

Your ideas and contributions are more than welcome! :)

  • Vote: I like it
  • +36
  • Vote: I do not like it

»
10 years ago, # |
  Vote: I like it +3 Vote: I do not like it

Nice stuff!

If you're looking for speed, you should probably use g++ with -O2 :)

And probably you could run, for every node, a number of processes equal to the number of cores.

I also parallelize my GCJ programs (not that it has ever actually come in useful), but on a single computer (one thread per core). The way I deal with splitting the testcases is the following: I have an Instance class with three methods: readInput(), compute() and writeOutput(). I create one such object per testcase. First, each one reads input sequentially; then they all compute in parallel; and then each one writes the output sequentially. So I get the splitting effect for free, by just letting thread 2 read from where thread 1 ended, etc. You could maybe consider doing this here to get rid of the splitter (although you couldn't have the programs be black-box anymore — they would at least have to let the runner know that they're done reading input).

→ Reply
  • »
    »
    10 years ago, # ^ |
      Vote: I like it +8 Vote: I do not like it

    I'll add the -O2 flag and parallelizing testcases on each node to the next release. Thank you :) Concerning the splitter, Your idea is very nice but in this case I didn't get how will I be able to share the reader object between different machines. What I have in mind is that every node will have a copy of the whole input file and the program, and every node instructs the program — through command line arguments maybe — to call readInput() n times before calling the compute() method to skip the first n testcases. But it's far more complicated for the developer than the splitter idea. But I guess this isn't what you meant.

    → Reply
    • »
      »
      »
      10 years ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      I agree that my local solution doesn't generalize very well :) Another idea to do this would be as follows: every node should read its input, let the caller script know that it's done reading the input (probably by outputting something, maybe to stderr, and flushing), and then start any time-intensive computations. Now I don't know if this is actually possible to do, but the caller script could be able to see how much input was read by the program. Then it will pass on the message to the next node, which will "fast-forward" the input file to that position, and call the program with that. This would be a rather faithful simulation of what happens locally in my routine (a difference being that in this case the first node can start working while the second one is reading its input — in my version first everyone reads, then everyone works). Of course it's also quite complicated and maybe impossible to accomplish ;)

      → Reply
    • »
      »
      »
      10 years ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      I finally understood what you wrote here :D I guess that would be the best solution! I actually don't think it's more complicated than the splitter idea.

      → Reply
      • »
        »
        »
        »
        10 years ago, # ^ |
          Vote: I like it +8 Vote: I do not like it

        After 8 months ? :D How did you get back to this post ? :D

        → Reply
        • »
          »
          »
          »
          »
          10 years ago, # ^ |
            Vote: I like it 0 Vote: I do not like it

          I looked at my old comments (there is a list in the user profile). I could have also googled :)

          → Reply
          • »
            »
            »
            »
            »
            »
            10 years ago, # ^ |
              Vote: I like it 0 Vote: I do not like it

            Aha, I mean how did you remember getting back to this post :D

            → Reply
            • »
              »
              »
              »
              »
              »
              »
              10 years ago, # ^ |
                Vote: I like it +14 Vote: I do not like it

              I thought about this for some reason, then came here to write it :) Then I saw that this is in fact what you meant.

              → Reply