Repository navigation
Create runner for CBBO #35
Description
Activity
An initial runner was created in https://github.com/deephyper/benchmark/blob/runner/src/deephyper_benchmark/benchmarks/cbbo/runner.py
To run it, move to
examples/cbbo_runnerthen run the command:python -m deephyper_benchmark.benchmarks.cbbo.runner --config cbbo.tomlThis will create a directory structure such as:
. ├── cbbo.toml └── runs └── ackley ├── cbo │ ├── 168c3143-794c-4d68-a30c-340ce47ccfb7 │ │ ├── context.yaml │ │ └── results.csv │ └── 86dae083-ca6d-4dfc-bdfb-6de7b936ac7c │ ├── context.yaml │ └── results.csv ├── random │ ├── 987a3d4d-b43d-4c2e-ba0b-08ce60a389d6 │ │ ├── context.yaml │ │ └── results.csv │ └── da83e868-0147-4c58-b110-a4c39ac510ab │ ├── context.yaml │ └── results.csv └── tpe ├── 9a0c667d-ca22-4a19-90b0-2e6fdb2620ab │ ├── context.yaml │ └── results.csv └── dd586a6c-c984-4b52-8776-2dced49fa41a ├── context.yaml └── results.csvExample printed outputs:
Running: CBBO Benchmark Starting benchmark: ackley Starting replica: ackley/cbo/0c9f10f3-643b-44c6-8a6a-33e80ecaeaae' ackley/cbo/0c9f10f3-643b-44c6-8a6a-33e80ecaeaae: 50%|███████████████████████████████████▌ | 250/500 [01:03<01:36, 2.60it/s, failures=0, objective=-0.034]Following the same logic, I added a
plotter:python -m deephyper_benchmark.benchmarks.cbbo.plotter --config cbbo.tomlThis command is to run from the same location as the runner.
It will creates the
plotsfolder such as:. ├── cbbo.toml ├── plots │ └── ackley │ └── average_simple_regret.png └── runs ├── ackley │ ├── cbo │ │ ├── 094e8e08-8b2e-4e14-b6dd-ed7fa83f84e4 │ │ │ ├── context.yaml │ │ │ └── results.csvand create figures such as:
There is no need for this:
python -m deephyper_benchmark.benchmarks.cbbo.plotter --config cbbo.tomlJust provide an entry point for the command line interface (CLI) and run it like this:
deephyper-benchmark cbbo runOr the plotter like this:
deephyper-benchmark cbbo plotIs the cbbo runner done? Once it's done you can merge it with the main branch. Then I can start another branch and do a merge request for the command line interface.
We will start by defining a
runner.pyscript in each benchmark folder to execute benchmarks.We will use TOML to define the configuration of a benchmark run.