Skip to content
  • STEVAN Antoine's avatar
    rework benchmarks: script and output format (!193) · ab0d9a8b
    STEVAN Antoine authored
    this is an attempt at making benchmarks easier to work with
    - `benchmarks run` will read benchmarks from NUON data and run them
    - `benchmarks plot` will plot benchmark results
    - the output format will be something 
      - a directory whose name is the hash of the CPU spec and the Komodo commit hash
      - contains `cpu.json` with the CPU info
      - contains `komodo.txt` with the Komodo commit hash
      - contains NDJSON result files
    
    > 💡 **Note**
    >
    > results will typically be uploaded to https://gitlab.isae-supaero.fr/dragoon/komodo-benchmark-results
    
    > 💡 **Note**
    >
    > this MR goes alongside the [`komodo-benchmark-results@restart`](https://gitlab.isae-supaero.fr/dragoon/komodo-benchmark-results/-/compare/main...restart) branch
    
    ## changelog
    - bump Nushell to 0.101.0
      - parallel `$in` => remove useless `let input = $in` when possible, e.g. still required when using the function's `$in` in a `for` loop
      - `group-by` changed => `group-by x --to-table` will now produce a table with columns `x` and `items` instead of `group` and `items` as in 0.95.0
    - add link to results repo
    - the Nushell benchmarks lib
      - rename `--force` to `--no-confirm (-y)`
      - add `--append` 
      - reject columns that GPLT will complain about, e.g. `$.points.k` for the FRI plots
      - add `--save` to the FRI plot
    - move the "field" and "curve group" benchmarks from `benchmarks/src/bin/operations/` to `benchmarks/src/bin/`
    - remove `benchmarks/params/fri.nu` because it's been uniformized with the other methods
    - rewrite the README
    - add main function to `benchmarks/` that runs the benchmarks from a NUON record specification
    - simplify the output of FRI run
    
    ## TODO
    - [x] fix "_atomic operations_" (done in 4f69a1d6)
    - [x] check that _plotting_ still works
    
    ## images
    
    ### Field
    ![complex_curve_group_operations](/uploads/57b36926cce041cf405a9b44f190b8b8/complex_curve_group_operations.png)
    ![complex_field_operations](/uploads/0747c85dbaff8980561aa9d922fcd5e7/complex_field_operations.png)
    ![simple_curve_group_operations](/uploads/974cf70fed68f8d8ac898d54be3f27be/simple_curve_group_operations.png)
    ![simple_field_operations](/uploads/a3d4d0dcdeb35d4c434eaa38fb51e7b5/simple_field_operations.png)
    
    ### Linear algebra
    ![linalg-inverse](/uploads/bc290ffa39459ce0f9bbd393b50b7b98/linalg-inverse.png)
    ![linalg-mul](/uploads/96d8c2a63ed48d6a0d3508b4a948153b/linalg-mul.png)
    ![linalg-transpose](/uploads/128e35eca91497d8aadb0130c05aeee3/linalg-transpose.png)
    
    ### FEC
    ![encoding](/uploads/405c4d3ef9ec5135ebdd7ce2e6c96bfe/encoding.png)
    ![decoding](/uploads/d793234d44e9fc6f34f0c2a9372863cd/decoding.png)
    ![recoding](/uploads/413021de997c86d45b1287fcfe7804c7/recoding.png)
    ![combined](/uploads/6d1c3ae6d3bf5547434ca29ae80b5536/combined.png)
    ![ratio](/uploads/6cebd7a0bcef57d1b1256bc3941c1b0a/ratio.png)
    
    ### ZK
    ![setup](/uploads/1feb169452aa3274dc924edf772c9a5b/setup.png)
    ![commit](/uploads/0a2775c2116ca7d3fa7b956b934d1565/commit.png)
    
    ### FRI
    ![commits_single](/uploads/4602725e551a025d42815183a61d11b2/commits_single.png)
    ![commits_single_normalized](/uploads/2ffebea940af1cbbe9bd64499343e0e9/commits_single_normalized.png)
    ![end_to_end](/uploads/46917abd2f5976dfa6d3039cc0ab2c0e/end_to_end.png)
    ![evaluating](/uploads/cf4dc496cd5144615bf5f9b06d27dccd/evaluating.png)
    ![proofs](/uploads/b0828bfa62c2226c8d63c9ecd387049f/proofs.png)
    ![proofs_normalized](/uploads/8fbf0d713884e2e147c62313a001f379/proofs_normalized.png)
    ![proving](/uploads/161539dc412330be1878cdda82c3d966/proving.png)
    ![verifying_single](/uploads/2df7e777481d7789478386f8e83e0783/verifying_single.png)
    ab0d9a8b