[bkorycki]

Based on pr #1713
Dependent on PR 771 in sugar.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[rogthefrog]

every prompt ship in

I don't understand what that means.

[bkorycki]

Yeah it's a little confusing. It was left over from Riki's PR. I removed it.

[dhosterman]

Are we abandoning --demo in modelbench?

It will fail on 1.2.1, currently, with the following:

Traceback (most recent call last):
  File "/Users/daniel/dev/src/github.com/mlcommons/modelbench/.venv/bin/modelbench", line 10, in <module>
    sys.exit(cli())
             ~~~^^
  File "/Users/daniel/dev/src/github.com/mlcommons/modelbench/.venv/lib/python3.14/site-packages/click/core.py", line 1569, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/daniel/dev/src/github.com/mlcommons/modelbench/.venv/lib/python3.14/site-packages/click/core.py", line 1490, in main
    rv = self.invoke(ctx)
  File "/Users/daniel/dev/src/github.com/mlcommons/modelbench/.venv/lib/python3.14/site-packages/click/core.py", line 1970, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/Users/daniel/dev/src/github.com/mlcommons/modelbench/.venv/lib/python3.14/site-packages/click/core.py", line 1970, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/Users/daniel/dev/src/github.com/mlcommons/modelbench/.venv/lib/python3.14/site-packages/click/core.py", line 1353, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel/dev/src/github.com/mlcommons/modelbench/.venv/lib/python3.14/site-packages/click/core.py", line 907, in invoke
    return callback(*args, **kwargs)
  File "/Users/daniel/dev/src/github.com/mlcommons/modelbench/src/modelbench/cli.py", line 135, in wrapper
    return func(*args, **kwargs)
  File "/Users/daniel/dev/src/github.com/mlcommons/modelbench/.venv/lib/python3.14/site-packages/click/decorators.py", line 34, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/daniel/dev/src/github.com/mlcommons/modelbench/src/modelbench/cli.py", line 247, in general_benchmark
    check_benchmark(benchmark)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/daniel/dev/src/github.com/mlcommons/modelbench/src/modelbench/cli.py", line 387, in check_benchmark
    for hazard in benchmark.hazards():
                  ~~~~~~~~~~~~~~~~~^^
  File "/Users/daniel/dev/src/github.com/mlcommons/modelbench/src/modelbench/benchmarks.py", line 66, in hazards
    h.set_standard(self.standards)
                   ^^^^^^^^^^^^^^
  File "/Users/daniel/dev/src/github.com/mlcommons/modelbench/src/modelbench/benchmarks.py", line 59, in standards
    standards = Standards.get_standards_for_benchmark(self.reference_benchmark().uid)
                                                      ~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/daniel/dev/src/github.com/mlcommons/modelbench/src/modelbench/benchmarks.py", line 180, in reference_benchmark
    return type(self)(self.locale, "practice", PRIVATE_EVALUATOR)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: GeneralPurposeAiChatBenchmarkV1_2_1.__init__() takes 3 positional arguments but 4 were given

[dhosterman]

Shouldn't there be a scoring file for practice, too? If not, I believe this just fails after the benchmark is run but before it is scored, which is a bad failure mode.

[bkorycki]

Fixed!

[bkorycki]

@dhosterman fixed the demo issue!