Lazyfying - #239
Lazyfying#239nikbpetrov wants to merge 6 commits into
Conversation
0a2dc33 to
d0914f1
Compare
…ble/reproducible
7de71bd to
ee90d58
Compare
| "API_KEY_TOGETHERAI": "API_KEY_TOGETHERAI", | ||
| "API_KEY_GOOGLE": "API_KEY_GEMINI", | ||
| "API_KEY_MISTRAL": "API_KEY_MISTRAL", | ||
| "API_KEY_XAI": "API_KEY_XAI", |
There was a problem hiding this comment.
these were removed in the llm forecaster rewrite, added back here. Why?
| df_bs = ( | ||
| df.groupby(["forecast_due_date", "source"]) | ||
| .apply(question_level_bootstrap, include_groups=False) | ||
| .apply( |
There was a problem hiding this comment.
nit: from the docs so can you just drop the lambda and do .apply(question_level_bootstrap, include_groups=False, random_state=random_state)
| ) | ||
| HUMAN_QUESTIONS.update(human_questions_of_question_type) | ||
|
|
||
| # Optional reproducibility: when QUESTION_SET_SEED is set, thread a single RandomState through |
There was a problem hiding this comment.
Can you explicitly mention "for testing"? Claude was confused about this and thought the seed was missing in deployment.
| # Optional reproducibility: when QUESTION_SET_SEED is set, thread a single RandomState through | ||
| # both sampling passes so the published set is deterministic. Unset (the default) preserves the | ||
| # historical unseeded behaviour. | ||
| seed_env = os.getenv("QUESTION_SET_SEED") |
There was a problem hiding this comment.
should we also check that seed_env is None when we're in prod to ensure it isn't accidentally ever set in deployment?
| WORKSPACE_BUCKET = os.environ.get("WORKSPACE_BUCKET") | ||
| # Names read as plain strings (``None`` if unset). | ||
| _STR_VARS = { | ||
| "PROJECT_ID": "CLOUD_PROJECT", |
There was a problem hiding this comment.
what about a commit renaming PROJECT_ID to CLOUD_PROJECT and simplifying this to a set of strings?
| ) | ||
| HUMAN_QUESTIONS.update(human_questions_of_question_type) | ||
|
|
||
| # Optional reproducibility: when QUESTION_SET_SEED is set, thread a single RandomState through |
There was a problem hiding this comment.
Instead of QUESTION_SET_SEED and the seed in the leaderboard code, what do you think about adding RANDOM_SEED to env.py? It would be an int and None by default and be referred to for any non-deterministic behavior in the codebase. Then the check for seed + prod can also be done in env.py
2 participants
meant to be merged as separate commits, just grouped here in the same PR
7de71bd is not lazy-related, sorry for sneaking it in, but it's a 1-line print -> logger.debug, change (preps the testing framework work)