Add evaluation file for code_llama2 model

[ReemaAlzaid]

Context

What is the purpose of this PR? Is it to

• add a new feature
• fix a bug
• update tests and/or documentation
• other (Clean Up)

Please link to any issues this PR addresses: #1810

Changelog

What are the changes made in this PR?

• Copied evaluation.yaml to /recipes/config/code_llama2/ directory
• Updated evaluation.yaml to point to CodeLlama-7b-hf model instantiations
• Updated the recipe registry class to pick up the new CodeLlama2 config

Test plan

Please make sure to do each of the following if applicable to your PR. If you're unsure about any one of these just ask and we will happily help. We also have a contributing page for some guidance on contributing.

• run pre-commit hooks and linters (make sure you've first installed via pre-commit install)
• add unit tests for any new functionality
• update docstrings for any new or updated methods or classes
• run unit tests via pytest tests
• run recipe tests via pytest tests -m integration_test
• manually run any new or modified recipes with sufficient proof of correctness
• include relevant commands and any other artifacts in this summary (pastes of loss curves, eval results, etc.)

CodeLlama-7b-hf evaluation recipe output:

(torchtune_env) ➜  torchtune git:(code_llama2_eval) ✗ tune run eleuther_eval --config code_llama2/evaluation
W1225 22:55:20.699000 62921 torch/distributed/elastic/multiprocessing/redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs.
Running EleutherEvalRecipe with resolved config:

batch_size: 8
checkpointer:
  _component_: torchtune.training.FullModelHFCheckpointer
  checkpoint_dir: /tmp/CodeLlama-7b-hf
  checkpoint_files:
  - pytorch_model-00001-of-00003.bin
  - pytorch_model-00002-of-00003.bin
  - pytorch_model-00003-of-00003.bin
  model_type: LLAMA2
  output_dir: ./
  recipe_checkpoint: null
device: cpu
dtype: bf16
enable_kv_cache: true
limit: null
max_seq_length: 4096
model:
  _component_: torchtune.models.code_llama2.code_llama2_7b
quantizer: null
resume_from_checkpoint: false
seed: 1234
tasks:
- truthfulqa_mc2
tokenizer:
  _component_: torchtune.models.llama2.llama2_tokenizer
  max_seq_len: null
  path: /tmp/CodeLlama-7b-hf/tokenizer.model

2024-12-25:22:55:21,296 INFO     [_utils.py:28] Running EleutherEvalRecipe with resolved config:

batch_size: 8
checkpointer:
  _component_: torchtune.training.FullModelHFCheckpointer
  checkpoint_dir: /tmp/CodeLlama-7b-hf
  checkpoint_files:
  - pytorch_model-00001-of-00003.bin
  - pytorch_model-00002-of-00003.bin
  - pytorch_model-00003-of-00003.bin
  model_type: LLAMA2
  output_dir: ./
  recipe_checkpoint: null
device: cpu
dtype: bf16
enable_kv_cache: true
limit: null
max_seq_length: 4096
model:
  _component_: torchtune.models.code_llama2.code_llama2_7b
quantizer: null
resume_from_checkpoint: false
seed: 1234
tasks:
- truthfulqa_mc2
tokenizer:
  _component_: torchtune.models.llama2.llama2_tokenizer
  max_seq_len: null
  path: /tmp/CodeLlama-7b-hf/tokenizer.model

Model is initialized with precision torch.bfloat16.
2024-12-25:22:55:40,938 INFO     [eleuther_eval.py:503] Model is initialized with precision torch.bfloat16.
2024-12-25:22:55:40,981 INFO     [huggingface.py:132] Using device 'cpu'
2024-12-25:22:55:41,861 INFO     [huggingface.py:369] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cpu'}
Running evaluation on the following tasks: ['truthfulqa_mc2']
2024-12-25:22:55:52,956 INFO     [eleuther_eval.py:540] Running evaluation on the following tasks: ['truthfulqa_mc2']
2024-12-25:22:55:52,957 INFO     [task.py:415] Building contexts for truthfulqa_mc2 on rank 0...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 817/817 [00:00<00:00, 2213.89it/s]

2024-12-25:22:55:53,352 INFO     [evaluator.py:496] Running loglikelihood requests
Running loglikelihood requests:  84%|
██████████████████████████████████████████████████████████████████████████████████████████████████████▋                   | 

4953/58Running loglikelihood requests:  84%|██████████████████████████████████████████████████████████████████████████████████████████████████████▉                   | 

4961/5882 [14:15:43<2:32:13,  9.92Running loglikelihood requests:  84%|███████████████████████████████████████████████████████████████████████████████████████████████████████                   | 

4969/5882 [14:17:04<2:31:57,  9.99Running loglikelihood requests: 100%|
████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5882/5882 [16:50:32<00:00, 10.31s/it]
Eval completed in 60635.12 seconds.
2024-12-26:15:46:28,080 INFO     [eleuther_eval.py:549] Eval completed in 60635.12 seconds.
Max memory allocated: 0.00 GB
2024-12-26:15:46:28,084 INFO     [eleuther_eval.py:550] Max memory allocated: 0.00 GB


|    Tasks     |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|--------------|------:|------|-----:|------|---|-----:|---|-----:|
|truthfulqa_mc2|      2|none  |     0|acc   |↑  |0.3715|±  |0.0142|


2024-12-26:15:46:28,366 INFO     [eleuther_eval.py:554] 

|    Tasks     |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|--------------|------:|------|-----:|------|---|-----:|---|-----:|
|truthfulqa_mc2|      2|none  |     0|acc   |↑  |0.3715|±  |0.0142|

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/2209

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 26126c6 with merge base aa8f365 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[ReemaAlzaid]

Hey @joecummings Kindly review this PR for this issue #1810

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 66.60%. Comparing base (aa8f365) to head (26126c6).

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #2209       +/-   ##
===========================================
+ Coverage   26.43%   66.60%   +40.17%     
===========================================
  Files         344      339        -5     
  Lines       20658    20104      -554     
===========================================
+ Hits         5460    13391     +7931     
+ Misses      15198     6713     -8485     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[joecummings]

Thanks!