[Fix] add more model examples#644
Merged
Merged
Conversation
- Set environment variables for Azure OpenAI API configuration. - Modify model arguments to use the new GPT-4o model version and enable Azure OpenAI support. - Clean up commented installation instructions for clarity.
- Consolidated import statements in `plm.py` and `utils.py` for better organization. - Removed redundant blank lines in `eval_utils.py`, `fgqa_utils.py`, `rcap_utils.py`, `rdcap_utils.py`, `rtloc_utils.py`, and `sgqa_utils.py` to enhance readability. - Ensured consistent import structure across utility files for improved maintainability.
- Revised installation instructions to facilitate direct package installation from Git. - Added detailed usage examples for various models including Aria, LLaVA, and Qwen2-VL. - Introduced new example scripts for model evaluations, enhancing user guidance for running specific tasks. - Improved clarity in environmental variable setup and common issues troubleshooting sections.
…ated evaluation instructions - Changed paths for model evaluation scripts to point to the new `examples/models` directory. - Added a note directing users to find more examples in the updated location. - Removed outdated evaluation instructions for LLaVA on multiple datasets to streamline the documentation.
…instructions - Replaced outdated evaluation commands with new script paths in the `examples/models` directory. - Updated sections for evaluating larger models, including the introduction of new scripts for tensor parallel and SGLang evaluations. - Streamlined instructions for model evaluation to improve clarity and usability.
dadwadw233
pushed a commit
to dadwadw233/lmms-eval
that referenced
this pull request
Apr 28, 2025
* Update OpenAI compatibility script for Azure integration - Set environment variables for Azure OpenAI API configuration. - Modify model arguments to use the new GPT-4o model version and enable Azure OpenAI support. - Clean up commented installation instructions for clarity. * Refactor imports and clean up code in various utility files - Consolidated import statements in `plm.py` and `utils.py` for better organization. - Removed redundant blank lines in `eval_utils.py`, `fgqa_utils.py`, `rcap_utils.py`, `rdcap_utils.py`, `rtloc_utils.py`, and `sgqa_utils.py` to enhance readability. - Ensured consistent import structure across utility files for improved maintainability. * Update README and add example scripts for model evaluations - Revised installation instructions to facilitate direct package installation from Git. - Added detailed usage examples for various models including Aria, LLaVA, and Qwen2-VL. - Introduced new example scripts for model evaluations, enhancing user guidance for running specific tasks. - Improved clarity in environmental variable setup and common issues troubleshooting sections. * Update README to reflect new example script locations and remove outdated evaluation instructions - Changed paths for model evaluation scripts to point to the new `examples/models` directory. - Added a note directing users to find more examples in the updated location. - Removed outdated evaluation instructions for LLaVA on multiple datasets to streamline the documentation. * Update README to reflect new script locations and enhance evaluation instructions - Replaced outdated evaluation commands with new script paths in the `examples/models` directory. - Updated sections for evaluating larger models, including the introduction of new scripts for tensor parallel and SGLang evaluations. - Streamlined instructions for model evaluation to improve clarity and usability.
6 tasks
kcz358
pushed a commit
that referenced
this pull request
Feb 16, 2026
* docs: restructure README and v0.6 release notes - Restructure v0.6 doc Section 1 from bottom-up to top-down architecture overview: Pipeline (1.1) -> Model Interface (1.2) -> API Concurrency (1.3) - Restore original "Why lmms-eval?" voice with v0.6 insights integrated naturally (statistical rigor, evaluation as infrastructure) - Reorder README sections by user journey: Why -> Quickstart -> Usage -> Advanced - Collapse i18n language links into expandable details tag - Simplify quick links from 3 rows to 2 rows - Update title to "LMMs-Eval: Probing Intelligence in the Real World" - Add comprehensive CHANGELOG.md covering v0.6 highlights and ~182 commits * fix: add CITATION.cff and remove dead repr_scripts.sh link - Copy CITATION.cff from feat/api-model-concurrency (enables GitHub's "Cite this repository" button and fixes broken FAQ reference) - Remove stale miscs/repr_scripts.sh link (file was renamed in #544, then deleted in #644; README was never updated) * Fix formatting of evaluation components diagram * Revise mermaid diagram for async pipeline with cache Updated mermaid diagram to improve clarity and formatting. --------- Co-authored-by: Pu Fanyi <FPU001@e.ntu.edu.sg>
Luodian
added a commit
that referenced
this pull request
Feb 28, 2026
* docs: restructure README and v0.6 release notes - Restructure v0.6 doc Section 1 from bottom-up to top-down architecture overview: Pipeline (1.1) -> Model Interface (1.2) -> API Concurrency (1.3) - Restore original "Why lmms-eval?" voice with v0.6 insights integrated naturally (statistical rigor, evaluation as infrastructure) - Reorder README sections by user journey: Why -> Quickstart -> Usage -> Advanced - Collapse i18n language links into expandable details tag - Simplify quick links from 3 rows to 2 rows - Update title to "LMMs-Eval: Probing Intelligence in the Real World" - Add comprehensive CHANGELOG.md covering v0.6 highlights and ~182 commits * fix: add CITATION.cff and remove dead repr_scripts.sh link - Copy CITATION.cff from feat/api-model-concurrency (enables GitHub's "Cite this repository" button and fixes broken FAQ reference) - Remove stale miscs/repr_scripts.sh link (file was renamed in #544, then deleted in #644; README was never updated) * Fix formatting of evaluation components diagram * Revise mermaid diagram for async pipeline with cache Updated mermaid diagram to improve clarity and formatting. --------- Co-authored-by: Pu Fanyi <FPU001@e.ntu.edu.sg>
stisiTT
pushed a commit
to bgoelTT/lmms-eval
that referenced
this pull request
Mar 6, 2026
* Update OpenAI compatibility script for Azure integration - Set environment variables for Azure OpenAI API configuration. - Modify model arguments to use the new GPT-4o model version and enable Azure OpenAI support. - Clean up commented installation instructions for clarity. * Refactor imports and clean up code in various utility files - Consolidated import statements in `plm.py` and `utils.py` for better organization. - Removed redundant blank lines in `eval_utils.py`, `fgqa_utils.py`, `rcap_utils.py`, `rdcap_utils.py`, `rtloc_utils.py`, and `sgqa_utils.py` to enhance readability. - Ensured consistent import structure across utility files for improved maintainability. * Update README and add example scripts for model evaluations - Revised installation instructions to facilitate direct package installation from Git. - Added detailed usage examples for various models including Aria, LLaVA, and Qwen2-VL. - Introduced new example scripts for model evaluations, enhancing user guidance for running specific tasks. - Improved clarity in environmental variable setup and common issues troubleshooting sections. * Update README to reflect new example script locations and remove outdated evaluation instructions - Changed paths for model evaluation scripts to point to the new `examples/models` directory. - Added a note directing users to find more examples in the updated location. - Removed outdated evaluation instructions for LLaVA on multiple datasets to streamline the documentation. * Update README to reflect new script locations and enhance evaluation instructions - Replaced outdated evaluation commands with new script paths in the `examples/models` directory. - Updated sections for evaluating larger models, including the introduction of new scripts for tensor parallel and SGLang evaluations. - Streamlined instructions for model evaluation to improve clarity and usability.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before you open a pull-request, please check if a similar issue already exists or has been closed before.
When you open a pull-request, please be sure to include the following
If you meet the lint warnings, you can use following scripts to reformat code.
Thank you for your contributions!