feat(huggingface): add audio and media generation tasks#1
Open
anishshiva7 wants to merge 11 commits into
Open
Conversation
|
👋 Thanks for your first contribution to Texera, @anishshiva7! You can drive common housekeeping tasks just by leaving a comment. Type the command on its own line. On issues
To find unclaimed work, search Linking sub-issues
You can write references as On pull requests (author only)
You can mention teams as
For the full contribution flow, see CONTRIBUTING.md. |
anishshiva7
force-pushed
the
hf/04-audio-mediagen
branch
from
June 7, 2026 20:54
6202d1b to
8a83dc2
Compare
Collaborator
Author
|
/request-review @Ma77Ball |
PG1204
force-pushed
the
hf/03-image-tasks
branch
from
June 16, 2026 00:50
dd644d4 to
5e0df3e
Compare
anishshiva7
force-pushed
the
hf/04-audio-mediagen
branch
from
June 16, 2026 18:20
2fab0ab to
b77eaec
Compare
…#5320) ### What changes were proposed in this PR? Adds the image task family — 9 HF pipeline tasks — as the second `TaskCodegen` plugged into the dispatcher established by apache#5278: image-only: image-classification, object-detection, image-segmentation, image-to-text image + prompt: visual-question-answering, document-question-answering, zero-shot-image-classification, image-text-to-text, image-to-image - `codegen/ImageTaskCodegen.scala` supplies the per-task payload + parse Python branches for all 9 tasks. - `TaskCodegen` trait gains a `tasks: Set[String]` default method (defaults to `Set(task)`) so a single codegen can register under multiple task strings; `ImageTaskCodegen` is the first multi-task codegen to use it. - `CodegenContext` extended with `imageInput` + `inputImageColumn` (`EncodableString`). - `HuggingFaceInferenceOpDesc.scala` gains 2 new `@JsonProperty` fields and registers `ImageTaskCodegen` via the new `tasks` flat-map. `PythonCodegenBase.scala` grows to host the shared image infrastructure: - Task-family tuples (`image_only_tasks`, `image_prompt_tasks`, `image_tasks`) + `image_headers` in `process_table`. - Per-row image-bytes resolution from upload or column with `_read_image_input` / `_read_binary_value` / `_compress_image_bytes`. - `_post_with_fallback` extended with `raw_binary_headers` + `use_raw_binary_body`; adds image-text-to-text chat-completions and model-author vision branches. - `_call_provider` gains zai-org, Replicate predictions + polling, Fal-ai, Wavespeed submit+poll branches, and image embedding for OpenAI-compatible / unknown-provider fallbacks. - Image content-type response handling returns `data:image/...;base64,...` URLs. - Image helpers added: `_read_image_input`, `_compress_image_bytes`, `_image_input_as_base64`, `_read_binary_value`, `_looks_like_html`, `_html_to_image_bytes`, `_extract_json_arg`, `_url_to_data_url`. Frontend integration (HF lines only — no agent / dataset noise): `HuggingFaceImageUploadComponent` declared in `app.module.ts`, `huggingface-image-upload` formly type registered, image upload component .ts/.html/.scss + `HuggingFace.png` + `sample-image.png` assets. User-input strings continue to flow through `pyb"..."` + `EncodableString` so they reach Python as `self.decode_python_template('<base64>')` rather than raw literals. `PythonCodeRawInvalidTextSpec` still passes (117/117 descriptors `py_compile` cleanly). ### Any related issues, documentation, or discussions? - Tracking issue: apache#5319 - Closes: apache#5319 - Stacked on: apache#5278 (operator + text-generation — issue apache#5277) - Parent issue: apache#5041 - Closed sibling issue: apache#5134 (REST resource — landed via apache#5124) ### How was this PR tested? - `sbt "WorkflowOperator/compile; WorkflowOperator/Test/compile"` clean. - `sbt scalafmtCheck` clean. - `sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.huggingFace.HuggingFaceInferenceOpDescSpec"` — 18/18 pass (PR 2's 13 spec tests + 5 new image-task tests: image-only routing, VQA / document-QA payload, image-text-to-text chat-completions, image-to-image data-URL parse, all-9-tasks dispatcher coverage). - `sbt "WorkflowOperator/testOnly org.apache.texera.amber.util.PythonCodeRawInvalidTextSpec"` — 117/117 descriptors `py_compile` cleanly with the new operator code paths, no marker leaks. - Generated Python verified via `python3 -m py_compile` on sample image-task outputs. ### Was this PR authored or co-authored using generative AI tooling? Yes, co-authored with Claude Opus 4.7. --------- Signed-off-by: Prateek Ganigi <91584519+PG1204@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
<!-- Thanks for sending a pull request (PR)! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: [Contributing to Texera](https://github.com/apache/texera/blob/main/CONTRIBUTING.md) 2. Ensure you have added or run the appropriate tests for your PR 3. If the PR is work in progress, mark it a draft on GitHub. 4. Please write your PR title to summarize what this PR proposes, we are following Conventional Commits style for PR titles as well. 5. Be sure to keep the PR description updated to reflect all changes. --> ### What changes were proposed in this PR? <!-- Please clarify what changes you are proposing. The purpose of this section is to outline the changes. Here are some tips for you: 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, you can clarify why it is a bug. 3. If it is a refactoring, clarify what has been changed. 3. It would be helpful to include a before-and-after comparison using screenshots or GIFs. 4. Please consider writing useful notes for better and faster reviews. --> Adds a unit test spec for `SessionUser`. Tests cover: - `getName` returns the wrapped `user.getName` (Principal contract) - `getUid`, `getEmail`, `getGoogleId` delegate to the wrapped User - `getUser` returns the same User instance - `isRoleOf(ADMIN)` returns `true` when role is ADMIN - `isRoleOf(ADMIN)` returns `false` when role is REGULAR Mirrors the existing `JwtParserSpec.scala` style (AnyFlatSpec with Matchers). ### Any related issues, documentation, discussions? <!-- Please use this section to link other resources if not mentioned already. 1. If this PR fixes an issue, please include `Fixes apache#1234`, `Resolves apache#1234` or `Closes apache#1234`. If it is only related, simply mention the issue number. 2. If there is design documentation, please add the link. 3. If there is a discussion in the mailing list, please add the link. --> Closes: apache#5779 ### How was this PR tested? <!-- If tests were added, say they were added here. Or simply mention that if the PR is tested with existing test cases. Make sure to include/update test cases that check the changes thoroughly including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future. If tests were not added, please describe why they were not added and/or why it was difficult to add. --> Spec verified with `sbt "common-auth/testOnly *SessionUserSpec"`. 7 tests passing. ### Was this PR authored or co-authored using generative AI tooling? <!-- If generative AI tooling has been used in the process of authoring this PR, please include the phrase: 'Generated-by: ' followed by the name of the tool and its version. If no, write 'No'. Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details. --> Coauthored-by: Claude Code (Anthropic)
anishshiva7
force-pushed
the
hf/04-audio-mediagen
branch
from
June 19, 2026 18:39
b77eaec to
1ca3ce2
Compare
anishshiva7
force-pushed
the
hf/04-audio-mediagen
branch
from
June 19, 2026 19:11
1ca3ce2 to
57a901c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
hf/03-image-tasks. Until that lands, the diff below may also include PR 3's image-task operator + codegen + spec changes depending on which base GitHub is showing. The new code in this PR iscodegen/AudioTaskCodegen.scala,codegen/MediaGenCodegen.scala, the audio/media-related additions tocodegen/PythonCodegenBase.scala, the new audio fields onHuggingFaceInferenceOpDesc.scala, and the audio/media-task tests inHuggingFaceInferenceOpDescSpec.scala. Once PR 3 merges and this PR is retargeted tomain, the diff should auto-clean to the PR 4 audio/media changes only.What changes were proposed in this PR?
Adds the audio and media-generation task families — 5 HF pipeline tasks — as new
TaskCodegens plugged into the dispatcher established by the text-generation PR:audio tasks:
automatic-speech-recognition,audio-classification,text-to-speechmedia-generation tasks:
text-to-image,text-to-videocodegen/AudioTaskCodegen.scalasupplies the per-task payload + parse Python branches for the 3 audio tasks.codegen/MediaGenCodegen.scalasupplies the per-task payload + parse Python branches for the 2 media-generation tasks.CodegenContextis extended withaudioInput+inputAudioColumn(EncodableString).HuggingFaceInferenceOpDesc.scalagains 2 new@JsonPropertyfields and registersAudioTaskCodegen+MediaGenCodegenin the dispatcher.PythonCodegenBase.scalagrows to host the shared audio/media infrastructure:audio_only_tasks) inprocess_table.automatic-speech-recognitionandaudio-classification.text-to-speech._call_provider, including OpenAI-compatible image/audio endpoints where supported.data:image/...,data:audio/..., ordata:video/...URLs where needed.User-input strings continue to flow through
pyb"..."+EncodableStringso they reach Python asself.decode_python_template('<base64>')rather than raw literals.PythonCodeRawInvalidTextSpecstill passes with 117/117 descriptors py_compile cleanly.Any related issues, documentation, or discussions?
Tracking issue: Add audio and media-generation task families to HuggingFace operator apache#5288
Closes: Add audio and media-generation task families to HuggingFace operator apache#5288
Stacked on: Add image task family (ImageTaskCodegen) to HuggingFace operator /
hf/03-image-tasksParent issue: Add Hugging Face inference operator apache#5041
Closed sibling issue: Add HuggingFaceModelResource REST endpoints for HF operator UI apache#5134
How was this PR tested?
sbt "WorkflowOperator/compile; WorkflowOperator/Test/compile"clean.sbt scalafmtCheckclean.sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.huggingFace.HuggingFaceInferenceOpDescSpec org.apache.texera.amber.util.PythonCodeRawInvalidTextSpec"— 26 focused tests pass, including HuggingFace audio/media task coverage and the raw Python descriptor scan.sbt "WorkflowOperator/testOnly org.apache.texera.amber.util.PythonCodeRawInvalidTextSpec"— 117/117 descriptors py_compile cleanly with the new operator code paths, no marker leaks.Was this PR authored or co-authored using generative AI tooling?
Yes, co-authored with generative AI tooling (Codex).