Skip to content

feat(flux2): add regional prompting for FLUX.2 Klein#9255

Draft
Pfannkuchensack wants to merge 1 commit into
invoke-ai:mainfrom
Pfannkuchensack:feat/flux2-klein-regional-guidance
Draft

feat(flux2): add regional prompting for FLUX.2 Klein#9255
Pfannkuchensack wants to merge 1 commit into
invoke-ai:mainfrom
Pfannkuchensack:feat/flux2-klein-regional-guidance

Conversation

@Pfannkuchensack
Copy link
Copy Markdown
Collaborator

Summary

FLUX.2 Klein's diffusers transformer already accepts an attention mask through joint_attention_kwargs, and its unified attention sequence [txt, img] matches FLUX 1, so regional prompting works without a forward override — we just build a restricted mask and pass it down.

Backend:

  • Flux2RegionalPromptingExtension builds a restricted [txt, img] mask from per-conditioning Qwen3 embeddings + image masks (FLUX 1 logic, no CLIP pooled). Unlike FLUX 1 the same mask is applied to every block (no per-block alternation, since we use the diffusers forward).
  • flux2 denoise() threads pos_joint_attention_kwargs to all positive transformer forwards; negative passes stay unmasked.
  • flux2_denoise invocation accepts list[FluxConditioningField] and skips masking when reference images extend the image stream.

Frontend:

  • buildFLUXGraph wires posCondCollect + addRegions for the FLUX.2 path.
  • addRegions clones flux2_klein_text_encoder edges (qwen3_encoder, max_seq_len, mask) for each regional prompt.
  • validators surface unsupported features (regional negative, auto-neg, reference images) so FLUX.2 regions are filtered consistently with FLUX 1.
  • schema.ts regenerated to reflect the list-or-single union.

Related Issues / Discussions

n/a

QA Instructions

  1. Load any FLUX.2 Klein model in the canvas.
  2. Create two or more Regional Guidance layers with distinct prompts and non-overlapping masks (e.g. left half = "a red car", right half = "a blue truck").
  3. Invoke txt2img — generated image should respect the regional split.
  4. Verify the unsupported-feature warnings show up on a FLUX.2 region that has:
    • a negative prompt
    • auto-negative enabled
    • a reference image attached
  5. Confirm a region with reference images is filtered out (no regional masking applied) — invocation should still run as a global prompt.
  6. Sanity-check that a global FLUX.2 Klein generation (no regions) is byte-identical to before this PR (no mask is built when no region masks are present, short-circuit returns None).
  7. Regression-check FLUX 1 and Z-Image regional prompting still works.

Merge Plan

Standard merge. No DB/schema migration. schema.ts is regenerated and committed.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • ❗Changes to a redux slice have a corresponding migration
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)

FLUX.2 Klein's diffusers transformer already accepts an attention mask
through joint_attention_kwargs, and its unified attention sequence
[txt, img] matches FLUX 1, so regional prompting works without a forward
override — we just build a restricted mask and pass it down.

Backend:
- Flux2RegionalPromptingExtension builds a restricted [txt, img] mask
  from per-conditioning Qwen3 embeddings + image masks (FLUX 1 logic,
  no CLIP pooled). Unlike FLUX 1 the same mask is applied to every
  block (no per-block alternation, since we use the diffusers forward).
- flux2 denoise() threads pos_joint_attention_kwargs to all positive
  transformer forwards; negative passes stay unmasked.
- flux2_denoise invocation accepts list[FluxConditioningField] and
  skips masking when reference images extend the image stream.

Frontend:
- buildFLUXGraph wires posCondCollect + addRegions for the FLUX.2 path.
- addRegions clones flux2_klein_text_encoder edges (qwen3_encoder,
  max_seq_len, mask) for each regional prompt.
- validators surface unsupported features (regional negative, auto-neg,
  reference images) so FLUX.2 regions are filtered consistently with
  FLUX 1.
- schema.ts regenerated to reflect the list-or-single union.
@github-actions github-actions Bot added python PRs that change python files invocations PRs that change invocations backend PRs that change backend files frontend PRs that change frontend files labels May 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend PRs that change backend files frontend PRs that change frontend files invocations PRs that change invocations python PRs that change python files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant