mono repo -> multi repos throwing Error: Duplicate key ...in UniqueKeyDict<models>

[lafirm]

We initially used a monorepo setup but recently decided to switch to a multi-repo structure. To support this, we introduced the project variable in config.yaml as described in the SQLMesh multi-repo guide. However, adding the project variable to our existing repository is causing the following error:

Error: Duplicate key '"iceberg"."schema"."table_1"' found in UniqueKeyDict. Call dict.update(...) if this is intentional.

Is there an easy or recommended way to migrate from a monorepo to a multi-repo setup? Thank you!

[tobymao]

can you provide a fully reproducible minimal example?

[lafirm]

can you provide a fully reproducible minimal example?

sure, the below steps will produce the error.

sqlmesh==0.167.0
python==3.13

sqlmesh init duckdb

sqlmesh plan

and apply the plan. Once done, introduce the project variable in the config.yaml like this,

project: demo

gateways:
  duckdb:
    connection:
      # For more information on configuring the connection to your execution engine, visit:
      # https://sqlmesh.readthedocs.io/en/stable/reference/configuration/#connections
      # https://sqlmesh.readthedocs.io/en/stable/integrations/engines/duckdb/#connection-options
      type: duckdb
      database: db.db
      # concurrent_tasks: 1
      # register_comments: True
      # pre_ping: False
      # pretty_sql: False
      # catalogs: 
      # extensions: 
      # connector_config: 
      # token: 


default_gateway: duckdb

model_defaults:
  dialect: duckdb
  start: 2025-05-21

then again run

sqlmesh plan

which will throw the below error,

Error: Duplicate key '"db"."sqlmesh_example"."incremental_model"' found in UniqueKeyDict. Call dict.update(...) if this is intentional.

Note:
Model name in the error output is different every time and the error occurs even with -p arg in the command, refer below for the sequence of tries and errors in th cli

.venv(bot)@K6W46PHGL7 sqlmesh-repo % sqlmesh plan
Error: Duplicate key '"db"."sqlmesh_example"."incremental_model"' found in UniqueKeyDict. Call dict.update(...) if this is intentional.
.venv(bot)@K6W46PHGL7 sqlmesh-repo % sqlmesh plan
Error: Duplicate key '"db"."sqlmesh_example"."full_model"' found in UniqueKeyDict. Call dict.update(...) if this is intentional.
.venv(bot)@K6W46PHGL7 sqlmesh-repo % sqlmesh -p /Users/bot/projects/sqlmesh-repo plan
Error: Duplicate key '"db"."sqlmesh_example"."seed_model"' found in UniqueKeyDict. Call dict.update(...) if this is intentional.
.venv(bot)@K6W46PHGL7 sqlmesh-repo % sqlmesh -p /Users/bot/projects/sqlmesh-repo plan
Error: Duplicate key '"db"."sqlmesh_example"."incremental_model"' found in UniqueKeyDict. Call dict.update(...) if this is intentional.
.venv(bot)@K6W46PHGL7 sqlmesh-repo %

[georgesittas]

Thanks @lafirm, I can reproduce this behavior against main. It does seem like a bug, we'll take a closer look.

[lafirm]

@georgesittas, @tobymao - I've a simple suggestion to fix the issue and created a PR for that, please have a look whenever you get some time.

[lafirm]

Additionally, the same error will occur if we change the project name.

MRE:

1. sqlmesh init duckdb
2. add the project var in config yaml
3. sqlmesh plan and then apply
4. change the project var in config yaml
5. sqlmesh plan and then the duplicateKey error raises

[codykonior-hsi]

I encountered this today trying to switch from a config with no project line, to one with a project line. So, this problem still exists.

(For anyone else that tries, sqlmesh destroy will fail in that state as well, you need to remove the project line again to be able to destroy your old config).

What's weird is, if you start with a config with a project name, you can remove the project name and run sqlmesh plan and it will remove it from all the models safely. So, you can go from named to nameless, but not the other way around. That's a real shame.

Anyway, I did notice in the multi-repo guide https://sqlmesh.readthedocs.io/en/stable/guides/multi_repo/ it says:

Even if you do not have a need for multiple repos now, consider adding a project key so that you can easily support multiple repos in the future.

But it's interesting that sqlmesh init doesn't create one in its default config. There's two potential changes you could make:

• If you aren't able to fix that, in the documentation you could explicitly state you can't add a project name to an existing config.
• sqlmesh init you could give configs a project a name by default, like "project: main". That way future users aren't caught out.