How to set up/scan C & C++ project via command line?

[davewichers]

I started with the following:

• codeql database create path/to/DBNAME --language=cpp --command="MYBUILDCOMMAND"
• codeql database analyze path/to/DBNAME codeql/cpp-queries --format=sarifv2.1.0 --output=FILENAME.sarif

And when the analyze completes, it says: "CodeQL scanned 2 out of 1562 C/C++ files in this invocation. Typically CodeQL is configured to analyze a single CodeQL language per invocation, so check other invocations to determine overall coverage information."

I looked at the documentation, and apparently you can set the language to multiple languages, e.g., --language=cpp,c

But when I do that, it returned an error during database create that I had to also specify --db-cluster. So, I added that, so now my first command is:

• codeql database create path/to/DBNAME --language=cpp,c --db-cluster --command="MYBUILDCOMMAND"

But then, when I run the database analyze step, I now get this error:

"A fatal error has occurred: path\to\DBNAME is not a recognized CodeQL database."

How do I fix that?

[jketema]

c and cpp are just aliases, so there's no reason to specify both of them, and your initial

codeql database create path/to/DBNAME --language=cpp --command="MYBUILDCOMMAND"

was correct.

If only 2 files were scanned, then we detected only 2 C/C++ compiler invocations during database creation. The most common reason for this is that build artifacts were not cleaned before creating the database, and subsequently build steps were skipped. Another common reason is the use of caching, e.g., with ccache, which is not supported. It is hard to tell though what is going on without further details.

[davewichers]

That didn't help. I deleted all the .obj files in the directory where the compile command is run, and I can see the compilation of every file happening. For each source file I'm seeing: [DATE/TIME] [build-stdout] FILENAME.c

The parameters to the cl invocation are: cl /I "path\to\supportclasses" /W3 /MT /GS /RTC1 /bigobj /EHsc /nologo /MP /c main.cpp STANDARD_PREFIX*.c STANDARD_PREFIX*.cpp (Note: /I is slash capital I, not L)

An .obj file is created for each source file during this process.

I'm using 'cl' as the compiler if that makes any difference. A single invocation of cl with wildcards for all the files to compile is used. i.e., STANDARD_PREFIX*.c STANDARD_PREFIX*.cpp along with paths to the supporting classes in other paths.

[jketema]

I think you hit a limitation in our tooling: we don't handle wildcard expansion on Windows. You should be able to confirm this by looking at the build-tracer.log file that occurs somewhere in the database directory. I'd expect you'll see something along the following lines:

Catastrophic error: cannot open source file "STANDARD_PREFIX*.c": Invalid argument

[davewichers]

Yes, I'm seeing those errors in the logs. Do you have an open bug report and are working on a fix? If so, any ETA for the fix?

In the meantime, is there a workaround on Windows that does work? I.e., can I use a bash shell, or Windows Subsystem for Linux shell, or some other shell that does work?

If you have no other suggestions, then I guess I can try it on Rocky 9 instead (or some Linux flavor)?

[jketema]

We have an internal issue open, but fixing this has not been prioritized, because this is pretty much an edge case (build systems generally specify all files explicitly). The workaround would be to specify all files explicitly. I would assume you can do this by using a Bash shell on Windows, but I have not tested that.

[davewichers]

The problem with that is that CL, which is the Windows compiler installed and made accessible in the Visual Studio Command prompt when you install Visual studio is NOT accessible via bash or WSL, so you can't run CL outside of a Visual Studio Command prompt. So, wildcard expansion in Windows shell really should be fixed to make this work.

[jketema]

The problem with that is that CL, which is the Windows compiler installed and made accessible in the Visual Studio Command prompt when you install Visual studio is NOT accessible via bash or WSL, so you can't run CL outside of a Visual Studio Command prompt.

That's not true. You can make the compiler accessible from any prompt. It's not tied to the command prompts that Visual Studio provides. I can understand that you don't want to go that route that, but then the question is: why can't specify all files explicitly?

[davewichers]

It's actually very complicated to make cl work in a non-VS command prompt as you have to run vcvarsall.bat to set up the proper environment for cl AND cl.exe in the same command line. So, while it is technically possible, it is very complicated/hard to get work. And when you have 1500+ source files in each folder, and I have over one hundred folders, listing each source file is a pain. So, it would be massively simpler for the user if you could get wildcard expansion to work in Windows. I'm trying to get the vcvarsall.bat plus cl.exe setup to work but no luck. Way too complicated.

[jketema]

I'm just trying to offer workarounds here, as this is not something we can fix instantaneously. Especially since this is a bit of an edge case from our perspective, as I mentioned already.

And when you have 1500+ source files in each folder, and I have over one hundred folders, listing each source file is a pain.

Why list them by hand? The CL supports response files, so you could just write a script that generates the appropriate response files.

[davewichers]

OK. I tried that but ran into an error. I created two .rsp files, the 1st is the standard parameters to CL, and the 2nd is the list of all the source files to compile. So, I can run this just fine: cl @standardParams.rsp @listOfAllSourceFilesToCompile.rsp

listOfAllSourceFilesToCompile.rsp lists1776 source files, 1 per line.

This works no problem, but when I pass that cl command as the --command parameter value to: 'codeql database create', it responds with the error: "(eventual cause: IOException "CreateProcess error=206, The filename or extension is too long")"

So, it doesn't look like you can compile all of the source files in a single invocation when creating the database.

When you invoke codeql database create, the is shows this output:

Initializing database at: path\to\DBNAME
Running build command: [cl, @standardParams.rsp, @listOfAllSourceFilesToCompile.rsp]
Running command in DIRNAME: [cl, then each parameter from the first rsp file followed by a COMMA and SPACE, then each filename from the 2nd rsp file also followed by COMMA and SPACE, and then the error:
(eventual cause: IOException "CreateProcess error=206, The filename or extension is too long")

[jketema]

I will need a reproducing test case if you want us to look into that further. From this I also suspect that fixing wildcard expansion will not be sufficient in you case; with wildcard expansion fixed, I expect we will see this error too.

[davewichers]

You can simply create a project with a source file to compile and duplicate that file 1700 times with different source file names so you can reproduce the same thing. Is there no way to expand the size of the command line to accommodate a much longer command string? Or do things in a different way so it doesn't have to expand it all into a single line, which runs into this line size error?

[jketema]

No, because I think we're running into Windows limitation with regard to size of what can be passed to CreateProcess.

You can simply create a project with a source file to compile and duplicate that file 1700 times with different source file names so you can reproduce the same thing.

Either you should be able to share the code, because you work on an open source project, or you have a GitHub advanced security license and should be able to get me a reproducing test case through a support ticket.

[davewichers]

My project is not in GitHub and it's not shareable. I'm doing this all at the command line locally using a local/private source code repo.

[jketema]

That means you have a GHAS license, and can open a support ticket with some kind of reproducer. I would not like to second guess this problem based on a somewhat vague description you're providing.

[davewichers]

As an alternative, can you run the database create a bunch of times, once per source file and build up the database that way so you don't try to compile everything all at once?

[jketema]

Yes, but then you need to use database init and database trace-command, but how would that be different from just calling the compiler multiple times from whatever build script you already have?

[davewichers]

My build script is a single invocation of CL. So, using the above commands, how would I would I 'build up' the database with multiple invocations? Can you give me an example of compiling 2 files and I assume the way you do the 2nd compile is just repeated for each of the rest of the files, and then is there a command at the end to finalize the database, or whatever, that you need to do before your run the analyze step?

[jketema]

The easiest way is to create a script, e.g. a batch script, with multiple cl invocations. Then do what you did before.

The hard way would be something like the following, but I'm not sure why you would make it hard on yourself by calling database trace-command multiple times:

# 1. Initialize the skeleton database (sets up tracing environment)
codeql database init --language=cpp --source-root . path/to/db

# 2. Call trace-command multiple times — one per build step/component
codeql database trace-command path/to/db -- gcc -c src/core/core.cpp -o build/core.o

codeql database trace-command path/to/db -- gcc -c src/utils/utils.cpp -o build/utils.o

codeql database trace-command path/to/db -- gcc -c src/main.cpp -o build/main.o

# 3. Finalize the database
codeql database finalize path/to/db