Ceedling V1.0.0 gcov:all Fails to Generate Code Coverage Files for Included .c Files

[jhcasco]

Summary: If file under test is #included using the *.c file rather than the standard *.h file method, gcov:all fails to generate coverage data files regardless of usage of TEST_SOURCE_FILE().

Context: Automotive MISRA guidelines and safety standards highly encourage the usage of static functions when the function is only used internal to the *.c file. This greatly complicates unit testing as many static functions may be called by a single global function. Testing via the global functions only would require a high level of getter access to the *.c file internal variables and state to verify these functions and in some cases testing is not possible or would require a high level of indirect checking for correct behavior (internal state machines are a nightmare scenario). For this reason, #including the *.c file directly gives Ceedling access to the static functions to be tested directly.

// test_foo.c
#ifdef TEST

#include "unity.h"

//#include "foo.h" // Standard way of testing via interfaces in Ceedling
#include "foo.c" // Grants access to static data, does not generate coverage data

void setUp(void)
{
}

void tearDown(void)
{
}

// Test functions here

#endif // TEST

Problem: No *.gcno or *.gcda are generated during the build step for any files under test #included directly as *.c files. Coverage files are correctly generated by files under test #included as *.h files.

Suspected source of issue: gcc flags "-fprofile-arcs -ftest-coverage" are only added during the link step, not during the compilation step. https://gcc.gnu.org/onlinedocs/gcc/Gcov-Data-Files.html suggests these flags can be added during both steps, but I am unsure of whether that will cause issues generating too much coverage data (e.g. test runner files). Regardless, I have manually tested recompiling the *.o files and linking with "-fprofile-arcs -ftest-coverage" in both steps and coverage data was generated for my *.c files under test.

Is it possible to detect files under test #included using *.c files and add these flags to both the compile and link steps of gcc?

Ceedling => 1.0.0

Build Frameworks
CMock => 2.6.0
Unity => 2.6.1
CException => 1.3.4

gcc version 13.2.0 (x86_64-posix-seh-rev0, Built by MinGW-Builds project)

[Letme]

Simple solution: create example_inline_impl.h with all static inline functions and have include of it protected with #ifdef TEST in example.h will lead you with flexibility to include the inline functions in test file by including inline impl file in tests where you do not want to mock them and give the option to mock them where needed by mocking example.h definitions.

You can also add flags to the :gcov_compiler: in your yml section if you think that should be a default option (which I agree should be added in all steps)

[mkarlesky]

Hi, @jhcasco. So sorry for the trouble and thank you for the report.

At the moment, I'm kinda stumped. I tried both #includeing a .c file and using TEST_SOURCE_FILE(). In both cases the extra source file was compiled with coverage and showed up in the coverage reporting.

Can you tell me anything more about your setup? Can you provide any code snippets?

If you look very closely at the build logging, the C files being compiled with coverage include “with coverage” in their progress messages. It's probably easiest to see this by limiting a test coverage build to just a single test file. If you crank up the verbosity with --verbosity=obnoxious (or -v 4) you can also see the compilation command lines for each file in your build. There should be coverage flags for compilation of the extra files there.

[jhcasco]

I will see if I can make a minimal example in the next day or two. The one I'm working on currently is not a small setup (real project) and my minimal setup has some mild quirks I would need to pare down to post. If I find one minimal setup works and another doesn't, I'll try to report the differences too.

Will try with the compiler flags workaround. I'm a bit of a stickler for not using custom "STATIC" keyword redefinition and maintaining support header files with duplicated function prototypes in the test code area seems problematic for a rapidly changing code base.

[jhcasco]

2025-01-07 Ceedling No Coverage Files for foo included as c file.zip

Here is an example where rs_tmr is generating coverage using #include rs_tmr.h in the test file, but foo is not since it is #include foo.c in the test file.

The log shows on line 263 rs_tmr.c is compiled with coverage. I think foo.c is not compiled separately since it is only #included in the test file and used nowhere else... Thoughts?

[mkarlesky]

Thank you for this example, @jhcasco. I need to actually run it and investigate further but now I understand your original report better. I'm pretty sure the problem is because the C source file you're including is the same as the file the test thinks it's exercising by convention (test_foo.c means foo.c is the file under test, if present). I'll have to dig in a bit. More than likely it is some kind of bug in filtering out multiple references to the same source file.

[jhcasco]

Thanks @mkarlesky and thank you @Letme . Adding

:flags:
  :test:
    :compile:
      '*':
        - -fprofile-arcs
        - -ftest-coverage

to my project.yml allowed test coverage generation that gcovr could pick up on, but as you suspected @mkarlesky, the gcov summary does not detect the coverage since it is hidden in the test_foo files. Hope that helps track it down. For now, this works as well since I work off of the gcovr report anyway.

[mkarlesky]

@jhcasco It took me a minute to figure this out, but I understand the essential problem. I'm not actually sure of the best way to handle your case.

Here are the two realities that are colliding:

1. The Gcov plugin compiles source code with coverage, not unit test code, test runners, or mock code. The basic idea is to instrument the code under test to see what your test code is exercising. Generally speaking, instrumenting the test code, runners, and mocks generates a lot of noise and can cause funny coverage results in some cases.
2. The C #include directive will slurp up any file you give it. Most commonly it's used for header files to collect symbol definitions, macros, and function declarations that the code of the file using the #include directives needs. During compilation, the contents of a file #included are actually expanded in place within the file that makes the reference. It's for this reason that include guards are a thing. Nested #include statements can cause the same file to be expanded multiple times. At any rate, by including a C source file, that code becomes part of the test file itself. The unit test code and the source code under test are effectively the same file.

So, in terms of coverage processing for a test build, the source file being #included in your test file does not exist. It's simply never encountered by the compiler — with or without coverage enabled. Conversely, the test file that effectively contains the code you want coverage metrics for is intentionally not compiled with coverage.

The simplest, brute force way to solve this is to do just what @Letme provided. Beyond that we get into philosophical discussions of how coverage testing should work and adding options to the Gcov plugin to support different philosophies (perhaps something that needs to happen).

[Letme]

@mkarlesky and @mvandervoord can you bump the version of these prereleases to 1.0.1 as now we have 1.0.0 released and the pre-releases come afterwards, making it strange...

[jhcasco]

@mkarlesky This may be ignorance on my part on how coverage is generated and the implications of generating it for the test code, but it seems like all the tools are there to allow some settings to pinpoint those test files that need to be compiled with coverage in this use case. The log's Collecting Test Context step correctly extracts *.h #include statements and *.c #include statements. Would simply need some conditional checks of extension to determine if the test_ file needs to be generated with or without coverage. This would prevent Ceedling from having to build all the test files with coverage and hopefully minimize noise on the output.

Again, I'm only looking at the end result from gcovr so I don't know gcov's intermediate step for generating the summary, but gcovr is able to ignore the test_ file coverage and extract only the coverage of the file under test so long as my report_include setting points to the source code paths and not any test code paths. Is such a directive available for gcov?

[mkarlesky]

@jhcasco The essential problem is that by including the C source file in the test file, the C source file that you want compiled with coverage is not ever compiled on its own. Its contents are part of the test file and get compiled as contents of the test file. There is functionality exposed in the plugin that should allow you to customize filters, etc., but I'm not sure how far that will take you for nice clean reporting.

While I could be wrong, I'm reasonably confident that you will not be able to filter the coverage results for the source file. Again, that's because the source file doesn't actually exist when including it directly in the test file. The content of your source file is now part of the test file. So, I believe you'll be able to filter coverage results by your test file, and those coverage results will include the functions that are in the source file (effectively copied into the test file).

We can certainly add an option to compile the test files with coverage, but I do not believe that will not cause your source functions to be listed in coverage results as part of your source file. The functions will be listed as part of your test file.

It might be possible to also compile the source file separately with coverage and the coverage results might correlate in such a way that source functions exercised in the test file could be filtered in the results to match the separately compiled source file. I'm not sure without playing with it. This is complicated and one of the reasons we generally discourage including C files in test files even though the C language allows it. It leads to difficult to deal with consequences and violates a general best practice of separating test and source code (though I do see the value of what you're doing for dealing with static functions!).

[jhcasco]

@mkarlesky Understood. It's a nonstandard use case and has a lot of concerns that can't be easily analyzed or maintained.

In the meantime, I will leave an updated configuration showing how brute forcing the coverage in the test files per @Letme and filtering gcovr for only source files (not test files) still generates a coverage report. This currently cannot be summarized by Ceedling, it is only visible in the gcovr report. If anyone has the same issue or wants to investigate further, this appears to be a viable workaround for now.

2025-01-08 Ceedling Brute Force gcovr Report for foo included as c file.zip