andreas.heigl.orgCode-Coverage with PCOV in a mono-repo
Today I finally managed to find and fix an issue that we had for some time with Code-Coverage generation with PCOV in github actions.
To give you a bit of background, imagine a project that uses 2 separate folders where one contains the business-logic and one contains the framework-related code. The framework-related part contains a lot of integration and end-to-end test that – of course – also use and therefore test the business-logic.
Every PullRequest runs the tests both from the bunsiness-logic part as well as from the framework-related part. And to give the developers feedback how well the changed code is covered with tests we collect coverage data and generate a patch-coverage report. I wrote about that some time back. So we now get a comment each in the PR that contains the untested lines in the business-logic related part and the framework-related part respectively.
That worked flawlessly for the business-logic related part as those were mostly unit-tests that do not rely upon the framework-related code.
The framework-related part though didn’t work that great. It always showed the business-logic related parts as untested even though I knew they were tested.
So… Why?
After some digging I realized that there is a difference in how the coverage is created between Xdebug and pcov – yes! There is obviously a difference as those are different tools. But there is also a difference in what is included in the reports.
And today I figured out what it was and why it broke our proces in our specific setup in GitHub actions.
For a start: We have our code in two folders: business and framework.
We also have a framework/phpunit.xml file that contains this code:
<?xml version="1.0" encoding="UTF-8"?><phpunit
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/12.1/phpunit.xsd"
processIsolation="false"
stopOnFailure="false"
>
<coverage/>
<source>
<include>
<directory>src</directory>
<directory>../business/src</directory>
</include>
</source>
</phpunit>So coverage shall be collected from the current src folder as well as from the business/src folder.
This works as expected when you run phpunit with Xdebug as coverage-generator.
With pcov as coverage-generator it worked locally as expected.
But not in our GitHub Actions…
Enter pcov.directory
After a bit of digging I learned that there is the pcov.directory ini-setting that determines which code is considered for coverage. So I tested my GitHub Actions by adding some test-code like this:
jobs:
tests:
steps:
- name: "Checkout"
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
ref: ${{ github.event.pull_request.head.ref }}
- name: "Install PHP"
uses: shivammathur/setup-php@44454db4f0199b8b9685a5d763dc37cbf79108e1 # v2
with:
extensions: "intl, pdo_mysql, zip, imap, xml, soap, apcu, redis"
php-version: "8.4"
coverage: "pcov"
- name: "check pcov.directory"
run: |
php -i | grep pcovRunning this in GitHub Actions provided me with output similar to this:
Run php -i | grep pcov
/etc/php/8.4/cli/conf.d/20-pcov.ini,
pcov
pcov.directory => /home/runner/work/<org-name>/<repo-name>
pcov.exclude => none
pcov.initial.memory => 65336 bytes
pcov.initial.files => 64/home/runner/work/<org-name>/<repo-name> was exactly what i expected! So why the heck does it seemingly not work?
After some more testing I realized that I was at one point doing a cd framework before running PHPUnit.
- name: "Run tests"
run: |
cd framework
vendor/phpunit/phpunit/phpunit --coverage-php /tmp/${{ github.sha }}_coverage.covShould that cause some issues?
So I added the php -i | grep pcov after the PHPUnit run like this:
- name: "Run tests"
run: |
cd framework
vendor/phpunit/phpunit/phpunit --coverage-php /tmp/${{ github.sha }}_coverage.cov
php -i | grep pcovAnd what was that? Suddenly I got
pcov.directory => /home/runner/work/<org-name>/<repo-name>/framework/src🤯
Suddenly the pcov directory was set to something totally different. And with that setting it was clear that I couldn’t get what I expected as now only content from framework/src/ would be considered as covered. And everything from the business folder was not considered for coverage.
So the only question left was whether that was something happening deep in PHPUnits logic or just based on changing the directory.
Long story short: It’s just based on the change directory. In essence setup-php sets up pcov so that the current folder is considered as pcov.directory. As pcov itself adds src when that folder is available that explains the changed folder.
But is there a way to change that? To use a fixed folder despite the current working directory being changed?
Yes! There is.
I added ini-values: "pcov.directory=$GITHUB_WORKSPACE" to the setup-php directive like this:
- name: "Install PHP"
uses: shivammathur/setup-php@44454db4f0199b8b9685a5d763dc37cbf79108e1 # v2
with:
extensions: "intl, pdo_mysql, zip, imap, xml, soap, apcu, redis"
ini-values: "pcov.directory=$GITHUB_WORKSPACE"
php-version: "8.4"
coverage: "pcov"And all of a sudden the pcov.directory stayed regardless of where I changed the working directory to.
And so with this little change I was again able to get the coverage-data from the business folder whenever i run tests in the framework folder.
Helpful links:
#coverage #pcov #php #phpunit
Greg Korba
Dan 🦺 Deboer
Sebastian Bergmann 
Shyim