[FeatureRequest] Error source classification through Skipped tests or alternative solution

[alastaire61]

Issue:
It's not possible to sort/filter out errors grouped by their causing source in the html report.

Current behavior:
Whenever I am using either Assume.assumeTrue(false) or throw new SkipException() inside a WebDriverException catch{} clause in a Step Definition method, scenario shows as Ignored, rather than Skipped, probably because it's coming from a previous exception.

Even though, the steps are marked correctly in the cucumber-reporter generated report, without failures:

But the scenario itself is marked as FAILED.

I do understand the design purpose of not passing an scenario which does not contain all the PASSED steps.

But, in my case, application bugs (often timeouts or inner application errors) are not related with expected tested subject (website or outter application being tested) bugs, so that's why I would like to remove/separate application bugs from final Assertion bugs, and the easiest way I thought in order to achieve so, was through Skipping those.

Request:
Is it possible to have a Status Ignored for such test cases, in order to use:
configuration.setNotFailingStatuses(Collections.singleton(Status.IGNORED));

Or, as an alternative, is it possible to have a custom exception or flag that cucumber-reporter will read in order to mark a given scenario as SKIPPED inside the report, instead of FAILED, when there is no step which failed?

Another alternative would maybe be to be able to filter/group errors that are coming from the same source inside the HTML report.

Edit: is there any workaround for this request currently or that could be achieved somehow?

Disclaimer:
I don't intend with this feature to fake false positive test cases, but rather have a method of separating different error sources within the report, since filtering by error type is currently not an option.

[alastaire61]

So digging up a bit, I found:

beforeStatus = new StatusCounter(before, configuration.getNotFailingStatuses()).getFinalStatus(); afterStatus = new StatusCounter(after, configuration.getNotFailingStatuses()).getFinalStatus(); stepsStatus = new StatusCounter(steps, configuration.getNotFailingStatuses()).getFinalStatus(); elementStatus = calculateElementStatus();

Where:

/** * If statuses for all items are the same then this finalStatus is returned, otherwise {@link Status#FAILED}. * * @return final status for this counter */ public Status getFinalStatus() { return finalStatus; }

For some reason, my scenario has PASSED Before and After hooks status, but the Steps status is returning FAILED, whereas in the JSON I can see that there is no Failed step, but only Passed and Skipped steps.

Would it be possible to have an option to change this behavior:

`stepsStatus = new StatusCounter(steps, configuration.getNotFailingStatuses()).getFinalStatus();`

By marking this counter as PASSED whenever a Scenario contains both step statuses as PASSED and SKIPPED, and not any FAILED step ?

[alastaire61]

I was able to solve the issue.
My problem was that I was trying to use:
configuration.setNotFailingStatuses(Collections.singleton(Status.SKIPPED)); configuration.setNotFailingStatuses(Collections.singleton(Status.PENDING)); configuration.setNotFailingStatuses(Collections.singleton(Status.UNDEFINED));

but there is an override for every status that you setup here:

public void setNotFailingStatuses(Set<Status> notFailingStatuses) { if (notFailingStatuses != null) { this.notFailingStatuses = notFailingStatuses; } }
I would change this to admit more than one status at once, to avoid conflicts.

[damianszczepanik]

Report is generated from JSON file then you need to probably work on the JSON and make sure it has SKIPPED status there and then the report should be generated correctly

[super-brian]

I added the following line and now 'skipped' scenarios are reported as 'passed'.
configuration.setNotFailingStatuses(Set.of(Status.SKIPPED, Status.PENDING, Status.UNDEFINED));