Testing RecyclerFlatList

[davebcn87]

I'd like to start a discussion on how are we going to test the library. I believe 1.0 should have basic testing to cover the supported use cases / props.

Things that I would consider:

• Unit tests to cover basic functionality of the props we implemented.
• E2E tests to measure performance and prevent regressions.
• E2E to test that fixture app builds and run.

Said that, what do you think is the best way to start: which tools should we use? How do measure coverage?

[fortmarek]

Unit tests to cover basic functionality of the props we implemented.

We can test what we input as props and what is passed down to RecyclerListView. Since we do not have a lot of business logic, the number of tests can stay limited.

E2E tests to measure performance and prevent regressions.

Yes, this would be nice-to-have and we can build on the exploration done by @ElviraBurchik

E2E to test that fixture app builds and run.

This is going to be covered by E2E tests to measure performance.

which tools should we use?

For unit tests, let's use jest. We also already have some tests in Swift and Kotlin which we should keep and potentially expand for the business logic done in the native layer.

For E2E, we shoul explore it further but most likely Detox will be the most straightforward choice. That being said, it's worth to consider appium, too, since it is what our apps are migrating to.

How do measure coverage?

Do we need to? I don't think it will bring a lot of value - especially in a UI-heavy codebase such as ours.

[naqvitalha]

I'd like to restart this discussion. While unit tests approach seems clear we need to decide on E2E tests. One that I definitely want to see covered is samples having identical rendering in both lists.

Detox seems simpler to use and has screenshot comparison as well. How do we feel about Detox vs Appium? I have worked with Appium in the past but not Detox.

[fortmarek]

Based on this internal meeting where we have discussed e2e testing in Shopify apps, I have the following notes:

Detox:

• grey testing
• works different based on how you drive your animations (doesn't work so well with reanimated)
• lots of gotchas, flaky (more than one would expect from e2e test framework)
• cannot interact with the rest of the device (it runs in a separated runtime)
  • you cannot test e.g. flow with email, today widget...

Appium:

• API:
  • weird, outdated
  • mainly designed for web (it's not that suitable for mobile)
  • Shop is working on a new layer over API to make it easier to use
• (a)sync
  • only sync mode will be available from Node 16 - it means having lots of await calls
• xPath
  • powerful but is dependent on the view tree hierarchy which should be an implementation detail
• how to work with login state
  • do what user would do - does it have tab? log out. If not, consider myself be logged out

I think Appium is a better choice if we wanted to test more complex user workflows but for our purposes, I think Detox is the winner. It's simpler and apart from flakiness, its shortcomings should not affect us.

One that I definitely want to see covered is samples having identical rendering in both lists.

Yes, I think we should start with that and possibly broaden the E2E test suite later.