Discuss integration, differential, fuzz testing in how-to guides

docs/build/guides/testing/coverage-testing.mdx

@@ -0,0 +1,9 @@
+---
+title: Coverage Testing
+hide_table_of_contents: true
+description: Coverage testing finds code not tested.
+sidebar_position: 8
+draft: true
+---
+
+**TODO: Fill in example.**

docs/build/guides/testing/detecting-changes-with-test-snapshots.mdx → docs/build/guides/testing/differential-tests-with-test-snapshots.mdx

@@ -1,16 +1,23 @@
 ---
-title: Detecting Unexpected Changes with Test Snapshots
+title: Differential Tests using Snapshots
 hide_table_of_contents: true
-description: Use test snapshots to detect unexpected changes in contract behavior
+description: Differential testing using automatic test snapshots.
+sidebar_position: 7
 ---
 
 Tests are written to ensure that contracts behave today as expected, and in the future as well. Over time a contract may change and in all software development there remains the possibility of changes causing side-effects that are unexpected. Testing is one of the ways that we identify unexpected changes.
 
 However tests are limited, as they only show changes to values that the tests assert on.
 
-## Test Snapshots
+:::tip
 
-The Soroban Rust SDK generates test snapshots on every test involving an `Env`. Test snapshots are enabled by default. At the end of the test the `Env` writes a JSON file to the `test_snapshots` directory with a full snapshot of all the events published, logged, and the final ledger storage state.
+Test snapshots are one tool for performing differential testing. See [Differential Testing] for other ways.
+
+:::
+
+### Test Snapshots
+
+The Soroban Rust SDK generates test snapshots on every test involving an `Env`. Test snapshots are enabled by default. At the end of the test the `Env` writes a JSON file to the `test_snapshots` directory with a full snapshot of all the events published, and the final ledger storage state.
 
 Most tests have a single `Env` and will result in a single test snapshot. Tests that have multiple `Env`s will write multiple test snapshots, one for each `Env`. Test snapshot files are named with a incrementing number on the end to separate the test snapshots for each `Env`.
 
@@ -32,8 +39,10 @@ Most tests have a single `Env` and will result in a single test snapshot. Tests
 
        assert_eq!(client.increment(), 1);
 
+       // highlight-start
        // At the end of the test the Env will automatically write a test snapshot
        // to the following directory: test_snapshots/test_abc.1.json
+       // highlight-end
    }
    ```
 
@@ -43,11 +52,16 @@ Most tests have a single `Env` and will result in a single test snapshot. Tests
 
 4. On future updates look out for changes to test snapshots in tests that are unexpected. For example, when changing one part of a contract if the test snapshots for other parts of the contract or unrelated end-to-end tests change, that could signal that side-effects have occurred.
 
-5. As needed, diff test snapshots as needed to look for hints to why the unexpected change may have occurred.
+5. Diff test snapshots as needed to look for hints to why an unexpected change has occurred.
+
+:::info
+
+Test snapshots files are verbose. Test snapshots are most useful when changes appear and can be diffed, such as a new event being published, or storage changing.
 
-Note that test snapshots are extremely verbose and thoroughly understanding each in isolation may not be realistic. Test snapshots are most useful when changes appear and can be diffed, such as a new event being published, or storage changing.
+:::
 
 To give this a go, check out the [Getting Started] contract or any of the [examples], run the tests, and look for the test snapshots on disk.
 
+[Differential Testing]: ./differential-testing
 [Getting Started]: ../../smart-contracts/getting-started/README.mdx
 [examples]: ../../smart-contracts/example-contracts/README.mdx

docs/build/guides/testing/differential-tests.mdx

@@ -0,0 +1,110 @@
+---
+title: Differential Tests
+hide_table_of_contents: true
+description: Differential testing detects unintended changes.
+sidebar_position: 6
+---
+
+Differential testing is the testing of two things to discover differences in their behavior.
+
+The goal is to prove that the two things behave consistently, and that they do not diverge in behavior except for some expected differences. The assertions should be as broad as possible, broadly testing that all observable outcomes do not change, except for any expected changes.
+
+This strategy is effective when building something new that should behave like something that already exists. That could be a new version of a contract that has unchanged behavior from it's previous version. Or it could be the same contract with an updated SDK or other dependency. Or it could be a refactor that expects no functional changes.
+
+This strategy can be used in the context of unit and integration tests, or in the context of fuzz tests as well.
+
+:::tip
+
+All contracts built with the Rust Soroban SDK have a form of differential testing built-in and enabled by default. See [Differential Testing with Test Snapshots].
+
+:::
+
+### How to Write Differential Tests
+
+To experiment with writing a differential test, open a contract that you've deployed, or checkout an example from the [soroban-examples] repository and deploy it.
+
+Assuming the contract has been deployed, and changes are being made to the local copy. We need to check that unchanged behavior in the contract hasn't changed compared to what is deployed.
+
+1. Use the [stellar contract fetch] command to fetch the contract that's already deployed. The contract already deployed will be used as a baseline that the local copy is expected to behave like.
+
+   ```console
+   $ stellar contract fetch --id C... --out-file contract.wasm
+   ```
+
+2. Write a test that runs the same logic for the deployed contract and the local contract, comparing the result. Assuming the [increment example] is in use, the test would look something like the following.
+
+   ```rust
+   #![cfg(test)]
+   use crate::{IncrementContract, IncrementContractClient};
+   use soroban_sdk::{testutils::Events as _, Env};
+
+   mod deployed {
+       soroban_sdk::contractimport!(file = "contract.wasm");
+   }
+
+   #[test]
+   fn differential_test() {
+       let env = Env::default();
+       assert_eq!(
+           // Baseline – the deployed contract
+           {
+               let contract_id = env.register(deployed::WASM, ());
+               let client = IncrementContractClient::new(&env, &contract_id);
+               (
+                   // Return Values
+                   (
+                       client.increment(),
+                       client.increment(),
+                       client.increment(),
+                   ),
+                   // Events
+                   env.events.all(),
+               )
+           },
+           // Local – the changed or refactored contract
+           {
+               let contract_id = env.register(IncrementContract, ());
+               let client = IncrementContractClient::new(&env, &contract_id);
+               (
+                   // Return Values
+                   (
+                       client.increment(),
+                       client.increment(),
+                       client.increment(),
+                   ),
+                   // Events
+                   env.events.all(),
+               )
+           },
+       );
+   }
+   ```
+
+3. Run the test to compare the baseline and local observable outcomes.
+
+This test uses the same patterns used in [unit tests] and [integration tests]:
+1. Create an environment, the `Env`.
+2. Import the Wasm contract to compare with.
+3. Register the local contract to be tested.
+4. Invoke functions using a client.
+5. Assert equality.
+
+:::tip
+
+Differential tests work best when less assumptions are made. Rather than asserting only on specific return values or on implementation details like specific state, asserting on all observable outcomes and including things like events published or return values of other read-only contract functions will help to discover unexpected changes.
+
+:::
+
+:::info
+
+Depending on the test complexity it can be desirable to use an independent `Env` for testing the deployed vs local. However at the moment it is only possible to compare host values, like `String`, `Bytes`, `Vec`, `Map`, if they've been created using the same `Env`. The tracking issue for supportin comparisons across environments is [stellar/rs-soroban-sdk#1360].
+
+:::
+
+[Getting Started]: ../../smart-contracts/getting-started
+[increment example]: https://github.com/stellar/soroban-examples/blob/main/increment/src/lib.rs
+[Differential Testing with Test Snapshots]: ./differential-testing-with-test-snapshots
+[stellar contract fetch]: ../../tools/developer-tools/cli/stellar-cli#stellar-contract-fetch
+[integration tests]: ./integration-tests
+[unit tests]: ./unit-tests
+[stellar/rs-soroban-sdk#1360]: https://github.com/stellar/rs-soroban-sdk/issues/1360

docs/build/guides/testing/fuzzing.mdx

@@ -0,0 +1,126 @@
+---
+title: Fuzz Tests
+hide_table_of_contents: true
+description: Fuzzing and property testing to find unexpected behavior.
+sidebar_position: 5
+---
+
+Fuzzing is the process of providing random data to programs to identify unexpected behavior, such as crashes and panics.
+
+Fuzz tests can also be written as property tests that instead of seeking to identify panics and crashes, assert on some property remaining true. Fuzzing as demonstrated here and elsewhere in these docs will use principles from both property testing and fuzzing, but will only use the term fuzzing to refer to both.
+
+The following steps can be used in any Stellar contract workspace. If experimenting, try them in the [increment example]. The contract has an `increment` function that increases a counter value by one on every invocation.
+
+### How to Write Fuzz Tests
+
+1. Install `cargo-fuzz`.
+
+   ```console
+   $ cargo install --locked cargo-fuzz
+   ```
+
+2. Initialize a fuzz project by running the following command inside your contract directory.
+
+   ```console
+   $ cargo fuzz init
+   ```
+
+2. Open the contract's `Cargo.toml` file. Add `lib` as a `crate-type`.
+
+   ```diff
+    [lib]
+   -crate-type = ["cdylib"]
+   +crate-type = ["lib", "cdylib"]
+   ```
+
+2. Open the generated `fuzz/Cargo.toml` file. Add the `soroban-sdk` dependency.
+
+   ```diff
+    [dependencies]
+    libfuzzer-sys = "0.4"
+   +soroban-sdk = { version = "*", features = ["testutils"] }
+   ```
+
+3. Open the generated `fuzz/src/fuzz_target_1.rs` file. It will look like the below.
+
+   ```rust
+   #![no_main]
+   use libfuzzer_sys::fuzz_target;
+
+   fuzz_target!(|data: &[u8]| {
+       // fuzzed code goes here
+   });
+   ```
+
+4. Fill out the `fuzz_target!` call with test setup and assertions. For example, for the [increment example]:
+
+   ```rust
+   #![no_main]
+   use libfuzzer_sys::fuzz_target;
+   use soroban_increment_with_fuzz_contract::{IncrementContract, IncrementContractClient};
+   use soroban_sdk::{
+       testutils::arbitrary::{self, Arbitrary},
+       Env,
+   };
+
+   #[derive(Debug, Arbitrary)]
+   pub struct Input {
+       pub by: u64,
+   }
+
+   fuzz_target!(|input: Input| {
+       let env = Env::default();
+       let id = env.register(IncrementContract, ());
+       let client = IncrementContractClient::new(&env, &id);
+
+       let mut last: Option<u32> = None;
+       for _ in input.by.. {
+           match client.try_increment() {
+               Ok(Ok(current)) => assert!(Some(current) > last),
+               Err(Ok(_)) => {} // Expected error
+               Ok(Err(_)) => panic!("success with wrong type returned"),
+               Err(Err(_)) => panic!("unrecognised error"),
+           }
+       }
+   });
+   ```
+
+5. Execute the fuzz target.
+
+   ```
+   cargo +nightly fuzz run --sanitizer=thread fuzz_target_1
+   ```
+
+   :::info
+
+   If you're developing on MacOS you need to add the `--sanitizer=thread` flag in order to work around a [known issue](https://github.com/stellar/rs-soroban-sdk/issues/1056).
+
+   :::
+
+This test uses the same patterns used in [unit tests] and [integration tests]:
+1. Create an environment, the `Env`.
+3. Register the contract to be tested.
+4. Invoke functions using a client.
+5. Assert expectations.
+
+:::tip
+
+For a full detailed example, see the [fuzzing example].
+
+:::
+
+:::info
+
+[`cargo-fuzz`] is reasonably popular in the Rust ecosystem. There also exists [`cargo-afl`]. See the [Rust Fuzz book] to learn more about both.
+
+:::
+
+[Getting Started]: ../../smart-contracts/getting-started
+[increment example]: https://github.com/stellar/soroban-examples/blob/main/increment/src/lib.rs
+[Differential Testing with Test Snapshots]: ./differential-testing-with-test-snapshots
+[stellar contract fetch]: ../../tools/developer-tools/cli/stellar-cli#stellar-contract-fetch
+[integration tests]: ./integration-tests
+[unit tests]: ./unit-tests
+[stellar/rs-soroban-sdk#1360]: https://github.com/stellar/rs-soroban-sdk/issues/1360
+[fuzzing example]: ../../smart-contracts/example-contracts/fuzzing
+[Rust Fuzz Book]: https://rust-fuzz.github.io/book

docs/build/guides/testing/integration-tests.mdx

@@ -0,0 +1,71 @@
+---
+title: Integration Tests
+hide_table_of_contents: true
+description: Integration testing uses dependency contracts instead of mocks.
+sidebar_position: 3
+---
+
+Integration tests are tests that include the integration between components, and so test a larger scope such as other contracts.
+
+The [Soroban Rust SDK] makes it just as easy to integration test by providing utilities for testing against real contracts fetched from mainnet, testnet, or the local file system.
+
+### How to Write Integration Tests
+
+The following is an example of a test that includes a dependency contract into the test, rather than mock it. The test is written to test the [increment-with-pause contract] and the [pause contract]. The contract has an `increment` function that increases a counter value by one on every invocation. The contract depends on the pause contract to control whether the increment functionality is paused.
+
+The following tests set up the `increment-with-pause` contract, as well as import and register the real pause contract using it's wasm file.
+
+If the pause contract was deployed on mainnet it could be retrieved with the stellar-cli:
+
+```
+stellar contract fetch --id C... > pause.wasm
+```
+
+And imported into the tests with:
+
+```rust
+mod pause {
+    soroban_sdk::contractimport!(file = "pause.wasm");
+}
+```
+
+The following test registers not only the increment-with-pause contract, but also the pause contract. The test checks that when the pause contract is not paused that the increment contract operates as expected. When it is paused, the increment function errors. Once it's unpaused the function operates as expected.
+
+```rust
+#[test]
+fn test() {
+    let env = Env::default();
+
+    // highlight-start
+    let pause_id = env.register(pause::WASM, ());
+    let pause_client = pause::Client::new(&env, &pause_id);
+    // highlight-end
+
+    let contract_id = env.register(
+        IncrementContract,
+        IncrementContractArgs::__constructor(&pause_id),
+    );
+    let client = IncrementContractClient::new(&env, &contract_id);
+
+    pause_client.set(&false);
+    assert_eq!(client.increment(), 1);
+
+    pause_client.set(&true);
+    assert_eq!(client.try_increment(), Err(Ok(Error::Paused)));
+
+    pause_client.set(&false);
+    assert_eq!(client.increment(), 2);
+}
+```
+
+Most tests, whether they're unit, mocks, or integration tests, will look very similar to the test above. The tests will do four things:
+1. Create an environment, the `Env`.
+2. Register the contract(s) to be tested.
+3. Invoke functions using the generated client.
+4. Assert the outcome.
+
+[increment-with-pause contract]: https://github.com/stellar/soroban-examples/blob/main/increment-with-pause/src/lib.rs
+[pause contract]: https://github.com/stellar/soroban-examples/blob/main/pause/src/lib.rs
+[Integration Tests]: ./integration-tests
+[Making Cross-Contract Calls]: ../conventions/cross-contract
+[Soroban Rust SDK]: ../../../tools/sdks/library#soroban-rust-sdk