Foundry Book

Fuzz Testing

Forge supports property based testing.

Property-based testing is a way of testing general behaviors as opposed to isolated scenarios.

Let’s examine what that means by writing a unit test, finding the general property we are testing for, and converting it to a property-based test instead:

pragma solidity 0.8.10;

import "forge-std/Test.sol";

contract Safe {
    receive() external payable {}

    function withdraw() external {
        payable(msg.sender).transfer(address(this).balance);
    }
}

contract SafeTest is Test {
    Safe safe;

    // Needed so the test contract itself can receive ether
    // when withdrawing
    receive() external payable {}

    function setUp() public {
        safe = new Safe();
    }

    function test_Withdraw() public {
        payable(address(safe)).transfer(1 ether);
        uint256 preBalance = address(this).balance;
        safe.withdraw();
        uint256 postBalance = address(this).balance;
        assertEq(preBalance + 1 ether, postBalance);
    }
}

Running the test, we see it passes:

$ forge test
Compiling 24 files with Solc 0.8.10
Solc 0.8.10 finished in 1.10s
Compiler run successful!

Ran 1 test for test/Safe.t.sol:SafeTest
[PASS] test_Withdraw() (gas: 19463)
Suite result: ok. 1 passed; 0 failed; 0 skipped; finished in 430.15µs (65.49µs CPU time)

Ran 1 test suite in 5.26ms (430.15µs CPU time): 1 tests passed, 0 failed, 0 skipped (1 total tests)

This unit test does test that we can withdraw ether from our safe. However, who is to say that it works for all amounts, not just 1 ether?

The general property here is: given a safe balance, when we withdraw, we should get whatever is in the safe.

Forge will run any test that takes at least one parameter as a property-based test, so let’s rewrite:

contract SafeTest is Test {
    // ...

    function testFuzz_Withdraw(uint256 amount) public {
        payable(address(safe)).transfer(amount);
        uint256 preBalance = address(this).balance;
        safe.withdraw();
        uint256 postBalance = address(this).balance;
        assertEq(preBalance + amount, postBalance);
    }
}

If we run the test now, we can see that Forge runs the property-based test, but it fails for high values of amount:

$ forge test
Compiling 1 files with Solc 0.8.10
Solc 0.8.10 finished in 1.06s
Compiler run successful!

Ran 1 test for test/Safe.t.sol:SafeTest
[FAIL. Reason: EvmError: Revert; counterexample: calldata=0x29facca700000000000000000000000000000000107f7b4131893fd3a3014927320c6d92 args=[21929569535135079389558657731225480594 [2.192e37]]] testFuzz_Withdraw(uint256) (runs: 2, μ: 19531, ~: 19531)
Suite result: FAILED. 0 passed; 1 failed; 0 skipped; finished in 793.74µs (411.87µs CPU time)

Ran 1 test suite in 5.40ms (793.74µs CPU time): 0 tests passed, 1 failed, 0 skipped (1 total tests)

The default amount of ether that the test contract is given is 2**96 wei (as in DappTools), so we have to restrict the type of amount to uint96 to make sure we don’t try to send more than we have:

    function testFuzz_Withdraw(uint96 amount) public {

And now it passes:

$ forge test
Compiling 1 files with Solc 0.8.10
Solc 0.8.10 finished in 1.05s
Compiler run successful!

Ran 1 test for test/Safe.t.sol:SafeTest
[PASS] testFuzz_Withdraw(uint96) (runs: 257, μ: 19213, ~: 19631)
Suite result: ok. 1 passed; 0 failed; 0 skipped; finished in 4.90ms (4.49ms CPU time)

Ran 1 test suite in 6.16ms (4.90ms CPU time): 1 tests passed, 0 failed, 0 skipped (1 total tests)

You may want to exclude certain cases using the assume cheatcode. In those cases, fuzzer will discard the inputs and start a new fuzz run:

function testFuzz_Withdraw(uint96 amount) public {
    vm.assume(amount > 0.1 ether);
    // snip
}

There are different ways to run property-based tests, notably parametric testing and fuzzing. Forge only supports fuzzing.

Interpreting results

You might have noticed that fuzz tests are summarized a bit differently compared to unit tests:

“runs” refers to the amount of scenarios the fuzzer tested. By default, the fuzzer will generate 256 scenarios, but this and other test execution parameters can be setup by the user. Fuzzer configuration details are provided here.
“μ” (Greek letter mu) is the mean gas used across all fuzz runs
“~” (tilde) is the median gas used across all fuzz runs

Configuring fuzz test execution

Fuzz tests execution is governed by parameters that can be controlled by users via Forge configuration primitives. Configs can be applied globally or on a per-test basis. For details on this topic please refer to 📚 Global config and 📚 In-line config.

Fuzz test fixtures

Fuzz test fixtures can be defined when you want to make sure a certain set of values is used as inputs for fuzzed parameters. These fixtures can be declared in tests as:

storage arrays prefixed with fixture and followed by param name to be fuzzed. For example, fixtures to be used when fuzzing parameter amount of type uint32 can be defined as

uint32[] public fixtureAmount = [1, 5, 555];

functions named with fixture prefix, followed by param name to be fuzzed. Function should return an (fixed size or dynamic) array of values to be used for fuzzing. For example, fixtures to be used when fuzzing parameter named owner of type address can be defined in a function with signature

function fixtureOwner() public returns (address[] memory)

If the type of value provided as a fixture is not the same type as the named parameter to be fuzzed then it is rejected and an error is raised.

An example where fixture could be used is to reproduce the DSChief vulnerability. Consider the 2 functions

    function etch(address yay) public returns (bytes32 slate) {
        bytes32 hash = keccak256(abi.encodePacked(yay));

        slates[hash] = yay;

        return hash;
    }

    function voteSlate(bytes32 slate) public {
        uint weight = deposits[msg.sender];
        subWeight(weight, votes[msg.sender]);
        votes[msg.sender] = slate;
        addWeight(weight, votes[msg.sender]);
    }

where the vulnerability can be reproduced by calling voteSlate before etch, with slate value being a hash of yay address. To make sure fuzzer includes in the same run a slate value derived from a yay address, following fixtures can be defined:

    address[] public fixtureYay = [
        makeAddr("yay1"),
        makeAddr("yay2"),
        makeAddr("yay3")
    ];

    bytes32[] public fixtureSlate = [
        keccak256(abi.encodePacked(makeAddr("yay1"))),
        keccak256(abi.encodePacked(makeAddr("yay2"))),
        keccak256(abi.encodePacked(makeAddr("yay3")))
    ];

Following image shows how fuzzer generates values with and without fixtures being declared: Fuzzer