I would avoid randomness in your automated tests. You want all of your automated tests to be predictable. Trying random inputs is certainly not predictable. That way if one of your tests fail, you will be able to more quickly determine why.

Instead, spend some time thinking about cases that might deserve testing and test those explicitly. Thinking about your code like this is more effective than using random tests and hoping that they catch cases you didn’t think of.

Let’s suppose that it works for email, but not name.

Now your acceptance test will fail half the time. Just re-running the test will potentially change the result. That’ll make it harder to notice errors and track down the problem with it occurs. It’ll be way easier if you just write two tests, one for each login method.

I won’t state randomness is a bad idea.

[Sav07] tested his algorithm implementation for binary search by writing a group of test which included some “theories” (hypotheses) about how the search algorithm should perform.

He then ran the “thorough” group of tests with a lot of random fixtures.

In Ruby:

require "test/unit"
require "binary_search" # binary_search gem: 
                        # <https://github.com/tyler/binary_search>

class BinarySearchTestTheories < Test::Unit::TestCase
    def before_all
       @rand = Random.new()
    end

    def test_theories
        experiments = 1000
        max_value = 1073741824 # Fixnum max value

        while (experiments-- > 0) do
            test_array = generateRandomSortedArray

            target_must_exist = [true, false].sample
            if target_must_exist
                target = test_array[@rand.rand(test_array.length)]
            elsif
                target = @rand.integer(max_value)
            end

            return_value = test_array.binary_search { |v| target <=> v }

            assert_theory1(test_array, target, return_value)
            assert_theory2(test_array, target, return_value)
            assert_theory3(test_array, target, return_value)
            assert_theory4(test_array, target, return_value)
        end
    end

    # Theories
    def assert_theory1(arr, target, expectation) ... end
    def assert_theory2(arr, target, expectation) ... end
    def assert_theory3(arr, target, expectation) ... end
    def assert_theory4(arr, target, expectation) ... end

    # Helper
    def does_array_contain_target?(arr, target) ... end
    def target_position(arr, target) ... end
end

I’m not certain his “test theories” qualify as acceptance tests—but they clearly are beyond unit or “boundary” tests.

My conclusion: random acceptance tests are useful for predictable algorithms. Sampling email and user name fixtures seems trivial enough.

[Sav07] Alberto Savoia: “Beautiful Tests”, in: Andy Oram and Greg Wilson (eds.): Beautiful Code, Beijing: O’Reilly, 2007.

See the full article at: http://cdn.ttgtmedia.com/searchSoftwareQuality/downloads/BeautifulTests.pdf

I sometimes write tests that depend on random occurrences for whether they succeed or not. I’ve learned that there are a few caveats, as described by other answers:

• You do not ideally want a test that only fails sometimes. If you break your code, you want to find out when you integrate your change, so you know which change to undo.
• If there are only a small number of inputs that cause failure, you need to know which inputs they are in order to fix the problem.

The former problem can be mostly solved, as long as you are willing to accept a slow test suite, by running the test many times. The latter problem requires you to log any random factors that may cause the failure.

These techniques also apply to situations where the randomness is unavoidable, e.g. you are testing for potential timing-related conditions in a multithreaded program.

As long as you are careful, it can be done. But the best thing to do is to isolate potential failure conditions and write a predictable test that always fails for them prior to fixing the problem, IMHO.