Is there any meaningful/behavioural difference between putting a ! inside or outside of square bracket tests?

e.g. Is there a difference between:

# if file exists but is empty, exit
[[ ! -s "${file}" ]] && exit 1

and

# if file exists but is empty, exit
! [[ -s "${file}" ]] && exit 1

I’m asking generally, not specific to the -s test. For example, does any of this change when working with numbers, or with multiple tests?