View source for How to make fish treat question mark and ampersand as normal characters

 set -U fish_features qmark-noglob

Then logout and log back in.

To get the simpler ampersand, you have to compile fish 3.4:

 cd ~/forks/fish-shell
 make -j4
 sudo make install

then the feature is:

 ampersand-nobg-in-token

Strangely this doesn't seem to work without running the shell from the built location.

When it works it looks like:

 fish-shell $ echo a&b?
 a&b?