Try this
Make a formula-text field called [Warning on too many decimals] like this
IF(Abs([price]-Round([price],0.01))>=0.001,"Too many decimals"). Put that field on the form near the price field and hide the label.
Then make a form rule that says
When the record is saved
and [warning on too many decimals is not equal to (blank)
abort the save with message "Too many decimals. Correct the decimals to two places and tab out of the field and then try saving again"
(note that like Excel, QuickBase does not see the input or the corrected input until the user tabs out of the field. That can be confusing to users as they may fix the price, and then keep trying unsuccessfully to Save. Try it and you will see what I mean.