A subnet mask is really just a length of bits: ones on the left, zeros on the right. You can't mix them.
Because of this binary nature, there are only nine possible values for any octet in a subnet mask:
Code:
1111 1111 = 255
1111 1110 = 254
1111 1100 = 252
1111 1000 = 248
1111 0000 = 240
1110 0000 = 224
1100 0000 = 192
1000 0000 = 128
0000 0000 = 0
This sort of segmentation is admittedly rigid, but critical to forming an efficient addressing hierarchy. IPv6 follows the same rules, but masks are expressed in much more user-friendly CIDR notation (/xx).