Alexandre Aja’s new movie could have been a cult success, but instead locks horns in a messy and noncommittal genre clash.