Books-A-Million said its largest shareholder group, the Anderson family, has withdrawn its non-binding offer to take the company private.