Linear optics quantum computing is a promising approach to implementing scalable quantum computation. However, this approach has very demanding physical resource requirements. Recently, Aaronson and Arkhipov (e-print arXiv:1011.3245) showed that a simplified model, which avoids the requirement for fast feed-forward and postselection, while likely not capable of solving BQP-complete problems efficiently, can solve an interesting sampling problem believed to be classically hard. Loss and mode mismatch are the dominant sources of error in such systems. We provide evidence that even lossy systems or systems with mode mismatch are likely to be classically hard to solve. This is of practical interest to experimentalists wishing to demonstrate such systems since it suggests that, even with errors in their implementation, they are likely implementing an algorithm that is classically hard to solve. Our results also equivalently apply to the multiwalker quantum walk model.