Broadcasting a request or challenge is a classic method of collecting local information in distributed wireless networks. Neighbor discovery is known to be a fundamental element in ad hoc and sensor networks topology formation, which takes advantage of such methods. Most of the current neighbor discovery protocols rely on a challenge or request broadcast by the discovering node called Hello. Hello flooding attack was specifically designed to exploit the broadcasting nature of these protocols in order to convince a large group of nodes that the sender is their neighbor by using very high transmission power. Several studies have been done to mitigate the effectiveness of the flooding threats but little effort has been made in modeling and analyzing this problem. Arguing that random channel access protocols must be inevitably employed in neighbor discovery, we propose an analytical approach for stochastic modeling of the challenge-broadcasting scenarios in networks using slotted carrier sense multiple access with collision avoidance (CSMA/CA) protocols. We model the nonstationary channel right after issuance of the request by a recursive method and then put forward an approach to find the broadcaster's approximate payoff. The model also supports the cases where the broadcaster is a malicious node with an abnormally high transmission and reception rang e, which is found in severe flooding attacks. We investigate the applications of the model in finding the optimal attack range for the flooding adversaries and deriving a flood-resilient medium access control (MAC) protocol design framework to increase the security of challenge-response protocols. The latter one is especially relevant to mobile networks as it provides a low-cost solution. This paper describes the detailed analysis of the proposed theoretical framework as well as the comprehensive evaluations that have been carried out via simulations.