Native speakers of a language can tell whether a speaker is native or non-native just by hearing one word or phrase in the language. It is expected that L2 learners will develop the ability to detect ‘good pronunciation’ as they establish the prototypes of the L2 sound system. However, it is not known what contributes to their judgement of good pronunciation. Therefore, this pilot study aims to clarify the mechanism of L2 listeners’ judgement of good pronunciation. In this study, we focus on the prosodic variations of timing and pitch-accent in Japanese. Four groups of informants participated in a perception experiment where they were asked to assess stimuli extracted from the recordings of L2 Japanese learners. These four groups are L2 Japanese learners at two levels of proficiency (beginner and advanced), and native Japanese speakers with and without formal teaching experience of L2 Japanese. All learner participants are native speakers of Australian English. We will report that the learners’ assessment of good pronunciation is not straightforward, being different from the logical expectation that learners will behave more like native speakers as their L2 Japanese proficiency develops. We will also discuss possible explanations for the results and implications for L2 Japanese education.