The goal of this paper is to present the development of a tracking technology to interact with a virtual object. This paper presents the general procedures of building a simple, low cost tracking system by using Wiimote (a remote of Nintendo game console) and the Open source Computer Vision (OpenCV) software library as well as interfacing the tracking system with an immersive virtual environment (Vizard). We used an iterative position and orientation estimation (POSIT algorithm) which is optimized as an OpenCV function for extracting position parameters. We filter out the noise in the coordinate values using Kalman filters. The orientation and translation of the tracked system are then used to manipulate a virtual object created in the virtual world of Vizard. Our results indicate that it is possible to implement an inexpensive and efficient application for interacting with virtual worlds using a Wiimote and appropriate digital filters.