Control X from the command line for scripts, and do "visual scraping" to find things on the screen. The control interface allows mouse movement, clicking, button up/down, key up/down, etc, using the XTest extension. The visgrep program find images inside of images and reports the coordinates, allowing programs to find buttons, etc, on the screen to click on.

If you just want to control X, you may find xdotool easier to use.