Computer programming based on screen shots

Researchers at MIT have developed a system that allows programmers to develop software using screen shots of the GUI (graphical user interface). The system, named Sikuli, enables novice computer users to write programs without mastering any programming languages. Computer vision algorithms are used to analyze any actions on the GUI.

Results from the research have been published in the 2009 User Interface Software and Technology conference, "Sikuli: Using GUI Screenshots for Search and Automation (Tom Yeh, Tsung-Hsiang Chang, Robert C. Miller)". The paper won the best student paper award. More information is available in a press release.