George is an API leveraging AI to make it easy to control a computer with natural language.
Unlike traditional frameworks which rely on predefined static selectors, this API uses AI vision to interpret the screen. This makes it more resilient to UI changes and able to automate interfaces that traditional tools can't handle.
george-api-demo.mp4
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let mut george = George::new("https://your-molmo-llm.com");
george.start().await?;
george.open_chrome("https://some-website.com").await?;
george.click("sign in link").await?;
george.fill_in("input Email text field", "[email protected]").await?;
george.fill_in("input Password text field", "super-secret").await?;
george.click("sign in button").await?;
george.close_chrome().await?;
george.stop().await?;
}- Rust
- Docker
- Molmo-7B-D-0924. details
George uses Molmo, a vision-based LLM, to identify UI elements by converting natural language descriptions into screen coordinates which are then used to execute computer interactions.
You can try the online Molmo demo and ask for the point coordinates of an element in an image.
To run Molmo within Docker, you can use the following command which requires a 24GB VRAM GPU:
docker run -d --name molmo_container --runtime=nvidia --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-p 8000:8000 \
--ipc=host \
vllm/vllm-openai:latest \
--model allenai/Molmo-7B-D-0924 \
--trust-remote-codeSee this script to easily install Docker with Nvidia support on Ubuntu
Alternatively, you can run Molmo on bare metal, which can reduce the GPU memory consumption down to ~18GB or even ~12GB by leveraging bitsandbytes. Here are some example projects:
You can run Molmo with Runpod.io via their vllm pod template. See the video below for a demo (youtube):
george-runpod-demo.mp4
template override:
--host 0.0.0.0 --port 8000 --model allenai/Molmo-7B-D-0924 --trust-remote-code --api-key your-api-key
- Create a UI to help build out the selectors. It can be time-consuming to come up with an accurate selector.
- Improve debugging and logging
- Create bindings for other languages
- Ruby
- Python
- JavaScript/Typescript
- Others?
This is George. Most of the time he does what he's supposed to, but sometimes he doesn't do the
right thing at all. He's a living embodiment of current AI expectations.
