Swift GUI Automation MCP Server

Browser AutomationSwift

Control macOS mouse and keyboard programmatically

Available Tools

moveMouse

Moves the mouse cursor to the specified coordinates

xy

mouseClick

Performs a mouse click at the current cursor position

button

sendKeys

Sends keyboard shortcuts or key combinations

keys

scroll

Performs scrolling in the specified direction

directionclicks

Swift GUI Automation provides a Model Context Protocol server that enables programmatic control of macOS through mouse movements, clicks, keyboard inputs, and scrolling. Built on SwiftAutoGUI, this tool allows AI assistants to interact with the macOS interface directly, automating repetitive tasks and enabling new interaction patterns between AI and desktop applications.

Overview

Swift GUI Automation is a Model Context Protocol (MCP) server that enables programmatic control of macOS through mouse and keyboard automation. This server leverages SwiftAutoGUI to provide a bridge between AI assistants and the macOS desktop environment.

Requirements

Before installing Swift GUI Automation, ensure your system meets these requirements:

macOS 15.0 or later
Swift 6.0 or later
Xcode 16.0 or later

Installation

Follow these steps to install and configure Swift GUI Automation:

Clone the repository:

git clone https://github.com/NakaokaRei/swift-mcp-gui.git
cd swift-mcp-gui

Install the package:

swift package experimental-install

This will install the executable to your Swift package manager's bin directory, typically at /Users/USERNAME/.swiftpm/bin/swift-mcp-gui.

Configure your MCP client to use the server by adding the appropriate configuration to your client's settings.

Usage

Once installed, the Swift GUI Automation server provides tools for controlling mouse movement, mouse clicks, keyboard input, and scrolling on macOS.

Security Considerations

This server has full control over your mouse and keyboard. Exercise caution when running it and only connect trusted MCP clients. Be aware that malicious use could potentially control your computer, so ensure you understand the commands being executed.

Troubleshooting

If you encounter issues with permissions, you may need to:

Grant accessibility permissions to the terminal or application running the server
Navigate to System Preferences > Security & Privacy > Privacy > Accessibility
Add your terminal application to the list of allowed applications

License

Swift GUI Automation is available under the MIT License, allowing for both personal and commercial use with proper attribution.