Initial commit: Alfred Mobile - AI Assistant Android App
- OAuth authentication via Authentik - WebSocket connection to OpenClaw gateway - Configurable gateway URL with first-run setup - User preferences sync across devices - Multi-user support with custom assistant names - ElevenLabs TTS integration (local + remote) - FCM push notifications for alarms - Voice input via Google Speech API - No hardcoded secrets or internal IPs in tracked files
This commit is contained in:
416
AGENT_TOOLS.md
Normal file
416
AGENT_TOOLS.md
Normal file
@@ -0,0 +1,416 @@
|
||||
# Agent Tools - Mobile Notifications & Alarms
|
||||
|
||||
**Alfred can send notifications, alerts, and alarms directly to your mobile device using FCM!**
|
||||
|
||||
## Overview
|
||||
|
||||
The Alfred mobile app supports receiving notifications even when the app is closed, screen is locked, or device is asleep. This enables:
|
||||
|
||||
- **Alarms** - High-priority notifications that wake the device
|
||||
- **Instant alerts** - Get notified when tasks complete
|
||||
- **Timers** - Set countdown notifications
|
||||
- **Reminders** - Schedule notifications for specific times
|
||||
- **Background work** - Alfred can notify you when long-running tasks finish
|
||||
|
||||
**Key Features:**
|
||||
- ✅ **FCM Push Notifications** - Delivery guaranteed even when app is closed
|
||||
- ✅ **Token Persistence** - Works after proxy restarts
|
||||
- ✅ **Dual Delivery** - WebSocket (if connected) + FCM (always)
|
||||
- ✅ **Wake on Alarm** - Device wakes for high-priority alarms
|
||||
|
||||
## Features
|
||||
|
||||
### ✅ What's Working
|
||||
|
||||
1. **Instant Notifications** - Alfred can send alerts immediately
|
||||
2. **System Notifications** - Show even when app is backgrounded
|
||||
3. **In-App Display** - Notifications appear in chat when app is open
|
||||
4. **Multiple Notification Types** - Alert (⚠️), Timer (⏰), Reminder (🔔)
|
||||
5. **Optional TTS** - Speak notifications when TTS is enabled (foreground only)
|
||||
6. **WebSocket Broadcast** - All connected mobile devices receive notifications
|
||||
|
||||
### 📱 Mobile App Features
|
||||
|
||||
- **Background Notifications** - Receive notifications even when app is closed
|
||||
- **Notification Icons** - Visual indicators for different notification types
|
||||
- **System Tray** - Notifications appear in Android notification shade
|
||||
- **Tap to Open** - Tapping notification opens the app
|
||||
- **Auto-dismiss** - Notifications clear when tapped
|
||||
|
||||
## Usage from Alfred
|
||||
|
||||
### Send Alarms
|
||||
|
||||
**Use `alfred-notify` for all notifications and alarms.**
|
||||
|
||||
```bash
|
||||
# High-priority alarm (wakes device)
|
||||
alfred-notify --alarm "Wake up!"
|
||||
|
||||
# Alarm with custom title
|
||||
alfred-notify --alarm --title "⏰ Morning Alarm" "Time to get up!"
|
||||
|
||||
# Silent alarm (vibrate only)
|
||||
alfred-notify --alarm --no-sound "Silent wake-up"
|
||||
```
|
||||
|
||||
### Send Notifications
|
||||
|
||||
```bash
|
||||
# Basic notification
|
||||
alfred-notify "Task completed successfully"
|
||||
|
||||
# Notification with custom title
|
||||
alfred-notify --title "Build System" "Compilation finished"
|
||||
|
||||
# Silent notification
|
||||
alfred-notify --no-sound --no-vibrate "Background update complete"
|
||||
```
|
||||
|
||||
### Schedule via Cron
|
||||
|
||||
For scheduled alarms/reminders, use the `cron` tool:
|
||||
|
||||
```javascript
|
||||
// Set alarm for 5 minutes from now
|
||||
{
|
||||
"name": "5 minute alarm",
|
||||
"schedule": {
|
||||
"kind": "at",
|
||||
"atMs": Date.now() + (5 * 60 * 1000)
|
||||
},
|
||||
"payload": {
|
||||
"kind": "agentTurn",
|
||||
"message": "Run: alfred-notify --alarm '⏰ 5 minute alarm!'",
|
||||
"deliver": false
|
||||
},
|
||||
"sessionTarget": "isolated"
|
||||
}
|
||||
```
|
||||
|
||||
**Important:** Always use `sessionTarget: "isolated"` + `agentTurn` for alarms, never `systemEvent`.
|
||||
|
||||
## Alfred Integration
|
||||
|
||||
Alfred uses `alfred-notify` from cron jobs or via exec tool:
|
||||
|
||||
**Example conversation:**
|
||||
```
|
||||
You: "Set an alarm for 5 minutes"
|
||||
Alfred: *schedules cron job* "I'll send an alarm in 5 minutes!"
|
||||
```
|
||||
|
||||
Behind the scenes, Alfred creates a cron job:
|
||||
```javascript
|
||||
cron.add({
|
||||
schedule: { kind: "at", atMs: Date.now() + 300000 },
|
||||
payload: {
|
||||
kind: "agentTurn",
|
||||
message: "Run: alfred-notify --alarm '⏰ 5 minute alarm!'"
|
||||
},
|
||||
sessionTarget: "isolated"
|
||||
})
|
||||
```
|
||||
|
||||
**Another example:**
|
||||
```
|
||||
You: "Let me know when the build finishes"
|
||||
Alfred: "Sure, I'll notify you when it completes"
|
||||
```
|
||||
|
||||
Alfred monitors the build and executes:
|
||||
```javascript
|
||||
exec(['alfred-notify', '--title', 'Build System', 'Build completed! ✅'])
|
||||
```
|
||||
|
||||
## Technical Details
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐
|
||||
│ Alfred │ (OpenClaw agent)
|
||||
│ exec/cron │
|
||||
└────────┬────────┘
|
||||
│ executes
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ alfred-notify │ (CLI wrapper)
|
||||
│ bash script │
|
||||
└────────┬────────┘
|
||||
│ HTTP POST
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Alfred Proxy │ (port 18790)
|
||||
│ + Firebase │
|
||||
│ Admin SDK │
|
||||
└────────┬────────┘
|
||||
│
|
||||
├──────────────────────┬─────────────────────┐
|
||||
│ WebSocket │ FCM Push │
|
||||
│ (if connected) │ (always) │
|
||||
▼ ▼ ▼
|
||||
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||
│ Mobile App(s) │ │ Firebase Cloud │ │ fcm-tokens │
|
||||
│ GatewayClient │ │ Messaging │ │ .json (disk) │
|
||||
└────────┬────────┘ └────────┬────────┘ └─────────────────┘
|
||||
│ │
|
||||
│ ▼
|
||||
│ ┌─────────────────┐
|
||||
│ │ Mobile Device │
|
||||
│ │ (even asleep) │
|
||||
│ └────────┬────────┘
|
||||
│ │
|
||||
└─────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Android OS │ (system notifications)
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
**Key Features:**
|
||||
- **Dual Delivery:** WebSocket (instant if connected) + FCM (guaranteed)
|
||||
- **Token Persistence:** FCM tokens saved to `fcm-tokens.json`, survive restarts
|
||||
- **Wake Device:** FCM can wake locked/sleeping devices for alarms
|
||||
- **Firebase Admin SDK:** Uses `cloudmessaging.messages.create` permission
|
||||
|
||||
### Event Format
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "event",
|
||||
"event": "mobile.notification",
|
||||
"payload": {
|
||||
"notificationType": "alert|timer|reminder",
|
||||
"title": "Alfred",
|
||||
"message": "Notification text",
|
||||
"priority": "default",
|
||||
"sound": true,
|
||||
"vibrate": true,
|
||||
"timestamp": 1706918400000,
|
||||
"action": null
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Mobile App Handling
|
||||
|
||||
1. **GatewayClient** receives `mobile.notification` event via WebSocket
|
||||
2. **MainScreen** `onNotification()` callback processes the event
|
||||
3. **Notification icons** added based on type (⏰ ⚠️ 🔔 📢)
|
||||
4. **System notification** shown if timer/reminder OR if app is backgrounded
|
||||
5. **Chat message** added if app is in foreground
|
||||
6. **Optional TTS** speaks the notification if enabled (foreground only)
|
||||
|
||||
## Testing
|
||||
|
||||
### Test Instant Notification
|
||||
|
||||
From terminal:
|
||||
```bash
|
||||
cd ~/.openclaw/workspace/alfred-proxy
|
||||
./alfred-notify "Test notification!"
|
||||
```
|
||||
|
||||
### Test Alarm
|
||||
|
||||
```bash
|
||||
./alfred-notify --alarm "Test alarm!"
|
||||
```
|
||||
|
||||
### Test with Custom Title
|
||||
|
||||
```bash
|
||||
./alfred-notify --title "Kitchen Timer" "Oven is ready!"
|
||||
```
|
||||
|
||||
### Test Silent Notification
|
||||
|
||||
```bash
|
||||
./alfred-notify --no-sound --no-vibrate "Silent test"
|
||||
```
|
||||
|
||||
### Test via Alfred
|
||||
|
||||
Ask Alfred:
|
||||
```
|
||||
"Send me a test alarm"
|
||||
"Set an alarm for 1 minute"
|
||||
"Notify me when this finishes"
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Check Proxy Status
|
||||
|
||||
```bash
|
||||
# Service status
|
||||
systemctl --user status alfred-proxy.service
|
||||
|
||||
# Watch logs
|
||||
tail -f /tmp/alfred-proxy.log
|
||||
|
||||
# Check for FCM token registration
|
||||
grep "fcm.*Registering" /tmp/alfred-proxy.log
|
||||
|
||||
# Check for successful sends
|
||||
grep "fcm.*Successfully" /tmp/alfred-proxy.log
|
||||
```
|
||||
|
||||
### Check Connected Clients & Tokens
|
||||
|
||||
When app connects, you'll see:
|
||||
```
|
||||
[proxy] New connection from ::ffff:192.168.1.20
|
||||
[auth] Token validated for user: shadow@dao-mail.com
|
||||
[fcm] Registering token for user a2b49b91...: ewqRvIsOTuiWJk...
|
||||
[fcm] Saved tokens to disk
|
||||
```
|
||||
|
||||
### Check Token Persistence
|
||||
|
||||
```bash
|
||||
# View persisted FCM tokens
|
||||
cat ~/.openclaw/workspace/alfred-proxy/fcm-tokens.json
|
||||
|
||||
# Should show:
|
||||
{
|
||||
"user-id-hash": [
|
||||
"fcm-token-string"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Test Notification Delivery
|
||||
|
||||
```bash
|
||||
# Via CLI (recommended)
|
||||
alfred-notify --alarm "Test"
|
||||
|
||||
# Via HTTP API
|
||||
curl -X POST http://localhost:18790/api/notify \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"notificationType": "alarm",
|
||||
"title": "Test",
|
||||
"message": "Hello!",
|
||||
"priority": "high",
|
||||
"sound": true,
|
||||
"vibrate": true
|
||||
}'
|
||||
```
|
||||
|
||||
Expected response:
|
||||
```json
|
||||
{"success":true,"clients":0,"fcm":1}
|
||||
```
|
||||
- `clients`: WebSocket connections (0 if app closed)
|
||||
- `fcm`: FCM devices notified (should be 1+)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### No notifications appearing
|
||||
|
||||
1. **Check FCM tokens persisted**:
|
||||
```bash
|
||||
cat alfred-proxy/fcm-tokens.json
|
||||
# Should show registered tokens
|
||||
```
|
||||
|
||||
2. **Check proxy logs for FCM success**:
|
||||
```bash
|
||||
tail -f /tmp/alfred-proxy.log | grep fcm
|
||||
# Should show: [fcm] Successfully sent X message(s)
|
||||
```
|
||||
|
||||
3. **If seeing "Permission denied" error**:
|
||||
```
|
||||
[fcm] Error: Permission 'cloudmessaging.messages.create' denied
|
||||
```
|
||||
- Check service account has correct role: **Firebase Admin SDK Administrator Service Agent**
|
||||
- Verify FCM API is enabled in Google Cloud Console
|
||||
- Regenerate service account key if needed
|
||||
|
||||
4. **If "No FCM tokens registered"**:
|
||||
- Open Alfred app to trigger token registration
|
||||
- Check logs for `[fcm] Registering token`
|
||||
- Verify app sends token on connect (should happen automatically)
|
||||
|
||||
5. **Test directly**:
|
||||
```bash
|
||||
alfred-notify --alarm "Test"
|
||||
```
|
||||
|
||||
### WebSocket vs FCM
|
||||
|
||||
- **WebSocket**: Instant delivery when app is open and connected
|
||||
- **FCM**: Always works, even when app is closed or device is asleep
|
||||
- Both should show in response: `{"clients":1,"fcm":1}`
|
||||
|
||||
If `clients:0, fcm:1` → App closed but FCM working (normal)
|
||||
If `clients:0, fcm:0` → No tokens registered (open app to fix)
|
||||
|
||||
### Cron alarms not firing
|
||||
|
||||
1. **Verify cron job uses correct format**:
|
||||
```javascript
|
||||
// ✅ CORRECT
|
||||
{
|
||||
"sessionTarget": "isolated",
|
||||
"payload": {
|
||||
"kind": "agentTurn",
|
||||
"message": "Run: alfred-notify --alarm 'Message'"
|
||||
}
|
||||
}
|
||||
|
||||
// ❌ WRONG
|
||||
{
|
||||
"sessionTarget": "main",
|
||||
"payload": {
|
||||
"kind": "systemEvent",
|
||||
"text": "Alarm message"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. **Check cron job exists**:
|
||||
```bash
|
||||
# Ask Alfred in chat
|
||||
/cron list
|
||||
```
|
||||
|
||||
3. **Check cron execution logs**:
|
||||
```bash
|
||||
tail -f /tmp/alfred-proxy.log
|
||||
# Look for cron execution and alfred-notify calls
|
||||
```
|
||||
|
||||
### Firebase Permission Errors
|
||||
|
||||
**Error:** `Permission 'cloudmessaging.messages.create' denied`
|
||||
|
||||
**Solution:**
|
||||
1. Verify service account role: **Firebase Admin SDK Administrator Service Agent** (NOT "Firebase Cloud Messaging Admin")
|
||||
2. Enable FCM API: https://console.cloud.google.com/apis/library/fcm.googleapis.com
|
||||
3. Download fresh service account key
|
||||
4. Place at: `alfred-proxy/service-account.json`
|
||||
5. Restart proxy: `systemctl --user restart alfred-proxy.service`
|
||||
|
||||
See `FCM_SETUP.md` for full setup instructions.
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
Potential additions:
|
||||
|
||||
- [ ] **Custom sounds** - Per-notification sound selection
|
||||
- [ ] **Action buttons** - "Snooze", "Mark as done", etc.
|
||||
- [ ] **Deep links** - Open specific app screens from notification
|
||||
- [ ] **Rich content** - Images, progress bars, etc.
|
||||
- [ ] **Notification groups** - Collapse similar notifications
|
||||
- [ ] **Do Not Disturb** - Respect user quiet hours
|
||||
- [ ] **Multi-device sync** - Mark as read across devices
|
||||
|
||||
## Version
|
||||
|
||||
1.0.0 - Initial release with alert/timer/reminder support
|
||||
Reference in New Issue
Block a user