Last summer my "smart" mower kept getting stuck — wheels spinning on wet grass, tearing up holes in the lawn. More than once I drove out to the summerhouse just to rescue it.
The mower's stock "intelligence" relied on one forecast API — no real ground read, and Danish weather shifts faster than forecasts capture. Not the vendor's fault: any single-source system fails the same way.
Those rescues became the spec sheet: redundant data sources, live-condition decisions, and a human checkpoint on every call.
My background shaped the approach. Telecom engineering taught redundancy as a first principle. A 2018 ML & Deep Learning diploma at the University of Copenhagen gave me the foundations. Nine years in financial IT — close to resilient, auditable systems — made human-in-the-loop feel natural, not optional. The architecture reflects all three.
System stack
Most of the hardware was already in the garden. Only the Pi Zero and Tuya soil sensors were added new.
🌱
Edge · In garden
Raspberry Pi Zero 2
Flask dashboard, live sensor reads, camera snapshots, mower control. The only hardware in the field.
🖥️
Local LLM · Home mini PC
Qwen3.5:4b via Ollama
Lighter local model over Tailscale mesh VPN — zero ports exposed to the internet. Handles Telegram natural-language control.
💬
Control interface
Telegram Bot
Natural-language garden control powered by the local LLM. Every agent decision arrives here with its reasoning — approve or reject.
👁️
Vision · Online
Gemini Vision
Analyses PTZ camera snapshots via Google APIs. Adds visual ground truth to irrigation and mowing decisions.
🔧
Sensors & Actuators
Tuya · Aqara · Robotic Mower
Tuya soil sensors and irrigation valves, Aqara climate sensors, a robotic mower, and a PTZ camera — orchestrated by the agent layer.
🔒
Networking
Tailscale Mesh VPN
Private overlay network between garden Pi and home mini PC. No open ports. The local LLM stays local.
PythonRaspberry Pi Zero 2Ollama · Qwen3.5Gemini VisionTuya CloudAqaraTailscaleFlaskTelegram Bot APIOpen-MeteoClaude Code
The hard parts
The hard parts had nothing to do with prompts. Getting a PoC out of the lab and running in the field required the kind of low-level work AI tools don't shortcut.
01
Reverse-engineering mobile app APIs
The mower, Aqara devices, and other hardware exposed no documented integration points. Capturing OAuth flows with mitmproxy, replacing broken SDKs with direct HTTP calls, reconstructing token refresh logic from traffic inspection.
02
Hand-writing ONVIF SOAP for live camera streams
No library worked reliably for the PTZ camera. Wrote ONVIF SOAP requests by hand — parsing WSDL, building XML envelopes, handling digest authentication — to pull live streams and trigger presets from the agent.
03
Edge constraints on Pi Zero 2
512 MB RAM and an ARMv7 core set hard limits on what runs locally. Careful separation of what lives on the edge (sensor reads, actuation, snapshots) versus what runs on the home mini PC (LLM inference, vision analysis).
04
Keeping the human loop fast enough to be useful
A human checkpoint only works if it's frictionless. Designing Telegram messages that give enough context to approve or reject in two seconds — not five paragraphs — took more iteration than the agent logic itself.
Architecture philosophy
The architecture splits roles deliberately. Conflating automation and agency is where IoT systems fail.
Layer 1 · Automation
Does the control
Deterministic thresholds trigger candidate actions. Testable, predictable, auditable. A soil sensor below a threshold always flags irrigation. No LLM involved. I would never let a language model drive a water valve independently.
Layer 2 · Agent
Does the judgment
Combines forecasts, camera snapshots, and sensor history into a plain-language proposal. "Soil is dry" is the threshold. The agent adds: "…but rain lands tonight — skip it." Or: "heatwave, moisture dropping fast — water extra now."
→
Scheduled baseline
Irrigation and mowing run autonomously on schedule and sensor thresholds — no agent, no approval needed for routine operations.
→
Agent proposal
Any deviation from the baseline — extra watering, skipping a mow, triggering a camera preset — is proposed by the agent with its reasoning attached.
→
Human checkpoint via Telegram
Proposal arrives on my phone. One tap to approve or reject. The system acts only after confirmation — or falls back to the scheduled baseline if I don't respond in time.
The models — Gemini Vision, a local 4B Qwen3.5 — are the standard part now. The tools, the guardrails, the trust built around them: that's the actual product.
Status & context
Running right now in my kolonihave — a small allotment garden with a summerhouse, common across Denmark. Built with Claude Code as a pair programmer: the vision became a working system in weeks, not months.
This is a PoC, not a scalable product — one garden, one user. Built to experiment around Agentic AI by doing. The engineering choices — redundancy, separation of concerns, human oversight — reflect the same principles I apply in financial IT at Nordea, just on a very different substrate.
What it demonstrates: the ability to take an AI architecture from concept to running hardware, outside the comfort of a dev environment, with all the messy integration work that entails.