Teaching my phone to put its own appointments in my calendar
The text that started it
It arrived on a Thursday afternoon, the way they always do — mid-task, mid-thought, mid-something-else.
A message from a medical centre. An appointment had been booked for me: a weekday, a specific time, an address. Reply Y to confirm, N to cancel.
Nothing unusual. I get a dozen messages a week like it. And that is exactly the problem. This one was a calendar entry wearing the costume of a text message. Somewhere in my phone’s notification shade sat a date, a time, and a place, and the only thing standing between that information and my actual calendar was me — reading it, remembering it, opening an app, and typing it in.
I have done that small chore thousands of times. Read it, decide it matters, open the calendar, type it in, close the tab. Forty seconds, give or take. Which does not sound like much until you multiply it by a few hundred a year and realise the total is larger than most side projects.
So I did the obvious thing and handed it to an agent. Not “write me an app” — just: deal with this. Watch the texts, work out which ones are real appointments, put them in my calendar, and only bother me when you are unsure.
What follows is the honest version. Four of the five steps fought back, in four completely different ways, and the last one nearly shipped a silent bug that would have made the whole thing look like it was working when it was not. But it did get finished — and along the way it turned into the clearest demonstration I have seen of why an agent with a real workspace is a different kind of tool from a service you subscribe to.
It got the appointment right, by the way. First live run, real message, no prompting. I will come back to that.
The shape of the problem
Strip it down and there are only a few moving parts: something that sees the texts, something that decides which ones matter, and something that writes to the calendar. Three boxes and two arrows. Nothing about that diagram is hard.
Everything around those boxes is. Where does the text-reading happen, given that Android has opinions about who is allowed to read SMS? Which machine does the deciding, and does it need the internet to do it? Whose calendar, and authenticated how? And what happens to the ninety-odd percent of messages that are not appointments at all?
Every one of those questions turned out to have a wall behind it.
So I set out to build it: watch my SMS, work out which messages are real appointments, and write those straight into my calendar. Local models only, no cloud.
What follows is the honest version — including the four places the stack actively resisted, because those turned out to be more interesting than the bits that just worked.
The shape of the thing
The design is simple to describe:
- A listener on the phone sees incoming messages.
- A machine at home polls that listener, over a private network.
- A local model classifies each message: is this an appointment with a real date, or is it noise?
- If it is confident, write the event to my calendar.
- Everything else gets ignored silently.
Five steps. Four of them fought back.
Problem one: Android does not want you reading SMS
The obvious first move is to read the SMS inbox directly. Android has an SMS permission group for exactly this, and the automation app on my phone asked for it.
The permission was denied, with no way to grant it.
Android has spent the last several releases locking down SMS access, because it is a favourite target for malware. From Android 10 onward, READ_SMS and SEND_SMS are restricted to the default messaging app. A regular app can’t just request them. If you’re not the default messenger, you can’t hold the permission — the toggle doesn’t even appear in the Permissions screen, which is genuinely confusing when you go looking for it.
There is a workaround: temporarily set the automation app as the default messaging app, grant the permission, then switch back. But here’s the thing — I didn’t want to change my default messaging app to support a hobby project. That is a lot of blast radius for a convenience feature.
What saved it: the phone’s app already had a notification listener enabled. When an SMS arrives, the Messages app posts a notification, and the listener reads that. It is read-only, it needs no SMS permission at all, and it gives you the message text, the sender, and a timestamp.
The trade-off is real and worth stating plainly: a notification listener is not a queue. It sees messages as they arrive, while it is running and connected. It cannot scroll back through your existing inbox. If the app is offline when a message lands and the notification is later dismissed, that message is simply gone from your pipeline’s point of view. For a “new appointments as they happen” feature, that is fine. For “process my whole history”, it is useless.
Problem two: the command I needed had been retired
With notifications flowing, the next job was writing to the calendar — and this is where the documentation and the running system disagreed with each other.
The platform’s command policy listed a calendar-write command as dangerous by default, requiring an explicit opt-in allowlist entry. Fine. I added it to the allowlist.
The error changed from “requires opt-in” to “the platform does not support this command.”
Digging into the shipped policy code, the command was in a set called RETIRED_PHONE_CONTROL_SEEDED_DENY_COMMANDS — sitting alongside camera capture, screen recording, and UI automation. It had not merely been disabled; it had been withdrawn. The phone could read calendars all day, but nothing in this build could write to one.
This is the kind of thing documentation drifts on. A doc comment says “gated”; the lockfile, or in this case the policy module, says “retired”. Always believe the running code.
The fix was to stop trying to write from the phone at all. Write from a machine that already has a proper API path. Which led directly to problem three.
Interlude: what the agent already had
Before I describe the auth failure, it is worth explaining why the fix was available at all — because this is the part that makes the whole exercise feel like a category change rather than a hack.
My agent runs inside a workspace that is, structurally, a developer’s environment. It has shell access, file access, a package manager, a credential store, and — crucially — the ability to write new tools for itself when the existing ones are not enough.
For the mail side of this project, that meant the agent did not have to wait for a vendor integration or a supported plugin. It could reach the mailbox the same way any mail client does: IMAP over OAuth2, with XOAUTH2 bearer auth instead of a password. Thunderbird does exactly this. So does every other desktop mail client. The difference is that instead of clicking through a settings dialog once, the flow became a small script the agent owns, runs, and can extend.
That script now supports:
| Capability | What it means |
|---|---|
status |
check whether the OAuth token is still valid |
folder list |
walk the entire mailbox hierarchy |
list --limit N --folder F |
recent messages from any folder |
read <uid> --format text/html/full |
pull one message, in whichever form is useful |
folder create |
make a new folder |
move <uid> --from-folder X --to-folder Y |
file messages server-side |
send |
submit mail over SMTP with the same token |
None of this is exotic. Individually, every one of those is something a mail app has done for twenty years. The point is what happens when they become composable, scriptable primitives inside an agent’s workspace: the agent can read a message, decide what it is, act on it, and file it — in one run, unprompted, with no UI in the loop.
The reason this matters for this project is that the calendar write needed an authenticated path into a personal Microsoft account. The agent already had a working OAuth device-code flow for that account, offline_access refresh, and a token cache outside the repository. Extending it was adding one scope and one function, not building an integration from scratch.
This is the quiet advantage of an agent with a real workspace: the mail client is not a black box you configure. It is a set of commands you can read, test, and change — and it degrades gracefully, in the sense that when something fails you can see exactly which layer failed.
Which is how I found the next problem.
Problem three: personal accounts are second-class citizens
I wanted events in a personal Outlook.com calendar. There were two candidate paths, and both failed for different reasons — which is itself the lesson.
Path A: SMTP. Send the event as a calendar invite to the mailbox and let the mail client file it. Microsoft’s SMTP endpoint accepts OAuth2, and the token I had carried a scope called SMTP.Send. It still returned an authentication error.
The reason is audience. That token was minted for the Graph API. The SMTP endpoint wants a token whose audience is the Outlook resource. A token with the right scope and the wrong audience is, to the SMTP server, a token with the wrong scope. The scope name matches. The token does not.
Path B: the Graph API’s send endpoint. This needs a scope called Mail.Send. My token had SMTP.Send — a different permission with a confusingly similar name — so the call came back 403 AccessDenied.
And you cannot fix either of these with a silent token refresh. When I tried requesting the missing scope, the identity platform answered plainly:
AADSTS70000: one or more scopes requested are unauthorized or expired. The user must first sign in and grant the client application access to the requested scope.
Scopes are consent, not configuration. You cannot programmatically grant yourself access a human never approved. The fix is a one-time interactive sign-in via device code — the same flow you use when adding a mail account to a desktop client.
There was a smaller trap worth recording too. The send endpoint returns an empty 202 on success. My HTTP helper assumed every successful response had a JSON body and crashed on the first real send. It had never worked, because it had never successfully sent anything. A crash on your first real success is a strange debugging moment.
The deeper lesson: work-tenanted Microsoft accounts and personal Microsoft accounts are genuinely different animals wearing the same API. Error messages rarely say which one you’re holding.
Problem four: the proxy that broke everything, silently
This one is my favourite, because it did not fail loudly.
The classifier — a local model on a home machine — was returning plausible-looking answers, but every single one was coming from a fallback path, never the model. The model call was erroring with 502 Bad Gateway and being quietly caught and replaced by a deterministic rule pass.
The cause: the host environment injects HTTP_PROXY and HTTPS_PROXY for controlled cloud egress. Python’s standard HTTP client honours those variables automatically. So my local call to a machine on my own private network was being routed through a proxy designed for the public internet — and that proxy, quite correctly, refused to reach a private address.
The same call worked perfectly from the command line, because the CLI bypasses the proxy for private hosts. So the behaviour differed by how you made the call, not what you asked for.
The fix is trivial once found — explicitly bypass the proxy for local addresses — but the failure mode is the dangerous part. Everything “worked”. Confidence scores looked reasonable. Results looked plausible. The only tell was a suspiciously uniform distribution and an error field I had been treating as noise.
A silent, plausible fallback is worse than a crash. If a fallback path produces output that looks like real output, you will trust it, and you will be wrong.
What actually shipped
- The phone exposes notifications; a machine at home polls them over a private network every few minutes.
- A local model (
Qwen-class, running on an Apple Silicon box) classifies each message with a schema-constrained prompt — it can only emit one of six labels, so it cannot wander. - A gate: the message must be labelled an appointment, score at or above a confidence floor, and contain a parseable date. A claimed appointment with no readable date is demoted. The date check is what stops a hallucinated time from ever reaching the calendar.
- Confident appointments are written to the calendar via the Graph API directly. Each event carries a note recording the sender, the confidence, and the original text, so any entry can be audited after the fact.
- Everything else — chit-chat, bills, delivery notices, one-time codes — is dropped without ceremony.
- A daily digest email lists what was added, what was ignored, and anything the model was unsure about.
- Outcomes are appended to a file and fed back into the prompt as few-shot examples. It gets better at your messages specifically, rather than messages in general.
The classification runs entirely on hardware I own, over a network I control. The only external service is the calendar API itself, because that is where the calendar lives.
Why this is an agent story, not a scripting story
It would be easy to file this under “I wrote some Python”. That undersells what changed.
The old shape of this problem is: identify a need, find a service that supports it, and adapt your workflow to whatever that service does. If no service supports it — and for “read my personal Outlook over an unsupported auth path, classify the contents with a local model, and write to my calendar” no service does — the request stops there.
The new shape is: the agent has a workspace, and the workspace is a real computer you can shape. So the request does not stop. The agent investigated, hit four hard walls, worked out what each wall actually was, built its own mail client from primitives, consented to one scope on my behalf, found and fixed a transport bug that was silently poisoning its own output, and left behind a documented, tested, reusable tool — all of which it can now operate on its own schedule.
That last part is the real shift. The finished pipeline does not need me at the keyboard. Every few minutes it checks the phone, classifies what it finds, writes the real appointments, quietly drops the noise, and mails me a daily summary. When it is unsure, it asks. When it is right, it learns — outcomes go back into the example store, so it gets better at my messages over time.
Compare that to the alternative: a cloud service that reads my SMS (which Android will not allow anyway), a handful of connected accounts, and a monthly fee. Instead, the classification runs on a machine in my office, the only external calls are to APIs I already use, and the whole thing is code I can read, test, and change.
And crucially — the agent could narrate its own failure. When the proxy silently degraded every result to a fallback, the fix was findable precisely because the agent could inspect its own layers: the HTTP client, the token contents, the response codes. A black-box integration gives you an error message. A workspace gives you the whole stack.
The bits I’d tell you to remember
- Believe the running system over the documentation. A doc comment said “gated”. The shipped policy said “retired”. They were both in the same repository.
- A permission with the right name and the wrong audience is not the right permission.
SMTP.SendandMail.Sendare not interchangeable, and the error message will not explain why. - Scopes are consent. No amount of token refreshing substitutes for a human approving access once.
- Your local calls are probably going through a proxy you forgot about. If the same request behaves differently from two different tools, suspect the transport before the server.
- Never let a fallback path look like a success path. If it can fail quietly and produce plausible output, add a loud marker so it can’t.
The whole thing took an afternoon, two of which were spent on the proxy.
Back to the text
Remember the message that started this? The medical centre, the weekday, the time, the address.
It was still sitting in the notification shade when the pipeline first went live. So that is what the first real run got handed — not a test fixture, not a sample string I had typed in to flatter myself. A genuine message from a genuine stranger, arriving through the real channel, at the real speed.
The classifier read it, labelled it an appointment, and pulled out the date, the time, and the location. Confidence: 0.98. For context, the deterministic fallback path — what runs when the model is unreachable — scores everything at a flat 0.5, because a rule engine does not really “decide”, it just fails safely.
So the first thing this system ever did, unasked, was put a real appointment in my calendar. Correctly.
I did not have to read it. I did not have to decide it mattered. I did not have to open an app, or remember a date, or type anything at all. The forty-second tax just… did not get charged.
And then it kept going. The next messages through were a payment confirmation, a food delivery notice, an ordinary conversation, and a credit-card alert. It ignored all four — verification code, delivery, chit-chat, finance — and bothered me about none of them. Which is, honestly, the harder half of the problem. Anyone can build something that shouts about appointments. Building something that knows when to shut up is what makes it liveable.
That is the whole point. Not that it is clever, but that it is quiet and right — and that it keeps running when I am not looking.
Next on the list: teaching it to tell the difference between “your appointment is Tuesday” and “would Tuesday suit you?” — which is a substantially harder problem than dates and labels, and exactly the sort of thing a small local model should be good at. And now that the agent can read and file mail as well as write calendar entries, the same shape applies to the inbox itself: classify, act, file, report — without me opening a mail client at all.