diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 0000000..c9abf82 --- /dev/null +++ b/PLAN.md @@ -0,0 +1,195 @@ +# Implementation plan — TODO #2–#6 + +Status: design only, nothing implemented. Branch `develop` created locally (TODO #1), not pushed. + +--- + +## Step 0 — Prerequisites (blocking for #2, #3, #4, #5) + +Two things in the current schema block almost everything else, so they come first. + +**0a. There is no migration mechanism.** `db.init_db()` (`db.py:7-21`) only runs `CREATE TABLE IF NOT EXISTS`, so it can never alter an existing deployment's table. Every item below adds columns. Add a `schema_version` table and run ordered migration steps: + +```python +async def migrate(db): + (v,) = await fetch_one("PRAGMA user_version") + if v < 2: ... # ALTER TABLE media ADD COLUMN ... + await db.execute(f"PRAGMA user_version = {TARGET}") +``` + +SQLite's `PRAGMA user_version` is enough here — no extra table needed. + +**0b. `file_unique_id` is globally UNIQUE** (`db.py:14`). That is correct for a single-owner bot and wrong for every multi-user mode in this TODO: two users cannot each save the same meme. It also makes `INSERT OR REPLACE` (`db.py:28`) silently *delete and reinsert* a colliding row, changing its `id` (AUTOINCREMENT + REPLACE), which quietly breaks any `/delete ` the user had noted. + +Change to `UNIQUE(owner_id, file_unique_id)`. SQLite can't drop a column constraint in place — this needs the create-new-table / copy / rename dance inside migration v2. + +**0c. Related, worth fixing while in there:** `search_media()` (`db.py:34`) never filters by `owner_id`, so today every allowed user sees every user's media. Harmless for a personal bot, unacceptable once #2 opens the bot up. Search needs an explicit scope argument from the start. + +--- + +## #6 — Full media coverage *(do this first: small, independent, immediate value)* + +Currently handled (`bot.py:157-190`): photo, animation, video, voice, sticker. +Missing: **audio (mp3), document (file), video_note (round video)**. + +**Refactor.** The five handlers are near-identical. Replace with one handler and an extractor table: + +```python +EXTRACTORS = { + "photo": lambda m: m.photo[-1], + "gif": lambda m: m.animation, + "video": lambda m: m.video, + "voice": lambda m: m.voice, + "sticker": lambda m: m.sticker, + "audio": lambda m: m.audio, + "document": lambda m: m.document, + "video_note":lambda m: m.video_note, +} +``` + +One `@dp.message(F.photo | F.animation | ... , StateFilter(None))` that walks the table. This also fixes the duplicated filter list at `bot.py:225`, which currently misses any newly added type. + +**Inline results** (`bot.py:395-416`) need matching branches: `InlineQueryResultCachedAudio` and `InlineQueryResultCachedDocument` (the latter requires a non-empty `title`). + +**Constraint to accept:** Telegram has **no cached inline result type for `video_note`**. Round videos can be stored and listed, but cannot be returned via inline search. Options: store them as `video` (loses the round presentation), or accept that they're `/list`-only. Recommend the latter and say so in the README. + +Also note `F.animation` catches most GIFs, but Telegram sometimes delivers a GIF as `document` with an image mime-type — the document branch will pick those up once it exists. + +--- + +## #3 — Moderation queue *(before #2, since public mode depends on it)* + +**Schema (migration v3):** add to `media`: +`status TEXT NOT NULL DEFAULT 'approved'` ∈ {`pending`, `approved`, `rejected`}, plus `submitted_by INTEGER`, `reviewed_by INTEGER`, `reviewed_at TIMESTAMP`. + +Existing rows default to `approved`, so current deployments are unaffected. Every read path (`search_media`, `list_media`, `count_media`) gains `WHERE status = 'approved'`. + +**Review flow — `/review` (moderators only):** pulls the *oldest pending* row, sends the media itself followed by its keywords and an inline keyboard: + +`[✅ Approve] [❌ Reject] [✏️ Fix keywords] [⏸ Stop]` + +**The "continue later if they are too much" requirement falls out for free**, and this is the key design choice: the queue position is not stored in FSM state — each button press just re-queries for the next `pending` row. A moderator can stop mid-queue, restart days later, or have several moderators working simultaneously, with no state to resume. Include a `(14 left)` counter in each message. + +Guard against two moderators grabbing the same item with a conditional update — `UPDATE media SET status=? WHERE id=? AND status='pending'` and check `rowcount`. + +--- + +## #2 — Public mode with membership conditions + +**Config:** `PUBLIC_MODE=true`, `REQUIRED_CHATS=@channel,-100123456`, `REQUIRE_MODE=all|any`. + +**Gate:** `getChatMember(chat_id, user_id)`, accepting status ∈ {`creator`, `administrator`, `member`, and `restricted` with `is_member=true`}. The bot must be an admin in each required chat for this call to work — that's a setup requirement for the README. + +**Caching is not optional here.** Inline queries fire on *every keystroke*. An uncached membership check means several Telegram API calls per typed character, which will hit rate limits fast. Cache per `(user_id, chat_id)` with a ~5 minute TTL in memory. + +**Rejection UX:** for inline queries, don't return an empty list — use `answerInlineQuery`'s `button` parameter (`InlineQueryResultsButton`, supported in aiogram 3.15) to show a tappable "Join @channel to use this bot" bar above the results. In direct chat, reply with the channel links. + +With `PUBLIC_MODE=true`, non-admin submissions land as `status='pending'` from #3 rather than being ignored. + +--- + +## #5 — Save vs. extract on forward + +On a forwarded message, ask with inline buttons: **[📥 Save whole post]** / **[✂️ Extract media]**. Extract = today's behavior. Save = the private vault. + +**Albums are the hard part.** A post with caption + multiple media arrives as *several separate messages* sharing a `media_group_id`, with no "album complete" signal. Standard approach: buffer by `media_group_id` and debounce ~1s after the last message before prompting. Ask once per album, not once per file. + +**Schema (migration v4)** — a separate table, not a `media` column, since one vault item holds N files: + +```sql +vault_items(id, owner_id, caption TEXT, keywords TEXT, source_chat TEXT, created_at) +vault_files(item_id, file_id, file_unique_id, media_type, position) +``` + +Vault rows are **always private** and never touched by the federation sync in #4 — enforce that structurally by keeping them in a different table, so no `visibility` flag can ever be mis-set. + +**Search should cover the caption, not just keywords** — the point of a vault is finding a post you saved without having tagged it. This is where the current `LIKE '%,kw%'` approach (`db.py:43`) stops being adequate; use an FTS5 virtual table over `caption + keywords`. + +**Open question — recall UX.** Telegram **inline mode cannot send an album**; one inline result sends exactly one message. So a multi-file vault item can't come back through `@bot query` intact. Two options: + +1. Inline returns the item's *first* media with the caption attached — works everywhere, but loses the other files. +2. A `/vault ` command in the bot's private chat that replies with the full album, which you then forward manually — preserves everything, costs one extra step. + +These aren't exclusive; doing both (1 for speed, 2 for fidelity) is probably right, but it's your call. Single-file vault items are unaffected either way. + +--- + +## #4 — Federation + +### The constraint that decides the design + +**A `file_id` is only valid for the bot that received it.** A different bot cannot send, download, or resolve another bot's `file_id`. `file_unique_id` *is* stable across bots, but cannot be used to send anything — it's an identity key only. + +So "sync the public database every hour" **cannot work by copying rows between instances.** Each peer bot must independently obtain its *own* `file_id` for every shared meme. Any design that just ships database rows around will produce a table full of file_ids that throw `wrong file identifier` on use. This is the single most important thing to get right before writing code. + +### Recommended architecture: shared channel + metadata hub + +Split media transport from metadata sync: + +**Media → a private Telegram "federation channel" that you own.** Every participating bot is added as an admin. When an instance marks media public, it copies that media into the channel. Peers, being admins, receive the `channel_post` update and extract a `file_id` *valid for them*, storing it against the shared `file_unique_id`. Telegram stores the bytes once; nobody hosts files; no media ever transits your server. + +**Metadata → a small hub API** (`POST /sync` / `GET /since?ts=`) carrying only `file_unique_id`, `media_type`, `keywords`, `channel_message_id`, `origin_instance`. Rows keyed by `file_unique_id`, which is the one identifier that means the same thing everywhere. + +**Backfill for a newly joined instance:** a bot can't read channel history, so it won't have seen older posts. It resolves them on demand — `forwardMessage(from_chat_id=, message_id=, chat_id=)` returns a full `Message` containing a `file_id` valid for that bot. Note this must be `forwardMessage`, not `copyMessage`: `copyMessage` returns only a `MessageId` with no file data. Do this lazily on first use, not as an hour-long startup storm. + +### Privacy and safety + +- `visibility TEXT NOT NULL DEFAULT 'private'` ∈ {`private`, `public`} on `media` (migration v5). The sync job reads **only** `visibility='public' AND status='approved'`. Private rows have no code path to the hub — assert this in a test. +- Marking public is explicit and per-item, and should be **irreversible in practice**: once shared, peers have copies. Say so in the confirmation prompt. +- Each instance gets an `INSTANCE_ID` + bearer token you issue; HTTPS only; an instance may only write rows it owns. You keep revocation and a takedown endpoint that tombstones a `file_unique_id` so peers delete on next sync. +- Rate-limit per instance. A compromised peer token should be able to spam, at worst, its own rows. +- Practical note: a federated pool of user-submitted media needs a takedown path for illegal content, and as the hub operator that lands on you. Worth deciding the policy before launching, not after. + +### Open question + +The hub is a separate deployable (small FastAPI + SQLite would do). Do you want it in this repo as `hub/`, or as its own repository? It has a different release cadence and a different threat model, which argues for separate. + +--- + +## Suggested order + +| # | Item | Depends on | Size | +|---|---|---|---| +| 0 | Migration harness + unique-constraint fix | — | S | +| 6 | Full media coverage | 0 (only for the refactor) | S | +| 3 | Moderation queue | 0 | M | +| 2 | Public mode + membership gate | 0, 3 | M | +| 5 | Save vs. extract + vault | 0 | L | +| 4 | Federation | 0, 3, hub built | XL | + +#6 is shippable on its own and touches nothing else — good first commit on `develop`. #4 should stay last: it's the only item that's hard to reverse once other people are running instances against your hub. + +## Decisions — settled 2026-09-01 + +1. **#5 recall UX** — inline cannot send an album. For multi-file vault items, the inline result sends the **link to the original post** where one exists; otherwise it tells the user to open the bot and pull the full album there. Single-file items send directly as today. +2. **#4 hub location** — **separate repository.** Channel name and hub hosting are the owner's call, not decided here. +3. **#6 video_note** — **store as `video`**, accepting the loss of round presentation, so round videos stay inline-searchable. See the risk note below. +4. **#4 architecture** — **shared federation channel accepted.** Media rides the channel, metadata rides the hub API. + +### Risk carried by decision 3 + +A Telegram `file_id` encodes its own file type. Storing a `video_note` file_id and later returning it inside an `InlineQueryResultCachedVideo` may be rejected with `wrong file identifier`, because the cached-video slot expects a file_id minted as a video. This cannot be verified without a live bot token — **send the bot a round video and try to recall it inline before trusting this path.** If Telegram rejects it, the fallback is to treat video notes as `/list`-only (the option not chosen), which needs a one-line change to the extractor table. + +### Process + +Each step below lands as its own git commit on `develop`, so any step can be reverted independently. + +--- + +## Verification + +The API claims above were checked against a real `aiogram==3.15.0` install (the version pinned in `requirements.txt`), not asserted from memory: + +| Claim | Result | +|---|---| +| `InlineQueryResultsButton` exists (used by #2) | ✅ present | +| `answerInlineQuery` accepts `button` | ✅ present | +| `InlineQueryResultCachedAudio` / `...CachedDocument` exist (#6) | ✅ both present | +| `InlineQueryResultCachedVideoNote` exists | ❌ **does not exist** — confirms the video_note limitation | +| `CachedDocument` requires a `title` | ✅ required fields are `id`, `title`, `document_file_id` | +| `copyMessage` returns file data (#4 backfill) | ❌ returns bare `MessageId` | +| `forwardMessage` returns file data (#4 backfill) | ✅ returns full `Message` | + +The two ❌ rows are why the plan routes around `video_note` inline results and specifies `forwardMessage` for federation backfill. + +Not verified here (needs a live bot token and a real chat): `getChatMember` behavior for #2, and the album `media_group_id` debounce timing for #5.