memes_bot/PLAN.md

# Implementation plan — TODO #2–#6

Status: design only, nothing implemented. Branch `develop` created locally (TODO #1), not pushed.

---

## Step 0 — Prerequisites (blocking for #2, #3, #4, #5)

Two things in the current schema block almost everything else, so they come first.

**0a. There is no migration mechanism.** `db.init_db()` (`db.py:7-21`) only runs `CREATE TABLE IF NOT EXISTS`, so it can never alter an existing deployment's table. Every item below adds columns. Add a `schema_version` table and run ordered migration steps:

```python
async def migrate(db):
    (v,) = await fetch_one("PRAGMA user_version")
    if v < 2: ...  # ALTER TABLE media ADD COLUMN ...
    await db.execute(f"PRAGMA user_version = {TARGET}")
```

SQLite's `PRAGMA user_version` is enough here — no extra table needed.

**0b. `file_unique_id` is globally UNIQUE** (`db.py:14`). That is correct for a single-owner bot and wrong for every multi-user mode in this TODO: two users cannot each save the same meme. It also makes `INSERT OR REPLACE` (`db.py:28`) silently *delete and reinsert* a colliding row, changing its `id` (AUTOINCREMENT + REPLACE), which quietly breaks any `/delete <id>` the user had noted.

Change to `UNIQUE(owner_id, file_unique_id)`. SQLite can't drop a column constraint in place — this needs the create-new-table / copy / rename dance inside migration v2.

**0c. Related, worth fixing while in there:** `search_media()` (`db.py:34`) never filters by `owner_id`, so today every allowed user sees every user's media. Harmless for a personal bot, unacceptable once #2 opens the bot up. Search needs an explicit scope argument from the start.

---

## #6 — Full media coverage *(do this first: small, independent, immediate value)*

Currently handled (`bot.py:157-190`): photo, animation, video, voice, sticker.
Missing: **audio (mp3), document (file), video_note (round video)**.

**Refactor.** The five handlers are near-identical. Replace with one handler and an extractor table:

```python
EXTRACTORS = {
    "photo":     lambda m: m.photo[-1],
    "gif":       lambda m: m.animation,
    "video":     lambda m: m.video,
    "voice":     lambda m: m.voice,
    "sticker":   lambda m: m.sticker,
    "audio":     lambda m: m.audio,
    "document":  lambda m: m.document,
    "video_note":lambda m: m.video_note,
}
```

One `@dp.message(F.photo | F.animation | ... , StateFilter(None))` that walks the table. This also fixes the duplicated filter list at `bot.py:225`, which currently misses any newly added type.

**Inline results** (`bot.py:395-416`) need matching branches: `InlineQueryResultCachedAudio` and `InlineQueryResultCachedDocument` (the latter requires a non-empty `title`).

**Constraint to accept:** Telegram has **no cached inline result type for `video_note`**. Round videos can be stored and listed, but cannot be returned via inline search. Options: store them as `video` (loses the round presentation), or accept that they're `/list`-only. Recommend the latter and say so in the README.

Also note `F.animation` catches most GIFs, but Telegram sometimes delivers a GIF as `document` with an image mime-type — the document branch will pick those up once it exists.

---

## #3 — Moderation queue *(before #2, since public mode depends on it)*

**Schema (migration v3):** add to `media`:
`status TEXT NOT NULL DEFAULT 'approved'` ∈ {`pending`, `approved`, `rejected`}, plus `submitted_by INTEGER`, `reviewed_by INTEGER`, `reviewed_at TIMESTAMP`.

Existing rows default to `approved`, so current deployments are unaffected. Every read path (`search_media`, `list_media`, `count_media`) gains `WHERE status = 'approved'`.

**Review flow — `/review` (moderators only):** pulls the *oldest pending* row, sends the media itself followed by its keywords and an inline keyboard:

`[✅ Approve] [❌ Reject] [✏️ Fix keywords] [⏸ Stop]`

**The "continue later if they are too much" requirement falls out for free**, and this is the key design choice: the queue position is not stored in FSM state — each button press just re-queries for the next `pending` row. A moderator can stop mid-queue, restart days later, or have several moderators working simultaneously, with no state to resume. Include a `(14 left)` counter in each message.

Guard against two moderators grabbing the same item with a conditional update — `UPDATE media SET status=? WHERE id=? AND status='pending'` and check `rowcount`.

---

## #2 — Public mode with membership conditions

**Config:** `PUBLIC_MODE=true`, `REQUIRED_CHATS=@channel,-100123456`, `REQUIRE_MODE=all|any`.

**Gate:** `getChatMember(chat_id, user_id)`, accepting status ∈ {`creator`, `administrator`, `member`, and `restricted` with `is_member=true`}. The bot must be an admin in each required chat for this call to work — that's a setup requirement for the README.

**Caching is not optional here.** Inline queries fire on *every keystroke*. An uncached membership check means several Telegram API calls per typed character, which will hit rate limits fast. Cache per `(user_id, chat_id)` with a ~5 minute TTL in memory.

**Rejection UX:** for inline queries, don't return an empty list — use `answerInlineQuery`'s `button` parameter (`InlineQueryResultsButton`, supported in aiogram 3.15) to show a tappable "Join @channel to use this bot" bar above the results. In direct chat, reply with the channel links.

With `PUBLIC_MODE=true`, non-admin submissions land as `status='pending'` from #3 rather than being ignored.

---

## #5 — Save vs. extract on forward

On a forwarded message, ask with inline buttons: **[📥 Save whole post]** / **[✂️ Extract media]**. Extract = today's behavior. Save = the private vault.

**Albums are the hard part.** A post with caption + multiple media arrives as *several separate messages* sharing a `media_group_id`, with no "album complete" signal. Standard approach: buffer by `media_group_id` and debounce ~1s after the last message before prompting. Ask once per album, not once per file.

**Schema (migration v4)** — a separate table, not a `media` column, since one vault item holds N files:

```sql
vault_items(id, owner_id, caption TEXT, keywords TEXT, source_chat TEXT, created_at)
vault_files(item_id, file_id, file_unique_id, media_type, position)
```

Vault rows are **always private** and never touched by the federation sync in #4 — enforce that structurally by keeping them in a different table, so no `visibility` flag can ever be mis-set.

**Search should cover the caption, not just keywords** — the point of a vault is finding a post you saved without having tagged it. This is where the current `LIKE '%,kw%'` approach (`db.py:43`) stops being adequate; use an FTS5 virtual table over `caption + keywords`.

**Open question — recall UX.** Telegram **inline mode cannot send an album**; one inline result sends exactly one message. So a multi-file vault item can't come back through `@bot query` intact. Two options:

1. Inline returns the item's *first* media with the caption attached — works everywhere, but loses the other files.
2. A `/vault <query>` command in the bot's private chat that replies with the full album, which you then forward manually — preserves everything, costs one extra step.

These aren't exclusive; doing both (1 for speed, 2 for fidelity) is probably right, but it's your call. Single-file vault items are unaffected either way.

---

## #4 — Federation

### The constraint that decides the design

**A `file_id` is only valid for the bot that received it.** A different bot cannot send, download, or resolve another bot's `file_id`. `file_unique_id` *is* stable across bots, but cannot be used to send anything — it's an identity key only.

So "sync the public database every hour" **cannot work by copying rows between instances.** Each peer bot must independently obtain its *own* `file_id` for every shared meme. Any design that just ships database rows around will produce a table full of file_ids that throw `wrong file identifier` on use. This is the single most important thing to get right before writing code.

### Recommended architecture: shared channel + metadata hub

Split media transport from metadata sync:

**Media → a private Telegram "federation channel" that you own.** Every participating bot is added as an admin. When an instance marks media public, it copies that media into the channel. Peers, being admins, receive the `channel_post` update and extract a `file_id` *valid for them*, storing it against the shared `file_unique_id`. Telegram stores the bytes once; nobody hosts files; no media ever transits your server.

**Metadata → a small hub API** (`POST /sync` / `GET /since?ts=`) carrying only `file_unique_id`, `media_type`, `keywords`, `channel_message_id`, `origin_instance`. Rows keyed by `file_unique_id`, which is the one identifier that means the same thing everywhere.

**Backfill for a newly joined instance:** a bot can't read channel history, so it won't have seen older posts. It resolves them on demand — `forwardMessage(from_chat_id=<fed_channel>, message_id=<from hub metadata>, chat_id=<own log chat>)` returns a full `Message` containing a `file_id` valid for that bot. Note this must be `forwardMessage`, not `copyMessage`: `copyMessage` returns only a `MessageId` with no file data. Do this lazily on first use, not as an hour-long startup storm.

### Privacy and safety

- `visibility TEXT NOT NULL DEFAULT 'private'` ∈ {`private`, `public`} on `media` (migration v5). The sync job reads **only** `visibility='public' AND status='approved'`. Private rows have no code path to the hub — assert this in a test.
- Marking public is explicit and per-item, and should be **irreversible in practice**: once shared, peers have copies. Say so in the confirmation prompt.
- Each instance gets an `INSTANCE_ID` + bearer token you issue; HTTPS only; an instance may only write rows it owns. You keep revocation and a takedown endpoint that tombstones a `file_unique_id` so peers delete on next sync.
- Rate-limit per instance. A compromised peer token should be able to spam, at worst, its own rows.
- Practical note: a federated pool of user-submitted media needs a takedown path for illegal content, and as the hub operator that lands on you. Worth deciding the policy before launching, not after.

### Open question

The hub is a separate deployable (small FastAPI + SQLite would do). Do you want it in this repo as `hub/`, or as its own repository? It has a different release cadence and a different threat model, which argues for separate.

---

## Suggested order

| # | Item | Depends on | Size |
|---|---|---|---|
| 0 | Migration harness + unique-constraint fix | — | S |
| 6 | Full media coverage | 0 (only for the refactor) | S |
| 3 | Moderation queue | 0 | M |
| 2 | Public mode + membership gate | 0, 3 | M |
| 5 | Save vs. extract + vault | 0 | L |
| 4 | Federation | 0, 3, hub built | XL |

#6 is shippable on its own and touches nothing else — good first commit on `develop`. #4 should stay last: it's the only item that's hard to reverse once other people are running instances against your hub.

## Decisions — settled 2026-09-01

1. **#5 recall UX** — inline cannot send an album. For multi-file vault items, the inline result sends the **link to the original post** where one exists; otherwise it tells the user to open the bot and pull the full album there. Single-file items send directly as today.
2. **#4 hub location** — **separate repository.** Channel name and hub hosting are the owner's call, not decided here.
3. **#6 video_note** — **store as `video`**, accepting the loss of round presentation, so round videos stay inline-searchable. See the risk note below.
4. **#4 architecture** — **shared federation channel accepted.** Media rides the channel, metadata rides the hub API.

### Risk carried by decision 3

A Telegram `file_id` encodes its own file type. Storing a `video_note` file_id and later returning it inside an `InlineQueryResultCachedVideo` may be rejected with `wrong file identifier`, because the cached-video slot expects a file_id minted as a video. This cannot be verified without a live bot token — **send the bot a round video and try to recall it inline before trusting this path.** If Telegram rejects it, the fallback is to treat video notes as `/list`-only (the option not chosen), which needs a one-line change to the extractor table.

### Process

Each step below lands as its own git commit on `develop`, so any step can be reverted independently.

---

## Verification

The API claims above were checked against a real `aiogram==3.15.0` install (the version pinned in `requirements.txt`), not asserted from memory:

| Claim | Result |
|---|---|
| `InlineQueryResultsButton` exists (used by #2) | ✅ present |
| `answerInlineQuery` accepts `button` | ✅ present |
| `InlineQueryResultCachedAudio` / `...CachedDocument` exist (#6) | ✅ both present |
| `InlineQueryResultCachedVideoNote` exists | ❌ **does not exist** — confirms the video_note limitation |
| `CachedDocument` requires a `title` | ✅ required fields are `id`, `title`, `document_file_id` |
| `copyMessage` returns file data (#4 backfill) | ❌ returns bare `MessageId` |
| `forwardMessage` returns file data (#4 backfill) | ✅ returns full `Message` |

The two ❌ rows are why the plan routes around `video_note` inline results and specifies `forwardMessage` for federation backfill.

Not verified here (needs a live bot token and a real chat): `getChatMember` behavior for #2, and the album `media_group_id` debounce timing for #5.