Add a vault for saving whole forwarded posts

Completes TODO #5. Forwarding a post now asks whether to save it whole
or extract its media; sending media directly is unchanged.

Built on forwarding rather than copying file_ids. The bot already has
the post in its private chat with the owner, so it stores message ids
and forwards them back. forwardMessages takes a list and keeps an album
intact, which is what made the caption-plus-album case tractable; ids
must be strictly ascending, so they are sorted on the way out.

- Vault rows live in their own table and every query is scoped to an
  owner_id, so a guessed id cannot reach someone else's saved post.
  Nothing in the vault is searchable inline, moderated or federated,
  and the separate table makes that structural rather than a flag.
- Albums arrive as separate messages with no completion signal, so they
  are buffered by media_group_id behind a 1s debounce and prompt once.
- The add path now takes a list of items, so extracting an album stores
  every piece under one set of keywords instead of asking repeatedly.
- /vault searches keywords and the original caption, since the point of
  a vault is finding something you never got round to tagging.
- Retrieval failures are reported: a message id is a pointer into the
  chat, not a self-contained file_id, so clearing history breaks it.

Three near-identical keyword collectors are now one helper.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
goyban
2026-09-01 22:01:30 +00:00
parent 8f80daaa9e
commit a35abb0541
8 changed files with 650 additions and 52 deletions
+33 -17
View File
@@ -7,10 +7,10 @@ Progress on branch `develop` (not pushed):
- [x] **#6** full media coverage — `efc2be5`
- [x] **#3** moderation queue — `0a8c81d`
- [x] **#2** public mode + membership gate — `82b5da4`
- [ ] **#5** save vs. extract (vault)
- [x] **#5** save vs. extract (vault) — forward-based
- [ ] **#4** federation
Test suites: `test_db.py`, `test_bot.py`, `test_access.py` — 20 cases, all passing.
Test suites: `test_db.py`, `test_bot.py`, `test_access.py`, `test_vault.py` — 29 cases, all passing.
---
@@ -98,29 +98,45 @@ With `PUBLIC_MODE=true`, non-admin submissions land as `status='pending'` from #
---
## #5 — Save vs. extract on forward
## #5 — Save vs. extract on forward — **implemented**
On a forwarded message, ask with inline buttons: **[📥 Save whole post]** / **[✂️ Extract media]**. Extract = today's behavior. Save = the private vault.
Built on forwarding rather than the file_id copy originally planned here, which
removed most of the work.
**Albums are the hard part.** A post with caption + multiple media arrives as *several separate messages* sharing a `media_group_id`, with no "album complete" signal. Standard approach: buffer by `media_group_id` and debounce ~1s after the last message before prompting. Ask once per album, not once per file.
**The bot already holds the post.** When you forward something, Telegram puts a
copy in your private chat with the bot, so the bot can forward *that* back later
by `message_id`. It never needs to reach the original channel, store media, or
re-upload anything.
**Schema (migration v4)** — a separate table, not a `media` column, since one vault item holds N files:
**`forwardMessages` (plural) preserves albums.** It takes up to 100 message ids
and returns them as one intact album, so the caption + N media case that made
this item large is handled by Telegram itself. Ids must be strictly ascending.
**One table, not two:**
```sql
vault_items(id, owner_id, caption TEXT, keywords TEXT, source_chat TEXT, created_at)
vault_files(item_id, file_id, file_unique_id, media_type, position)
vault_items(id, owner_id, chat_id, message_ids TEXT, caption, keywords, created_at)
```
Vault rows are **always private** and never touched by the federation sync in #4 — enforce that structurally by keeping them in a different table, so no `visibility` flag can ever be mis-set.
Every vault query takes an `owner_id` and filters on it, so a guessed id cannot
reach another user's post. Vault rows live in their own table and are never
searched by inline mode, never moderated, and never federated.
**Search should cover the caption, not just keywords** — the point of a vault is finding a post you saved without having tagged it. This is where the current `LIKE '%,kw%'` approach (`db.py:43`) stops being adequate; use an FTS5 virtual table over `caption + keywords`.
**Flow.** A forwarded message prompts *Save whole post* / *Extract media*
(extract only appears when there is media). Save collects keywords and stores
the pointer; extract runs the existing add path, now generalised to a list so a
whole album shares one set of keywords. Albums are buffered by `media_group_id`
with a 1s debounce, since Telegram gives no "album complete" signal.
**Open question — recall UX.** Telegram **inline mode cannot send an album**; one inline result sends exactly one message. So a multi-file vault item can't come back through `@bot query` intact. Two options:
**Recall is inside the bot only**, as decided: `/vault [search]` lists matching
posts with a retrieve button and a delete button. Matching covers keywords and
the original caption. The save confirmation tells the user plainly that vault
posts do not appear in inline search.
1. Inline returns the item's *first* media with the caption attached — works everywhere, but loses the other files.
2. A `/vault <query>` command in the bot's private chat that replies with the full album, which you then forward manually — preserves everything, costs one extra step.
These aren't exclusive; doing both (1 for speed, 2 for fidelity) is probably right, but it's your call. Single-file vault items are unaffected either way.
**The tradeoff accepted:** a `message_id` is a pointer into the bot's chat with
the owner, not self-contained like a `file_id`. If that history is cleared, the
pointer breaks; retrieval catches the failure and tells the user to forward the
post again.
---
@@ -164,14 +180,14 @@ The hub is a separate deployable (small FastAPI + SQLite would do). Do you want
| 6 | Full media coverage | 0 (only for the refactor) | S |
| 3 | Moderation queue | 0 | M |
| 2 | Public mode + membership gate | 0, 3 | M |
| 5 | Save vs. extract + vault | 0 | L |
| 5 | Save vs. extract + vault | 0 | M (was L) |
| 4 | Federation | 0, 3, hub built | XL |
#6 is shippable on its own and touches nothing else — good first commit on `develop`. #4 should stay last: it's the only item that's hard to reverse once other people are running instances against your hub.
## Decisions — settled 2026-09-01
1. **#5 recall UX** — inline cannot send an album. For multi-file vault items, the inline result sends the **link to the original post** where one exists; otherwise it tells the user to open the bot and pull the full album there. Single-file items send directly as today.
1. **#5 recall UX** — settled further during implementation: the vault is **forward-based and retrievable only inside the bot**. Inline search does not surface vault posts at all; the save confirmation says so.
2. **#4 hub location** — **separate repository.** Channel name and hub hosting are the owner's call, not decided here.
3. **#6 video_note** — **store as `video`**, accepting the loss of round presentation, so round videos stay inline-searchable. See the risk note below.
4. **#4 architecture** — **shared federation channel accepted.** Media rides the channel, metadata rides the hub API.