Record what may be shared, ahead of federation
The bot-side half of TODO #4. Nothing federates yet; this establishes what would be eligible to, and proves the boundary before there is a hub that could leak across it. - Schema v5 adds visibility, defaulting to private. Upgrading an existing library therefore opts nothing into sharing, which a test on the legacy upgrade path pins down. - list_syncable() is the only function that will ever feed a hub, and it requires both public and approved. Tests force rows into states set_visibility refuses, so the filter is what protects the boundary rather than the caller being careful. - set_visibility refuses anything not approved: media still in the moderation queue cannot be marked shared. - /share lists memes with their current state and asks before making one public, since un-sharing stops future copies but cannot recall those already taken. The wording describes what sharing will mean once a hub exists, because by then warning is too late. - /list shows the same markers. Vault posts have no path into the sync set at all; being in a separate table is what guarantees it, and there is a test saying so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
goyban
@@ -8,9 +8,9 @@ Progress on branch `develop` (not pushed):
|
||||
- [x] **#3** moderation queue — `0a8c81d`
|
||||
- [x] **#2** public mode + membership gate — `82b5da4`
|
||||
- [x] **#5** save vs. extract (vault) — forward-based
|
||||
- [ ] **#4** federation
|
||||
- [~] **#4** federation — bot-side visibility done; hub, channel and sync client outstanding
|
||||
|
||||
Test suites: `test_db.py`, `test_bot.py`, `test_access.py`, `test_vault.py` — 29 cases, all passing.
|
||||
Test suites: `test_db.py`, `test_bot.py`, `test_access.py`, `test_vault.py` — 34 cases, all passing.
|
||||
|
||||
---
|
||||
|
||||
@@ -166,6 +166,18 @@ Split media transport from metadata sync:
|
||||
- Rate-limit per instance. A compromised peer token should be able to spam, at worst, its own rows.
|
||||
- Practical note: a federated pool of user-submitted media needs a takedown path for illegal content, and as the hub operator that lands on you. Worth deciding the policy before launching, not after.
|
||||
|
||||
### Status
|
||||
|
||||
Done in this repo: the `visibility` column (private by default), `/share` to
|
||||
change it, and `list_syncable()` — the single function that defines what is
|
||||
ever allowed to leave this instance. It is guarded by tests that force rows
|
||||
into states the writer would refuse, so the filter is proven rather than
|
||||
assumed. Nothing calls it yet.
|
||||
|
||||
Outstanding, and all of it depends on decisions only the owner can make:
|
||||
the federation channel, the hub deployment, the instance tokens, and the sync
|
||||
client that pushes to and pulls from the hub.
|
||||
|
||||
### Open question
|
||||
|
||||
The hub is a separate deployable (small FastAPI + SQLite would do). Do you want it in this repo as `hub/`, or as its own repository? It has a different release cadence and a different threat model, which argues for separate.
|
||||
|
||||
@@ -61,6 +61,7 @@ docker compose up -d --build
|
||||
| `/list` | نمایش همه رسانه‌های ذخیره‌شده |
|
||||
| `/vault [search]` | بازیابی پستی که کامل ذخیره کرده‌اید |
|
||||
| `/review` | بررسی یک‌به‌یک ارسال‌های در انتظار تأیید (فقط ادمین) |
|
||||
| `/share [search]` | مشخص‌کردن اینکه یک میم به اشتراک گذاشته شود یا خصوصی بماند (فقط ادمین) |
|
||||
| `/delete <id>` | حذف یک مورد (ID را از `/list` بگیرید) |
|
||||
| `/cancel` | لغو عملیات جاری |
|
||||
|
||||
@@ -110,6 +111,16 @@ REQUIRE_MODE=all
|
||||
|
||||
> **تا وقتی مطمئن نیستید این گزینه را خاموش بگذارید.** وقتی `ALLOWED_USERS` خالی باشد همه کاربرِ مجاز حساب می‌شوند، پس روشن‌کردن این گزینه یعنی پذیرفتن رسانه از هر کسی که ربات را پیدا کند.
|
||||
|
||||
### اشتراک‌گذاری (مقدمات)
|
||||
|
||||
هر میم به‌صورت پیش‌فرض **خصوصی** است. با `/share [search]` فهرست میم‌ها را با نشان 🔒 یا 🌐 می‌بینید و می‌توانید وضعیت هرکدام را عوض کنید. در `/list` هم همین نشان‌ها دیده می‌شود.
|
||||
|
||||
> **هنوز هیچ‌چیزی به اشتراک گذاشته نمی‌شود.** این فقط خواستهٔ شما را ثبت می‌کند. فدراسیون — جایی که ربات‌هایی که جدا اجرا می‌شوند میم‌های عمومی‌شان را روی هم می‌گذارند — هنوز ساخته نشده و تا آن‌موقع هیچ میمی از این ربات بیرون نمی‌رود. متن تأیید، معنای اشتراک‌گذاری را در آینده توضیح می‌دهد، چون بعد از راه‌افتادن هاب دیگر برای هشدار دادن دیر است.
|
||||
|
||||
دو قاعده در همان تک‌کوئری‌ای که قرار است به هاب داده شود اعمال می‌شود: فقط رسانه‌ای که هم **تأییدشده** و هم **عمومی** باشد واجد شرایط است. رسانه‌ای که هنوز در انتظار بررسی است اصلاً قابل اشتراک‌گذاری نیست، و پست‌های گاوصندوق در جدولی جدا هستند و هیچ مسیری به آنجا ندارند.
|
||||
|
||||
خصوصی‌کردن دوباره، جلوی کپی‌های بعدی را می‌گیرد اما کپی‌هایی که قبلاً گرفته شده را برنمی‌گرداند؛ به همین دلیل قبل از عمومی‌کردن تأیید گرفته می‌شود، نه بعد از آن.
|
||||
|
||||
### انتقال به سرور دیگر
|
||||
|
||||
فقط کافی است این دو فایل را منتقل کنید:
|
||||
|
||||
@@ -68,6 +68,7 @@ Telegram suggests these commands as you type `/`, with a short description besid
|
||||
| `/vault [search]` | Retrieve a whole post you saved |
|
||||
| `/review` | Review pending submissions one by one (admin only) |
|
||||
| `/edit [search]` | Select a meme and edit its keywords (admin only) |
|
||||
| `/share [search]` | Mark a meme as shared or private (admin only) |
|
||||
| `/delete <id>` | Remove an entry (get ID from `/list`; admin only) |
|
||||
| `/cancel` | Cancel current add operation |
|
||||
|
||||
@@ -117,6 +118,16 @@ The queue keeps no cursor. `/review` always fetches the oldest pending item, so
|
||||
|
||||
> **Leave this off unless you mean it.** With `ALLOWED_USERS` empty, everyone is an allowed user, so turning submissions on would accept media from anyone who finds the bot.
|
||||
|
||||
### Sharing (groundwork)
|
||||
|
||||
Every meme is **private** by default. `/share [search]` lists your memes with 🔒 or 🌐 and lets you flip one. `/list` shows the same markers.
|
||||
|
||||
> **Nothing is shared yet.** This only records your intent. Federation — where independently run bots pool their shared memes — is not built, and no meme leaves this bot until it is. The wording in the confirmation describes what sharing *will* mean, because once a hub exists it is too late to start warning.
|
||||
|
||||
Two rules are enforced in the one query that will ever feed a hub: only **approved** and **public** media is eligible. Media awaiting moderation cannot be marked shared at all, and vault posts are in a separate table with no path there whatsoever.
|
||||
|
||||
Un-sharing stops future copies but cannot recall copies already taken, which is why the confirmation asks before making something public rather than after.
|
||||
|
||||
### Moving to another server
|
||||
|
||||
The only files you need to transfer are:
|
||||
|
||||
@@ -86,6 +86,7 @@ ADMIN_COMMANDS = [
|
||||
*USER_COMMANDS,
|
||||
BotCommand(command="review", description="Review pending submissions"),
|
||||
BotCommand(command="edit", description="Edit meme keywords"),
|
||||
BotCommand(command="share", description="Choose what is shared"),
|
||||
BotCommand(command="delete", description="Remove a meme by ID"),
|
||||
]
|
||||
|
||||
@@ -252,6 +253,7 @@ def menu_text(user_id: int) -> str:
|
||||
"/vault [search] — retrieve a whole post you saved\n"
|
||||
"/review — review pending submissions\n"
|
||||
"/edit [search] — edit keywords of a meme\n"
|
||||
"/share [search] — mark a meme shared or private\n"
|
||||
"/delete <id> — remove a meme by ID\n"
|
||||
"/cancel — cancel current operation"
|
||||
)
|
||||
@@ -447,7 +449,10 @@ async def list_page(page: int) -> tuple[str, InlineKeyboardMarkup | None]:
|
||||
total_pages = (total + LIST_PAGE_SIZE - 1) // LIST_PAGE_SIZE
|
||||
page = max(0, min(page, total_pages - 1))
|
||||
items = await db.list_media(page, LIST_PAGE_SIZE)
|
||||
lines = [f"<code>{item['id']}</code> [{item['media_type']}] — {item['keywords']}" for item in items]
|
||||
lines = [
|
||||
f"{visibility_icon(item)} <code>{item['id']}</code> [{item['media_type']}] — {item['keywords']}"
|
||||
for item in items
|
||||
]
|
||||
text = f"Page {page + 1} of {total_pages}\n" + "\n".join(lines)
|
||||
|
||||
buttons = []
|
||||
@@ -832,6 +837,102 @@ async def on_vault_action(callback: CallbackQuery):
|
||||
)
|
||||
|
||||
|
||||
# ── /share ───────────────────────────────────────────────────────────────────
|
||||
#
|
||||
# Marking media public only records intent: nothing leaves this bot until a
|
||||
# federation hub is configured. The confirmation is worded for what sharing
|
||||
# will mean once one is, because by then it is too late to warn.
|
||||
|
||||
def visibility_icon(item: dict) -> str:
|
||||
return "🌐" if item.get("visibility") == "public" else "🔒"
|
||||
|
||||
|
||||
@dp.message(Command("share"))
|
||||
async def cmd_share(msg: Message):
|
||||
if not is_admin(msg.from_user.id):
|
||||
await msg.answer("Only admins can change what is shared.")
|
||||
return
|
||||
|
||||
parts = msg.text.split(maxsplit=1)
|
||||
query = parts[1].strip() if len(parts) > 1 else ""
|
||||
items = await db.search_media(query)
|
||||
if not items:
|
||||
no_match = f" matching '{query}'" if query else ""
|
||||
await msg.answer(f"No memes found{no_match}.")
|
||||
return
|
||||
|
||||
rows = [
|
||||
[InlineKeyboardButton(
|
||||
text=f"{visibility_icon(item)} #{item['id']} {item['keywords'][:35]}",
|
||||
callback_data=f"share:pick:{item['id']}",
|
||||
)]
|
||||
for item in items[:10]
|
||||
]
|
||||
public_count = await db.count_public()
|
||||
await msg.answer(
|
||||
f"🔒 private · 🌐 shared — tap to change.\n"
|
||||
f"{public_count} of your memes are marked shared.",
|
||||
reply_markup=InlineKeyboardMarkup(inline_keyboard=rows),
|
||||
)
|
||||
|
||||
|
||||
@dp.callback_query(F.data.startswith("share:"))
|
||||
async def on_share_action(callback: CallbackQuery):
|
||||
if not is_admin(callback.from_user.id):
|
||||
await callback.answer("Not allowed.", show_alert=True)
|
||||
return
|
||||
|
||||
_, action, raw_id = callback.data.split(":", 2)
|
||||
media_id = int(raw_id)
|
||||
item = await db.get_media_by_id(media_id)
|
||||
if item is None:
|
||||
await callback.answer("Not found.", show_alert=True)
|
||||
return
|
||||
|
||||
if action == "pick":
|
||||
await callback.message.edit_reply_markup(reply_markup=None)
|
||||
if item["visibility"] == "public":
|
||||
await db.set_visibility(media_id, "private")
|
||||
await callback.message.answer(
|
||||
f"🔒 <b>#{media_id}</b> is private again. Any instance that already "
|
||||
"took a copy keeps it — this stops future sharing, it does not recall "
|
||||
"what has gone.",
|
||||
parse_mode="HTML",
|
||||
)
|
||||
await callback.answer()
|
||||
return
|
||||
|
||||
await callback.message.answer(
|
||||
f"Share <b>#{media_id}</b> [{item['media_type']}]\n"
|
||||
f"Keywords: <b>{item['keywords'].replace(',', ', ')}</b>\n\n"
|
||||
"Shared media is copied by every other instance that syncs with it. "
|
||||
"You can stop sharing later, but you cannot take back copies already "
|
||||
"made. Nothing is shared until a federation hub is configured.",
|
||||
parse_mode="HTML",
|
||||
reply_markup=InlineKeyboardMarkup(inline_keyboard=[[
|
||||
InlineKeyboardButton(text="🌐 Share it", callback_data=f"share:yes:{media_id}"),
|
||||
InlineKeyboardButton(text="Cancel", callback_data=f"share:no:{media_id}"),
|
||||
]]),
|
||||
)
|
||||
await callback.answer()
|
||||
return
|
||||
|
||||
await callback.message.edit_reply_markup(reply_markup=None)
|
||||
|
||||
if action == "no":
|
||||
await callback.message.answer("Left private.")
|
||||
await callback.answer()
|
||||
return
|
||||
|
||||
shared = await db.set_visibility(media_id, "public")
|
||||
if shared:
|
||||
await callback.message.answer(f"🌐 <b>#{media_id}</b> is marked shared.", parse_mode="HTML")
|
||||
else:
|
||||
# set_visibility refuses anything not approved.
|
||||
await callback.message.answer("Only approved media can be shared.")
|
||||
await callback.answer()
|
||||
|
||||
|
||||
# ── /review ──────────────────────────────────────────────────────────────────
|
||||
|
||||
# Sending a stored item back needs one call per media type. Stickers cannot
|
||||
|
||||
@@ -4,7 +4,7 @@ import os
|
||||
DB_PATH = os.getenv("DB_PATH", "/data/media.db")
|
||||
|
||||
# Bump this and add a matching _migrate_to_vN() when the schema changes.
|
||||
SCHEMA_VERSION = 4
|
||||
SCHEMA_VERSION = 5
|
||||
|
||||
|
||||
# ── schema / migrations ──────────────────────────────────────────────────────
|
||||
@@ -87,10 +87,18 @@ async def _migrate_to_v4(db) -> None:
|
||||
await db.execute("CREATE INDEX IF NOT EXISTS idx_vault_owner ON vault_items(owner_id)")
|
||||
|
||||
|
||||
async def _migrate_to_v5(db) -> None:
|
||||
"""Mark what may leave this instance. Everything defaults to private, so an
|
||||
upgrade never opts an existing library into sharing."""
|
||||
await db.execute("ALTER TABLE media ADD COLUMN visibility TEXT NOT NULL DEFAULT 'private'")
|
||||
await db.execute("CREATE INDEX IF NOT EXISTS idx_visibility ON media(visibility)")
|
||||
|
||||
|
||||
MIGRATIONS = {
|
||||
2: _migrate_to_v2,
|
||||
3: _migrate_to_v3,
|
||||
4: _migrate_to_v4,
|
||||
5: _migrate_to_v5,
|
||||
}
|
||||
|
||||
|
||||
@@ -160,7 +168,7 @@ async def delete_media(media_id: int) -> bool:
|
||||
|
||||
# ── reads ────────────────────────────────────────────────────────────────────
|
||||
|
||||
COLUMNS = "id, owner_id, file_id, file_unique_id, media_type, keywords, status"
|
||||
COLUMNS = "id, owner_id, file_id, file_unique_id, media_type, keywords, status, visibility"
|
||||
|
||||
|
||||
async def search_media(query: str, owner_id: int | None = None,
|
||||
@@ -267,6 +275,49 @@ async def set_status(media_id: int, status: str, reviewer_id: int) -> bool:
|
||||
return cur.rowcount > 0
|
||||
|
||||
|
||||
# ── visibility / federation ──────────────────────────────────────────────────
|
||||
|
||||
async def set_visibility(media_id: int, visibility: str) -> bool:
|
||||
"""Only approved media can be shared; something still awaiting moderation
|
||||
or already rejected must never become publishable."""
|
||||
async with aiosqlite.connect(DB_PATH) as db:
|
||||
cur = await db.execute(
|
||||
"UPDATE media SET visibility = ? WHERE id = ? AND status = 'approved'",
|
||||
(visibility, media_id),
|
||||
)
|
||||
await db.commit()
|
||||
return cur.rowcount > 0
|
||||
|
||||
|
||||
async def list_syncable(limit: int = 500) -> list[dict]:
|
||||
"""The only rows permitted to leave this instance.
|
||||
|
||||
Both conditions matter: private media is nobody else's business, and
|
||||
unreviewed or rejected media must not be pushed to peers. Everything that
|
||||
ever talks to the hub goes through this function.
|
||||
"""
|
||||
async with aiosqlite.connect(DB_PATH) as db:
|
||||
db.row_factory = aiosqlite.Row
|
||||
cursor = await db.execute(
|
||||
f"""
|
||||
SELECT {COLUMNS} FROM media
|
||||
WHERE visibility = 'public' AND status = 'approved'
|
||||
ORDER BY added_at ASC, id ASC
|
||||
LIMIT ?
|
||||
""",
|
||||
(limit,),
|
||||
)
|
||||
return [dict(r) for r in await cursor.fetchall()]
|
||||
|
||||
|
||||
async def count_public() -> int:
|
||||
async with aiosqlite.connect(DB_PATH) as db:
|
||||
cursor = await db.execute(
|
||||
"SELECT COUNT(*) FROM media WHERE visibility = 'public' AND status = 'approved'"
|
||||
)
|
||||
return (await cursor.fetchone())[0]
|
||||
|
||||
|
||||
# ── vault ────────────────────────────────────────────────────────────────────
|
||||
#
|
||||
# Every query here takes an owner_id and filters on it. A vault item belongs to
|
||||
|
||||
+80
-1
@@ -56,6 +56,10 @@ async def test_legacy_upgrade(tmp: str):
|
||||
tables = {r[0] for r in c.execute("SELECT name FROM sqlite_master WHERE type='table'")}
|
||||
assert statuses == {"approved"}, f"upgraded rows must stay live: {statuses}"
|
||||
assert "vault_items" in tables, "upgrading should create the vault table too"
|
||||
with sqlite3.connect(path) as c:
|
||||
visibilities = {r[0] for r in c.execute("SELECT visibility FROM media")}
|
||||
assert visibilities == {"private"}, \
|
||||
f"upgrading must not opt an existing library into sharing: {visibilities}"
|
||||
print(" legacy v1 database upgrades with ids, keywords and visibility intact")
|
||||
|
||||
|
||||
@@ -168,6 +172,78 @@ async def test_rejected_stays_hidden(tmp: str):
|
||||
print(" rejected items leave the queue and never surface in search")
|
||||
|
||||
|
||||
async def test_new_media_is_private(tmp: str):
|
||||
path = os.path.join(tmp, "visibility.db")
|
||||
db = load_db(path)
|
||||
await db.init_db()
|
||||
|
||||
await db.add_media(111, "F1", "U1", "photo", ["kw"])
|
||||
assert (await db.search_media("kw"))[0]["visibility"] == "private"
|
||||
assert await db.list_syncable() == [], "nothing should be shared by default"
|
||||
print(" media is private unless explicitly shared")
|
||||
|
||||
|
||||
async def test_only_public_and_approved_can_sync(tmp: str):
|
||||
path = os.path.join(tmp, "syncable.db")
|
||||
db = load_db(path)
|
||||
await db.init_db()
|
||||
|
||||
await db.add_media(111, "F1", "U1", "photo", ["private_approved"])
|
||||
await db.add_media(111, "F2", "U2", "photo", ["public_approved"])
|
||||
await db.add_media(111, "F3", "U3", "photo", ["public_pending"], status="pending")
|
||||
await db.add_media(111, "F4", "U4", "photo", ["public_rejected"], status="rejected")
|
||||
|
||||
# Force the states set_visibility would refuse, to prove the query itself
|
||||
# filters rather than relying on the writer being careful.
|
||||
with sqlite3.connect(path) as c:
|
||||
c.execute("UPDATE media SET visibility='public' WHERE file_unique_id IN ('U2','U3','U4')")
|
||||
|
||||
syncable = [r["keywords"] for r in await db.list_syncable()]
|
||||
assert syncable == ["public_approved"], f"leaked: {syncable}"
|
||||
assert await db.count_public() == 1
|
||||
print(" only approved AND public media is ever syncable")
|
||||
|
||||
|
||||
async def test_sharing_requires_approval(tmp: str):
|
||||
path = os.path.join(tmp, "share_guard.db")
|
||||
db = load_db(path)
|
||||
await db.init_db()
|
||||
|
||||
await db.add_media(111, "F1", "U1", "photo", ["waiting"], status="pending")
|
||||
pending = (await db.search_media("", status="pending"))[0]
|
||||
assert await db.set_visibility(pending["id"], "public") is False, \
|
||||
"unreviewed media must not be shareable"
|
||||
assert await db.list_syncable() == []
|
||||
print(" media awaiting moderation cannot be marked shared")
|
||||
|
||||
|
||||
async def test_unsharing_stops_future_syncs(tmp: str):
|
||||
path = os.path.join(tmp, "unshare.db")
|
||||
db = load_db(path)
|
||||
await db.init_db()
|
||||
|
||||
await db.add_media(111, "F1", "U1", "photo", ["kw"])
|
||||
item = (await db.search_media("kw"))[0]
|
||||
assert await db.set_visibility(item["id"], "public") is True
|
||||
assert len(await db.list_syncable()) == 1
|
||||
|
||||
assert await db.set_visibility(item["id"], "private") is True
|
||||
assert await db.list_syncable() == [], "unsharing should remove it from the sync set"
|
||||
print(" unsharing takes an item back out of the sync set")
|
||||
|
||||
|
||||
async def test_vault_never_syncs(tmp: str):
|
||||
path = os.path.join(tmp, "vault_sync.db")
|
||||
db = load_db(path)
|
||||
await db.init_db()
|
||||
|
||||
await db.add_vault_item(111, 500, [1, 2], "a private post", ["personal"])
|
||||
# The vault lives in its own table, so the sync query cannot reach it at all.
|
||||
assert await db.list_syncable() == []
|
||||
assert await db.count_public() == 0
|
||||
print(" vault posts have no path into the sync set")
|
||||
|
||||
|
||||
async def main():
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
@@ -175,7 +251,10 @@ async def main():
|
||||
test_readd_keeps_id, test_fresh_and_idempotent,
|
||||
test_pending_is_hidden, test_queue_is_oldest_first,
|
||||
test_two_moderators_cannot_double_review,
|
||||
test_rejected_stays_hidden):
|
||||
test_rejected_stays_hidden, test_new_media_is_private,
|
||||
test_only_public_and_approved_can_sync,
|
||||
test_sharing_requires_approval,
|
||||
test_unsharing_stops_future_syncs, test_vault_never_syncs):
|
||||
await test(tmp)
|
||||
print("\nAll database tests passed.")
|
||||
|
||||
|
||||
Reference in New Issue
Block a user