Skip to main content

Serhii Talks - Centralized Content Persistence & Telegram Broadcast Architecture

Overview​

Serhii Talks (@serhiiTalks) is the central content delivery hub and immutable backup archive that I engineered for all technical articles, system architecture write-ups, video releases, and micro-thoughts I produce across the web.

Rather than relying on third-party algorithms or fragmented social platforms to maintain an audience connection, I built this system to operate as an automated content ingestion, normalization, and fallback pipeline. Whenever I publish a new post, video, or thread on platforms like Medium, X (Twitter), YouTube, or my personal site Hrekov.com, my custom pipeline captures the source payload, normalizes the data, persists it into my managed serverless database, and dispatches an optimized broadcast to my @serhiiTalks Telegram channel.

       [ My Content Sources ]
 Medium | X (Twitter) | YouTube | Blog
                  |
                  v
       [ Automated Ingestion ]
      (Webhooks / RSS / APIs)
                  |
                  v
       [ Central Database ]
   (Normalized Content Backup)
                  |
                  v
       [ Telegram Dispatcher ]
    (Rate Limiting & Formatting)
                  |
                  v
          [@serhiiTalks]

High-Level Architecture & Personal Design Rationale​

As a backend software engineer publishing developer tutorials and system architecture guides, I found that relying solely on third-party platforms introduced significant content fragility. Platform algorithm adjustments, API breaking changes, or account restrictions could orphan my historical posts and fragment my reader base.

To solve this, I engineered the Serhii Talks pipeline around three core personal design principles:

  1. Complete Data Ownership & Backup: Before any post reaches a subscriber, my backend immediately sanitizes, structures, and saves the raw Markdown/HTML payload into my central database. This guarantees I maintain a 100% vendor-independent backup of my work.
  2. Unified Reader Feed: I wanted my followers to have a single, clean chronological stream on Telegram containing executive summaries, code snippets, and direct links without forcing them to check multiple platforms.
  3. Decoupled Asynchronous Processing: I decoupled content ingestion from broadcasting using asynchronous task queues. If Telegram experiences transient API drops or network glitches, my data is already safely persisted in PostgreSQL and will retry automatically.

My Tech Stack & Infrastructure Decisions​

I selected a modern, battle-tested Python stack optimized for asynchronous I/O, low latency, and operational resilience:

  • Application Backend: Python 3.12 running FastAPI inside Docker containers hosted on my dedicated Linux VPS.
  • Database & Storage: PostgreSQL hosted on Neon (serverless Postgres). I use SQLAlchemy 2.0 (AsyncIO) for asynchronous ORM queries and Alembic for schema migrations.
  • Task Queue & Scheduling: Celery with Redis as the message broker and result backend to handle background feed polling and broadcast queues.
  • HTTP Engine & API Clients: HTTPX for async HTTP calls, python-telegram-bot for Telegram Bot API communication, and pydantic v2 for strict payload validation.
  • CI/CD & Monitoring: Docker Compose deployment orchestrated via GitHub Actions CI/CD workflows.
+-----------------------------------------------------------------------------------+
|                                MY CONTENT SOURCES                                 |
|   +------------------+   +----------------------+   +--------------------------+  |
|   |  Blog (RSS/Atom) |   | YouTube Data API v3  |   |  X (Twitter) v2 Webhooks |  |
|   +--------+---------+   +----------+-----------+   +------------+-------------+  |
+------------|------------------------|----------------------------|----------------+
             |                        |                            |
             v                        v                            v
+-----------------------------------------------------------------------------------+
|                            FASTAPI INGESTION ENDPOINTS                            |
|   - Request signature validation & webhook authentication                         |
|   - Payload parsing with Pydantic v2 schemas                                      |
|   - Enqueuing jobs to Redis task queue                                           |
+-------------------------------------+---------------------------------------------+
                                      |
                                      v
+-----------------------------------------------------------------------------------+
|                              CELERY WORKER PIPELINE                               |
|   1. HTML / Markdown Content Sanitization (BeautifulSoup4 / Bleach)               |
|   2. Canonical URL & Metadata Extraction                                          |
|   3. SHA-256 Content Deduplication & Hash Verification                            |
+-------------------------------------+---------------------------------------------+
                                      |
                                      v
+-----------------------------------------------------------------------------------+
|                        NEON POSTGRESQL PERSISTENCE LAYER                          |
|   - Stores normalized post records: [id, title, body, source_url, status, hash]    |
|   - Maintains state transitions: [PENDING -> PROCESSING -> PUBLISHED / FAILED]     |
+-------------------------------------+---------------------------------------------+
                                      |
                                      v
+-----------------------------------------------------------------------------------+
|                        TELEGRAM DISPATCHER & BROADCASTER                          |
|   - Rate limit governor (Enforces Telegram 30 msg/sec & group thresholds)        |
|   - Dynamic HTML formatting & inline CTA keyboard builder                         |
|   - Exponential backoff retry engine for transient HTTP 429 / 5xx errors          |
+-------------------------------------+---------------------------------------------+
                                      |
                                      v
                           +--------------------+
                           |  @serhiiTalks Feed |
                           +--------------------+

Detailed Data Flow & Implementation Details​

1. Multi-Source Ingestion & Polling​

I designed the ingestion layer to combine passive webhooks with active background polling:

  • RSS/Atom Feed Monitor: A periodic Celery beat worker polls my blog RSS feeds every 10 minutes, extracting new entries and feeding them into my processing pipeline.
  • YouTube API Integration: Polling jobs query the YouTube Data API v3 for my newly uploaded videos, capturing video IDs, thumbnail URIs, and descriptions.
  • Webhook Listeners: FastAPI endpoints receive real-time webhooks, verifying HMAC request signatures before accepting payloads.

2. Payload Normalization & SHA-256 Deduplication​

To prevent duplicate posts during feed updates or retries, I implement strict payload validation and hashing:

  • Sanitization: HTML payloads are cleaned with BeautifulSoup4, converting formatted text into Telegram-compliant tags (<b>, <i>, <code>, <pre>).
  • Deduplication Hash: I compute a SHA-256 fingerprint from the canonical URL and publication timestamp: 
    import hashlib

    def generate_post_hash(canonical_url: str, published_at_iso: str) -> str:
        payload = f"{canonical_url.strip().lower()}:{published_at_iso.strip()}"
        return hashlib.sha256(payload.encode("utf-8")).hexdigest()
  • Database Upsert: The record is saved in Neon PostgreSQL with a PENDING status. If the hash exists, my worker skips execution to prevent duplicate broadcasts.

3. Telegram Broadcasting & Rate Limit Control​

My broadcast worker fetches PENDING records and dispatches them via the Telegram Bot API:

  • Message Layout: I format clean executive summaries with inline action buttons pointing directly to the canonical source link.
  • Rate Limit Governor: Telegram enforces a global cap of 30 messages per second. I implemented a Redis-backed token bucket rate limiter to prevent 429 Too Many Requests errors.
  • Failure Recovery: Network or API errors trigger Celery's exponential backoff policy (retrying from 5 seconds up to 1 hour). Successful dispatches store the returned Telegram message_id and mark the status as PUBLISHED.

Database Schema Design​

I modeled the primary persistence layer in SQLAlchemy for Neon PostgreSQL:

CREATE TABLE content_posts (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    content_hash VARCHAR(64) UNIQUE NOT NULL,
    source_platform VARCHAR(32) NOT NULL, -- 'medium', 'youtube', 'blog', 'twitter'
    canonical_url TEXT NOT NULL,
    title VARCHAR(255) NOT NULL,
    body_markdown TEXT,
    media_urls JSONB DEFAULT '[]'::jsonb,
    status VARCHAR(20) NOT NULL DEFAULT 'PENDING', -- 'PENDING', 'PUBLISHED', 'FAILED'
    telegram_message_id BIGINT,
    published_at TIMESTAMP WITH TIME ZONE NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_posts_status ON content_posts(status);
CREATE INDEX idx_posts_hash ON content_posts(content_hash);

Key Takeaways & Practical Results​

  • 100% Content Ownership: I am never vulnerable to a single platform's algorithm changes or outages.
  • Zero Overhead: Publishing once on any supported source automatically mirrors my content across my entire network.
  • Direct Subscriber Connection: My readers receive clean, instant updates in Telegram formatted specifically for mobile and desktop reading.

Join My Telegram Channel​

If you want to follow my real-time engineering write-ups, system design breakdowns, and open-source releases, subscribe to my channel:

👉 @serhiiTalks on Telegram

Last updated on