crossroad-companion

Go service that fetches RSS/Atom feeds, extracts readable article content, and generates a morning digest — built as the content pipeline for the Crossroad e-reader firmware.

Architecture

┌─────────────────────────────────────────────────┐
│                  RSS / Atom feeds                │
│  (blogs, news, substack, anything with a feed)   │
└──────────────────────┬──────────────────────────┘
                       │ fetch (gofeed)
                       ▼
┌─────────────────────────────────────────────────┐
│                   Fetcher                        │
│  Parses feeds, returns Items (title, link,       │
│  content, author, published date, GUID)          │
└──────────────────────┬──────────────────────────┘
                       │ each item
                       ▼
┌─────────────────────────────────────────────────┐
│                  Extractor                       │
│  Fetches full article from link                  │
│  → go-readability (content extraction)           │
│  → html-to-markdown (clean .md output)           │
└──────────────────────┬──────────────────────────┘
                       │ markdown
                       ▼
┌─────────────────────────────────────────────────┐
│                  Stitcher                        │
│  Orchestrates fetch→extract→write pipeline:      │
│  • Deduplicates by GUID (bbolt + in-memory)      │
│  • 4-worker concurrent article processing        │
│  • Writes to output/<feed-name>/<date-title>.md  │
│  • read-later pipeline: POST /convert            │
└──────────────────────┬──────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────┐
│              Morning Paper Generator             │
│  • Triggered at 6am + on every fetch cycle       │
│  • Table of contents + per-feed summaries        │
│  • Writes morning-paper/<date>-morning-paper.md  │
│  • Symlinks morning-paper/latest.md              │
│  • Optional weather line via WEATHER_TEXT env    │
└──────────────────────┬──────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────┐
│                 HTTP Server                      │
│  Serves articles + paper to Crossroad devices    │
│  on :8080 (configurable via PORT)                │
└─────────────────────────────────────────────────┘

Quick Start

# Clone
git clone https://durandal.exe.xyz/starsetbyte/crossroad-companion.git
cd crossroad-companion

# Build
go build -o crossroad-companion ./cmd/crossroad/

# Configure
export FEED_URLS="https://example.com/feed.xml,https://blog.example.org/rss"
export OUTPUT_DIR="./output"
export PORT=8080
export FETCH_INTERVAL="1h"
export WEATHER_TEXT="☀  72°F, clear"   # optional

# Run
./crossroad-companion

Configuration

All config is via environment variables — no config file needed:

Variable	Default	Description
FEED_URLS	(required)	Comma-separated RSS/Atom feed URLs
OUTPUT_DIR	./output	Where articles and morning papers land
PORT	8080	HTTP server port
FETCH_INTERVAL	1h	How often to poll feeds (Go duration format)
WEATHER_TEXT	(empty)	Optional weather line in morning paper header

HTTP API

Method	Path	Description
GET	/	Service info + endpoint listing
GET	/health	Health check ({"status":"ok"})
GET	/feeds/	List available feeds with article counts
GET	/output/<feed>/	List articles in a feed directory
GET	/output/morning-paper/latest.md	Today's morning paper
POST	/feeds/fetch?url=<feed_url>	Trigger an immediate feed fetch
POST	/convert	Read-later: extract any URL to markdown

Read-later pipeline

curl -X POST http://localhost:8080/convert \
  -H "Content-Type: application/json" \
  -d '{"url": "https://longform.org/article", "title": "My Title"}'

Articles land in output/readlater/<date-title>.md.

Directory Layout

output/
├── <feed-name>/           # One directory per feed
│   ├── 2026-07-01-article-slug.md
│   ├── 2026-07-02-another-article.md
│   └── ...
├── readlater/             # Saved-via-convert articles
│   └── 2026-07-03-my-title.md
├── morning-paper/
│   ├── 2026-07-03-morning-paper.md
│   └── latest.md → 2026-07-03-morning-paper.md   # symlink
└── data/
    └── seen-guids.bolt    # bbolt DB of processed GUIDs

Crossroad Integration

The companion is the content pipeline. The device is the reader. Here's how they talk:

1. Feed Sync Activity (on-device)

The ESP32-C3 connects to WiFi, hits the companion's HTTP API, and pulls markdown files to the SD card. A new FeedSync IExtra activity in src/activities/extras/:

┌──────────────────────────────────────────────────┐
│  Crossroad (ESP32-C3)                            │
│                                                  │
│  FeedSync::onEnter()                             │
│    ├─ WiFi connect (existing stack)              │
│    ├─ GET /feeds/          → list of feeds       │
│    ├─ GET /output/<feed>/  → list of articles    │
│    ├─ For each new article:                      │
│    │    GET /output/<feed>/<article>.md          │
│    │    → write to SD: /feeds/<feed>/<article>.md│
│    └─ GET /output/morning-paper/latest.md        │
│         → write to SD: /morning/latest.md        │
│                                                  │
│  FeedReader::onEnter()                          │
│    ├─ Scan /feeds/ directory on SD               │
│    ├─ Render markdown via existing IF parser     │
│    ├─ Button nav: next/prev article, next feed   │
│    └─ Log FEED_READ event → pet gets happier     │
└──────────────────────────────────────────────────┘

2. What Already Exists on Firmware

The Crossroad firmware already has:

• Markdown parser — The IF engine parses .md with YAML frontmatter, links, and formatting. Feed articles are simpler markdown (no branching logic), so the renderer handles them natively.
• WiFi stack — ESP32-C3 has built-in WiFi. The SDK provides WiFiClient for HTTP GETs.
• SD card — Already mounted for books and stories. Feed articles land in a new /feeds/ directory.
• Event log — Reading feeds generates events → feeds into achievements, pet state, and stats.
• Extras menu — Feed Sync and Feed Reader slot in as IExtra activities behind the Extras bridge point.

3. Memory Budget

ESP32-C3 has 380KB usable RAM, no PSRAM. The feed sync path is read-only (GET requests, stream to SD) — no need to buffer full articles in RAM:

Operation	RAM needed
HTTP GET response buffer	~2KB (ring buffer)
SD write buffer	~512B (SdFat sector)
Directory listing (JSON parse)	~1KB (stack string_view into response)
Total per sync	< 4KB

The markdown renderer already handles files from SD — feeds are just smaller markdown files with simpler structure than IF stories.

4. Companion Address Discovery

The device needs to find the companion on the local network. Two options:

Option A: Hardcoded IP — Configure companion IP in a settings file on the SD card (/config/companion.conf → 192.168.1.100:8080). Simplest, zero code.

Option B: mDNS — Companion advertises as crossroad-companion.local (add an mDNS library to the Go service). Device resolves the hostname. More resilient when IPs change.

Start with Option A. It's one WiFiClient call.

5. Sync Flow (end to end)

┌──────────┐     WiFi      ┌──────────────┐    cron    ┌──────────┐
│ Crossroad │ ◄─────────── │  companion    │ ◄──────── │  Feeds   │
│ (ESP32)   │   HTTP GET   │  (Go, :8080)  │           │  (RSS)   │
└─────┬─────┘              └──────┬───────┘           └──────────┘
      │                           │
      │   GET /feeds/             │
      │ ◄───────────────────────  │
      │   [{"name":"HN","articles":12},...]
      │                           │
      │   GET /output/HN/         │
      │ ◄───────────────────────  │
      │   [{"name":"2026-07-03-...md","size":4521},...]
      │                           │
      │   GET /output/HN/2026-07-03-....md
      │ ◄───────────────────────  │
      │   # Article Title         │
      │   *July 3, 2026 — by ...* │
      │   Content markdown...     │
      │                           │
      │   → SD: /feeds/HN/2026-07-03-....md
      │                           │
      │   GET /output/morning-paper/latest.md
      │ ◄───────────────────────  │
      │   # ☀ Morning Paper       │
      │   ...                     │
      │                           │
      │   → SD: /morning/latest.md
      │                           │
      │   Done. Pet: "eager, alert"

Companion ↔ Carin

Carin is the agent dispatch board running on the same machine. The companion integrates as a data source:

Carin dashboard ─── GET /feeds/ ───► crossroad-companion
                   ◄─── feed list ──

A future Carin card/panel could show recently fetched articles and trigger on-demand fetches. The companion is standalone — it doesn't import Carin, it just exposes an HTTP API that Carin (or anything else) can call.

Project Status

Ready for device integration. The fetch→extract→stitch→serve pipeline is complete. The firmware side (FeedSync + FeedReader activities) is designed and budgeted but not yet implemented.

See the Crossroad firmware repo for device-side development and the full IDEAS.md backlog.