RSSからAIニュースを自動収集・重複排除・要約するPythonバッチを作る #自動化 - Qiita
@aidocs

Build a Python batch process to automatically collect AI news from RSS, remove duplicates, and summarize it

  • Python
  • RSS
  • AI
  • Automation
  • SQLite
0
Last updated at 2026-11-10Posted at 2026-11-10
Although AI-related news can be obtained through RSS, the same article appears repeatedly when you subscribe to multiple media. In addition, it is difficult to judge the content from the title alone, and reading every article every day is also a chore.

This time, we will create a Python batch process that regularly retrieves articles from RSS, removes duplicates using SQLite, and summarizes them in Japanese with a local LLM.

In conclusion, combining RSS retrieval, duplicate detection, summarization, and storage in a single script results in fewer dependencies and a structure that is easy to run with cron or a task scheduler. Since Ollama, which runs locally, is used for summarization, you can try it without sending the article bodies to an external API.

Configuration and preparation

The processing flow from RSS retrieval to duplicate detection, summarization, and SQLite storage is as follows:

The configuration for this project is as follows:

  • RSS retrieval: feedparser
  • Duplicate removal and storage: SQLite
  • Summarization: Ollama HTTP API
  • Execution: Python command and, if necessary, cron

First, install the Python packages:

python -m venv .venv source .venv/bin/activate pip install feedparser

On Windows, activate the virtual environment as follows:

.venv\Scripts\Activate.ps1 pip install feedparser

Start Ollama and prepare the model to use for summarization. Change the model to suit your environment.

ollama pull qwen2.5:7b

Fetch RSS and save it to SQLite

Below is the main batch script. The RSS URLs are examples, so change them to the feeds you actually want to subscribe to.

# rss_ai_news.py import hashlib import html import os import re import sqlite3 import urllib.request import json from datetime import datetime, timezone import feedparser RSS_FEEDS = [ "https://hnrss.org/newest?q=AI", "https://feeds.feedburner.com/TechCrunch/", ] DB_PATH = "ai_news.db" OLLAMA_URL = "http://localhost:11434/api/generate" OLLAMA_MODEL = os.getenv ("OLLAMA_MODEL", "qwen2.5:7b")def clean_text(value: str) -> str: """Remove HTML tags and extra whitespace.""" value = html.unescape(value or "") value = re.sub(r"<[^>]+>", " ", value) value = re.sub(r"\s+", " ", value) return value.strip() def normalize_title(title: str) -> str: """Create a string for comparing titles.""" return re.sub(r"[\W_]+", "", title.lower()) def init_db(conn: sqlite3.Connection) -> None: conn.execute( """ CREATE TABLE IF NOT EXISTS items ( id INTEGER PRIMARY KEY AUTOINCREMENT, url TEXT NOT NULL UNIQUE, title TEXT NOT NULL, title_key TEXT NOT NULL, published TEXT, body TEXT, summary TEXT, created_at TEXT NOT NULL ) """ ) conn.execute( "CREATE INDEX IF NOT EXISTS idx_items_title_key " "ON items(title_key)" ) conn.commit() def is_duplicate(conn: sqlite3.Connection, url: str, title_key: str, ) -> bool: row = conn.execute( "SELECT 1 FROM items WHERE url = ? OR title_key = ? LIMIT 1", (url, title_key), ).fetchone() return row is not None def summarize(title: str, body: str) -> str: prompt = f""" Please summarize the following article in Japanese. Include only the following three items in the output: - Key points: 2–3 sentences - Technical implications: 1 sentence - Cautions: 1 sentence Article title: {title} Article body:{body[:4000]} """.strip() request_body = json.dumps( { "model": OLLAMA_MODEL, "prompt": prompt, "stream": False, } ).encode("utf-8") request = urllib.request.Request( OLLAMA_URL, data=request_body, headers={"Content-Type"": "application/json"},method="POST", ) with urllib.request.urlopen(request, timeout=120) as response: result = json.loads(response.read().decode("utf-8")) return result.get("response", "").strip() def collect() -> None: conn = sqlite3.connect(DB_PATH) init_db(conn) inserted = 0for feed_url in RSS_FEEDS: print(f"fetch: {feed_url}") feed = feedparser.parse(feed_url) if getattr(feed, "bozo", False): print(" warning: A warning occurred while parsing the RSS") for entry in feed.entries: title = clean_text(entry.get("title", "Untitled"))url = entry.get("link", "").strip() if not url: continue body = clean_text( entry.get("summary") or entry.get("description") or "" ) title_key = normalize_title(title) if is_duplicate(conn, url, title_key): continue try:summary = summarize(title, body) except Exception as exc: print(f" summarize failed: {title} ({exc})") summary = "Summarization failed. Please check the original article." conn.execute( """ INSERT INTO items (url, title, title_key, published, body, summary, created_at) VALUES (?, ?, ?, ?, ?, ?, ?) """, ( url, title, title_key,entry.get("published", ""), body, summary, datetime.now(timezone.utc).isoformat(), ), ) conn.commit() print(f" added: {title}") inserted += 1 conn.() print(f"Done:{inserted} items were added") if __name__ == "__main__":collect()

Deduplication approach

Both the URL and the title are used for duplicate detection.

When the same article is distributed through multiple RSS feeds, the URLs will usually match. However, tracking parameters and URLs may differ depending on the publisher, so title_key, which is created by removing punctuation and whitespace from the title, is also compared.

For example, titles such as the following are more likely to be considered identical:

Announcing a new generative AI model Announcing a new generative AI model!

However, separate articles may also be merged if their titles are merely similar. In actual operation, adjustments are needed, such as including the publisher name, comparing publication dates, and introducing a similarity calculation.

In this implementation, we first used simple rules that make misclassifications easy to spot. Another benefit is that the SQLite tables can be inspected directly, making it easier to improve the detection logic later.

Summarizing with a local LLM

Ollama is accessed over HTTP without using the Python SDK. This makes it easy to swap out the summarization component.

For example, if you want to use a different local model, you can specify it as follows:

OLLAMA_MODEL=llama3.1:8b python rss_ai_news.py

In Windows PowerShell, use the following format:

$env:OLLAMA_MODEL = "llama3.1:8b" python rss_ai_news.py

Only the portion of the article body included in the RSS summary or description is used. Since the article body may be short depending on the RSS feed, if you want to improve summary quality, add a process that fetches the HTML at the URL and extracts the article body.

However, the HTML structure differs from site to site, and robots.txt and the terms of use must also be taken into consideration. It is safer to begin operations using only the information included in the RSS feed.

Checking execution results and running on a schedule

Run it for the first time.

python rss_ai_news.py

The data is saved in ai_news.db in the same directory. To check the saved contents, run the following in the SQLite CLI.

sqlite3 ai_news.db \ "SELECT title, summary, url FROM items ORDER BY id DESC LIMIT 10;"

From the second run onward, URLs or titles that have already been saved are ignored. Therefore, even if you fetch the RSS feed every day, the number of previously read articles will not keep increasing.

To run it every morning at 8:00 a.m. on Linux, add the following to cron.

0 8 * * * cd /path/to/project >> /path/to/project/.venv/bin/python rss_ai_news.py >> rss.log 2>&1

I fetched the same RSS feed multiple times locally and confirmed that articles were added only on the first run, while the number added on the second run was zero. I also confirmed that when Ollama is stopped, the article is saved as a summary failure, so even the RSS fetch results are not lost.

Summary

The process of collecting AI news from RSS is easier to implement when divided into the following four steps:

  • Fetch the RSS feed with feedparser
  • Detect duplicates using the URL and normalized title
  • Save the articles and summaries in SQLite
  • Summarize them using an LLM such as Ollama

Rather than building a full-featured news platform from the start, it is recommended to start small with RSS and SQLite. Once the number of articles grows, you can add per-outlet weighting, summary reruns, keyword classification, and output to Markdown or email, expanding the system to suit your needs.