Open Source Web Scraping Software
Web Scraping tools automate the extraction of structured data and media from websites. These tools convert unstructured page content into usable formats like RSS feeds, knowledge bases, or local files. This type of software is for researchers and developers who need to collect video streams, audio files, and text without relying on official APIs. Self-hosting these apps ensures that scraping schedules and extracted media libraries remain on private hardware. You avoid the limitations and costs associated with managed data extraction services.
This page lists 18 open source tools in the Web Scraping category. The most popular are RSSHub, Maxun and SurfSense. Most use the GPL-3.0 or MIT license, and 15 offer an official Docker image.
RSSHub
Generate RSS feeds from various websites and social media platforms that do not provide them natively.
Maxun
A no-code platform to extract structured data from websites using AI, recording, and automated crawling tools.
SurfSense
Build searchable knowledge bases from diverse data sources with custom LLM integration and automated AI agents.
VidBee
Download video and audio content from over 1000 websites with automated RSS feed subscription capabilities.
RSS-Bridge
Generates web feeds for websites that do not provide their own RSS or Atom feeds.
MediaGo
Download videos from YouTube and other platforms by sniffing m3u8 and HLS streams via desktop.
OmniGet
A cross platform media downloader and manager for online courses, videos, music, and electronic books.
Streambert
Stream and download movies, tv series, and anime through a cross-platform desktop application without ads.
BiliTools
Download videos, audio, subtitles, and metadata from Bilibili using a lightweight cross-platform desktop interface.
Pinchflat
Self-hosted media manager that automatically downloads YouTube channels and playlists using yt-dlp for local archiving.
bitmagnet
A self-hosted BitTorrent indexer and DHT crawler with content classification and a web interface.
YaCy
A decentralized peer-to-peer search engine that allows users to host their own search index.
yt-dlp Web UI
A self-hosted web interface and RPC server for managing video downloads using the yt-dlp tool.
Wayback
A web archiving and playback tool that captures and preserves web content via an IM-style interface.
Fredy
A self-hosted real estate crawler that monitors German listing platforms and sends instant notifications for new properties.
Fess
An enterprise search server that crawls websites, file systems, and data stores for full text indexing.
Discount Bandit
Track product prices across multiple e-commerce stores and receive notifications when specific price criteria are met.
Kibitzr
A self hosted personal web assistant for automating routine tasks and monitoring web activity.
Join our newsletter to get shiny new open source software delivered to your inbox. Unsubscribe anytime.