Skip to main content

Crawler

Generic crawler platform infrastructure β€” Laravel Crawler plugin plus Python runtime adapter contract.

Status​

KeyValue
Layercore
TierL1
Statuswip
Version0.12.0
PriceFree (free)
CategoryInfrastructure

Overview​

Crawler is a Layer 1 Core Plugin for multi-saas-kit. It provides the standard contract between Laravel applications and an external Python crawler runtime, plus common models for DataSource / CrawlDefinition / CrawlJob / CrawlSchedule / Channel / Item.

As of 0.12.0, Crawler has three boundaries:

  1. Laravel plugin β€” CrawlableInterface, CrawlApiClient, standard Eloquent models, and preservation-safe migrations.
  2. Platform runtime β€” platform/service/crawler-runtime, a FastAPI + APScheduler runtime. Its default adapter is none and it does not know project-specific tables.
  3. Project adapter package β€” mounted by each project at /app/project_adapters and loaded with CRAWLER_RUNTIME_ADAPTER_FACTORY=<module>:<factory>. scout.how book/ranking/AI task behavior now lives behind this boundary.

Runtime Adapter Contract​

Base runtime defaults:

CRAWLER_RUNTIME_ADAPTER=none
CRAWLER_RUNTIME_ADAPTER_FACTORY=

Project adapter example:

CRAWLER_RUNTIME_ADAPTER=scout
CRAWLER_RUNTIME_ADAPTER_FACTORY=project_adapters.scout_runtime:create_adapter
CRAWLER_RUNTIME_ENABLE_SCOUT_AI_ROUTES=true
CRAWLER_RUNTIME_ENABLE_SCOUT_POSTPROCESSING=true

Storage Boundary​

config('crawler.tables.*') and shared PLG_CRAWLER_*_TABLE values can override crawler core table names. Migrations skip existing tables and only drop tables on rollback when the plugin created them and wrote the ownership comment marker.

Domain Boundary​

Crawler owns crawler core tables and the API/runtime contract. books, book_keywords, keyword_rankings, keyword_ranking_items, keyword_score_tiers, daily_stats, and ai_tasks are project adapter domain tables.

License​

MIT

Demos​

πŸ›’ View on Plugin Store: store.codebase.how/plugins/crawler