Jeriel’sTechnology RadarMain Site

github.com

Crawlee for Python

A Python framework for web scraping and browser automation.

https://github.com/apify/crawlee-python

About This Resource

A Python framework for web scraping and browser automation. It provides tools for scheduling crawls, handling page interactions, and extracting data from websites.

Saved Notes

Original notes saved with this link. They may describe an earlier version.

GitHub - apify/crawlee-python: Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

A web scraping and browser automation library

Crawlee covers your crawling and scraping end-to-end and helps you build reliable scrapers. Fast.

Assessment: Hold

I’ve given this a try locally and on a colaboratory and I’ve encountered setup issues despite their own documentation and guides. I’ve no idea why the setup is in such a broken state, but because of it I’d rather try other crawlers.

Skill issue? Perhaps. Maybe I should try the JS version instead because after all the attempts I did, I only found out that it’s JS first and not Python. Oof.

GitHub - apify/crawlee: Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Next