examples-of-web-crawlers
A collection of beginner-friendly Python web crawler examples for several Chinese websites.
- Stars
- 14.7k
- Forks
- 3.8k
- Licence
- MIT
- Language
- HTML
- Last update
- 1.4 years ago
- Type
- Learning resource
No commits in over a year — check for forks or alternatives before adopting.
What it is
This is a set of Python scripts that show how to collect data from websites such as Taobao, Tmall, WeChat, Douban and QQ. According to its description, the examples are meant to be friendly to beginners. The README was empty in the provided material, so the exact contents of each example could not be verified.
- Study how a Python script fetches and parses pages from a shopping or social site
- Learn Selenium-based browser automation for pages that load content dynamically
- Explore multithreaded crawling patterns as a starting point for your own projects
Great for
- Beginners who want to see working examples of web scraping in Python
- Students learning how crawlers, Selenium or PyQuery are used in practice
- Developers who want a reference for scraping Chinese websites
Not ideal for
- Anyone needing a production-ready scraping system, since the README was not available to confirm robustness or maintenance
- People who need to scrape sites outside the ones listed, since coverage beyond the description is unverified
- Users who need guidance on legal or terms-of-service compliance, which is not addressed in the provided material
How to get it
- 1
Clone and build from source
You'll need: Python 3 · Git
Clone the repository
git clone https://github.com/shengqiangzhang/examples-of-web-crawlers - 2
Run it
- Clone the repository to your computer
- Open the folder and look for the example you are interested in
- Read the code in that example's file or folder before running it
- Install any libraries it imports, such as Selenium or PyQuery, and run it
- 3
Get the most out of it
- Start with one example and modify the target selectors to learn how parsing works
- Compare the Selenium-based examples with the plain request-based ones to see the trade-offs
- Check the repository topics for related areas such as agent pools and multithreading to explore further
Downloads
Strengths
- Described as beginner-friendly examples in Python
- Covers several different websites as separate examples
- Released under the MIT license, which permits reuse with attribution
Trade-offs
- The README content was not available, so setup and usage details are unknown
- Examples target specific sites that may have changed since the code was written
- Scraping sites may conflict with their terms of service; check before running
How it compares
| Project | Stars | Type | Level | Updated | Best for |
|---|---|---|---|---|---|
| examples-of-web-crawlers (this) | 14.7k | Learning resource | beginner | 1.4 years ago | Beginners who want to see working examples of web scraping in Python |
| proxy_pool | 23.8k | App | intermediate | 5 months ago | Developers building web scrapers who want to experiment with rotating proxies |
| spider-flow | 11.4k | App | intermediate | 3.4 years ago | Non-programmers who need to collect data from websites and prefer visual flow definitions |
| akshare | 23.2k | Library | beginner | 4 days ago | Python users doing academic or personal research on Chinese financial markets |
| newspaper | 15.2k | Library | intermediate | 11 days ago | Python developers building news aggregation or article collection scripts |
| requests-html | 13.8k | Library | beginner | 2.6 years ago | Python developers who already use requests and want to parse HTML in the same style |
Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.