02 · Completed
Polite
Scraper.
A production-minded Node.js web scraper for collecting structured book data from Books to Scrape.
CATALOGUE
PARSE
OUTPUT
{
books: [
...
]
}The problem
Turn catalogue pages into useful data.
I wanted a way to collect book information from Books to Scrape and bring the results together as structured data instead of working with individual pages separately.
The solution
A Node.js scraper that visits catalogue pages, extracts book information, processes and normalizes the data, validates records, and produces a JSON file.
My role
I built the scraper and worked through the implementation and troubleshooting required to get the different parts working together.
Responsible scraping
The project includes request throttling and robots.txt handling as part of the approach to responsible web scraping.
Automation
Automated testing and GitHub Actions CI help make the scraper easier to check and run consistently.
How it works
From catalogue to structured output.
Resolve catalogue URL
Fetch HTML
Parse book data
Normalize records
Validate schema
Save JSON
Parsing
HTML is parsed so the scraper can extract the book information it needs.
Normalization
The extracted values are processed into a consistent structure.
Validation
Records are checked against a schema before the final JSON is produced.
What I learned
Troubleshooting is part of the build.
The hardest part for me was troubleshooting. I learned how to work through issues until the pieces of the scraper worked reliably together, while gaining practical experience with responsible web scraping, data processing, validation, testing, and automation.