Chewy web scraper

Overview

This project scrapes dog food product listings from chewy.com and stores the results as JSON. The pipeline runs in three stages: collect product links, scrape each product page, and load the resulting data for analysis.

Files

File	Role
chewy_functions.py	Shared library. Defines every function the other scripts import.
get_links.py	Entry point for stage 1. Collects product URLs from a category page.
scrape_pages.py	Entry point for stage 2. Visits each product URL and extracts data.
load_data.py	Entry point for stage 3. Loads the JSON output and runs a sample query.
chewy_links.txt	Output of stage 1: one product URL per line.
test_dump.json	Output of stage 2: a dictionary of scraped product records.
sample.json	A hand-picked example of the test_dump.json schema.
dry-food-output.html	A saved copy of a category page’s HTML, used to develop the selectors in chewy_functions.py.

Pipeline

Collect product links

get_links.py calls get_links() with the starting category URL, https://www.chewy.com/b/dry-food-294. The function does the following:

1. Requests the page with a spoofed User-Agent header and parses it with BeautifulSoup.
2. Reads the “showing X - Y of Z results” text to determine how many product cards are on the page and how many exist in total.
3. Finds each product card, extracts its link, and appends it to the module-level prod_links list.
4. If more products remain and the page count is under the stop limit, finds the next-page link and calls get_links() again to continue from there.
5. Writes prod_links to chewy_links.txt through save_links().

The count and stop parameters limit how many pages get_links() visits. get_links.py calls the function with stop=1, so it scrapes only the first page of results and doesn’t follow pagination.

Scrape product pages

scrape_pages.py loads the saved links with load_links(), which reads chewy_links.txt into prod_links. It then passes a random sample of 10 links to scrape_pages().

For each link, scrape_pages() does the following:

1. Requests the product page and parses it with BeautifulSoup.
2. Extracts the product name and a specifications table (weight, lifestage, breed size, food form, special diet, and so on) into a dictionary.
3. Adds the ingredients list, if the page has an ingredients section.
4. Adds the recommendation rate, star rating, and review count, if the page has review data.
5. Stores the dictionary in the module-level prod_dict, keyed by the product’s item number.
6. Retries a request after a five-second wait if it fails, so one network error doesn’t stop the run.

scrape_pages.py then serializes prod_dict to JSON and writes it to test_dump.json through save_json().

Load and query the data

load_data.py loads test_dump.json with load_json() into a dictionary of product records, then runs a sample filter. The current script prints the item numbers of every product whose Special Diet field contains "Weight Control".

Data schema

Each entry in test_dump.json is keyed by item number and contains the fields scraped from that product’s specifications table, plus these fields:

Field	Contains
name	The product title.
Ingredients	The full ingredients text, if present.
Reccomendation Rate	The percentage of reviewers who recommend the product, if the page has review data. The key name matches the misspelling in the scraper’s output.
Rating	The average star rating, as a float.
Review Count	The number of reviews.

sample.json shows this schema for two products.

Known limitations

• The functions in chewy_functions.py use module-level lists and dictionaries (prod_links, prod_dict) instead of return values. Running a stage twice in the same session accumulates data instead of replacing it.
• save_links(), load_links(), save_json(), and load_json() use the relative path chewy\.... Run these scripts from the parent of the chewy directory, not from inside it, or update the paths.
• All page requests depend on the current HTML structure of chewy.com. The BeautifulSoup selectors use specific CSS class names, which will break if the site’s markup changes.
• Network requests retry indefinitely on failure in get_links(), and after a five-second wait in scrape_pages(), with no maximum retry count.

Run the pipeline

1. Run get_links.py to populate chewy_links.txt with product URLs from the category page.
2. Run scrape_pages.py to scrape a random sample of those URLs into test_dump.json.
3. Run load_data.py to load test_dump.json and query the results.