PROJECT

Scrape Websites with Configurable Python Automation

Custom software development, Website data scraping, Configurable Python automation, Structured CSV and JSON export, JavaScript page rendering, Recurring scraper runs, Database and API delivery

  • Industry: Research services
  • Germany Location: Germany
  • Delivery time: 10 weeks
  • Technologies: Python, JavaScript / TypeScript
3 months

This client spends time collecting product, listing, and contact data from public websites. Manual or ad hoc collection makes repeatable research work harder. The software accepts target URLs or URL patterns, selectors, and page rules. It then returns structured records in CSV, JSON, or Excel. It gives a research organization one configurable tool for recurring website data extraction, including pages that require browser-based rendering.

Summary

Problem

The research organization spends time replacing manual or ad hoc website collection with repeatable extraction.

Solution

The solution became a configurable scraper that follows page rules, handles dynamic content, and exports structured records.

Result

The software is made to reduce manual collection and inconsistent extraction.

Client story

The client is a research organization from Germany. It collects product, listing, and contact data from public websites for research work.

Manual and ad hoc collection costs the team time, especially when target websites use pagination, changing page structures, and dynamic tables.

The request was simple: accept target URLs or URL patterns, configurable selectors or configuration files, and return selected website content as structured records.

The solution was shaped around configurable field rules, multi-page crawling, and browser-based rendering. Scheduled runs and clear logs supported repeatable collection.

The client asked for

  1. Accept URLs, URL patterns, selectors, XPaths, or configuration files
  2. Extract selected content into structured records
  3. Follow pagination across pages and websites
  4. Handle JavaScript pages, dynamic tables, and lazy loading

Why ready-made software did not fit

Ready-made data extraction software covers many source types, while managed scraping services focus on APIs, browser infrastructure, and proxy services. This client needed project-specific fields, selectors, and pagination rules. The solution also needed local operation, documentation, and defined output requirements.

Project overview

The solution was designed around a scraper configuration screen where the research team defines target URLs, fields, and selectors. Pagination rules and output formats are also available.

To handle different page structures, the scraper uses configurable extraction rules for fields, visible text, and tables. It crawls connected pages and follows pagination where required.

Browser-based rendering was provided for JavaScript pages, dynamic tables, and lazy-loaded content. Static pages can use direct HTTP requests and HTML parsing instead.

The solution was also shaped for repeatable operation. It runs from the command line or a local machine, supports scheduling, and records failures and retries.

As a result, collected records can move to CSV, JSON, or Excel. PostgreSQL, a lightweight REST JSON endpoint, and project documentation support further use by the client team.

Key features delivered

  • Configurable target URLs and URL patterns
  • Field selector and XPath editor
  • Multi-page crawling and pagination
  • JavaScript and lazy-loaded content rendering
  • CSV, JSON, and Excel export
  • Rate limits, delays, retries, and user-agent settings
  • Run history and scraping logs
  • PostgreSQL or REST JSON delivery

Can be added

+ Robots.txt compliance controls + Cloud scheduling + Data archiving + Project-specific documentation + Additional database mappings

Who else can use this solution

These businesses have a similar task, and the same solution can be adapted to their public website data collection.

A market research agency

Task It collects product, market, and contact data from public websites.
How the solution helps The solution helps it apply repeatable extraction rules and export structured research records.
Business services

A data analysis team

Task It gathers listings, tables, and selected content across public websites.
How the solution helps The solution helps it follow pagination, render dynamic pages, and deliver records to files or databases.
Professional services

A research consultancy

Task It archives or analyzes visible text and structured content from public webpages.
How the solution helps The solution helps it schedule runs, review failures, and keep generated files with run logs.
Business services

Technology stack

Python

Automation language

Python runs the configurable scraper, extraction rules, data handling, and exports.

JavaScript / TypeScript

Browser automation language

JavaScript or TypeScript supports browser automation where browser-based extraction is used.

Requests

Request HTML pages

Requests (a Python library for sending web requests) retrieves pages for direct extraction.

BeautifulSoup

Parse page HTML

BeautifulSoup (a Python library for reading HTML) finds selected fields and visible content in page markup.

Scrapy

Crawl multiple pages

Scrapy (a Python framework for web crawling) follows connected pages and supports multi-page extraction.

Playwright

Render browser pages

Playwright is a high-level API for automating web browsers during browser-based extraction.

Pandas

Handle extracted data

Pandas organizes extracted records before export or delivery.

openpyxl

Write Excel files

openpyxl (a Python library for reading and writing Excel files) supports XLSX output.

CSV

Export table records

CSV stores extracted records in a simple tabular file.

JSON

Export structured records

JSON stores extracted fields as structured records for files or APIs.

Integrations

PostgreSQL

The scraper writes collected records to PostgreSQL when database delivery is required.

REST JSON endpoint

The solution exposes collected records through a lightweight REST JSON endpoint when requested.

Excel

The solution exchanges extracted records through XLSX files.

Command line and local scheduling

The scraper runs from the command line or local machine with cron or Windows Task Scheduler.

The result

Set extraction rules for each website

The configuration screen keeps target websites, fields, and selectors together. Pagination and dynamic-content settings are also available.

This gives the research team a clear place to define what the scraper collects before a run.

How it works

  1. The configuration stores website URLs, required fields, and selectors or XPaths.
  2. The scraper applies field rules to selected page content.
  3. Pagination and dynamic-content modules prepare the run for connected pages and browser-rendered content.
  • Target URLs can be single addresses or URL patterns.
  • Outputs include CSV, JSON, or Excel.
Note: Rate limits, delays, user-agent settings, retries, and robots.txt compliance controls support responsible collection.
Scraper configuration

Scraper configuration

Functions on this screen
Extract selected fields Handle Pagination Render dynamic content

Review structured website records

The results table presents extracted fields, visible text, and tables as structured records.

CSV, JSON, and Excel export keeps the collected data ready for research files and further analysis.

How it works

  1. The extraction module creates records from selected fields, visible text, tables, or listings.
  2. The results table displays extracted fields and record count.
  3. The export module creates CSV, JSON, or XLSX files.
  • JSON is suitable for structured exchange.
  • Excel is available where the extraction scope requires it.
Extracted records and export

Extracted records and export

Functions on this screen
Extract structured data Export data to CSV or JSON

Track runs and investigate failures

The run history shows target websites, timestamps, and status. It also records errors, retries, and generated-file paths.

Logs give the team a practical way to review scraper activity when a page changes or a run needs attention.

How it works

  1. The scraper records run timestamps, target websites, and success or failure status.
  2. The log viewer displays error messages and retry information.
  3. Generated-file links connect each run with its CSV, JSON, or log paths.
  • Run history records recurring collection activity.
  • Error messages show issues recorded during a run.
Run history and scraper logs

Run history and scraper logs

Functions on this screen
Log scraping activity

Send records to a database or API

The delivery screen prepares collected records for database or API use instead of leaving them only in export files.

Field-to-schema mapping and a JSON payload preview help align extracted data with the destination structure.

How it works

  1. The destination module stores database settings or destination configuration.
  2. The mapping module matches extracted fields with the target table schema.
  3. The integration module records schema-matched JSON, database records, and run logs.
  • The solution supports PostgreSQL delivery when required.
  • A lightweight REST JSON endpoint is available for requested integrations.
Database or API delivery

Database or API delivery

What the solution gives

With this solution, the research organization can:

  • Define extraction rules for each target website
  • Collect selected content across paginated pages
  • Handle browser-rendered and lazy-loaded content
  • Export structured records to CSV, JSON, or Excel
  • Review failures, retries, and generated files
  • Send collected records to a database or API

Could ready-made systems do the job?

Before the design, the task was compared with ready-made products by their own descriptions.

What the client needs Data extraction software Data Extraction Web scraping APIs Oxylabs, Zyte, Scrapfly No-code scraping tools Octoparse Custom solution
Accept URLs and extraction rules
Extract selected website content
Follow pagination across pages
Handle dynamic browser content partly
Fits the client's own rules: stages, roles, reports — — —
All steps in one place —
No manual steps between programs partly partly

By the public descriptions of the products and their category.

The main parts of the task exist in data extraction software, web scraping APIs, and no-code scraping tools. A ready-made product is enough when only one part of the task is needed.

This client needed its own software because the extraction rules, local operation, documentation, output formats, logs, and destination requirements had to fit the research work in one place. Ready-made options also differ in how they handle manual steps between programs.

In the end

In the end, the solution was made to take manual collection out of the path from public websites to structured research data.

FAQ

Thinking about building a similar system? Read the answers below to see how configurable scraping, exports, browser rendering, and delivery can fit a research workflow.

Need something similar? Let's talk
Or browse more projects