AI-based scrapping API designer

[DrWebDeveloper]

To summarize my idea for an AI-based scraping API designer tool:

• Feature: The tool should test a URL and save the scraping rules.
• Functionality: Use these saved rules to scrape data in the future without hitting the website every time.
• AI Role: Utilize AI to create and optimize these scraping rules.

I mean the AI should only be used until confirmation, once it is confirmed that the columns are getting correct data, the rules should be stored and used for scrapping with some Python library to be used for free further scrapping.

[raznem]

I am experimenting with the generation of scraping code instead of always passing pages through LLM.
Do you mean anything different to it by scraping rules?

[DrWebDeveloper]

No @raznem you got it right, best of luck