API Scraping is an application developed in Python using FastAPI, Selenium, and Docker technologies. With this API, you can perform automations, scrape data, and create customized workflows for collecting and manipulating information directly from web pages.
This repository provides a scalable structure for building an API that performs scraping via HTTP requests. It is easily integrable into automation workflows, webhooks, and other services.
- Python: The main language for development.
- FastAPI: A framework for building fast, modern, scalable, and simple APIs.
- Selenium: A browser automation tool used to interact with web pages.
- Docker: For packaging and running the environment in an isolated and consistent way.
- Navigation Automation: Full control over web pages using Selenium.
- Data Collection: Extract information from websites efficiently and at scale.
- API Execution: Define endpoints to start and control scraping and automation processes.
- Webhooks: Easy integration with other systems to trigger automated events.
- Modular Structure: Easily expand scraping logic by adding new scripts and endpoints.
- Ensure Docker and Docker Compose are installed on your machine.
- Clone the project repository:
git clone <REPOSITORY_URL> cd <PROJECT_DIRECTORY>
- Start the Docker environment:
docker-compose up --build
- Access the application via a browser or tools like cURL/Postman:
http://0.0.0.0:8000/
main.py: Contains the main logic of the FastAPI application with defined endpoints.scraping.py: Example automation script using Selenium.docker-compose.yml: Configuration for running the Docker environment.requirements.txt: Python dependencies for the application.tests/: (Optional) Suggested directory to add your automated tests.
- Start automations with Selenium through FastAPI endpoints.
- Execute customized tasks, such as extracting information from dynamic web pages.
- Integrate the API with other systems by consuming the exposed endpoints.
- Use the API as a base to create complex automation workflows.
💡 Tip: Be creative and adapt the structure to meet your scraping and automation needs!
Feel free to contribute to the project! To do so:
- Fork the repository.
- Create a branch for your feature/your-feature:
git checkout -b my-feature
- Submit a Pull Request (PR) with your changes.
- Or get in touch to exchange ideas and share knowledge! 👽
🚀 API Scraping: A versatile solution for web automation and data collection.
🌟 Automate, collect, and explore the potential of the web with ease!