From 8b37b39b1475e60ccc052b5fd46cd2d945cf1eb0 Mon Sep 17 00:00:00 2001 From: agatha Date: Mon, 6 Nov 2023 14:53:50 -0500 Subject: [PATCH] Add usage example --- README.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/README.md b/README.md index 1fda595..cf37413 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,33 @@ # Harvester Python package for harvesting commonly available data, such as free proxy servers. +## Modules +### Proxy +The `proxy` module will harvest proxies from URLs with the `fetch_list` function. + +It functions by running a regular expression against the HTTP response, looking for +strings that match a `username:password@ip:port` pattern where username and password +are optional. + +```python +from harvester.proxy import fetch_list + + +URLS = [ + 'https://api.openproxylist.xyz/socks4.txt', + 'https://api.openproxylist.xyz/socks5.txt', + 'https://api.proxyscrape.com/?request=displayproxies&proxytype=socks4', +] + + +def main(): + """Main entry point.""" + for url in URLS: + proxies = fetch_list(url) + print(proxies) + +``` + ## Testing ``` pip install -r requirements.txt