diff --git a/README.md b/README.md index 1fda595..cf37413 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,33 @@ # Harvester Python package for harvesting commonly available data, such as free proxy servers. +## Modules +### Proxy +The `proxy` module will harvest proxies from URLs with the `fetch_list` function. + +It functions by running a regular expression against the HTTP response, looking for +strings that match a `username:password@ip:port` pattern where username and password +are optional. + +```python +from harvester.proxy import fetch_list + + +URLS = [ + 'https://api.openproxylist.xyz/socks4.txt', + 'https://api.openproxylist.xyz/socks5.txt', + 'https://api.proxyscrape.com/?request=displayproxies&proxytype=socks4', +] + + +def main(): + """Main entry point.""" + for url in URLS: + proxies = fetch_list(url) + print(proxies) + +``` + ## Testing ``` pip install -r requirements.txt