DanMcInerney/xsscrapy

Go to file

DanMcInerney 2287da4725 made UA and referer header testing into one request

2014-08-14 03:05:36 -04:00

made UA and referer header testing into one request

2014-08-14 03:05:36 -04:00

.gitignore

lots of cleanup

2014-07-25 06:28:24 -04:00

loginform.py

log in now works

2014-06-24 05:50:32 -04:00

README.md

readme

2014-08-12 10:19:16 -04:00

scrapy.cfg

initial

2014-06-23 07:24:43 -04:00

README.md

xsscrapy

From within the main folder run:

scrapy crawl xsscrapy -a url='http://something.com'

If you wish to login then crawl:

scrapy crawl xsscrapy -a url='http://something.com/login_page' -a login=username -a pw=secret_password

Output is stored in formatted-urls.txt.