DanMcInerney/xsscrapy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request #58 from shelld3v/patch-1

Browse Source 
Bug fix
This commit is contained in:
Dan McInerney
2022-01-14 10:31:31 -07:00
committed by GitHub
parent e1d8ff73fa b8def30f55
commit 2d142844f4
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
xsscrapy/middlewares.py
View File

@@ -3,7 +3,7 @@ from urlparse import unquote
from pybloom import BloomFilter from pybloom import BloomFilter
import random import random
import re import re
from settings import bloomfilterSize from xsscrapy.settings import bloomfilterSize
# Filter out duplicate requests with Bloom filters since they're much easier on memory # Filter out duplicate requests with Bloom filters since they're much easier on memory
#URLS_FORMS_HEADERS = BloomFilter(3000000, 0.00001) #URLS_FORMS_HEADERS = BloomFilter(3000000, 0.00001)