[Web Crawling] Web Crawler Container(Docker)
less than 1 minute read
Web Crawling Container(Docker)
- Run Container
- Start Scrapy Project
Run Container
- scrapy-kidnews image를 활용하여 scrapy container 생성
- 생성 후 scrapy 정상 실행을 위해, lzma 오류 해결해야함 * 🔗 lzma 해결
docker container run -it -d --network airflownet -v $(pwd):/home/scrapy/scrapy -e LC_ALL=C.UTF-8 --name scrapy scrapy-kidnews:latest
Start Scrapy Project
su - scrapy
source ./.venv/bin/activate
pip3 install scrapy
pip3 install pymongo
cd scrapy
scrapy startproject kidnewscrawling
