scrapy-ctl(1)
NAME
scrapy-ctl - Python Scrapy control script
SYNOPSIS
scrapy-ctl [command] [OPTIONS] ...
DESCRIPTION
Scrapy is controlled through the scrapy-ctl control script. The script
provides several commands, for different purposes. Each command supports its own particular syntax. In other words, each command supports
a different set of arguments and options.
OPTIONS
- fetch [OPTION] URL
- Fetch a URL using the Scrapy downloader
- --headers
Print response HTTP headers instead of body
- runspider [OPTION] spiderfile
- Run a spider
- --output=FILE
Store scraped items to FILE in XML format
- settings [OPTION]
- Query Scrapy settings
- --get=SETTING
Print raw setting value
- --getbool=SETTING
Print setting value, intepreted as a boolean
- --getint=SETTING
Print setting value, intepreted as an integer
- --getfloat=SETTING
Print setting value, intepreted as an float
- --getlist=SETTING
Print setting value, intepreted as an float
- --init Print initial setting value (before loading extensions and spi
ders)
- shell URL | file
- Launch the interactive scraping console
- startproject projectname
- Create new project with an initial project template
- --help, -h
- Print command help and options
- --version
- Print Scrapy version and exit
- --logfile=FILE
- Log file. if omitted stderr will be used
- --loglevel=LEVEL, -L LEVEL
- Log level (default: None)
- --nolog
- Disable logging completely
- --spider=SPIDER
- Always use this spider when arguments are urls
- --profile=FILE
- Write python cProfile stats to FILE
- --lsprof=FILE
- Write lsprof profiling stats to FILE
- --pidfile=FILE
- Write process ID to FILE
- --set=SET
- Set/override setting (may be repeated)
- --settings=MODULE
- Python path to the Scrapy project settings
AUTHOR
Scrapy-ctl was written by the Scrapy Developers <scrapy-developers@googlegroups.com>.
- This manual page was written by Ignace Mouzannar <mouzannar@gmail.com>,
for the Debian project (but may be used by others).