Scroll to navigation

SCRAPY(1) General Commands Manual SCRAPY(1)


scrapy - the Scrapy command-line tool


scrapy [command] [OPTIONS] ...


Scrapy is controlled through the scrapy command-line tool. The script provides several commands, for different purposes. Each command supports its own particular syntax. In other words, each command supports a different set of arguments and options.


fetch [OPTION] URL

Fetch a URL using the Scrapy downloader
Print response HTTP headers instead of body

runspider [OPTION] spiderfile

Run a spider
Store scraped items to FILE in XML format

settings [OPTION]

Query Scrapy settings
Print raw setting value
Print setting value, interpreted as a boolean
Print setting value, interpreted as an integer
Print setting value, interpreted as a float
Print setting value, interpreted as a float
Print initial setting value (before loading extensions and spiders)

shell URL | file

Launch the interactive scraping console

startproject projectname

Create new project with an initial project template

--help, -h

Print command help and options


Log file. if omitted stderr will be used

--loglevel=LEVEL, -L LEVEL

Log level (default: None)


Disable logging completely


Always use this spider when arguments are urls


Write python cProfile stats to FILE


Write lsprof profiling stats to FILE


Write process ID to FILE


Set/override setting (may be repeated)


Scrapy was written by the Scrapy Developers.

This manual page was written by Ignace Mouzannar <>, for the Debian project (but may be used by others).

October 17, 2009