When executing Scrapy, we can follow these rules for managing the crawler execution:
- If the scraping process fails, you can look in the console log for lines that include [scrapy] DEBUG.
- If you want to stop Scrapy while it is still processing, just press the key combination Ctrl+C.
- When Scrapy has finished processing data, it will display the following information in the log console: [scrapy] INFO: Spider closed (finished).
- By default, Scrapy will append new data to the end of the output file if it already exists. If the file does not exist, it will create one. So, if you want to only get new data, then you should first remove the old file.