Deploy & Run Scrapy Spiders On AWS Server Using ScrapeOps

Question

i have Deployed & Run Scrapy Spiders On AWS Server Using ScrapeOps. the issue is that it is not saving the scrapped data at AWS server at the specific path. when i have checked the logged at scrapeOps it is showing the following error:

ERROR: Disabled feed storage scheme: s3. Reason: missing botocore library

well i have installed following commands at AWS console pip install boto3 botocore pip install botocore

but still the issue is same. please clarify do i need to pip install botocore at AWS console? or i have to install it anywhere else?

Bobby · Answer

Hey! The error message “ERROR: Disabled feed storage scheme: s3. Reason: missing botocore library” suggests that your Scrapy project cannot access the AWS S3 storage because the required ‘botocore’ library is either missing or not installed correctly. You need to make sure botocore and boto3 are installed correctly in the same environment where your Scrapy spider is running. Use the following command in your AWS console: pip install boto3 botocore Test the installation by running a simple Python script to access S3: import boto3 s3 = boto3.client('s3') response = s3.list_buckets() print(response) If this script works, you’ve installed the libraries successfully. After that, modify your Scrapy project settings to enable S3 storage. Here’s an example configuration in your settings.py:Python FEED_URI = 's3://your-bucket-name/path/to/store/data' FEED_FORMAT = 'json' FEED_EXPORTERS = { 'json': 'scrapy.exporters.JsonItemExporter' } Replace your-bucket-name with the actual name of your S3 bucket. If you were already attempting to use S3 within your spider code, double-check the import statements and the way you’re interacting with S3. Sometimes, restructuring imports or method calls can help if the library installations are correct. If this still does not work, make sure to check the ScrapeOps logs as they provide valuable insights into errors during execution. Carefully examine the logs for additional clues about missing dependencies or configuration errors specific to your project. On another note, make sure botocore, boto3, and Scrapy have compatible versions. Refer to their documentation for specific compatibility details. Also consider using a virtual environment (e.g., virtualenv) to isolate your project’s dependencies from other Python installations on your AWS server. Let me know if you encounter any issues along the way. Provide more details about your Scrapy project setup, relevant code snippets, and complete error messages if further troubleshooting is necessary! Best, Bobby

Become a contributor for community

DigitalOcean Documentation

Resources for startups and AI-native businesses

The developer cloud

Start building today

Report this

Deploy & Run Scrapy Spiders On AWS Server Using ScrapeOps

Related questions

GitHub Student Pack DigitalOcean benefit

Forgot to destroy droplet, now have $10 charge – any help?

I'm getting this ERROR: you dont' have permission to complete this action

Digital Ocean Preview Deploy on PR getting github user permission error

Advanced Django CI/CD Pipeline with GitHub Actions

Github student pack

External image App fails deployment, App image succeeds

Deploying app from digital ocean using github education

Automatic redeployments continue to run in GitHub Actions despite being completed in DigitalOcean

How do I exclude a file from being updated?

Saved files cannot be found.

Cannot rsync to my droplet from githubaction

Report this

Deploy & Run Scrapy Spiders On AWS Server Using ScrapeOps

Related questions

GitHub Student Pack DigitalOcean benefit

Forgot to destroy droplet, now have $10 charge – any help?

I'm getting this ERROR: you dont' have permission to complete this action

Digital Ocean Preview Deploy on PR getting github user permission error

Advanced Django CI/CD Pipeline with GitHub Actions

Github student pack

External image App fails deployment, App image succeeds

Deploying app from digital ocean using github education

Automatic redeployments continue to run in GitHub Actions despite being completed in DigitalOcean

How do I exclude a file from being updated?

Saved files cannot be found.

Cannot rsync to my droplet from githubaction