Jessica Brown

Overview

On Thursday, February 6, 2025, multiple Cloudflare services, including R2 object storage, experienced a significant outage lasting 59 minutes. This incident resulted in complete operational failures against R2 and disruptions to dependent services such as Stream, Images, Cache Reserve, Vectorize, and Log Delivery. The root cause was traced to human error and inadequate validation safeguards during routine abuse remediation procedures.

Impact Summary

• Incident Duration: 08:14 UTC to 09:13 UTC (primary impact), with residual effects until 09:36 UTC.

• Primary Issue: Disabling of the R2 Gateway service, responsible for the R2 API.

• Data Integrity: No data loss or corruption occurred within R2.

Affected Services

1. R2: 100% failure of operations (uploads, downloads, metadata) during the outage. Minor residual errors (<1%) post-recovery.

2. Stream: Complete service disruption during the outage.

3. Images: Full impact on upload/download; delivery minimally affected (97% success rate).

4. Cache Reserve: Increased origin requests, impacting <0.049% of cacheable requests.

5. Log Delivery: Delays and data loss (up to 4.5% for non-R2, 13.6% for R2 jobs).

6. Durable Objects: 0.09% error rate spike post-recovery.

7. Cache Purge: 1.8% error rate increase, 10x latency during the incident.

8. Vectorize: 75% query failures, 100% insert/upsert/delete failures during the outage.

9. Key Transparency Auditor: Complete failure of publish/read operations.

10. Workers & Pages: Minimal deployment failures (0.002%) for projects with R2 bindings.

Incident Timeline

• 08:12 UTC: R2 Gateway service inadvertently disabled.

• 08:14 UTC: Service degradation begins.

• 08:25 UTC: Internal incident declared.

• 08:42 UTC: Root cause identified.

• 08:57 UTC: Operations team begins re-enabling the R2 Gateway.

• 09:10 UTC: R2 starts to recover.

• 09:13 UTC: Primary impact ends.

• 09:36 UTC: Residual error rates recover.

• 10:29 UTC: Incident officially closed after monitoring.

Root Cause Analysis

The incident stemmed from human error during a phishing site abuse report remediation. Instead of targeting a specific endpoint, actions mistakenly disabled the entire R2 Gateway service. Contributing factors included:

• Lack of system-level safeguards.

• Inadequate account tagging and validation.

• Limited operator training on critical service disablement risks.

The Risks of CDN Dependencies in Critical Systems

Content Delivery Networks (CDNs) play a vital role in improving website performance, scalability, and security. However, relying heavily on CDNs for critical systems can introduce significant risks when outages occur:

• Lost Revenue: Downtime on e-commerce platforms or SaaS services can result in immediate lost sales and financial transactions, directly affecting revenue streams.

• Lost Data: Although R2 did not suffer data loss in this incident, disruptions in data transmission processes can lead to lost or incomplete data, especially in logging and analytics services.

• Lost Customers: Extended or repeated outages can erode customer trust and satisfaction, leading to churn and damage to brand reputation.

• Operational Disruptions: Businesses relying on real-time data processing or automated workflows may face cascading failures when critical CDN services are unavailable.

Remediation Steps

Immediate Actions:

• Deployment of additional guardrails in the Admin API.

• Disabling high-risk manual actions in the abuse review UI.

In-Progress Measures:

• Improved internal account provisioning.

• Restricting product disablement permissions.

• Implementing two-party approval for critical actions.

• Enhancing abuse checks to prevent internal service disruptions.

Cloudflare acknowledges the severity of this incident and the disruption it caused to customers. We are committed to strengthening our systems, implementing robust safeguards, and ensuring that similar incidents are prevented in the future.

For more information about Cloudflare's services or to explore career opportunities, visit our website.

Uploading large files to a website can fail due to server-side limitations on file size. This issue is typically caused by default configurations of web servers like Nginx or Apache, or by PHP settings for sites using PHP.

This guide explains how to adjust these settings and provides detailed examples for common scenarios.

For Nginx

Nginx limits the size of client requests using the client_max_body_size directive. If this value is exceeded, Nginx will return a 413 Request Entity Too Large error.

Step-by-Step Fix

1. Locate the Nginx Configuration File

   • Default location: /etc/nginx/nginx.conf
   • For site-specific configurations: /etc/nginx/sites-available/ or /etc/nginx/conf.d/.

2. Adjust the client_max_body_size Add or modify the directive in the appropriate http, server, or location block. Examples:

   Increase upload size globally:

   http {
       client_max_body_size 100M;  # Set to 100 MB
   }
   

   Increase upload size for a specific site:

   server {
       server_name example.com;
       client_max_body_size 100M;
   }
   

   Increase upload size for a specific directory:

   location /uploads/ {
       client_max_body_size 100M;
   }
   

3. Restart Nginx Apply the changes:

   sudo systemctl restart nginx
   

4. Verify Changes

   • Upload a file to test.
   • Check logs for errors: /var/log/nginx/error.log.

For Apache

Apache restricts file uploads using the LimitRequestBody directive. If PHP is in use, it may also be restricted by post_max_size and upload_max_filesize.

Step-by-Step Fix

1. Locate the Apache Configuration File

   • Default location: /etc/httpd/conf/httpd.conf (CentOS/Red Hat) or /etc/apache2/apache2.conf (Ubuntu/Debian).
   • Virtual host configurations are often in /etc/httpd/sites-available/ or /etc/apache2/sites-available/.

2. Adjust LimitRequestBody Modify or add the directive in the <Directory> or <VirtualHost> block.

   Increase upload size globally:

   <Directory "/var/www/html">
       LimitRequestBody 104857600  # 100 MB
   </Directory>
   

   Increase upload size for a specific virtual host:

   <VirtualHost *:80>
       ServerName example.com
       DocumentRoot /var/www/example.com
       <Directory "/var/www/example.com">
           LimitRequestBody 104857600  # 100 MB
       </Directory>
   </VirtualHost>
   

3. Update PHP Settings (if applicable)

   • Edit the php.ini file (often in /etc/php.ini or /etc/php/7.x/apache2/php.ini).

   • Modify these values:

     upload_max_filesize = 100M
     post_max_size = 100M
     

   • Restart Apache to apply changes:

     sudo systemctl restart apache2  # For Ubuntu/Debian
     sudo systemctl restart httpd    # For CentOS/Red Hat
     

4. Verify Changes

   • Upload a file to test.
   • Check logs: /var/log/apache2/error.log.

Examples for Common Scenarios

1. Allow Large File Uploads to a Specific Directory (Nginx): To allow uploads up to 200 MB in a directory /var/www/uploads/:

   location /uploads/ {
       client_max_body_size 200M;
   }
   

2. Allow Large File Uploads for a Subdomain (Apache): For a subdomain uploads.example.com:

   <VirtualHost *:80>
       ServerName uploads.example.com
       DocumentRoot /var/www/uploads.example.com
       <Directory "/var/www/uploads.example.com">
           LimitRequestBody 209715200  # 200 MB
       </Directory>
   </VirtualHost>
   

3. Allow Large POST Requests (PHP Sites): Ensure PHP settings align with web server limits. For example, to allow 150 MB uploads:

   upload_max_filesize = 150M
   post_max_size = 150M
   max_execution_time = 300   # Allow enough time for the upload
   max_input_time = 300
   

4. Handling Large API Payloads (Nginx): If your API endpoint needs to handle JSON payloads up to 50 MB:

   location /api/ {
       client_max_body_size 50M;
   }
   

General Best Practices

1. Set Reasonable Limits: Avoid excessively high limits that might strain server resources.
2. Optimize Server Resources:
   • Use gzip or other compression techniques for file transfers.
   • Monitor CPU and memory usage during large uploads.
3. Secure Your Configuration:
   • Only increase limits where necessary.
   • Validate file uploads on the server-side to prevent abuse.
4. Test Thoroughly:
   • Use files of varying sizes to confirm functionality.
   • Check server logs to troubleshoot unexpected issues.