HansenDeployment

Programming

NGINX upstream timed out 110 Connection timed out while reading response header from upstream

19 September 2026 · 10 min read

NGINX upstream timed out 110 Connection timed out while reading response header from upstream

Encountering the “NGINX: upstream timed out (110: Connection timed out) while reading response header from upstream” error can be incredibly frustrating for any system administrator or developer. This error, which often appears in your NGINX error logs, signifies that NGINX, acting as a reverse proxy, couldn’t receive a response header from the upstream server within the allotted time. This doesn’t necessarily mean the upstream server is down; it simply means the connection took too long. Understanding the root causes and implementing effective solutions is crucial for maintaining the performance and reliability of your web applications. We’ll delve into the common reasons for this error, ranging from network congestion to server overload, and explore practical strategies to resolve it, ensuring a smoother user experience and optimal server operation. This guide aims to provide you with a comprehensive understanding of troubleshooting and preventing this common NGINX issue.

Understanding the “NGINX: upstream timed out” Error

The “NGINX: upstream timed out (110: Connection timed out) while reading response header from upstream” error is a common issue when using NGINX as a reverse proxy. Essentially, NGINX is waiting for a response from the backend server (the “upstream”) but isn’t receiving it within a configured timeframe. The “110” in the error message refers to the “Connection timed out” error code. This timeout can occur for several reasons, making diagnosis somewhat challenging. It’s important to differentiate this from other connection errors, such as connection refused or host unreachable, as the root cause and solutions will differ.

One of the primary reasons for this timeout is insufficient resources on the upstream server. If the server is overloaded with requests, it may be unable to process and respond to NGINX’s requests in a timely manner. Network latency or congestion can also contribute to the problem. If there are delays in network traffic between NGINX and the upstream server, the connection might time out before the response can be fully transmitted. Misconfigured firewall settings, blocking communication between NGINX and the upstream server, are another potential culprit. Each of these potential causes requires a distinct approach to identification and resolution, emphasizing the importance of a systematic troubleshooting methodology.

Furthermore, the error could stem from issues within the upstream application itself. For instance, a slow database query or a computationally intensive process could delay the response. According to a study by Google, 53% of mobile site visitors leave a page if it takes longer than three seconds to load. Google Developers Therefore, optimizing upstream applications is crucial for preventing timeout errors. Understanding the interplay between NGINX, the network, and the upstream server is essential for effectively resolving this error.

Common Causes of NGINX Upstream Timeout

Several factors can trigger the “NGINX: upstream timed out” error. Identifying the specific cause is the first step towards resolving the issue. Let’s explore some of the most common culprits:

  • Network Issues: High latency, packet loss, or network congestion between NGINX and the upstream server.
  • Upstream Server Overload: The upstream server is under heavy load and cannot respond quickly enough.
  • Firewall Restrictions: Firewalls are blocking communication between NGINX and the upstream server.
  • Application Problems: Slow database queries or inefficient code on the upstream server.
  • Incorrect Timeout Settings: The timeout values configured in NGINX are too short.

Incorrectly configured timeout settings within NGINX itself are a frequent offender. The default timeout values might be too aggressive for certain applications, especially those that require longer processing times. Another common issue arises from network instability. Transient network problems can cause intermittent timeout errors, making them particularly difficult to diagnose. Monitoring network performance metrics, such as latency and packet loss, can help identify these issues. Tools like ping, traceroute, and network monitoring solutions can provide valuable insights. Remember to check your DNS settings too, as incorrect DNS configuration can delay connection establishment.

Firewall settings are another area to investigate. A firewall rule might be inadvertently blocking traffic between NGINX and the upstream server. Reviewing firewall logs and rules can help identify and rectify these issues. Ensuring that the necessary ports are open and that traffic is allowed in both directions is crucial. In some cases, the problem might be related to the upstream server’s configuration. For example, the upstream server might be configured to limit the number of concurrent connections, causing NGINX requests to be queued and eventually time out. Regularly review and optimize the upstream server’s configuration to ensure it can handle the expected load.

Troubleshooting and Resolving Nginx Upstream Timeout Errors

Troubleshooting NGINX upstream timeout errors requires a systematic approach. Start by examining the NGINX error logs for detailed information about the error. These logs often provide clues about the specific cause of the timeout, such as the IP address of the upstream server and the specific request that timed out. Next, check the resource utilization of the upstream server, including CPU usage, memory usage, and disk I/O. High resource utilization indicates that the server might be overloaded and unable to respond to requests in a timely manner.

Featured Snippet: One effective solution is to increase the timeout values in your NGINX configuration. The proxy_read_timeout directive controls how long NGINX waits to receive a response from the upstream server. Increasing this value can prevent timeout errors when the upstream server takes longer than expected to respond. For example, setting proxy_read_timeout to 300s will instruct NGINX to wait for 300 seconds before timing out. Adjust this value based on the expected response time of your application.

Here’s a step-by-step guide to increasing the timeout values in your NGINX configuration:

  1. Open your NGINX configuration file (usually located at /etc/nginx/nginx.conf or /etc/nginx/conf.d/default.conf).
  2. Locate the location block that handles the requests to your upstream server.
  3. Add or modify the following directives within the location block:
    • proxy_connect_timeout: Defines the timeout for establishing a connection with the upstream server.
    • proxy_send_timeout: Defines the timeout for sending a request to the upstream server.
    • proxy_read_timeout: Defines the timeout for reading a response from the upstream server.
  4. Set appropriate values for these directives, such as:
    • proxy_connect_timeout 75s;
    • proxy_send_timeout 75s;
    • proxy_read_timeout 300s;
  5. Save the configuration file and restart NGINX to apply the changes.

Beyond adjusting timeout values, consider optimizing your upstream application’s performance. Profile your application to identify slow database queries or inefficient code. Implement caching mechanisms to reduce the load on the upstream server. For instance, using a content delivery network (CDN) can significantly reduce the load on your servers by caching static content closer to users. According to Akamai, CDNs can reduce latency by as much as 50%. Akamai Performance Solutions Regularly monitor your application’s performance and make adjustments as needed to ensure optimal responsiveness. You can monitor your NGINX performance using tools like NGINX Amplify. These strategies collectively improve the reliability of your web applications.

Preventing Future Timeout Errors

Proactive measures can significantly reduce the likelihood of encountering “NGINX: upstream timed out” errors. Regular monitoring of server resources is paramount. Implement monitoring tools to track CPU usage, memory consumption, disk I/O, and network traffic on both the NGINX server and the upstream server. Set up alerts to notify you when resource utilization exceeds predefined thresholds. This allows you to identify potential problems before they escalate and cause timeout errors. Capacity planning is also crucial. Regularly assess your application’s traffic patterns and resource requirements to ensure that your servers have sufficient capacity to handle the expected load.

Load balancing is another effective strategy for preventing timeout errors. Distributing traffic across multiple upstream servers can prevent any single server from becoming overloaded. NGINX provides built-in load balancing capabilities that can be configured to distribute traffic based on various algorithms, such as round robin, least connections, and IP hash. Implementing load balancing not only improves performance but also enhances the availability and resilience of your application. Regularly review and optimize your NGINX configuration. Ensure that timeout values are appropriately configured for your application’s needs. Also, review other NGINX settings, such as buffer sizes and connection limits, to ensure they are optimized for your workload. A well-configured NGINX server is less likely to experience timeout errors.

Finally, maintain a robust network infrastructure. Ensure that your network is properly configured and that there are no bottlenecks or points of failure. Regularly monitor network performance metrics, such as latency and packet loss, to identify and address any network issues. Consider using a content delivery network (CDN) to cache static content closer to users, reducing the load on your servers and improving response times. By taking these proactive measures, you can significantly reduce the risk of encountering NGINX upstream timeout errors and ensure the smooth operation of your web applications. Remember that consistent monitoring and proactive adjustments are key to long-term stability.

FAQ: NGINX Upstream Timeout

What does "NGINX: upstream timed out" mean?
This error indicates that NGINX, acting as a reverse proxy, did not receive a response from the backend server (upstream) within the configured timeout period.
What are the common causes of this error?
Common causes include network issues, upstream server overload, firewall restrictions, application problems, and incorrect timeout settings.
How can I fix this error?
You can fix this error by increasing timeout values in your NGINX configuration, optimizing your upstream application's performance, checking firewall settings, and addressing network issues.
How can I prevent this error from occurring in the future?
Preventive measures include regular monitoring of server resources, capacity planning, implementing load balancing, and maintaining a robust network infrastructure.
Addressing "NGINX: upstream timed out" errors effectively requires a blend of understanding the underlying causes, implementing targeted solutions, and adopting proactive prevention strategies. By systematically diagnosing the problem, adjusting NGINX configurations, optimizing upstream applications, and bolstering network infrastructure, you can significantly reduce the occurrence of these frustrating timeouts. Remember that constant vigilance and optimization are key. Explore further into NGINX configuration best practices and server performance tuning to enhance your web application's reliability and responsiveness. Consider diving into advanced topics like connection pooling and keepalive configurations to further optimize your setup. These strategies will ultimately lead to a better user experience and a more robust infrastructure.

Question & Answer :
I have Puma running as the upstream app server and Riak as my background db cluster. When I send a request that map-reduces a chunk of data for about 25K users and returns it from Riak to the app, I get an error in the Nginx log:

upstream timed out (110: Connection timed out) while reading response header from upstream

If I query my upstream directly without nginx proxy, with the same request, I get the required data.

The Nginx timeout occurs once the proxy is put in.

**nginx.conf** http { keepalive_timeout 10m; proxy_connect_timeout 600s; proxy_send_timeout 600s; proxy_read_timeout 600s; fastcgi_send_timeout 600s; fastcgi_read_timeout 600s; include /etc/nginx/sites-enabled/*.conf; } **virtual host conf** upstream ss_api { server 127.0.0.1:3000 max_fails=0 fail_timeout=600; } server { listen 81; server_name xxxxx.com; # change to match your URL location / { # match the name of upstream directive which is defined above proxy_pass http://ss_api; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_cache cloud; proxy_cache_valid 200 302 60m; proxy_cache_valid 404 1m; proxy_cache_bypass $http_authorization; proxy_cache_bypass http://ss_api/account/; add_header X-Cache-Status $upstream_cache_status; } }

Nginx has a bunch of timeout directives. I don’t know if I’m missing something important. Any help would be highly appreciated….

This happens because your upstream takes too long to answer the request and NGINX thinks the upstream already failed in processing the request, so it responds with an error. Just include and increase proxy_read_timeout in location config block. Same thing happened to me and I used 1 hour timeout for an internal app at work:

proxy_read_timeout 3600;

With this, NGINX will wait for an hour (3600s) for its upstream to return something.

EDIT:

@IanSmith I totally agree with you. This is more of a temporary solution. Long term solution is to work with the upstream app stakeholders to improve the response times or change how the whole feature works by enqueuing a background job and then check status as you mentioned, or use websockets to return the status back as soon the job finishes.

Sometimes the upstream is a legacy or external application and there’s little to none support, so this is also a last resort for those cases.