Best Practices for HTTP Servers

How can web servers be optimised to perform faster? Can resources be cached, compressed, etc.
There are a number of HTTP headers that one needs to understand in order to deal with performance optimisations.

1. Last-Modified – Origin server indicates when the resource was believed to have been last modified. Given by both date and time.
2. Etag – Entity tag or unique ID for each version of a resource which is usually comprised of the file-location, file-size and last modified date
3. Expires – Origin server indicates to the browser when the resource becomes stale or how long to keep in cache. Applicable to HTTP/1.0; apparently  deprecated in HTTP/1.1
4. Cache-control – Origin server indicates to the browser and intermediaries whether or not to cache the resource and if to cache, for how long (cache-control:max-age). Applicable to HTTP/1.1

Notice the similarity in some definitions.
In practice, Last-Modified and Etag are similar validators and can be grouped together. Expires and cache-control can similarly be grouped together.
Expires and cache-control would indicate to the browser that a cached resource is still valid and therefore, does not need to be fetched.
Last-Modified and Etag however, will require the browser to attempt to validate with the server every-time. If the origin server ascertains that the resource has not changed, status 304 is returned (indicating that the cache is still valid). The actual resource would be returned otherwise.
It is a good practice to use both groups where appropriate:

1. For content that seldom changes (e.g. CSS, images) – use Expires and Cache-control
2. For others (e.g. javascript files, HTML, dynamic pages) – use Last-Modified and Etag

Note that use of Etag in a web farm environment can be tricky as the Etag computed value may be distinct for different machines in the web farm.
Apache allows the Etag value to be computed using only last modified date and size without the inode value.
IIS 6 allows administrators to set the Etag_change_number – e.g. to 0 – to synchronise the servers in a web farm (incidentally, IIS 7 already has 0 as the immutable default value).

More information regarding Cache-control can be found here

Sending SMS

There are different means of sending SMS that I’m aware of, namely:

1. via GSM modem
2. via SMSC (SMS Centre)
3. via 3rd-party gateways

Given that the 3rd-party gateways have got different offerings from vendor to vendor, here are some differences between the 1st two:

	GSM Modem	SMSC
Cost	Usually lower and is fixed per SMS	Normally higher unless there is bulk discount
Use of short code (4-5 digit numbers)?	No	Yes
Use of TPOA	No	Yes
Latency per SMS (or throughput)	5-8 secs or 13-17 secs with DR (up to 15-20 SMS/sec)	Less than 1 sec (20-40 or up to 100s SMS/ sec)
Interface/ protocol	USB/ COM ports	Usually SMPP (TCP/IP)

TPOA - Stands for Transmission Path Originating Address. It is a feature to mask the originating number with a alphanumeric string (up to 11 characters). E.g. Citi, HSBC


To understand how all these entities/ components come together, see the following UML diagrams:

Troubleshooting Common .NET HTTP Connection Errors

The first is to identify whether the error is with the client or server (or even intermediaries). Most of the errors begin with “The underlying connection was closed: “…

Indications of client error

• An unexpected error occurred on a send – Could be due to:
  • antivirus software installed on the client machine

 

Indications of intermediary error

• The remote name could not be resolved or The proxy name could not be resolved – Could be due to:
  • DNS issue
  • inability to access the hosts file
• Unable to connect to the remote server – Likely to have gotten through the DNS but hit a connection glitch due to:
  • proxy
  • firewall
  • network authentication

 

Indications of server error or intermediaries (e.g. load balancer, proxy, etc.)

• An unexpected error occurred on a receive – Server or intermediary unexpectedly closes the TCP connection. May be due to:
  • Server or intermediary timeout values set too low (TODO: increase the client’s request timeout & also the server’s execution timeout such that client’s request timeout < intermediary’s < server’s)
  • Server crashes (TODO: check the server logs)
• The request was canceled or The connection was closed unexpectedly – Server or intermediary cancels the TCP connection (TCP RSET). May be due to:
  • low (execution) timeout setting on the server or intermediary (TODO: increase the client’s request timeout & also the server’s execution timeout such that client’s request timeout < intermediary’s < server’s)
  • low keep-alive timeout setting on the server or intermediary (TODO: set client’s keep-alive timeout < intermediary’s < server’s)
  • Although improbable, it could be the result of insufficient concurrent connections at the client machine (typically defaulted to 2 by ServicePointManager.DefaultConnectionLimit). This is aggravated by slow processing at the server side
• Could not establish trust relationship for the SSL/TLS secure channel -
  • Root and chain certificates are not installed correctly (TODO: install in the Local Machine store)
• A connection that was expected to be kept alive was closed by the server – Server or intermediary closed the connection that is to be kept-alive due to:
• client’s keep-alive timeout is longer than that of the server or intermediary (TODO: set client’s keep-alive timeout < intermediary’s < server’s)
• The operation has timed out – Could be due to:
  • stale DNS entry for the proxy
  • socket timed-out before the response is returned (TODO: increase the client’s request timeout & read-write timeout. Increase the server’s execution timeout)
  • insufficient connection at the server such that no connection thread is available (TODO: increase server’s max connection to 12x number of processors)

Best Practices for running ASP.NET on IIS 7

When should application pools be turned into web gardens?

Web gardens should only be used if the application doesn’t use in-process session variables but rather out-of-process ones (e.g. session state service or database session state). Reason is that a web garden would have at least 2 worker processes which do not share in-process (session) memory.
Drivers to using web gardens are:

1. Application makes long-running synchronous requests
2. Application is low in availability and crashes often
3. Application creates high CPU load on worker process

Best Practices Systems Settings

1. Optimum paging file size setting:
   1. 1.5x the RAM for 32-bit OS
   2. system-managed for 64-bit OS
2. Disk queue length should always average less than 2
3. Processor queue length should be less than the number of processors 
4. Network utilisation should be less than 50%

Application Isolation Policy

1. Some applications should be deployed into their own application pool
   1. mission critical and should be highly available
   2. needs to be isolated for troubleshooting (e.g leaks memory)
   3. expected to incur high throughput/ load. putting undue stress on server resources
   4. has non-standard requirements (e.g. configuration)

Application Pool Settings

1. Enable 32-bit Applications (WOW64) where necessary (e.g. when application uses 32-bit libraries)
2. Always use Managed Pipeline Mode unless application runs into issues otherwise
3. Increase Queue Length if required (especially if server hits error 503 – Service Unavailable)
4. Run using Network Service as the Identity
5. For HA requirement, disable Idle Timeout of the work process
6. Disable Rapid Fail Protection for PRD environment
7. For generating event log under the pool recycling settings, set all events to true
8. Set to 0 minutes the recycling of the application pool at Regular Time Interval

IIS Configuration Settings

1. Enable at least static compression
2. Turn off directory browsing

Handling Session States

1. Use InProc in standalone mode (where web farms are not used). This performs the best.
2. Use StateServer (perferably on a different machine) to share session state in web farms. However, the performance penalty is 15% slower than InProc. A more serious consequence is that the StateServer would now be a single point of failure.
3. Use SQLServer to share session state in web farms introduces 25% penalty over InProc. However, if set-up as a failover cluster, it will work reliably with HA.
4. Use CustomProvider for other in-memory, distributed and equally reliable session replication. These tend to perform faster than SQLServer and almost equally fast as StateServer. Some popular providers are: Windows Server AppFabric, ScaleOut SessionServer, and memcached.

Handling MachineKey

1. The machineKey is used to encrypt and decrypt the following: ViewState, Forms authentication, out-of-process Session data.
2. By default, the encryption and decryption keys are auto-generated by ASP.NET whenever the application pool restarts.
3. For web farms, do ensure that the machineKey for all the servers in the web farm are configured to use the same values (esp. validationKey, decryptionKey). Auto-generation or inconsistent values would cause unnecessary issues.

Solving Mixed-Content Warnings

Web applications can be written to work with HTTP as well as HTTPS schemes.

To do so, URLs to resources should be encoded as relative URLs.

For example, on a site (softwarehard.blogspot.com), all URLs to resources should be encoded as “/images/imageResource.jpg” or “images/imageResource.jpg” 
instead of “http://softwarehard.blogspot.com/images/imageResource.jpg”

Doing so will allow for the resource to be served in both HTTP and HTTPS schemes depending on the the page request.

However, if an external resource is required, the authority (domain name) needs to be included. The most portable way to include the external resource is to use a scheme-less URL.

For example: “//cdn.blogger.com/js/jquery.js”

Note that neither HTTP nor HTTPS have been specified and doing so will allow us to skirt the mixed-content issue.

This is valid under the URI RFC

However, the caveats are that:

1. this works well in web browsers but will likely break in email clients!
2. there is also a minor downside to using this in IE 7 & 8 for stylesheet in that the CSS resource will be downloaded twice (explained here)