XSendFileContrib

A viewfile replacement to send static files efficiently

Introduction

This package implements a more efficient way to send static files using a web application like Foswiki. A standard configuration of a Foswiki uses the viewfile service to send static files over to the browser thus enforcing access rights on them. In doing so it reads the static file into memory completely before handing it over to the HTTP web server which takes over responsibility to contact the browser. This of course is far from optimal as it introduces a lot of overhead by copying the static file several times in memory. The impact on the overall system performance is even higher for large files being downloaded from a Foswiki server. For now the only alternative solution to this problem is not to protect static files by Foswiki's access control at all, and let the HTTP web server do its job on static files all on its own. Web servers are in fact quite good in sending static files, even large ones. So the best you can do is to leave this job to them.

Still you might require access control, even on large static files.

There's a quite unknown yet very important feature in Apache2, Nginx and Lighttpd, called "xsendfile", to interact with an upstream web application before sending a static file. This feature can be used to solve the above problem: it lets you control access to static files while still using the HTTP web server to do the heavy lifting.

This basically works like this:

• client requests an url
• the web server calls an upstream web application for this url
• the web app performs its specific actions and returns a specific HTTP header in its response, e.g. X-Lighttp-send-file, but not the data being requested, when access is granted, or an HTTP error otherwise
• the web server parses the HTTP headers returned by the web app looking for the X-Send-File header
• if it finds the X-Send-File header then the will the web serve transfer the static file this header points to over to the client
• otherwise it will return the response generated by the web app as usual

Configuring Foswiki

XSendFileContrib comes with a separate service script .../bin/xsendfile replacing the standard .../bin/viewfile. It differs from viewfile only in two respects

• any HTTP error is delivered immediately, that is without requiring Foswiki to render an error page while processing an exception.
• instead of copying over the static file to the downstream HTTP web server, it generates an empty response with the approrpiate X-Send-File header set

There are two parameters that need to be adjusted according to your web server's type and configuration:

• $Foswiki::cfg{XSendFileContrib}{Header} … the name of the HTTP header to trigger the X-Send-File feature in your web server (see below)
• $Foswiki::cfg{XSendFileContrib}{Location} … the uri prefix that the web server serves the actual static file from
• $Foswiki::cfg{XSendFileContrib}{Locations}{pdf} … the uri prefix for pdf files

Configuring Lighttpd

(see http://redmine.lighttpd.net/projects/1/wiki/X-LIGHTTPD-send-file for lighttpd)

The X-Send-File header for Lighttpd is called X-LIGHTTPD-send-file and points to the file path on disk that the web server should send to the client.

If you are already using Foswiki behind a lighttpd web server, all you need to do is to add "allow-x-send-file" => "enable" to your fastcgi stanza

url.rewrite-once += ( "^/pub/((?!System|Applications|images|cache).*)/(.*?)$" => "/bin/xsendfile/$1/$2" )

$HTTP["url"] =~ "^/bin/" {
  alias.url += ( "/bin" => "/path/to/bin/foswiki.fcgi" )

  fastcgi.server = ( ".fcgi" => ((
...
      "allow-x-send-file" => "enable"
    ),
  ))
}
$HTTP["url"] =~ "^/bin/xsendfile)" {
  expire.url = ( "" => "access 12 hours")
}

Configuring Nginx

(see http://wiki.nginx.org/XSendfile, http://wiki.nginx.org/X-accel)

In Nginx this feature is called X-Accel-Redirect. Yet the idea is the same. Requirements on the upstream web applications are almost identical besides using a different X-Send-File header called X-Accel-Redirect.

In addition Nginx requires a special "location" stanza for internal use only. Protected files will be served from there one the web app checked the original uri the.

location ~ ^/pub/(System|Applications|images|cache)/ {
  root /path/to/foswiki;
  expires 12h;
  gzip_static on;
}

location /pub {
  rewrite ^/pub/(.*)$ /bin/xsendfile/$1;
}

location /protected_files {
   internal;
   alias /path/to/foswiki/pub/;
}

location /protected_files/pdf {
   internal;
   alias /path/to/foswiki/pub/;

   # disable Accept-Ranges header as it breaks cookie authentication with pdf.js
   max_ranges 0;
}

Configuring Apache2

Apache itself requires an additional simple module that processes an X-Sendfile header. In a classical Foswiki Apache2 installation you have to modify your web server configuration as follows:

XSendFile on
XSendFilePath /path/to/pub
RewriteRule ^/+pub/+(.*)$  /bin/xsendfile/$1 [L,PT]

See https://tn123.org/mod_xsendfile/ for more information on how to compile mod_xsendfile and how to configure apache.

Installation

You do not need to install anything in the browser to use this extension. The following instructions are for the administrator who installs the extension on the server.

Open configure, and open the "Extensions" section. "Extensions Operation and Maintenance" Tab → "Install, Update or Remove extensions" Tab. Click the "Search for Extensions" button. Enter part of the extension name or description and press search. Select the desired extension(s) and click install. If an extension is already installed, it will not show up in the search results.

You can also install from the shell by running the extension installer as the web server user: (Be sure to run as the webserver user, not as root!)

cd /path/to/foswiki
perl tools/extension_installer <NameOfExtension> install

If you have any problems, or if the extension isn't available in configure, then you can still install manually from the command-line. See https://foswiki.org/Support/ManuallyInstallingExtensions for more help.

Dependencies

Name	Version	Description
Foswiki::Contrib::FastCGIEngineContrib	>=0.9.5	Optional.
File::MMagic	>0	Required.

Change History

09 Oct 2020	added {Locations} feature to be able to work around a bug in pdf.js not forwarding cookies in byte-range requests
11 Jun 2018	display reason why access to a file was denied
12 Dec 2017	fixed content encoding of a 403 warning; made file extension configurable that must be delivered in an "attachment" disposition mode
11 Dec 2017	return correct http error code when not authorized to access an attachment; optionally redirect to a login in case of an unauthorized access
25 Sep 2017	remove txt files from default dispositioning
30 Nov 2016	fixed encoding under Foswiki-2.x; detect 404 file not found properly; added support for rev parameter to access previous versions of an attachment; allow to specify content disposition (inline or attachment); allow to deliver and protect other content paths organized in parallel to the normal pub directory tree, for instance /pub/images for thumbnails
17 Jul 2015	added support for Foswiki-2.0
29 Aug 2014	improved decoding and untainting url components
28 May 2014	fixed file check on wrong location
15 Apr 2014	untaint attachment filenames
18 Mar 2014	return a proper 404 for invalid filename parameters
06 Nov 2013	fixed filenames with spaces not being found
01 Nov 2013	added support for if-modified-since http headers
22 May 2013	using mime-magic as a fallback in case file extensions don't unveil the mime-type
28 Mar 2013	implemented {AccessRules} to allow any kind of access control list for attachments