Squid Proxy Cache OSX

From Catznip
Jump to navigation Jump to search
Catznip
Are you reading the correct page for you operating system? Windows - OSX - Linux

The Second Life viewer can download a significant amount of data for every location you visit (easily hundreds of MB). In order to save on repeatedly fetching the same data and to speed up the rendering of places you visit frequently the viewer has a local cache that can be up to 10GB in size.

The viewer cache is less than ideal and while it tries very hard to be smart, doesn't get it right nearly as often as it should and can not be shared between different computers / viewers. How often have you teleported home after exploring only to find you have to wait while it downloads again.

A squid proxy cache acts as an intermediary between the viewer and Linden Lab. It fetches the data requested by the viewer and stores it; should it be requested again, it provides the copy it has already downloaded almost instantly.

This makes up for the viewer cache's tendency to forget all the things you would rather it didn't and allows you to run with a much smaller local cache. It can easily be shared by multiple computers in the same household.

The size of a squid proxy cache is only limited by the amount of disk space you can spare, and it can run quite happily on older / recycled hardware. The software is free open source.

This guide based on one first published on the Phoenix / Firestorm Wiki and requires some technical ability, It will work with any recent Second Life viewer.

Advantages

  • Caches textures, mesh, materials and web content (profiles, search images etc).
  • Faster more consistent loading when visiting locations repeatedly.
  • Smaller local cache (perhaps even a RAMDisk).
  • Reduced bandwidth (useful if you are on a metered service).
  • More responsive experience following teleports.
  • May enable HTTP texture usage in situations where it has caused problems previously.
  • Makes up for viewer cache shortcomings.
  • Easily shared by multiple computers, different viewers and other local Second Life users.
  • Recycle an older computer and give it a new lease of life!

Caveats

  • This solution is entirely unsupported by the Catznip viewer team, while we would be interested to know if you have problems getting it to work, we wont be able to help you.
  • The first time a texture / sculpt loads may feel a little slower, it will however load in at a higher resolution.
  • Some technical ability is required.

Second Life Squid Proxy for OSX

Use the terminal for all the instructions in this guide. You will need administrator access and your password. This was tested on a Mid 2007 Mac Mini running Mac OS X 10.7.5 "Lion".

MacPorts

First step is to install MacPorts.

If you already have MacPorts installed, now is a good time to make sure it's all up to date.

sudo port -d selfupdate
sudo port upgrade outdated

Install Squid 2.7

Newer versions of squid wont work, search MacPorts and see what versions it has:

sudo port search squid

You will see output similar to this.

squid @2.7.STABLE9_3 (net)
    advanced proxy caching server for http, https, ftp, gopher

squid3 @3.4.2 (net)
    advanced proxy caching server for http, https, ftp, gopher

squid3-devel @3.2.0.4_1 (net)
    this port is only a stub and has been made obsolete by squid3

volta @0.2.0 (www, net)
    a high performance Squid compatible URI redirector

Found 4 ports.

In our case we want the first one which is version 2.7, this may take a few moments.

sudo port install squid

Create store_url_rewrite

We need to create a little script to handle the fetching.

sudo nano /opt/local/etc/squid/store_url_rewrite

Copy and paste the following :

#!/usr/bin/perl
 
$| = 1;
 
while (<>) {
        chomp;
        if (m/sim(.*?)\.agni\.lindenlab\.com(.*?)\/cap\/(.*?)\/\?texture_id=(.*)/) {
                print "http://agni.lindenlab.com/texture/" . $4 . "\n";
        } else {
                print $_ . "\n";
        }
}

Then press CTRL+X to exit nano and save the changes.

When back at the console, make the file executable with :

sudo chmod +x /opt/local/etc/squid/store_url_rewrite

Editing squid.conf

This is a bit of a pain as the config file is very long.

In nano you can press CTRL+_ to go to a specific line number. Line numbers quoted are correct at time of writing.

sudo nano /opt/local/etc/squid/squid.conf

Go to line 653 and comment out the following two entries.

http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports

So it looks like

#http_access deny !Safe_ports
#http_access deny CONNECT !SSL_ports

Scroll down a few lines and to where it says

http_access allow localnet

and add the following line

http_access allow localhost

Go to the refresh pattern section, roughly line 2718 and add "refresh_pattern /cap/ 259200 20% 302400" so it looks like

#Suggested default:
refresh_pattern /cap/ 259200 20% 302400
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern .               0       20%     4320

Almost done, now go to the very end of the file (press page down a lot) and paste the following

# 32 GB cache size, can make larger later.
cache_dir ufs /opt/local/var/squid/cache/ 32000 16 256

# disable range requests due to LL fail. No Last Modify header.
range_offset_limit -1

# this is a local cache, prevent possible error
visible_hostname MyBigCache

# this is the magic, convert the URL into something more static.
storeurl_rewrite_program /opt/local/etc/squid/store_url_rewrite

storeurl_rewrite_children 32

# what domains to apply the store url rewrite too.
acl store_rewrite_list dstdomain .agni.lindenlab.com
storeurl_access allow store_rewrite_list
storeurl_access deny all

This creates a 32GB cache with the files stored in /opt/local/var/squid/cache/, you can make the cache as large as you like.

Complete file can be seen on Pastebin.

Start Squid

To start Squid and have it run when the computer starts

sudo port load squid

Configure the viewer & Final Steps

The following instructions are Catznip specific, for other viewers please refer to their documentation. These steps are required. Once completed relog.

Proxy Settings

In Catznip, open the preferences (CTRL+P), select the 'Setup' tab and press the 'Adjust proxy settings' button. Enter the network address of the machine now running squid, in this case it's named 'MiniMac', If you are running the proxy on the same machine as you use for secondlife it will be 'localhost'. Make sure the port is '3128', and most importantly then at the bottom set 'Other HTTP traffic..' to use the proxy.

Catznip proxy cache.png

Setting the Viewer Cache

You can now reduce the viewer cache. Preferences, 'Advanced tab, move the slider to the desired amount. We do not recommend you set this smaller than 1GB. While here, press the clear cache button.

Catznip cache prefs.png

Enabling HTTP Textures

Catznip ships with HTTP textures disabled by default due to some long standing issues with some domestic routers.

Enable the 'Developer menu' by pressing CTRL+ALT+Q or via preferences.

From the top menu check Develop > HTTP Texures.

Menu develop http textures.png

Relog the viewer

Go on .. it wont take a moment :)