StoreID – Support Caching Youtube dan Facebook HTTPS


Ini catatan pribadi, isinya storeid untuk squid-3.x Support caching youtube (thanks Ces Pun), Facebook HTTPS, serta beberapa web lainnya (playstore, reverbnation, dan sebagainya). Untuk mengaktifkan log storeid, silakan set $debug=1; pada storeid, dan buat file “/tmp/storeid.log” yang sudah di chown ke user proxy/squid.

Untuk mengecek log, silakan “tail -f /tmp/storeid.log”.

Note : untuk konfigurasi membutuhkan squid-3.4.7 yang sudah di patch untuk bug # 3937 (storeid loop).

Konfigurasi

# storeid *test*
acl urlrewrite dstdomain .fbcdn.net .akamaihd.net
acl speedtest url_regex -i speedtest\/.*\.(jpg|txt)\?.*
acl reverbnation url_regex -i reverbnation.*audio_player.*ec_stream_song.*$
acl utmgif url_regex -i utm.gif.*
acl playstoreandroid url_regex -i c.android.clients.google.com.market.GetBinary.GetBinary.*
acl idyoutube url_regex -i youtube.*(ptracking|stream_204|player_204).*(v\=|docid\=|video_id\=).*$
acl videoyoutube url_regex -i (youtube|googlevideo).*videoplayback\?
acl videoyoutube url_regex -i (youtube|googlevideo).*videoplayback\?
acl loop_302 http_status 302
acl getmethod method GET
 
store_id_program /usr/bin/perl /etc/squid/storeid
store_id_children 20 startup=0 idle=1 concurrency=5
store_id_access deny !getmethod
store_id_access allow urlrewrite
store_id_access allow speedtest
store_id_access allow reverbnation
store_id_access allow utmgif
store_id_access allow playstoreandroid
store_id_access allow idyoutube
store_id_access allow videoyoutube
store_id_access deny all

store_miss deny videoyoutube loop_302
send_hit deny videoyoutube loop_302

refresh_pattern (akamaihd|fbcdn)\.net 14400 99% 518400  ignore-no-store ignore-private ignore-reload ignore-must-revalidate store-stale
refresh_pattern -i squid\.internal 14400 99% 518400  ignore-no-store ignore-private ignore-reload ignore-must-revalidate store-stale
refresh_pattern \.(jpg|png|gif|css|ico)($|\?) 14400 99% 518400  ignore-no-store ignore-private reload-into-ims ignore-must-revalidate store-stale
refresh_pattern . 0 99% 518400  ignore-no-store ignore-private reload-into-ims store-stale

Isi storeid.pl

#!/usr/bin/perl
#
# storeid.pl with debug opt - based on storeurl.pl
# @ http://www2.fh-lausitz.de/launic/comp/misc/squid/projekt_youtube/
#
# mods by cespun and fajar @ ipfire.id
#  
 
use IO::File;
$|=1;
STDOUT->autoflush(1);
$debug=0;			## recommended:0
$bypassallrules=0;		## recommended:0
$sucks="";			## unused
$sucks="sucks" if ($debug>=1);
$timenow="";
$printtimenow=1;  		## print timenow: 0|1
my $logfile = '/tmp/storeid.log';

open my $logfh, '>>', $logfile
    or die "Couldn't open $logfile for appending: $!\n" if $debug;
$logfh->autoflush(1) if $debug;

while (<>) {
$timenow=time()." " if ($printtimenow);
print $logfh "$timenow"."in : $_" if ($debug>=1);
chop; 
my $myURL = $_;
@X = split(" ",$myURL);
$a = $X[0]; ## channel id
$b = $X[1]; ## url
$c = $X[2]; ## ip address
$u = $b; ## url

if ($bypassallrules){
 $out="$u"; ## map 1:1

} elsif ($u=~ m/http.*\.(fbcdn|akamaihd)\.net\/h(profile|photos).*[\d\w].*\/([\w]\d+x\d+\/.*\.[\d\w]{3}).*/) {
	$out="OK store-id=http://fbcdn.net.squid.internal/" . $2 . "/" . $3 ;

} elsif ($u=~ m/^http(.*)static(.*)(akamaihd|fbcdn).net\/rsrc.php\/(.*\/.*\/(.*).(js|css|png|gif))(\?(.*)|$)/) {
	$out="OK store-id=http://fbcdn.net.squid.internal/static/" . $5 . "." . $6 ;

} elsif ($u=~ m/^https?\:\/\/.*utm.gif.*/) {
	$out="OK store-id=http://google-analytics.squid.internal/__utm.gif";
 
} elsif ($u=~ m/^https?\:\/\/.*\/speedtest\/(.*\.(jpg|txt)).*/) {
	$out="OK store-id=http://speedtest.squid.internal/" . $1;
 
} elsif ($u=~ m/^https?\:\/\/.*\/(.*\..*(mp4|3gp|flv))\?.*/) {
	$out="OK store-id=http://video-file.squid.internal/" . $1;

} elsif ($u=~ m/^https?\:\/\/c2lo\.reverbnation\.com\/audio_player\/ec_stream_song\/(.*)\?.*/) {
	$out="OK store-id=http://reverbnation.squid.internal/" . $1;
 
} elsif ($u=~ m/^https?\:\/\/.*\.c\.android\.clients\.google\.com\/market\/GetBinary\/GetBinary\/(.*\/.*)\?.*/) {
	$out="OK store-id=http://playstore-android.squid.internal/" . $1;
 
} elsif ($u=~ m/^https?\:\/\/.*youtube.*ptracking.*/){
	@video_id = m/[&?]video_id\=([^\&\s]*)/;
	@cpn = m/[&?]cpn\=([^\&\s]*)/;
	unless (-e "/tmp/@cpn"){
	open FILE, ">/tmp/@cpn";
	print FILE "@video_id";
	close FILE;
	}
	$out="ERR";
 
} elsif ($u=~ m/^https?\:\/\/.*youtube.*stream_204.*/){
	@docid = m/[&?]docid\=([^\&\s]*)/;
	@cpn = m/[&?]cpn\=([^\&\s]*)/;
	unless (-e "/tmp/@cpn"){
	open FILE, ">/tmp/@cpn";
	print FILE "@docid";
	close FILE;
	}
	$out="ERR";
 
} elsif ($u=~ m/^https?\:\/\/.*youtube.*player_204.*/){
	@v = m/[&?]v\=([^\&\s]*)/;
	@cpn = m/[&?]cpn\=([^\&\s]*)/;
	unless (-e "/tmp/@cpn"){
	open FILE, ">/tmp/@cpn";
	print FILE "@v";
	close FILE;
	}
	$out="ERR";
 
} elsif ($u=~ m/^https?\:\/\/.*(youtube|googlevideo).*videoplayback.*/){
	@itag = m/[&?](itag\=[0-9]*)/;
	@range = m/[&?](range\=[^\&\s]*)/;
	@cpn = m/[&?]cpn\=([^\&\s]*)/;
	@mime = m/[&?](mime\=[^\&\s]*)/;
	@id = m/[&?]id\=([^\&\s]*)/;
 
	if (defined(@cpn[0])){
		if (-e "/tmp/@cpn"){
		open FILE, "/tmp/@cpn";
		@id = <FILE>;
		close FILE;}
	}
	$out="OK store-id=http://video-srv.squid.internal/id=@id@mime@range";

} else {
	$out="ERR";
}
	print $logfh "$timenow"."out: $a $out\n" if ($debug>=1);
	print "$a $out\n";
}
close $logfh if ($debug);

Tested untuk cache Facebook HTTPS dan youtube. Silakan di copas kalo pengen :D


Related Posts

28 thoughts on “StoreID – Support Caching Youtube dan Facebook HTTPS

  1. Znet

    mas, store id + store pl ny itu copy semuanya ke squid.conf begitu kan ?

    Reply
    1. fazar Post author

      storeid.pl di buat terpisah mas, jangan di gabungin ke squid.conf. ^^

      Reply
  2. ibrahim

    Om nanya, pengen tau Aja bukannya YouTube/google ITU udah pake https, squidnya bisa nangkep https ? Apa harus diintercept dulu httpsnya dulu baru bisa caching?

    Thx om.

    Reply
    1. fazar Post author

      yup.. harus di intersep https nya… btw, storeid yg ini mungkin outdated. :)

      Reply
  3. Hen

    master,
    kirimin email ke saya storeid dan squid.conf terbaru supaya bisa cache youtube https dong, pls

    Thanks

    Reply
  4. shr

    bagian ini untuk pfsense error, kiranya diganti apa gan ?
    store_miss deny videoyoutube loop_302
    send_hit deny videoyoutube loop_302

    Reply

Tinggalkan Balasan

Alamat email Anda tidak akan dipublikasikan. Ruas yang wajib ditandai *