network

StoreID – Support Caching Youtube dan Facebook HTTPS

Ditulis oleh Beri komentar Lihat komentar

Ini catatan pribadi, isinya storeid untuk squid-3.x Support caching youtube (thanks Ces Pun), Facebook HTTPS, serta beberapa web lainnya (playstore, reverbnation, dan sebagainya). Untuk mengaktifkan log storeid, silakan set $debug=1; pada storeid, dan buat file “/tmp/storeid.log” yang sudah di chown ke user proxy/squid.

Untuk mengecek log, silakan “tail -f /tmp/storeid.log”.

Note : untuk konfigurasi membutuhkan squid-3.4.7 yang sudah di patch untuk bug # 3937 (storeid loop).

Konfigurasi

# storeid *test*
acl urlrewrite dstdomain .fbcdn.net .akamaihd.net
acl speedtest url_regex -i speedtest\/.*\.(jpg|txt)\?.*
acl reverbnation url_regex -i reverbnation.*audio_player.*ec_stream_song.*$
acl utmgif url_regex -i utm.gif.*
acl playstoreandroid url_regex -i c.android.clients.google.com.market.GetBinary.GetBinary.*
acl idyoutube url_regex -i youtube.*(ptracking|stream_204|player_204).*(v\=|docid\=|video_id\=).*$
acl videoyoutube url_regex -i (youtube|googlevideo).*videoplayback\?
acl videoyoutube url_regex -i (youtube|googlevideo).*videoplayback\?
acl loop_302 http_status 302
acl getmethod method GET
 
store_id_program /usr/bin/perl /etc/squid/storeid
store_id_children 20 startup=0 idle=1 concurrency=5
store_id_access deny !getmethod
store_id_access allow urlrewrite
store_id_access allow speedtest
store_id_access allow reverbnation
store_id_access allow utmgif
store_id_access allow playstoreandroid
store_id_access allow idyoutube
store_id_access allow videoyoutube
store_id_access deny all

store_miss deny videoyoutube loop_302
send_hit deny videoyoutube loop_302

refresh_pattern (akamaihd|fbcdn)\.net 14400 99% 518400  ignore-no-store ignore-private ignore-reload ignore-must-revalidate store-stale
refresh_pattern -i squid\.internal 14400 99% 518400  ignore-no-store ignore-private ignore-reload ignore-must-revalidate store-stale
refresh_pattern \.(jpg|png|gif|css|ico)($|\?) 14400 99% 518400  ignore-no-store ignore-private reload-into-ims ignore-must-revalidate store-stale
refresh_pattern . 0 99% 518400  ignore-no-store ignore-private reload-into-ims store-stale

Isi storeid.pl

#!/usr/bin/perl
#
# storeid.pl with debug opt - based on storeurl.pl
# @ http://www2.fh-lausitz.de/launic/comp/misc/squid/projekt_youtube/
#
# mods by cespun and fajar @ ipfire.id
#  
 
use IO::File;
$|=1;
STDOUT->autoflush(1);
$debug=0;			## recommended:0
$bypassallrules=0;		## recommended:0
$sucks="";			## unused
$sucks="sucks" if ($debug>=1);
$timenow="";
$printtimenow=1;  		## print timenow: 0|1
my $logfile = '/tmp/storeid.log';

open my $logfh, '>>', $logfile
    or die "Couldn't open $logfile for appending: $!\n" if $debug;
$logfh->autoflush(1) if $debug;

while (<>) {
$timenow=time()." " if ($printtimenow);
print $logfh "$timenow"."in : $_" if ($debug>=1);
chop; 
my $myURL = $_;
@X = split(" ",$myURL);
$a = $X[0]; ## channel id
$b = $X[1]; ## url
$c = $X[2]; ## ip address
$u = $b; ## url

if ($bypassallrules){
 $out="$u"; ## map 1:1

} elsif ($u=~ m/http.*\.(fbcdn|akamaihd)\.net\/h(profile|photos).*[\d\w].*\/([\w]\d+x\d+\/.*\.[\d\w]{3}).*/) {
	$out="OK store-id=http://fbcdn.net.squid.internal/" . $2 . "/" . $3 ;

} elsif ($u=~ m/^http(.*)static(.*)(akamaihd|fbcdn).net\/rsrc.php\/(.*\/.*\/(.*).(js|css|png|gif))(\?(.*)|$)/) {
	$out="OK store-id=http://fbcdn.net.squid.internal/static/" . $5 . "." . $6 ;

} elsif ($u=~ m/^https?\:\/\/.*utm.gif.*/) {
	$out="OK store-id=http://google-analytics.squid.internal/__utm.gif";
 
} elsif ($u=~ m/^https?\:\/\/.*\/speedtest\/(.*\.(jpg|txt)).*/) {
	$out="OK store-id=http://speedtest.squid.internal/" . $1;
 
} elsif ($u=~ m/^https?\:\/\/.*\/(.*\..*(mp4|3gp|flv))\?.*/) {
	$out="OK store-id=http://video-file.squid.internal/" . $1;

} elsif ($u=~ m/^https?\:\/\/c2lo\.reverbnation\.com\/audio_player\/ec_stream_song\/(.*)\?.*/) {
	$out="OK store-id=http://reverbnation.squid.internal/" . $1;
 
} elsif ($u=~ m/^https?\:\/\/.*\.c\.android\.clients\.google\.com\/market\/GetBinary\/GetBinary\/(.*\/.*)\?.*/) {
	$out="OK store-id=http://playstore-android.squid.internal/" . $1;
 
} elsif ($u=~ m/^https?\:\/\/.*youtube.*ptracking.*/){
	@video_id = m/[&?]video_id\=([^\&\s]*)/;
	@cpn = m/[&?]cpn\=([^\&\s]*)/;
	unless (-e "/tmp/@cpn"){
	open FILE, ">/tmp/@cpn";
	print FILE "@video_id";
	close FILE;
	}
	$out="ERR";
 
} elsif ($u=~ m/^https?\:\/\/.*youtube.*stream_204.*/){
	@docid = m/[&?]docid\=([^\&\s]*)/;
	@cpn = m/[&?]cpn\=([^\&\s]*)/;
	unless (-e "/tmp/@cpn"){
	open FILE, ">/tmp/@cpn";
	print FILE "@docid";
	close FILE;
	}
	$out="ERR";
 
} elsif ($u=~ m/^https?\:\/\/.*youtube.*player_204.*/){
	@v = m/[&?]v\=([^\&\s]*)/;
	@cpn = m/[&?]cpn\=([^\&\s]*)/;
	unless (-e "/tmp/@cpn"){
	open FILE, ">/tmp/@cpn";
	print FILE "@v";
	close FILE;
	}
	$out="ERR";
 
} elsif ($u=~ m/^https?\:\/\/.*(youtube|googlevideo).*videoplayback.*/){
	@itag = m/[&?](itag\=[0-9]*)/;
	@range = m/[&?](range\=[^\&\s]*)/;
	@cpn = m/[&?]cpn\=([^\&\s]*)/;
	@mime = m/[&?](mime\=[^\&\s]*)/;
	@id = m/[&?]id\=([^\&\s]*)/;
 
	if (defined(@cpn[0])){
		if (-e "/tmp/@cpn"){
		open FILE, "/tmp/@cpn";
		@id = <FILE>;
		close FILE;}
	}
	$out="OK store-id=http://video-srv.squid.internal/id=@id@mime@range";

} else {
	$out="ERR";
}
	print $logfh "$timenow"."out: $a $out\n" if ($debug>=1);
	print "$a $out\n";
}
close $logfh if ($debug);

Tested untuk cache Facebook HTTPS dan youtube. Silakan di copas kalo pengen :D


Related Posts

Filed under

Networking

| Tags:

,

Leave a comment ?

28 Comments.

  1. mas, store id + store pl ny itu copy semuanya ke squid.conf begitu kan ?

  2. Om nanya, pengen tau Aja bukannya YouTube/google ITU udah pake https, squidnya bisa nangkep https ? Apa harus diintercept dulu httpsnya dulu baru bisa caching?

    Thx om.

  3. Thx om…

  4. master,
    kirimin email ke saya storeid dan squid.conf terbaru supaya bisa cache youtube https dong, pls

    Thanks

  5. ini bisa diterapkan di pfsense ?

  6. bagian ini untuk pfsense error, kiranya diganti apa gan ?
    store_miss deny videoyoutube loop_302
    send_hit deny videoyoutube loop_302

  7. Masih Manjur kah wat chache youtube / https ??

  8. amazing! thanks for your work!

Leave a Comment