3.4 Where did you come from ? - The Science of Referring Urls by Silversandstorm This article is the result of a discussion at the deny.de Basic Knowledge Millenium Edition forum. The initial question was raised by Lightson, a question regarding "M-Spoofer" which ignited a 30+ post thread. Basically in this article, I'll cover what a Referer is, where its used, why its insecure, and hence why sites which use it as a means of security have gaping huge holes in them. A few basic concepts. ==================== This knowledge is essential, your foundation. Referer urls are essentially a core feature of HTTP - Hyper Text Transfer Protocol. What is HTTP ? Well, its a protocol, which is more or less a fancy way of saying that its a format, or a set of rules which are followed while transferring data over WWW - the World Wide Web. These rules, if not followed, will result in HTTP errors. HTTP data transfer is carried out in Request + Response pairs. The person who wants the information (called the client) requests the data, and the person / site which is providing the information (also called the server) sends a response, along with that information, or an error code which indicates why it is unable to provide that information. Each request has a part called the http HEADER. This header consist of the actual request for the information followed by a series of HEADER FIELDS. A header field is merely a line in this format - Field: Value. For example, Host: www.google.com. For example GET http://www.microsoft.com/ HTTP/1.0 Accept: image/jpeg, image/pjpeg, application/vnd.ms-powerpoint, application/vnd.ms-excel, application/msword, */* Accept-Language: en-us Accept-Encoding: gzip, deflate User-Agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows 98) Host: www.microsoft.com Proxy-Connection: Keep-Alive Header fields are used to provide data to the server about the kind of application requesting the data, what data it can handle (for example, Windows media player cant handle text files) and additional parameters about the request. On to the hunt ! ================ One particular header field is the Referer field. Basically it looks like this... Referer: http://www.deny.de/programs.htm Whenever you surf to any site in your browser (be it Internet Explorer, Netscape, Opera, NeoPlanet, Hotjava, Mozilla or CubicEye), the browser sets the Referer: field in the request to the last url you visited. So if you first surf to http://www.google.com, and then click on the "Add your ad to google today" link, the referer for the new page you are visiting is http://www.google.com. This is done automatically, and invisibly. The only case in which you *dont* have a referer url is when you have just opened your browser OR if you enter a url / paste a url into the url box of your browser. When you click on a button / link, a referer url is ALWAYS sent. Q: Why use a referer ? A: Well, according to the HTTP protocol.... "The Referer header in a request provides the server with the URL of the resource from which the request-URL was obtained. This gives the server information about the user's previous page-view." So this is basically used for stats, to analyze the path a user takes through the site, maybe to ensure that people are not linking directly to certain parts of your site. It can also be used to ensure that a user visits a certain part of the site only from a link from a certain page or by redirection from a certain page. The H0le ======== Great, now we understand why a referer is used. But why is a referer such a trivial issue in most peoples' minds ? Well its very simple. A browser generates a Request using a request-URL and HTTP headers. Why cant your own program do the same, and merely change the HTTP headers to whatever values it wants ? Of course it can ! So if you want to make a site believe you came from page http://www.google.com/ when you didnt really come from there, all you have to do is write a program that creates the Request and enters http://www.google.com in the Referer field. Like this GET http://www.microsoft.com/directx HTTP/1.0 Accept: */* Referer: http://www.google.com/ Accept-Language: en-us Accept-Encoding: gzip, deflate User-Agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows 98) Host: www.microsoft.com See the Referer here ? :) No link to http://www.microsoft.com/directx from Googles main page, is there ? :) Referer urls used for security ============================== Some sites unfortunately use Referer urls for security. They seem to be unaware of the fact that with a little effort, a user could specify the Referer url of his/her choice and enter restricted areas of the site which allow requests only with a specific Referer url. Some sites, NOT ALL sites work like this. You enter your login and password on a form (text boxes) on the page, and click a button to submit the login information. The data is sent to a cgi script on the server, which processes the information. If the data is correct, it redirects you to a *secret* url on the server. For example, http://www.somesite.com/login.cgi =======> http://www.somesite.com/confidentialdata/ Now, http://www.somesites.com/confidentialdata/ -> When this url is requested, all the site does is check if the Referer field contains the cgi script url. Why ? Because ONLY THE CGI SCRIPT IS ALLOWED TO REFER YOU TO THE SECRET URL, which it does only if your login and password are accepted. Getting past referer based security =================================== Very simple. All you need to do is find the correct Referer url, and the *secret* url, and voila ! You use your own custom program with Internet Explorer or whatever browser you use to modify the Referer field to to the cgi script url, and surf to the *secret* url. Bingo ! The secret url sees the correct Referer url and welcomes you in :). The most important and often toughest thing to do is find the correct Referer and secret urls. The correct Referer isnt that hard to find, its almost always the login cgi script. So all you need to do is click on login in the login form after entering ANYTHING in the username and password fields, and note the url which is loaded in the browser. Thats your cgi script ! To find the secret url, the easiest method to use is login once with a valid username and password if you have one and note down the final url you reach after logging in. However, in many cases, you might not have a working login and password. What do you do then ? Id suggest you use Intellitamper, from http://www.multimania.com/intellitamper. Download it and scan the site for common directory names, like /secret/ , /content/ etc. Note: Use the referer faking method which Ill outline below while scanning, or the site will reject your request, and Intellitamper will think the directory isnt present. Putting all this theory into practise...actually doing it ========================================================= There are programs already out there which allow you to modify requests sent by your browser to allow you to send the Referer url of your choice. One of them is A4Proxy (http://www.inetprivacy.com). Another is RefCheat from Ksoze (I wonder where that guy is), http://www.deny.de/kz. A third is Mspoofer. Id recommend A4proxy, its the easiest. All these programs do is listen on a particular port. Now you have to set up your browser to use localhost (your own computer) and that particular port as your proxy server. The request -----> Localhost:port (modified) -------> Proxy or Site directly. If you use A4proxy, you can also use a public proxy. Otherwise, the data is sent directly to the site. In A4proxy how do you configure Referer faking ? Goto A4proxy, in the options look for a section titled Browser Options. In the field titled Variable, enter Referer. In the field titled Replace With, enter the full referer url you wish to use. Now click on Add and your ready to go ! Conclusion ========== Referer spoofing is very useful when sites use such a security system. However it has its limitations and Referer based security must be recognized as a lame method of securing your site. Signed SilverSandStorm