This post was most recently updated on December 11th, 2018<\/p>
Downloading content at a specific URL is common practice on the internet, especially due to increased usage of web services and APIs offered by Amazon, Alexa, Digg, etc. PHP’s cURL library, which often comes with default shared hosting configurations, allows web developers to complete this task.<\/p>\n
\/* gets the data from a URL *\/function get_data($url){\t$ch = curl_init();\t$timeout = 5;\tcurl_setopt($ch,CURLOPT_URL,$url);\tcurl_setopt($ch,CURLOPT_RETURNTRANSFER,1);\tcurl_setopt($ch,CURLOPT_CONNECTTIMEOUT,$timeout);\t$data = curl_exec($ch);\tcurl_close($ch);\treturn $data;}<\/pre>\ncopy<\/a>\/* gets the data from a URL *\/<\/span>function<\/span> get_data<\/span>(<\/span>$url<\/span>)<\/span>{<\/span> $ch<\/span> = curl_init<\/span>(<\/span>)<\/span>; $timeout<\/span> = 5<\/span>; curl_setopt<\/span>(<\/span>$ch<\/span>,CURLOPT_URL<\/span>,$url<\/span>)<\/span>; curl_setopt<\/span>(<\/span>$ch<\/span>,CURLOPT_RETURNTRANSFER<\/span>,1<\/span>)<\/span>; curl_setopt<\/span>(<\/span>$ch<\/span>,CURLOPT_CONNECTTIMEOUT<\/span>,$timeout<\/span>)<\/span>; $data<\/span> = curl_exec<\/span>(<\/span>$ch<\/span>)<\/span>; curl_close<\/span>(<\/span>$ch<\/span>)<\/span>; return<\/span> $data<\/span>;}<\/span><\/pre>\nThe Usage<\/h2>\n$returned_content = get_data('http:\/\/davidwalsh.name');<\/pre>\ncopy<\/a>$returned_content<\/span> = get_data<\/span>(<\/span>'http:\/\/davidwalsh.name'<\/span>)<\/span>;<\/pre>\nAlternatively, you can use the file_get_contents<\/a> function remotely, but many hosts don’t allow this.<\/p>\n\u00a0<\/p>\n
Set the User Agent With PHP cURL<\/h1>\n\u00a0<\/div>\nA few months back, I shared with you how to download the contents of a URL<\/a> and execute a HTTP POST transmission<\/a> using PHP cURL. For security purposes, some hosts require that a common user agent be present in the POST. If an unacceptable user agent is given, the POST is ignored. Luckily, cURL allows us to “spoof” the server using any user agent we choose:<\/p>\ncurl_setopt($ch,CURLOPT_USERAGENT,'Mozilla\/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko\/20080311 Firefox\/2.0.0.13');<\/pre>\ncopy<\/a>curl_setopt<\/span>(<\/span>$ch<\/span>,CURLOPT_USERAGENT<\/span>,'Mozilla\/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko\/20080311 Firefox\/2.0.0.13'<\/span>)<\/span>;<\/pre>\nA little sneaky but it could get you out of a jam at some point!
\u00a0<\/p>\n