From: Antonio Ospite <ao2@ao2.it>
Date: Sat, 27 Jul 2019 20:06:15 +0000 (+0200)
Subject: src/Tweeper.php: enable cookie handling to fix scraping twitter.com
X-Git-Tag: v1.4.1~2
X-Git-Url: https://git.ao2.it/tweeper.git/commitdiff_plain/2efcaf768f68d35872c0d06136279e673128c46f?ds=inline

src/Tweeper.php: enable cookie handling to fix scraping twitter.com

When the user agent used by a client matches an actual browser,
twitter.com enables content-security-policy and redirects the client on
the first request to make it reload the content.

After the redirection, the server assumes that the client sets cookies
appropriately, however cURL does not do that by default.

Enable cookie handling in cURL to fix scraping twitter.com.

NOTE: the CURLOPT_COOKIEFILE option is set to an empty string to enable
in-memory handling of the cookies, removing the need for a temporary
file on the filesystem, see:
https://www.php.net/manual/en/function.curl-setopt.php
---

diff --git a/src/Tweeper.php b/src/Tweeper.php
index 09bd7cc..7ecbf2f 100644
--- a/src/Tweeper.php
+++ b/src/Tweeper.php
@@ -121,6 +121,7 @@ class Tweeper {
       CURLOPT_CONNECTTIMEOUT => Tweeper::$maxConnectionTimeout,
       // Follow http redirects to get the real URL.
       CURLOPT_FOLLOWLOCATION => TRUE,
+      CURLOPT_COOKIEFILE => "",
       CURLOPT_RETURNTRANSFER => TRUE,
       CURLOPT_SSL_VERIFYHOST => FALSE,
       CURLOPT_SSL_VERIFYPEER => FALSE,