Bug #2829
Updated by Gregory Magarshak about 2 years ago
When we try to call scrape() on a new URL such as microsoft.com/word, it just hangs sometimes.
Try to debug it, identify the root cause and update this issue. Then fix it.
It may be that the external sites are blocking our servers. In this case, let's implement @Websites_Scrape@ class, with the Adapter Pattern. So we can use @Websites_Scrape_ScraperAPI@ for instance, as an adapter that will use the Scraper API. The credentials should be stored under "Websites"/"scrape"/"api"/... in @local/app.json@ config. (Use https://www.scraperapi.com/)
h2. Example
https://new.freetalklive.com/action.php/Websites/scrape?Q.ajax=json&Q.slotNames=result&Q.method=POST&Q.nonce=e781bbfc7f4939189ae68e5a3be2272cf595c4a21ce46bc302326990df7142d3
<pre>
Query String Parameters
Q.ajax: json
Q.slotNames: result
Q.method: POST
Q.nonce: e781bbfc7f4939189ae68e5a3be2272cf595c4a21ce46bc302326990df7142d3
Form Data:
url: microsoft.com/word
</pre>
Stack trace:
<pre>
Operation timed out after 30859 milliseconds with 0 out of -1 bytes received
in /live/Q/platform/classes/Q/Utils.php (1164)
#0 /live/Q/platform/classes/Q/Utils.php(964): Q_Utils::request()
#1 /live/Q/platform/plugins/Websites/classes/Websites/Webpage.php(129): Q_Utils::get()
#2 /live/Q/platform/plugins/Websites/handlers/Websites/scrape/post.php(23): Websites_Webpage::scrape()
#3 /live/Q/platform/classes/Q.php(1170): Websites_scrape_post()
#4 /live/Q/platform/classes/Q.php(1050): Q::handle()
#5 /live/Q/platform/handlers/Q/post.php(30): Q::event()
#6 /live/Q/platform/classes/Q.php(1170): Q_post()
#7 /live/Q/platform/classes/Q.php(1050): Q::handle()
#8 /live/Q/platform/classes/Q/Dispatcher.php(343): Q::event()
#9 /live/Q/platform/classes/Q/ActionController.php(69): Q_Dispatcher::dispatch()
#10 /live/Q/platform/classes/Q/WebController.php(33): Q_ActionController::execute()
#11 /live/FTL/web/index.php(12): Q_WebControlle
</pre>