Qbix-Beginners

When we try to call scrape() on a new URL such as microsoft.com/word, it just hangs sometimes.

Try to debug it, identify the root cause and update this issue. Then fix it.

It may be that the external sites are blocking our servers. In this case, let's implement Websites_Scrape class, with the Adapter Pattern. So we can use Websites_Scrape_ScraperAPI for instance, as an adapter that will use the Scraper API. The credentials should be stored under "Websites"/"scrape"/"api"/... in local/app.json config. (Use https://www.scraperapi.com/)

Example¶

https://new.freetalklive.com/action.php/Websites/scrape?Q.ajax=json&Q.slotNames=result&Q.method=POST&Q.nonce=e781bbfc7f4939189ae68e5a3be2272cf595c4a21ce46bc302326990df7142d3

Query String Parameters
Q.ajax: json
Q.slotNames: result
Q.method: POST
Q.nonce: e781bbfc7f4939189ae68e5a3be2272cf595c4a21ce46bc302326990df7142d3

Form Data:
url:  microsoft.com/word

Stack trace:

Operation timed out after 30859 milliseconds with 0 out of -1 bytes received

in /live/Q/platform/classes/Q/Utils.php (1164)
#0 /live/Q/platform/classes/Q/Utils.php(964): Q_Utils::request()
#1 /live/Q/platform/plugins/Websites/classes/Websites/Webpage.php(129): Q_Utils::get()
#2 /live/Q/platform/plugins/Websites/handlers/Websites/scrape/post.php(23): Websites_Webpage::scrape()
#3 /live/Q/platform/classes/Q.php(1170): Websites_scrape_post()
#4 /live/Q/platform/classes/Q.php(1050): Q::handle()
#5 /live/Q/platform/handlers/Q/post.php(30): Q::event()
#6 /live/Q/platform/classes/Q.php(1170): Q_post()
#7 /live/Q/platform/classes/Q.php(1050): Q::handle()
#8 /live/Q/platform/classes/Q/Dispatcher.php(343): Q::event()
#9 /live/Q/platform/classes/Q/ActionController.php(69): Q_Dispatcher::dispatch()
#10 /live/Q/platform/classes/Q/WebController.php(33): Q_ActionController::execute()
#11 /live/FTL/web/index.php(12): Q_WebControlle