How to login to any website using Curl from the command line or shell script

tomo — Fri, 08 Jan 2016 10:04:11 +0000

January 8, 2016

There are times you need to scrape/crawl some field on a page but the page requires authentication (logging in). Unless the site is using Basic Auth, where you can have the username and password in the url like http://username:1234paSSwoRd@target.site/ then you'll need to curl with more sophistication. Besides curl, there are other web tools which you can use on the command line such as links/elinks (elinks is an enhanced version of links which also supports JavaScript to a very limited extent). Links and curl will not execute JavaScript though, so if that's necessary to get...

Selenium IDE vs Selenium Webdriver vs CasperJS

tomo — Tue, 24 Nov 2015 03:50:23 +0000

November 24, 2015

webcrawler scrape selenium casperjs phantomjs javascript chrome firefox

Or more specifically: Selenium IDE (Firefox plugin) vs Selenium Webdriver (Python and other languages) vs CasperJS (and PhantomJS or SlimerJS)

Selenium allows you, a programmer or non-programmer, to control a web browser and make it do things that you would otherwise do manually. With that ability, you can test your website over and over (and automatically from cron), similate users, or visit any number of web pages and read data (web scraping) on them and save to a file for processing.

If you go to the Selenium website you will...