I hate pretty much every website related to paying bills, and it’s not only because they make use of irrelevant stock photos:

“Would it be possible for you to tell me if there is an online bill pay there, and if so sir, may I converse with her briefly?”

Being able to pay bills online is a huge convenience, and once you get passwords into a system like 1Password, it makes avoiding missed bills and phone calls like this one much easier. I also use numerous reminders from my GMail calendar so that I check bills in the right time frame for those bills which refuse to ever land on the same day of the month. I also do not give service providers access to my bank account to avoid having to fight them to get my money back when they overcharge me. Instead, I use my bank’s online pay system to send them checks/wire money/whatever they do based on the recipient. That still leaves a large chunk of time spent on manual work visiting each of the websites that I can see my balance.

At work, we have used Geb to do some automated browser testing. I really like its mix of Groovy syntax and jQuery style selectors for DOM manipulation. One afternoon, I decided to take what I’d learned from the brief exposure to Geb I had and tried throwing it at a few of these websites.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63

@Grapes([
    @Grab("org.codehaus.geb:geb-core:0.7.0"),
    @Grab("org.seleniumhq.selenium:selenium-firefox-driver:2.21.0"),
  @Grab("org.seleniumhq.selenium:selenium-chrome-driver:2.21.0"),
    @Grab("org.seleniumhq.selenium:selenium-support:2.21.0")
])

import geb.Browser
import geb.Configuration
import org.openqa.selenium.firefox.FirefoxDriver
import org.openqa.selenium.chrome.ChromeDriver
import org.openqa.selenium.Keys

gebConfig = """
/*
  This is the Geb configuration file.

  See: http://www.gebish.org/manual/current/configuration.html
*/

import org.openqa.selenium.firefox.FirefoxDriver
import org.openqa.selenium.chrome.ChromeDriver

reportsDir = "page-reports"


driver = { driver = new FirefoxDriver() }

waiting {
    timeout = 20
    retryInterval = 0.5
}
"""

rawGebConfig = new ConfigSlurper().parse(gebConfig)
configuration = new Configuration(rawGebConfig)

println "Starting bill page scrapes at ${new Date()}"

// ...

void check_bill_website_foo() {
  Browser.drive(configuration) {
      go "https://..."

      // Make sure we actually got to the page
      assert title == "Pay Your Foo Bills Online!"

      $("#login").userid = "someone@somewhere.com"
      $("#login") << Keys.TAB // Some sites have REALLY shitty javascript form validation
      $("#login").password = "my-long-1password-string"
      def loginLink = $("input[title=Login]")
      // Click the button, because submitting the form breaks aforementioned JS
      loginLink.click()
      report "foo_bill"
  }
}

// ...

println "Bill page scrapes finished at ${new Date()}"

Browser.drive(configuration){}.quit()

I have several of these methods, each one meticulously designed to fit the terrible ID-less DOM that most of these sites employ. Naturally, many of these sites like to redesign, making a script here or there fail, but from month to month this happens less than 10% of the time. The report call at the end of the method creates a PNG snapshot of the page at the time of execution and also captures the raw HTML of the current page. I have a cron job that runs the entire script nightly and the script only notifies me if any of the assertions fail, but I can check the directory that the scripts output to and see if there are any non-zero balances.

There are numerous improvements I would like to make still:

• Hook into my 1Password DB to query the site password instead of hardcoding it.
• Run some sort of diff on the files to detect an updated balance.
• E-Mail or some other notification rather than just a log file in a directory.
• Have the script pay the bills for me! ☺