Can't get past two-step security with HtmlUnit - htmlunit

Question

I'm new to HtmlUnit (using version 2.30). Working in Eclipse on a Mac. I'm trying to create a stock data scraper by logging onto my Ameritrade account and manipulating the watch lists I've created there. First login form leads to the two-step security page where the challenge question is asked. I don't know why/how the site knows that it wants to challenge my userid/password in the first place. Because it looks like a new browser?
But anyway I fill out the form on the second page with the answer to the challenge question and submit. Instead of taking me to my account's home page, it once again takes me to the two-step security page with the same challenge question. Here is the relevant code:
final int sleepMinSeconds = 1;
final int sleepRandomSeconds = 4;
final long javascriptTimeout = 10000;
System.out.println("HtmlUnitTest");
String applicationName = "Mozilla";
String applicationVersion = "5.0 (Windows NT 6.3; WOW64; rv:56.0) Gecko/20100101 Firefox/56.0";
final String userAgent = applicationName + "/" + applicationVersion;
BrowserVersion browserVersion = new BrowserVersion.BrowserVersionBuilder(BrowserVersion.FIREFOX_52)
.setApplicationName(applicationName)
.setApplicationVersion(applicationVersion)
.setUserAgent(userAgent)
.build();
WebClient webClient = new WebClient(browserVersion);
java.util.logging.Logger.getLogger("com.gargoylesoftware.htmlunit").setLevel(java.util.logging.Level.ALL);
java.util.logging.Logger.getLogger("org.apache.commons.httpclient").setLevel(java.util.logging.Level.ALL);
webClient.setIncorrectnessListener(new com.gargoylesoftware.htmlunit.IncorrectnessListener() {
#Override public void notify(String arg0, Object arg1) {}
});
webClient.setJavaScriptErrorListener(new com.gargoylesoftware.htmlunit.javascript.JavaScriptErrorListener() {
#Override public void timeoutError(HtmlPage arg0, long arg1, long arg2) {}
#Override public void scriptException(final HtmlPage arg0, final com.gargoylesoftware.htmlunit.ScriptException arg1) {}
#Override public void malformedScriptURL(HtmlPage arg0, String arg1, java.net.MalformedURLException arg2) {}
#Override public void loadScriptError(HtmlPage arg0, java.net.URL arg1, Exception arg2) {}
});
webClient.setCssErrorHandler(new com.gargoylesoftware.htmlunit.SilentCssErrorHandler());
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setDoNotTrackEnabled(true);
webClient.getOptions().setActiveXNative(true);
webClient.getOptions().setRedirectEnabled(true);
webClient.getOptions().setPrintContentOnFailingStatusCode(true);
webClient.getCookieManager().setCookiesEnabled(true);
webClient.getOptions().setDownloadImages(true);
String loginURL = "https://www.tdameritrade.com/home.page";
System.out.println("Connecting to " + loginURL + " (" + webClient.getBrowserVersion() + ")");
System.out.print(" Waiting to avoid being detected as a robot...");
Thread.sleep((long)(Math.random()*sleepRandomSeconds) * 1000);
System.out.print(" Done waiting.\n");
HtmlPage page = webClient.getPage(loginURL);
System.out.println("title text: " + page.getTitleText());
System.out.print(" \nWaiting for Javascript to complete...");
webClient.waitForBackgroundJavaScript(javascriptTimeout);
System.out.println("\nOK");
System.out.print(" Waiting to avoid being detected as a robot...");
Thread.sleep((long)(sleepMinSeconds + Math.random()*sleepRandomSeconds) * 1000);
System.out.print(" Done waiting.\n");
System.out.println("Logging in...");
HtmlForm form = page.getFormByName("form-login");
HtmlTextInput useridField = form.getInputByName("tbUsername");
HtmlPasswordInput passwordField = form.getInputByName("tbPassword");
useridField.type("<userid>");
passwordField.type("<password>");
HtmlButton button = form.getButtonByName("");
System.out.println("button value: " + button.getValueAttribute());
// Did this to make sure I had right button, which was unnamed.
// Value is "Log in", so I proceed.
HtmlPage page2 = button.click();
System.out.print(" \nWaiting for Javascript to complete...");
webClient.waitForBackgroundJavaScript(javascriptTimeout);
System.out.println("\nOK");
System.out.print(" Waiting to avoid being detected as a robot...");
Thread.sleep((long)(sleepMinSeconds + Math.random()*sleepRandomSeconds) * 1000);
System.out.print(" Done waiting.\n");
HtmlElement element = page2.getHtmlElementById("loginBlock");
HtmlForm form2 = element.getEnclosingForm();
HtmlPasswordInput challengeField = form2.getInputByName("challengeAnswer");
if(page2.asXml().contains("boss")) {
System.out.println("boss question...");
challengeField.type("<answer to boss question>");
}
else if(page2.asXml().contains("street")) {
System.out.println("street question...");
challengeField.type("<answer to street question>");
}
else {
System.out.println("What?");
}
HtmlCheckBoxInput checkBox = form2.getInputByName("rememberDevice");
checkBox.setChecked(true);
HtmlInput button2 = form2.getInputByName("mAction");
System.out.println("button2 value: " + button2.getValueAttribute());
// value here is "submit" - so I proceed
HtmlPage page3 = button2.click();
System.out.print(" \nWaiting for Javascript to complete...");
webClient.waitForBackgroundJavaScript(javascriptTimeout);
System.out.println("\nOK");
webClient.close();
In other words, page2 and page3 are the same, i.e., the two-step security page. I expected page3 to be my account's home page. (I confirmed this by writing them both out as XML to separate files.) I would appreciate any help I can get on this! Thanks!

Answer 1

OK, lets start with some comments to your code.
Not sure what you like to archive by using a different browser setup instead of using the build in default ones. There is nothing wrong with this but be aware that changing the browser setup will NOT have any effect on the browser behavior (e.g. the supported js functionality).
Second: if you are hunting for bugs/problems i fear it is a bad idea to disable all the listeners. It might be worth to have this listener output...
Regarding all the options: Why not starting with the default setup, it is really close to real browsers.
Now some words about the login process:
It will be a great help to try to understand the login process of the real application. All this 'modern' web applications are doing a lot of strange things (async/javascript) to simulate a rich ui on the not that rich platform.Tools like Charles WebProxy are of real help to get an idea of the communication done behind the scenes.
One common problem with this HtmlPage page3 = button2.click(); API is that the click method returns the sync result of the click. If the button is one of this fancy Ajax buttons, this is usually the page of the button itself. You are already waiting for the Ajax stuff to finish but the page will not change if there is a ajax redirect to a new page. In this case you have to do something like this after the wait call.
// there is an ajax redirect that loads a new page into this window
page3 = (HtmlPage) page.getEnclosingWindow().getEnclosedPage();
Hope that helps a bit...