build a Java application to log into a website

@rubaFeb 24.2006

Hello All,

I’m trying to programmatically log into a web site.
In-other-words, I would like to build an application/applet/script (whatever) so that when it’s executed it uses my username & password to log into a web server and then allow me to perform searches and display the results (currently this is possible by loggin through a web browser).

I have experience with c++, Java, perl and a little javascript. I wouldn’t mind trying C# if it will get me to where I want to go.

I have tried the Java.net package (URL & sockets) with no success.

ray326 suggested I try java.net.HttpURLConnection and place a post here for help. I now have some java code running using HttpURLConnection but I still can’t login through my java application.

I’m also playing with PHP right now to see if I can get what I need.

Please understand I’m trying to not put any code on the server, at this point I’m trying to work through the “normal” webtop interface (not around it).

Thank you for any help or suggestions.

p.s. please feel free to move this thread if it’s in the wrong area.

to post a comment

Java

13 Comments(s) _↴

@Khalid_AliFeb 24.2006 — #well you are not providing enough info. What happens when you open a connection to the webpage and try to login?

See essentially java does not know whats on that page and have no idea if you want to login. What you will need to do is open the connection. read the page and submit to the page that actually logs in with required field with thier name and values so that server can performa login and when it returns the logged in page then you will have parse it again and...etc...etc

@BigDogFeb 24.2006 — #There are ALOT of ways to do what you are asking for. Your description is kind of vague though, so it is hard to give concrete examples.

From what you have said though, it really sounds like a web service would be the best choice. Do you have access to the website or is it something that is black box? I would look to see if the site offers some sort of SOAP interfaces for you to get your info from, or maybe they offer an XML page. Parsing HTML for info would be a nightmare, especially if the page changes frequently.

@rubaauthorFeb 25.2006 — #First, thanks for your time.

Currently there are 2 main layers to this system, the top layer is a proprietary commercial web interface (the black box) that accesses a SQL database underneath. I’m trying to add a third layer to the top for anonymous users. I could write code to directly access the database. But I want to go through the proprietary commercial web interface if possible.

What happens when you open a connection to the webpage and try to login?[/QUOTE]

If I’m using a web browser I type in the URL, then enter my username and password into the fields and press the submit button, it then logs me in (it installs a cookie so it knows that I’m logged in for this session) I can now perform searches for various documents and then open & read those documents. What I’m trying to do is provide a way for an anonymous user to get in and perform the same searches but instead of actually opening the documents, the anonymous user would only see a brief description of the document(s) and then would be required to contact the author of the document(s) for permission.

I chose Java because I have some background with it, but I’ll try any language if it will help me solve my problem.

I have several versions of my Java code:

1) this version uses a socket I can open a connection but then I get no response back from the server. I used the below code:

socket = new Socket("url.of.the.server", 80);// [COLOR=Navy]URL of the login page gives errors[/COLOR]

out = new PrintWriter(socket.getOutputStream(), true);

in = new BufferedReader(new InputStreamReader(socket.getInputStream()));

out.write("various statements");

BufferedReader stdIn = new BufferedReader(new nputStreamReader(System.in));

String userInput;

while ((userInput = in.readLine()) != null) {

System.out.println(userInput);

}

I think the reason I get no response back is because there is no code listening to my socket at the other end.

2) this version I use java URL:

URL u = new URL("URL of the login page");

URLConnection connection = u.openConnection();

connection.setDoOutput(true);

with this version I get the login page source code back from the server (which is good) But I don’t know how to go beyond this to actually send my username & password to server so it can login.

3) this version I use HttpURLConnection:

HttpURLConnection huc = (HttpURLConnection)u.openConnection();

huc.setDoOutput(true);

huc.setRequestMethod("POST");

with this version I can get response codes (e.g. 200 ok) and the login page source code as with version 2.

What you will need to do is open the connection. read the page and submit to the page that actually logs in with required field with thier name and values so that server can performa login and when it returns the logged in page then you will have parse it again and...etc...etc[/QUOTE]
Yes, the problem is that I am not sure how to submit my login information to the "black box" web software through my java program.

Parsing HTML for info would be a nightmare, especially if the page changes frequently.[/QUOTE]
The page will change depending on the search queries.

sorry for being so long winded....

@Khalid_AliFeb 25.2006 — #its not a good solution however to answer ur question your connection url don't have to be what a user sees as a login page, as far as the webserver is concern it requires and user name (whatever name u have for that field) and a password field.

so you will have to append these to the url

something line

"ur.server.com?userName=khalid&password=opensesame"

now in some cases depending upon the server they do not respond back unless the querying tool tells them its an http complinat browser, so to be safe alwasy set the USER-AGENT property for the connection to a valid user-agent string

@rubaauthorFeb 26.2006 — #
"ur.server.com?userName=khalid&password=opensesame"
[/QUOTE]
I've tried that. I've even tried several different variable names(password, pwd, passwd, etc). I've tried it in various ways with all three versions (from my previous post). What I get back is the source code for the login page plus some javascript. When I paste the exact same url string into a web browser I get the same thing back as my java program and I notice that my username is properly entered into the field but the password field is empty. If I hit enter I again I get a message: "The password applet is not fully loaded yet, please try again in a few seconds."

I've also looked at the packets between my machine and the server using Etherreal. I've compared the packets for logging in from a web browser versus my java program. In the packet from the java program I can see the password, but in the web browser the password is encrypted. Obviously, I thought about copying the packet from the web browser and sending that in my java program, but I've noticed the values for the password in the login packet changes each time I login through the web browser.

Is there some place that explains what communication occurs between a web browser and a server when logging into a website with a password (i.e. yahoo, hotmail, etc...)?

It seems my hitch is the password. Other than the method discussed above how can I enter the password into the password field of the form code and submit the document....in the java program, not the web browser (web browser is easy ?)? Or is there something I'm missing from the url method mentioned above.

@Khalid_AliFeb 26.2006 — #if you like, you can send me the login pages html and the url where u need to login..I'll see whats going on....and ofcourse if there is a temp user with password thats even better...let me know...

@stevooMar 06.2008 — #I have the same problem .... i am wondering if you could just point me out what i am doing so wrong.

All i want to do is login, get the data from the site and analyze them in the way that i want.

But login in is just a pain.

So login form is :

[CODE]<form name="login" action="login.php" method="post" target="_top"><input type="hidden" name="PHPSESSID" value="166c592c10598cfc0e8d2be5226cf33f" />
 E-mail: <input type="text"     size="27" maxlength="65" class="custom" name="email" title="Your E-Mail"   value="Your E-Mail"   onfocus="javascript:handleFocus(this);"><br><br>
 Password: <input type="password" size="27" maxlength="20" class="custom" name="pass"  title="Your Password" value="Your Password" onfocus="javascript:handleFocus(this);"><br><br>
 <input type="image" src="img/login2.jpg?PHPSESSID=166c592c10598cfc0e8d2be5226cf33f" value="Submit" name="submit" style="margin-right:60px;">
 <input type="hidden" name="submit" value="Submit">
 </form>[/CODE]

And i am trying the following

[CODE] public void tryLog() {
 try {
 // setup url connection. use POST to send forms data
 //url = new URL(HOUSE);
 
   urlConn = (HttpURLConnection) url.openConnection();
 
   urlConn.setRequestMethod("POST");
   urlConn.setDoInput(true);
   urlConn.setDoOutput(true);
   urlConn.setUseCaches(false);
   urlConn.setAllowUserInteraction(true);
   urlConn.setFollowRedirects(true);
   urlConn.setInstanceFollowRedirects(true);
   urlConn.setRequestProperty("content-type-Type", "text/html; charset=iso-8859-1");
   out = new DataOutputStream(urlConn.getOutputStream());
   //  String content = "email=" + URLEncoder.encode(EMAIL_ID, "ISO-8859-1") + "&pass=" + URLEncoder.encode(PASS_WORD, "ISO-8859-1") + "&action=login";
   String content = "email=" + EMAIL_ID + "&pass=" + PASS_WORD + "&action=login.php";
   System.out.println(content + "n" + "sending form to HTTP server ...");
   out.writeBytes(content);
   System.out.println(urlConn.getPermission());
   setCookie(urlConn.getHeaderField("Set-Cookie"));
   //System.out.println("sessionid: " + session_id);
 
   out.flush();
   out.close();
   // get input connection
   in = new BufferedReader(new InputStreamReader(urlConn.getInputStream()));
   } catch (IOException e) {
   e.printStackTrace();
   }
  }[/CODE]

I am trying to login with this and then get the website content

[CODE] public String getString() {
 String all = "";
 try {
 BufferedReader in = new BufferedReader(
 new InputStreamReader(url.openStream()));
 String inputLine;
 
 
   while ((inputLine = in.readLine()) != null) {
   all += inputLine + "n";
   // System.out.println(inputLine);
   }
 
   in.close();
 
   } catch (MalformedURLException e) {     // new URL() failed
   System.out.println(e);
   } catch (IOException e) {               // openConnection() failed
   System.out.println(e);
   }
   return all;
  }[/CODE]

And to read the content of the site ?

Anyway, what ever i try all i get is the login page.

How can i by pass this and actually get the first page when i login.

@chazzyMar 07.2008 — #Have you thought of using Apache's Commons Http Client? It makes all of this a lot easier. What about JMeter?

@nguefeuAug 07.2008 — #How Ali,

I have the solution of problem.If you want into website with Java Application,you can the HttpClient API of Java.

You can obtain the tutorial at this link:http://hc.apache.org/httpclient-3.x/tutorial.html

An example is at this link:http://svn.apache.org/viewvc/httpcomponents/oac.hc3x/trunk/src/examples/FormLoginDemo.java?view=co

This example shows how pass parameter to formula and submit it allowing you to log into website.

Please excuse me i don't speak very well english.

@LuisMRNov 29.2012 — #What version of HttpClient is use for the example that you put?

I used httpClient 3.0 and 3.0.1 but throw a exception:

java.lang.NoClassDefFoundError

java.lang.ClassNotFoundException

@Alexia_ANov 30.2012 — #What version of HttpClient is use for the example that you put?

I used httpClient 3.0 and 3.0.1 but throw a exception:

java.lang.NoClassDefFoundError

java.lang.ClassNotFoundException[/QUOTE]
Is this related to the version of HttpClient?

@LuisMRNov 30.2012 — #I think maybe I have a newer version and java not found the necessary classes for this reason.Or just my CLASSPATH is wrong.

@Silvia_YDec 06.2012 — #Yeah, maybe. Can you tell us the exact reason when you figure it out. Looking forward to knowing.

Also in #Java _↴

How can I replicate this concept easily?How to save webpages without modifying their original charset?jave program to read a file

Success!

Help @ruba spread the word by sharing this article on Twitter...

Tweet This

build a Java application to log into a website

13 Comments(s) _↴

Also in #Java _↴

Success!

Social

Version

build a Java application to log into a website

13 Comments(s) ↴

Also in #Java ↴

Success!

The web is an endless sea of information. Don't miss the boat... Subscribe!

Social

Version

13 Comments(s) _↴

Also in #Java _↴