[RESOLVED] reading xml files and memory usage

[FearedAtheist]

I'm using the following function to read in a (currently 1.3 mb XML file) and return it to a socket for writing to a remote PHP script. When the function appends the buffer (tempData) to the String containing the file data (data), my memory usage spikes up to 1.1 gigs. I knew Java was a memory hog but I just HAVE to be doing something wrong.

Alternatively, I've tried reading the file using a char array. It's much faster, but as I eventually have to convert it to a String, I still encounter the same memory problem.

Am I working too high level with this amount of data? Do I need to start thinking bytes [] ? I can't understand where Java comes up with 1.1 gigs...

Thanks for any tips or suggestions.


public static String getPlaylist(String path) {
//open file
BufferedReader in = null;
try {
in = new BufferedReader(new InputStreamReader(new FileInputStream(path)));
} catch (IOException e) {
System.out.println("Couldn't create file reader!");
System.exit(-1);
}

String data = "";
try {
String tempData = null;
while ((tempData = in.readLine()) != null) {
data += tempData;
}

} catch (IOException e) {
System.out.println("Couldn't read file!");
System.exit(-1);
}

return data;
}

[zimonyi]

Well, since a String is a byte[] it hardly won't matter, however you might want to read the whole file in one go.

Since Strings are immutable, you create a new String object each time you concatenate, which might explain the spike (although perhaps not 1.1 Gigs). I think in that short timespan, the garbage collecting hasn't kicked in.

But if you know the size of the file you should be able to read it all in one go.

You could also try using one of the XML parsers. They might also be less memory heavy when loading the contents of the XML, then you need to use some method to return the whole XML as one String (since that is what you want to do from what I understand).

Hope it helps,

Archie

[FearedAtheist]

Thanks for you help Archie. The concatening Strings was definitely not a good idea. I got my memory usage down to 10 megs. It add 3 megs during each XML read, but I can debug that later. When the PHP script pulls in that data from the socket, it went from a 47 second read to 2.7 seconds.

What a ridiculous improvement!

My function ended up like this:


public void writePlaylistToSocket(String path) {
//open file
BufferedReader in = null;
try {
in = new BufferedReader(new InputStreamReader(new FileInputStream(path)));
} catch (IOException e) {
System.out.println("Couldn't create file reader!");
System.exit(-1);
}

long fileSize = new File(path).length();
char buff[] = new char[(int)fileSize];
try {
in.read(buff, 0, (int)fileSize);
} catch (IOException e) {
System.out.println("Couldn't read file!");
System.exit(-1);
}
//System.out.print(buff[charIndex]);

//send something back
PrintWriter out = null;
try {
out = new PrintWriter(clientSocket.getOutputStream());
} catch (IOException e) {
System.out.println("Read failed!");
System.exit(-1);
}

out.write(buff);
out.flush();
}


Thanks again!

[zimonyi]

No problem.

Just remember one thing. Since you can only read int number of bytes or characters you might not read the complete file (since the length of the file is returned as a long and you type cast), so you might need to loop so that you get all characters (and thus your offset will the first time start at 0 but the second time start at the number of read characters/bytes).

Also, I would most likely define a File variable and use that as the argument to your FileInputStream.

Also, since you are not buffering anything, I think you only need the FileInputStream, no need for the others really, since FileInputStream also contains the read() method you use.

Archie

[FearedAtheist]

Yeah, I was definitely thinking today about ways to bypass the readers and interface more directly with the stream. I'm definitely going to try that out.