Reading Large ASCII Datasets

Ever tried reading large geophysical ASCII data sets with a standard text editor only to get angry when it hangs, crashes or runs out of memory? Chances are you don't even want to edit the document but rather want to read the first one or two lines in the header. There are several solutions to solve this problem. First of all you can load the data into a software package which imports the data directly into a database (i.e., Oasis Montaj), but that requires you to load hundreds of megabytes or even  gigabytes of data, wasting minutes in the process. Secondly, if you are 'fortunate' enough to use a Unix based OS, you can use the "less" command, great for Linux users but pretty much worthless for Windows users.

If you want to try another solution out see below.

The premise

The header in a large ASCII text file can be read by storing one line at a time in memory. This avoids memory issues and consumes mere kilobytes of memory rather than trying to import gigabytes at a time.

The Solution

I have written some code using the JDK 8 developers release SDK containing the latest JavaFX libraries. This can be run under the Windows OS. The aptly named perpetual ASCII file viewer reads a file in blocks. Using Java this is accomplished by using the lines,

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

BufferedReader in;
public final TextArea t = new TextArea(); //Text area displaying the read text
private int preload, bufferSize;	//preload 'preload' lines and load bufferSize number of subsequent lines on request 
 
public void load(File f) throws Exception{
  BufferedReader in = new BufferedReader(new FileReader(f));
  this. in = in;
  preload();
}
private void preload() throws Exception{
  Platform.runLater(new Thread(new Runnable() {
    @Override
    public void run() {
      appendLine(bufferSize);
      t.positionCaret(0); //scroll to end
    }
  }));
}
 
private void appendLine(int n) {
  try {
    String append = "";				
    for(int i = 0 ; i < n ; i++){					
      String line = in.readLine();				
      if(line == null) {
        in.close();
	i = n;
	break;
      } else {
        append += (line + "\n");
      }
    }
    t.appendText(append);
  } catch (IOException e) {
    e.printStackTrace();
  }
}

The program is initialised by calling the load(File f) function. This creates a buffered reader (in). Upon loading the file into the buffered reader, the function preload() is called. The preload function loads n=bufferSize lines into memory/text area. The appendLine(int n) function reads in subsequent lines.

This is only the gist of the program, but you can view the complete source code from my Bitbucket repository.

Installing and Running the Program

Run the installer. This will install a local version of Java (150mb) along with the Perpetual File Viewer Program (55kb). Yes I know, so much required for such a small program.

Step 1: Install the Software

Run the executable which has just installed. This will bring up a file selection dialogue. Once you have selected the file click the open button.

Step 2: Run the program PFV.exe. This will bring up a File Chooser dialogue box. Select the file you wish to view.

Finally you can view the file.

Step 3: View the File. The data will be perpetually read on MOUSE SCROLL WHEEL ONLY!

The software currently cannot save edited documents yet. Possibly a feature to create if enough support is generated.

Download the Installer

Perpetual File Viewer Installer 0.00 KB

Download

Download the JAR File only (must have JDK8)

Perpetual File Viewer JAR 0.00 KB

Download