What does FlatPack do?

It parses or creates Delimited files (e.g. CSV) or Fixed Width 'files'.

• CSV, TAB, Fixed Width, etc Readers and Writers.
• It is reasonable and fast.
• It can handle large files/records (see BuffReaderDelimiterFactory) or read everything in memory.
• You do not need to know ALL the column names, columns can be ignored/added/removed without breaking your code.
• You do not need to know the order of columns, columns order can be modified without breaking your code.
• The column names are case insensitive by default: record.getString("StATUs") is ok.
• If you parse everything in memory, you can restart the dataSet, skip records, go back etc.
• Java 8 stream like interface: e.g. CsvParserFactory.newInMemoryParser(..reader...).parseAsStream().filter(xxx).map(yyy).collect(zzz).

Why should you use FlatPack?

There is more to it than one could imagine at first sight:

• Problem Sorting Data Prior To File Processing?
• FlatPack provides a sorting mechanism for your flat files. This allows for your data to be properly sorted before processing your business logic.
• Bad Numeric Data?
• Do you often receive numeric data with dollar signs, or commas? FlatPack has the ability to strip these out and return as a double or int. Even empty elements or hard spaces can be converted to a numeric without issue.
• Problems With One Bad Record Blowing Up The Entire Parse?
• Bad records get excluded from the DataSet and added to an error collection. This error collection can be checked at the end of the parse to report on bad records. A description of the error and the line number in the file is provided. Why blow up an entire import for one bad record?

Which JDK?

1.8 and up

New JDK8 to Read Delimited File

Handling a LARGE TAB Delimited File

New way to Read Delimited File With Immutable Record

// Obtain the proper parser for your needs: basic CSV Parser with column Header
Parser parser = CsvParserFactory.newForwardCsvParser(new FileReader("DataFile.csv"));

//obtain DataSet
StreamingDataSet ds = parser.parseAsStream();

while (ds.next()){ //loop through file
    Record record = ds.getRecord(); // immutable object, you can give it to a thread.
    String value = record.getString("mycolumnName"); // columns name are case INsensitive
}
                    

Reading Delimited File With Column Mapping

//Obtain the proper parser for your needs
Parser parser = DefaultParserFactory.getInstance().newDelimitedParser(
    new FileReader("map.fpmap.xml"), //xml column mapping
    new FileReader("DataFile.txt"), //txt file to parse
    ',', //delimiter
    '"', //text qualfier
    false); //ignore the first record (may need to be done if first record contain column names)

//obtain DataSet
DataSet ds = parser.parse();

while (ds.next()){ //loop through file
    ds.getString("mycolumnName");
}
                    

Reading Delimited File Column Names In First Record Of File

//Obtain the proper parser for your needs
Parser parser = DefaultParserFactory.getInstance().newDelimitedParser(
    new FileReader("DataFile.txt"), //txt file to parse
    ',', //delimiter
    '"'); //text qualifier

//obtain DataSet
DataSet ds = parser.parse();

while (ds.next()){ //loop through file
    ds.getString("mycolumnName");
}
                    

Reading Fixed Width File

//Obtain the proper parser for your needs
Parser parser = DefaultParserFactory.getInstance().newFixedLengthParser(
    new FileReader("map.fpmap.xml"), //fixed with column map
    new FileReader("DataFile.txt")); //txt file to parse

//obtain DataSet
DataSet ds = parser.parse();

while (ds.next()){ //loop through file
    ds.getString("mycolumnName");
}
                    

Reading a Message of Fixed Width

final String msg = "data data2 data3 data4"
//Obtain the proper parser for your needs
Parser parser = DefaultParserFactory.getInstance().newFixedLengthParser(
    new FileReader("map.fpmap.xml"), //fixed with column map to bind col names
    new StringReader(mag)); //data to parse

//obtain DataSet
DataSet ds = parser.parse();

while (ds.next()){ //loop through file
    ds.getString("mycolumnName");
}
                    

Using SLF4J With FlatPack

Slf4j is utilized to capture logging information that occurred during a parse.
By default, this information will be logged to the console. SLF supports the following loggers; log4j, jcl, nop, and jdk1.4.
Please go to http://www.slf4j.org to download the proper jar for your logging preference. Here is a typical SLF4j setup:

SLF-api-XXX.jar + SLF-MyLoggerPreference.jar + MyLogger.jar (lo4j.jar for example)

Here are the steps that would need to be taken for log4j:

1. Install log4j.jar
2. Setup log4j.properties or log4j.xml
3. Install SLF-api-XXX.jar
4. Install SLF-log4j12-XXX.jar