News

ObjectLab Open Source News

Grab this Headline Animator

March 03, 2010: FlatPack 3.2.0 is released, it is mainly a bug-fix release, see the change log.

FlatPack Introduction

FlatPack came out of the frustration of having to mix file parsing logic with business logic.

FlatPack on SourceForge: a Java (1.4+) flat file parser that handles CSV, fixed length and custom delimiters. The formats are configured in XML, it is fast and released under Apache license 2.0.

Starting at release 3.1, FlatPack introduces exporters via the writer package. This mechanism allows you to export a DataSet to a fixed length or delimited format. A big thank you to Dirk and Holger from the Mule project for the initial code contribution. FlatPack should integrate nicely with Mule, the open source choice for integration and SOA.

Substrings in a fixed width parse can be daunting to deal with when trying to analyze what existing code is doing, and what about when you have no comments...

We also provide delimited file parsing; works with any delimiter / qualifier, multiline records, delimiter or qualifier allowed in column value.

A manual is available as Word document or a PDF.

FlatPack is part of the ObjectLab Kit family. Digg!

Why should you use FlatPack?

There is more to it than one could imagine at first sight:

  • Problem Sorting Data Prior To File Processing?
    • FlatPack provides a sorting mechanism for your flat files. This allows for your data to be properly sorted before processing your business logic.
  • Bad Numeric Data?
    • Do you often receive numeric data with dollar signs, or commas? FlatPack has the ability to strip these out and return as a double or int. Even empty elements or hard spaces can be converted to a numeric without issue.
  • Problems With One Bad Record Blowing Up The Entire Parse?
    • Bad records get excluded from the DataSet and added to an error collection. This error collection can be checked at the end of the parse to report on bad records. A description of the error and the line number in the file is provided. Why blow up an entire import for one bad record?

Which JDK?

1.4 and up

UML?

We are using yDoc to generate a 'javadoc-like' documentation per module with the added twist of UML diagrams:

How do I use it?

Basic Steps

  1. Obtain a Parser from a parser factory (DefaultParserFactory) witha File, InputStream, or a reader(coming soon)
  2. Set additional Parser options, Parser.setAnOption()
  3. Obtain a DataSet Parser.parse()
  4. Call DataSet.next() to advance record pointer
  5. Process columns; DataSet.getString("colname"), getInt("colname"), getDouble("colname"), getDate("colname"), etc
  6. Check for parse errors; DataSet.getErrors()

Reading Delimited File With Column Mapping

    	//Obtain the proper parser for your needs
    	Parser parser = DefaultParserFactory.getInstance().newDelimitedParser(
                new FileReader("map.fpmap.xml"), //xml column mapping
                new FileReader("DataFile.txt"),  //txt file to parse
                ',', //delimiter
                '"', //text qualfier
                false); //ignore the first record (may need to be done if first record contain column names)

        //obtain DataSet
        DataSet ds = parser.parse();

        while (ds.next()){ //loop through file
            ds.getString("mycolumnName");
        }
    

Reading Delimited File Column Names In First Record Of File

    	//Obtain the proper parser for your needs
    	Parser parser = DefaultParserFactory.getInstance().newDelimitedParser(
                new FileReader("DataFile.txt"),  //txt file to parse
                ',', //delimiter
                '"'); //text qualifier

        //obtain DataSet
        DataSet ds = parser.parse();

        while (ds.next()){ //loop through file
            ds.getString("mycolumnName");
        }
    

Reading Fixed Width File

	    //Obtain the proper parser for your needs
        Parser parser = DefaultParserFactory.getInstance().newFixedLengthParser(
        		new FileReader("map.fpmap.xml"), //fixed with column map
                new FileReader("DataFile.txt"));  //txt file to parse

        //obtain DataSet
        DataSet ds = parser.parse();

        while (ds.next()){ //loop through file
            ds.getString("mycolumnName");
        }
    

Reading a Message of Fixed Width

    	final String msg = "data     data2     data3     data4"
	    //Obtain the proper parser for your needs
        Parser parser = DefaultParserFactory.getInstance().newFixedLengthParser(
        		new FileReader("map.fpmap.xml"), //fixed with column map to bind col names
                new StringReader(mag));  //data to parse

        //obtain DataSet
        DataSet ds = parser.parse();

        while (ds.next()){ //loop through file
            ds.getString("mycolumnName");
        }
    

Using SLF4J With FlatPack

		Slf4j is utilized to capture logging information that occurred during a parse.
		By default, this information will be logged to the console.  SLF supports the following loggers; log4j, jcl, nop, and jdk1.4.
		Please go to http://www.slf4j.org to download the proper jar for your logging preference.  Here is a typical SLF4j setup:

		SLF-api-XXX.jar + SLF-MyLoggerPreference.jar + MyLogger.jar (lo4j.jar for example)

		Here are the steps that would need to be taken for log4j:

		1.	Install log4j.jar
		2.	Setup log4j.properties or log4j.xml
		3.	Install SLF-api-XXX.jar
		4.	Install SLF-log4j12-XXX.jar