October 2006 Commit Log

Number of Commits:
96
Number of Active Developers:
2
Benoit Xhenseval 2006-10-31 22:47

Paul, some interesting new methods, the BX parser (sorry about the name) seems to be significantly faster than the current one when there are NO qualifiers, If there are qualifiers, it is reasonably faster. But the interesting bits start when I created a new method that uses a StringBuffer rather than using "chunks".

The new method is flying when there are qualifiers all over the place... but is slower (albeit not slower than the current one) when there are no qualifier... and that is a bit of a mystery... anyway... getting too late... Have a look and let me know.

126 lines of code changed in:

  • PZFileReader/src/main/java/net/sf/pzfilereader/util: BXParser.java (+110 -5)
  • PZFileReader/src/test/java/net/sf/pzfilereader/parserutils: BXParserTest.java (+5 -5)
  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples: Examples.java (+11 -4)
Benoit Xhenseval 2006-10-31 20:47

do not use null for empty fields, use an empty String.

11 lines of code changed in:

  • PZFileReader/src/main/java/net/sf/pzfilereader/util: BXParser.java (+3 -5)
  • PZFileReader/src/test/java/net/sf/pzfilereader/parserutils: BXParserTest.java (+8 -8)
Paul Zepernick 2006-10-31 20:02

modified the last check in testSomeExtremeCases()

it appears that it should have 2 " in the result of the parse

Benoit, please lmk if this is incorrect.

3 lines of code changed in:

Paul Zepernick 2006-10-31 20:00

- handle null's for lTrim(), lTrimKeepTabs, and splitLine()



- splitLine now returns nulls for elements which are empty and have not been qualified



- Added a trimToNull method.



- All current tests pass with these changes

39 lines of code changed in:

  • PZFileReader/src/main/java/net/sf/pzfilereader/util: ParserUtils.java (+39 -2)
Benoit Xhenseval 2006-10-31 17:33

Added an option (17) to compare the BX parser and the current parser.

It should be noted that the current parser fails on some tests (Paul could you fix?)

Just select the number of repeat, the number of columns and whether the column should be qualified or not... tell me your results. ta

40 lines of code changed in:

  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples: Examples.java (+40)
Benoit Xhenseval 2006-10-31 17:12

Paul, I've added some basic tests for null, empty, ",,," kind of things. I've also had a go at a parser, the regular expression is a dead-end or will become **extremely** complex due to our special and whacky cases... The basic tests make quite a few things break in the current version. I'll run a couple of speed tests to see where we're going...

28 lines of code changed in:

Benoit Xhenseval 2006-10-31 16:07

One more funny test....

1 lines of code changed in:

Benoit Xhenseval 2006-10-31 16:05

I have added a few very basic tests and they all seem to fail... Paul, could you investigate? thanks.

9 lines of code changed in:

Paul Zepernick 2006-10-31 15:04

converted to factory classes

74 lines of code changed in:

  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/csvperformancetest: CSVPerformanceTest.java (+7 -3)
  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/delimitedcolumnnamesinfile: DelimitedColumnNamesInFile.java (+9 -3)
  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/delimiteddynamiccolumns: DelimitedWithPZMap.java (+6 -2)
  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/delimiteddynamiccolumnswitherrors: DelimitedWithPZMapErrors.java (+11 -15)
  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/exporttoexcel: DelimitedFileExportToExcel.java (+11 -7)
  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/fixedlengthdynamiccolumns: FixedLengthWithPZMap.java (+7 -5)
  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/fixedlengthheaderandtrailer: FixedLengthHeaderAndTrailer.java (+6 -3)
  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/jsptableexample/pzfilereaderwebsamples: index.jsp (+4 -8)
  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/multilinedelimitedrecord: DelimitedMultiLine.java (+7 -4)
  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/numericsanddates: NumericsAndDates.java (+6 -3)
Paul Zepernick 2006-10-31 14:38

converted to IDataSet interface

3 lines of code changed in:

Paul Zepernick 2006-10-31 14:25

converted to factory classes

9 lines of code changed in:

  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/csvheaderandtrailer: CSVHeaderAndTrailer.java (+9 -6)
Benoit Xhenseval 2006-10-31 13:38

added a few more tests.

69 lines of code changed in:

Benoit Xhenseval 2006-10-31 11:44

Added a few examples for the website.

116 lines of code changed in:

Paul Zepernick 2006-10-31 11:24

first fixed width test

36 lines of code changed in:

Benoit Xhenseval 2006-10-30 16:46

first cut at reg expressions...

3 lines of code changed in:

Paul Zepernick 2006-10-30 15:57

added a couple more tests

3 lines of code changed in:

Paul Zepernick 2006-10-30 12:20

added 2 more extreme tests. Possible bug on the last test.

Needs more discussion.

9 lines of code changed in:

Benoit Xhenseval 2006-10-30 12:05

removed code that was commented out since the new code is now the accepted version.

46 lines of code changed in:

  • PZFileReader/src/main/java/net/sf/pzfilereader/util: ParserUtils.java (+46 -125)
Benoit Xhenseval 2006-10-30 11:57

Just to keep note.

11 lines of code changed in:

Paul Zepernick 2006-10-30 11:56

documented addition to splitLine

1 lines of code changed in:

Paul Zepernick 2006-10-30 11:55

added to javadoc description for splitLine

3 lines of code changed in:

Paul Zepernick 2006-10-30 11:51

Trim left and right space for unqualified elements.

7 lines of code changed in:

  • PZFileReader/src/main/java/net/sf/pzfilereader/util: ParserUtils.java (+7 -3)
Benoit Xhenseval 2006-10-27 14:14

General clean Up by Eclipse (cleanup, organise imports and format).

523 lines of code changed in:

Benoit Xhenseval 2006-10-27 14:06

First cut at re-org to use Factory mechanisms. Converted 2 unit tests and they seem happy...

Still using IDataSet for the interface.

LargeSet not covered at this stage.

258 lines of code changed in:

Benoit Xhenseval 2006-10-27 11:55

the DataError should be immutable (i.e. no Set method)

15 lines of code changed in:

  • PZFileReader/src/main/java/net/sf/pzfilereader: DataError.java (+15 -1)
Benoit Xhenseval 2006-10-27 11:52

Removed some throw Exception

Only non-runtime exceptions should be declared and never at the 'Exception' level, which is far too generic and forces every caller to deal with something which is 'unknonwn'

23 lines of code changed in:

Paul Zepernick 2006-10-27 01:05

starting to go through extreme tests

13 lines of code changed in:

Paul Zepernick 2006-10-27 01:05

removed system out

1 lines of code changed in:

  • PZFileReader/src/main/java/net/sf/pzfilereader/util: ParserUtils.java (+1 -1)
Paul Zepernick 2006-10-27 01:04

fixed bug, should not trim off qualifier unless the element

began with a qulifier

3 lines of code changed in:

  • PZFileReader/src/main/java/net/sf/pzfilereader/util: ParserUtils.java (+3 -1)
Paul Zepernick 2006-10-27 00:25

ParserUtils for fixed width files

15 lines of code changed in:

Paul Zepernick 2006-10-27 00:23

moved parse to FixedWidthParserUtils

10 lines of code changed in:

  • PZFileReader/src/main/java/net/sf/pzfilereader: DataSet.java (+10 -8)
Paul Zepernick 2006-10-27 00:22

added a new method to add a collection of columns to the row

14 lines of code changed in:

  • PZFileReader/src/main/java/net/sf/pzfilereader/structure: Row.java (+14)
Paul Zepernick 2006-10-27 00:21

- fixed line count bug

- moved parse to FixedWidthParserUtils

- moved constants

28 lines of code changed in:

Paul Zepernick 2006-10-27 00:20

moved fixed width method to FixedWidthParserUtils



deprecated method

2 lines of code changed in:

Paul Zepernick 2006-10-27 00:07

moved some constants in from LargeDataSet

34 lines of code changed in:

  • PZFileReader/src/main/java/net/sf/pzfilereader/util: PZConstants.java (+34 -1)
Paul Zepernick 2006-10-27 00:06

added a better comment to getColumns()

3 lines of code changed in:

Benoit Xhenseval 2006-10-26 15:54

Added a heuristic test that proves that using a StringBuffer delete is better than creating a new one...

60 lines of code changed in:

  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples: Examples.java (+60 -31)
Benoit Xhenseval 2006-10-26 15:53

Try to reduce the number of trimmings but Paul, could you check the comments //+ as I believe that those tests are redudant...

7 lines of code changed in:

  • PZFileReader/src/main/java/net/sf/pzfilereader: DataSet.java (+7 -7)
Benoit Xhenseval 2006-10-26 15:51

try to reduce memory requirements by trimming to size the list.

5 lines of code changed in:

  • PZFileReader/src/main/java/net/sf/pzfilereader/util: ParserUtils.java (+5 -3)
Benoit Xhenseval 2006-10-26 15:50

Added a couple of whacky tests, some fail (on purpose); Paul could you check what results you expect and create a few more?

Thanks

32 lines of code changed in:

Benoit Xhenseval 2006-10-26 14:04

First cut at some interfaces. Paul, could you review and tell me if you think that they are well separated.

I think that PZParserFactory.java and PZParser.java are ok but have I put everything that is required for the manipulation

of a DataSet in IDataSet.java?

54 lines of code changed in:

Benoit Xhenseval 2006-10-26 08:58

Removed System.out

0 lines of code changed in:

  • PZFileReader/src/main/java/net/sf/pzfilereader: DataSet.java (-1)
Paul Zepernick 2006-10-25 11:36

removed freeMemory() call and updated constructor



I have not tested the changes yet. I go back through and make sure they are still okay

11 lines of code changed in:

  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/delimitedcolumnnamesinfile: DelimitedColumnNamesInFile.java (+1 -4)
  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/delimiteddynamiccolumns: DelimitedWithPZMap.java (+1 -3)
  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/delimiteddynamiccolumnswitherrors: DelimitedWithPZMapErrors.java (+1 -4)
  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/exporttoexcel: DelimitedFileExportToExcel.java (+1 -4)
  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/fixedlengthdynamiccolumns: FixedLengthWithPZMap.java (-3)
  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/fixedlengthheaderandtrailer: FixedLengthHeaderAndTrailer.java (-3)
  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/jsptableexample/pzfilereaderwebsamples: index.jsp (+2 -5)
  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/largedataset/delimiteddynamiccolumns: LargeDelimitedWithPZMap.java (+1 -1)
  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/largedataset/largecsvperformancetest: CSVLarge.java (+1 -1)
  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/lowlevelparse: LowLevelParse.java (+1 -1)
  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/multilinedelimitedrecord: DelimitedMultiLine.java (+1 -4)
  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/numericsanddates: NumericsAndDates.java (+1 -5)
Paul Zepernick 2006-10-25 11:27

removed freeMemmory call and updated constructor

2 lines of code changed in:

  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/csvheaderandtrailer: CSVHeaderAndTrailer.java (+1 -4)
  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/csvperformancetest: CSVPerformanceTest.java (+1 -3)
Paul Zepernick 2006-10-25 11:26

added missing char version of constructor

31 lines of code changed in:

  • PZFileReader/src/main/java/net/sf/pzfilereader: DataSet.java (+31)
Benoit Xhenseval 2006-10-24 23:33

Optimised the ParserUtils to use char for delimiter and qualifier.

I have added deprecated methods for Strings (using only the first character). Could you find out where these are used and remove the call to those and use the char instead.

All tests are passed but we should add more... especially with regards to the multi line one...

Time to hit the sack!

381 lines of code changed in:

Benoit Xhenseval 2006-10-24 21:36

Forgot to append the actual element.

2 lines of code changed in:

Paul Zepernick 2006-10-24 17:33

expanded upon the tests. Made a little more generic. There is an

array of delimiters and qualifiers which we can fill in for whatever

we want to test. Implemented Benoit's formating suggestions.

127 lines of code changed in:

Paul Zepernick 2006-10-24 11:36

started splitline test. Publishing so I can work on it futher from work

20 lines of code changed in:

Paul Zepernick 2006-10-24 11:12

Renamed to follow naming convention "ClassNameMethodToTest"

20 lines of code changed in:

Paul Zepernick 2006-10-24 11:08

added CSV with hdr and trailer file to make #5 work okay

0 lines of code changed in:

Paul Zepernick 2006-10-24 11:03

Added code to print the errors found in the file if there were any.



Pointed to a text file with no header and trailer to corrispond with

the mapping.

9 lines of code changed in:

Paul Zepernick 2006-10-23 11:34

added header and trailer checks

20 lines of code changed in:

  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/delimiteddynamiccolumns: DelimitedWithPZMap.java (+1 -1)
  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/delimiteddynamiccolumnswitherrors: DelimitedWithPZMapErrors.java (+19 -1)
Paul Zepernick 2006-10-23 10:58

Keep the header and trailer in the same order when moving to the bottom

9 lines of code changed in:

  • PZFileReader/src/main/java/net/sf/pzfilereader/ordering: OrderBy.java (+9 -3)
Benoit Xhenseval 2006-10-20 12:14

Fixed the test.

1 lines of code changed in:

  • PZFileReader/src/test/java/net/sf/pzfilereader/parserutils: LTrimTest.java (+1 -3)
Benoit Xhenseval 2006-10-20 09:53

Took liberty to make the tests more explicit in order to detect any potential side effect, say the lTrim would correctly remove the leading space and leave the last one but mangle the text in between, the original tests would not have spotted that.

I have also added a space in the middle of the word to detect more potential issues.

Finally, I have added a method at the bottom and this has raised a question about the exacts spec... Paul, please have a look.

i.e. lTrimWithKeepTabs, what if the string starts with a tab and then a space and then some text "\t blabla" what should the result be??? "\t blabla" (now) or "\tblabla" ???

19 lines of code changed in:

  • PZFileReader/src/test/java/net/sf/pzfilereader/parserutils: LTrimTest.java (+19 -11)
Benoit Xhenseval 2006-10-20 09:49

Fixed link to download page.

1 lines of code changed in:

Paul Zepernick 2006-10-19 22:22

bad test on keep leading tabs...corrected

1 lines of code changed in:

  • PZFileReader/src/test/java/net/sf/pzfilereader/parserutils: LTrimTest.java (+1 -1)
Paul Zepernick 2006-10-19 15:28

Test cases for lTrim and lTrimKeepTabs

11 lines of code changed in:

  • PZFileReader/src/test/java/net/sf/pzfilereader/parserutils: LTrimTest.java (new 11)
Paul Zepernick 2006-10-19 07:27

added checks for header and trailer records

17 lines of code changed in:

  • PZFileReaderSamples/src/main/java/net/sf/pzfilereader/examples/delimiteddynamiccolumns: DelimitedWithPZMap.java (+17)
Paul Zepernick 2006-10-19 07:16

added default system type to get around JDOM parse error. More notes in task manager

7 lines of code changed in:

  • PZFileReader/src/main/java/net/sf/pzfilereader/xml: PZMapParser.java (+7 -2)
Paul Zepernick 2006-10-19 06:03

backed out a 1.5 only method Integer.valueOf

2 lines of code changed in:

  • PZFileReader/src/main/java/net/sf/pzfilereader/util: ParserUtils.java (+2 -1)
Benoit Xhenseval 2006-10-18 20:51

Uploaded new site.

2 lines of code changed in:

Benoit Xhenseval 2006-10-18 13:40

Some serious kicking...

1/ use a map for finding the column index; this makes the fetch of the first or last column consistent

2/ removed SOME of the substring which are causing dramatic performance degradation when once has a fair amount of columns.

3/ optimised some string manipulation code (getDelimiterOffset, lTrim, lTrimKeepTabs, removeChar

4/ I would suggest the creation of a suite of unit tests for all those methods.... Paul, do you want to take this on?

320 lines of code changed in:

Paul Zepernick 2006-10-18 11:27

javadoc package

0 lines of code changed in:

  • PZFileReader/src/main/java/net/sf/pzfilereader: package.html (new)
  • PZFileReader/src/main/java/net/sf/pzfilereader/structure: package.html (new)
Paul Zepernick 2006-10-18 11:24

package javadoc

0 lines of code changed in:

  • PZFileReader/src/main/java/net/sf/pzfilereader/ordering: package.html (new)
Paul Zepernick 2006-10-18 11:21

javadoc package docs

0 lines of code changed in:

  • PZFileReader/src/main/java/net/sf/pzfilereader/util: package.html (new)
  • PZFileReader/src/main/java/net/sf/pzfilereader/xml: package.html (new)
Benoit Xhenseval 2006-10-18 08:44

Fix the links to Word and PDF doco.

6 lines of code changed in:

Paul Zepernick 2006-10-18 00:39

updated package structure to net.sf on bat file

1 lines of code changed in:

Benoit Xhenseval 2006-10-17 23:00

tiny amount of formatting.

0 lines of code changed in:

  • PZFileReader/src/main/java/net/sf/pzfilereader: DataSet.java (-1)
  • PZFileReader/src/main/java/net/sf/pzfilereader/util: ParserUtils.java (-1)
Benoit Xhenseval 2006-10-17 22:28

scoping and using PreparedStatement (always better).

81 lines of code changed in:

Benoit Xhenseval 2006-10-17 21:39

Reduce the scope of some variables that now can be declared 'final', this in turn helps the JVM to optimize the runtime code, as well as keeping the memory requirement to a minimum.

38 lines of code changed in:

Benoit Xhenseval 2006-10-17 21:23

Fix the homepage and reduced the scope of some variables.

39 lines of code changed in:

Benoit Xhenseval 2006-10-17 20:30

Link to the documentation (somehow it had been removed...)

6 lines of code changed in:

Benoit Xhenseval 2006-10-17 20:26

Avoid a loop with string addition.

6 lines of code changed in:

  • PZFileReader/src/main/java/net/sf/pzfilereader: DataSet.java (+5 -11)
  • PZFileReader/src/main/java/net/sf/pzfilereader/util: ParserUtils.java (+1 -3)
Benoit Xhenseval 2006-10-17 20:09

Couple of changes for website.

40 lines of code changed in:

Benoit Xhenseval 2006-10-17 19:36

Few site changes, also preparing a press release.

171 lines of code changed in:

Benoit Xhenseval 2006-10-17 17:26

Final items for move to net.sf.pzfilereader.

15 lines of code changed in:

Benoit Xhenseval 2006-10-17 17:19

Moved to net.sf.pzfilereader

7 lines of code changed in:

Benoit Xhenseval 2006-10-17 17:09

Moved to net.sf.pzfilereader

51 lines of code changed in:

Benoit Xhenseval 2006-10-17 16:51

Moved remotely

155 lines of code changed in:

Benoit Xhenseval 2006-10-17 16:35

Paul's main page.

113 lines of code changed in:

Benoit Xhenseval 2006-10-17 15:05

Work on the examples to use a Console menu that the user can launch and then pick which test to run interactively.

312 lines of code changed in:

Paul Zepernick 2006-10-17 14:22

documented change

1 lines of code changed in:

Paul Zepernick 2006-10-17 14:21

removed exception for DataSets containing <record> mappings when an orderBy was called



Added logic to push all non-detail records to the bottom of the DataSet on a sort

14 lines of code changed in:

  • PZFileReader/src/main/java/com/pz/reader: DataSet.java (+5 -3)
  • PZFileReader/src/main/java/com/pz/reader/ordering: OrderBy.java (+9 -1)
Benoit Xhenseval 2006-10-16 23:27

A few Checkstyle and FindBugs resolutions.

54 lines of code changed in:

Benoit Xhenseval 2006-10-16 21:19

One step closer to have the examples working. Tidy up website too.

393 lines of code changed in:

Benoit Xhenseval 2006-10-16 19:52

Add some documentation

1923 lines of code changed in:

Benoit Xhenseval 2006-10-16 16:39

Suggest that we aim the next release at 2.3 or 3.0, make our maven changes and modify the package structure for iBiblio.org (that will also have an impact on Maven build).

4 lines of code changed in:

Benoit Xhenseval 2006-10-16 16:21

Converted changes.txt to the changes.xml that will be automatically included in the website generation.

Only this file should be maintained.

174 lines of code changed in:

Benoit Xhenseval 2006-10-16 15:35

Applied the Eclipse cleanUp, organise imports and format to help reduce the Checkstyle issues, from 1696 to 32.

slightly modified a few lines as well (ensured encapsulation).

2259 lines of code changed in:

Repository Tag: release_2.2.0
Benoit Xhenseval 2006-10-16 13:44

Added unit tests in pzfilereader (quite limited).

241 lines of code changed in:

Benoit Xhenseval 2006-10-16 12:59

May look a bit scary but there was no code change. I still need to think further about the examples but now the project is split in 2 modules, the main library pzfilereader and the examples pzfilereader-examples. Both follow the same version numbering.

Each can be built at their level (e.e the PZFileReader directory or PZFileReaderSamples directory.

But preferably at the top level using:

maven pz:build

maven pz:rebuild

maven pz:site

maven pz:releasebuild



The Site is the generated in target/pzfilereader-site-2.2.0/bin/pzfilereader-2.2.0/docs

More to follow...

2979 lines of code changed in:

Paul Zepernick 2006-10-16 10:55

Adding updated examples which are packaged with the current PZFileReader distro

0 lines of code changed in:

Paul Zepernick 2006-10-15 19:52

put pacakge structure back so version history could be saved before moving over to subversion.

1984 lines of code changed in:

Paul Zepernick 2006-10-13 22:32

Refactored for net.sf.pzfilereader packaging

7 lines of code changed in:

September 2006 »

Generated by StatSVN 0.3.2-SNAPSHOT