csvread 3.0.1 exponentially slower than v 3.0.0

[gOS]

I've not sat through all the way, but on 2 seperate windows machines, one with a pentium 4 and one with a much older processor, I get similar results.

I can confirm that in 3.0.0 the script took about 5-10 seconds to complete.

In 3.0.1 I've sat for 10 minutes waiting for csvread to finish and it hasn't. Ctrl+c causes Octave to exit without warning and does not return control to the prompt.

I'm reading in 3mb of text. I've attached said file.

Command that hangs indefinately:

data = csvread('myFile.dat');

Help would be appreciated. I've confirmed that rolling back to 3.0.0 fixes the problem. I'm using the most up to date version from the Octave Forge site. Can anyone confirm that this is an Octave problem and not specific to windows? Can anyone one fix it? Thank you.

I couldn't get a bug report to generate after calling csvread because it never finished. Here's my bug report for just starting up Octave though. Maybe it will help. :-/

#! /bin/sh -
#
# Copyright (C) 1994, 1995, 1996, 1997, 1998, 2000, 2001, 2002, 2004,
#               2005, 2006 John W. Eaton
#
# This file is part of Octave.
#
# Octave is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the
# Free Software Foundation; either version 3 of the License, or (at
# your option) any later version.
#
# Octave is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
# for more details.
#
# You should have received a copy of the GNU General Public License
# along with Octave; see the file COPYING.  If not, see
# <http://www.gnu.org/licenses/>.

# octave-bug - create a bug report and mail it to the bug-octave
# mailing list.
#
# Patterned after the bashbug script from bash 1.14.

# Configuration:  these variables are filled in when running make to
# compile Octave.

config_opts="'--build=i686-pc-msdosmsvc' '--prefix=/c/Software/VCLibs/local/octave-3.0.1' '--with-zlib=zlib' '--with-curl=libcurl' 'build_alias=i686-pc-msdosmsvc' 'CC=cc-msvc' 'CFLAGS=-O2' 'CXX=cc-msvc' 'CXXFLAGS=-O2' 'F77=fc-msvc' 'FFLAGS=-O2'"
VERSION="3.0.1"
SED="/usr/bin/sed"
MACHINE="i686-pc-msdosmsvc"
F77="fc-msvc"
FFLAGS="-O2"
FPICFLAG=""
FLIBS="-lhdf5 -lzlib -lf2c -lkernel32"
F2C="@F2C@"
F2CFLAGS="@F2CFLAGS@"
CPPFLAGS="-I.  -Ic:/Software/VCLibs/include"
INCFLAGS="-I. -I. -I./liboctave -I./src -I./libcruft/misc"
CC="cc-msvc"
CC_VERSION=""
CFLAGS="-O2 -MD"
CPICFLAG=""
CXX="cc-msvc"
CXX_VERSION=""
CXXFLAGS="-O2 -EHs -MD"
CXXPICFLAG=""
LD_CXX="cc-msvc"
LDFLAGS=""
LIBFLAGS="-L."
RLD_FLAG=""
LIBS="-lreadline  -lncurses -lhdf5 -lzlib  -lws2_32 -lkernel32"
BLAS_LIBS="-llapack -lblas"
FFTW_LIBS="-lfftw3"
LEXLIB=""
LIBGLOB="-lglob"
DEFS="-DPACKAGE_NAME=\"\" -DPACKAGE_TARNAME=\"\" -DPACKAGE_VERSION=\"\" -DPACKAGE_STRING=\"\" -DPACKAGE_BUGREPORT=\"\" -DOCTAVE_SOURCE=1 -D_GNU_SOURCE=1 -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_UNISTD_H=1 -DSEPCHAR=';' -DSEPCHAR_STR=\";\" -D__NO_MATH_INLINES=1 -DCXX_NEW_FRIEND_TEMPLATE_DECL=1 -DCXX_ISO_COMPLIANT_LIBRARY=1 -DCXX_ABI=unknown -DHAVE_QHULL=1 -DHAVE_PCRE=1 -DHAVE_ZLIB_H=1 -DHAVE_ZLIB=1 -DHAVE_HDF5_H=1 -DHAVE_HDF5=1 -DHAVE_H5GGET_NUM_OBJS=1 -D_HDF5USEDLL_=1 -DHAVE_FFTW3=1 -DHAVE_GLPK_H=1 -DHAVE_GLPK=1 -DHAVE_CURL_CURL_H=1 -DHAVE_CURL=1 -DHAVE_IEEE754_DATA_FORMAT=1 -DF77_FUNC(name,NAME)=name ## _ -DF77_FUNC_(name,NAME)=name ## __ -DHAVE_BLAS=1 -DHAVE_SUITESPARSE_UMFPACK_H=1 -DHAVE_UMFPACK=1 -DUMFPACK_SEPARATE_SPLIT=1 -DHAVE_SUITESPARSE_COLAMD_H=1 -DHAVE_COLAMD=1 -DHAVE_SUITESPARSE_CCOLAMD_H=1 -DHAVE_CCOLAMD=1 -DHAVE_SUITESPARSE_CHOLMOD_H=1 -DHAVE_CHOLMOD=1 -DHAVE_SUITESPARSE_CS_H=1 -DHAVE_CXSPARSE=1 -Dmode_t=int -Dpid_t=int -Duid_t=int -Dgid_t=int -DHAVE_DEV_T=1 -DHAVE_INO_T=1 -DHAVE_LONG_LONG_INT=1 -DHAVE_UNSIGNED_LONG_LONG_INT=1 -DHAVE_SIG_ATOMIC_T=1 -DSIZEOF_SHORT=2 -DSIZEOF_INT=4 -DSIZEOF_LONG=4 -DSIZEOF_LONG_LONG=8 -DHAVE_ALLOCA=1 -DNPOS=std::string::npos -DHAVE_PLACEMENT_DELETE=1 -DSTDC_HEADERS=1 -DHAVE_ASSERT_H=1 -DHAVE_DIRECT_H=1 -DHAVE_FCNTL_H=1 -DHAVE_FLOAT_H=1 -DHAVE_LIMITS_H=1 -DHAVE_LOCALE_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_UTIME_H=1 -DHAVE_UNISTD_H=1 -DHAVE_VARARGS_H=1 -DHAVE_SSTREAM=1 -DHAVE_GLOB_H=1 -DHAVE_FNMATCH_H=1 -DHAVE_CONIO_H=1 -DHAVE_ATEXIT=1 -DHAVE_CHMOD=1 -DHAVE_DUP2=1 -DHAVE_EXECVP=1 -DHAVE_GETCWD=1 -DHAVE_GETPID=1 -DHAVE__KBHIT=1 -DHAVE_MEMMOVE=1 -DHAVE_MKDIR=1 -DHAVE_PUTENV=1 -DHAVE_RAISE=1 -DHAVE_RENAME=1 -DHAVE_RMDIR=1 -DHAVE_SETLOCALE=1 -DHAVE_SETVBUF=1 -DHAVE_STAT=1 -DHAVE_STRDUP=1 -DHAVE_STRERROR=1 -DHAVE_STRICMP=1 -DHAVE_STRNICMP=1 -DHAVE_TEMPNAM=1 -DHAVE_UMASK=1 -DHAVE_UNLINK=1 -DHAVE_UTIME=1 -DHAVE_VFPRINTF=1 -DHAVE_VSPRINTF=1 -DHAVE_VSNPRINTF=1 -DHAVE__CHMOD=1 -DHAVE__SNPRINTF=1 -DHAVE__UTIME32=1 -DOCTAVE_HAVE_BROKEN_STRPTIME=1 -D_WIN32_WINNT=0x0403 -D_USE_MATH_DEFINES=1 -DHAVE_LOADLIBRARY_API=1 -DENABLE_DYNAMIC_LINKING=1 -DHAVE__FINITE=1 -DHAVE__ISNAN=1 -DHAVE__COPYSIGN=1 -DHAVE_DECL_SIGNBIT=0 -DHAVE_STRUCT_STAT_ST_RDEV=1 -DHAVE_DECL_TZNAME=1 -DHAVE_TZNAME=1 -DCLOSEDIR_VOID=1 -DMKDIR_TAKES_ONE_ARG=1 -DUSE_READLINE=1 -DRETSIGTYPE=void -DHAVE_DECL_SYS_SIGLIST=0 -DMUST_REINSTALL_SIGHANDLERS=1 -DRETSIGTYPE_IS_VOID=1 -DGNUPLOT_BINARY=\"pgnuplot\""

: ${USER=$LOGNAME}

CC_AND_VERSION=
if test -n "$CC_VERSION"; then
  CC_AND_VERSION="$CC, version $CC_VERSION"
fi

CXX_AND_VERSION=
if test -n "$CXX_VERSION"; then
  CXX_AND_VERSION="$CXX, version $CXX_VERSION"
fi

PATH=/bin:/usr/bin:/usr/ucb:/usr/local/bin:$PATH
export PATH

TEMP=/tmp/octave-bug.$$

if [ -z "$DEFEDITOR" ] && [ -z "$EDITOR" ]; then
  if [ -x /usr/bin/editor ]; then
    DEFEDITOR=editor
  elif [ -x /usr/local/bin/ce ]; then
    DEFEDITOR=ce
  elif [ -x /usr/local/bin/emacs ]; then
    DEFEDITOR=emacs
  elif [ -x /usr/contrib/bin/emacs ]; then
    DEFEDITOR=emacs
  elif [ -x /usr/bin/emacs ]; then
    DEFEDITOR=emacs
  elif [ -x /usr/bin/xemacs ]; then
    DEFEDITOR=xemacs
  elif [ -x /usr/contrib/bin/jove ]; then
    DEFEDITOR=jove
  elif [ -x /usr/local/bin/jove ]; then
    DEFEDITOR=jove
  elif [ -x /usr/bin/vi ]; then
    DEFEDITOR=vi
  else
    echo "octave-bug: No default editor found: attempting to use vi" >&2
    DEFEDITOR=vi
  fi
fi

: ${EDITOR=$DEFEDITOR}

if [ -z "$DEFPAGER" ] && [ -z "$PAGER" ]; then
  if [ -x /usr/bin/pager ]; then
    DEFPAGER=pager
  elif [ -x /usr/bin/less ]; then
    DEFPAGER=less
  elif [ -x /bin/less ]; then
    DEFPAGER=less
  elif [ -x /usr/local/bin/less ]; then
    DEFPAGER=less
  elif [ -x /usr/bin/more ]; then
    DEFPAGER=more
  elif [ -x /bin/more ]; then
    DEFPAGER=more
  elif [ -x /usr/bin/pg ]; then
    DEFPAGER=pg
  elif [ -x /bin/pg ]; then
    DEFPAGER=pg
  else
    echo "octave-bug: No default pager found: attempting to use more" >&2
    DEFPAGER=more
  fi
fi

: ${PAGER=$DEFPAGER}

trap 'rm -f $TEMP $TEMP.x; exit 1' 1 2 3 13 15
trap 'rm -f $TEMP $TEMP.x' 0

UN=
if (uname) > /dev/null 2>&1; then
  UN=`uname -a`
fi

HAVE_FMT=false
if (fmt < /dev/null) > /dev/null 2>&1; then
  HAVE_FMT=true
fi

# Check whether to use -n or \c to keep echo from printing a newline
# character.  Stolen from autoconf, which borrowed the idea from dist 3.0.

if (echo "testing\c"; echo 1,2,3) | grep c >/dev/null; then
  # Stardent Vistra SVR4 grep lacks -e, says [e-mail].
  if (echo -n testing; echo 1,2,3) | $SED s/-n/xn/ | grep xn >/dev/null; then
    echo_n=
    echo_c='
'
  else
    echo_n=-n
    echo_c=
  fi
else
  echo_n=
  echo_c='\c'
fi

ss_p=`echo $VERSION | grep "^ss-"`
if test -n "$ss_p"; then
  BUGADDR="[e-mail]"
else
  pretest_p=`echo $VERSION \
    | $SED 's,.*\.\([0-9]*\).*,\1,' \
    | grep -v '\.' \
    | grep '[0-9]'`

  if test -n "$pretest_p" && test "$pretest_p" -ge 90; then
    BUGADDR="[e-mail]"
  else
    BUGADDR="[e-mail]"
  fi
fi

SUBJECT="[50 character or so descriptive subject here (for reference)]"
if test $# -gt 0; then
  case "$1" in
    -s)
      shift
      if test $# -gt 0; then
        SUBJECT="$1"
        shift
      else
        echo "usage: octave-bug [-s subject]"
        exit 1
      fi
    ;;
  esac
fi

cat > $TEMP << EOF
To: $BUGADDR
EOF
if test -n "$USER"; then
cat >> $TEMP << EOF
Cc: $USER
EOF
fi
cat >> $TEMP << EOF
Subject: $SUBJECT
--------
Bug report for Octave $VERSION configured for $MACHINE

Description:
-----------

  * Octave stops recognizing packages and files in its path with out
    much warning. It may have to do with the error handling system,
    but I can't be sure.

    The main problem is that there appears to be no reliable way to
    cause this to happen, even with the same input. It happens between,
    1 out of 5 and 1 out of 10 times, depending on the day or hour.

    Windows XP machine. Pentium 4.

Repeat-By:
---------

  *

        1) Compile qpParseSum.c
        2) run testQpLoadAnalyzeData(true, true)
                This should cause an error no matter what on input.
                When it asks you to pick a column just pick UTC each time, case matters.
                Occasionally this causes Octave to stop recognizing files.

%%%%%%%%%%%%%%%%%%%%%%%
% You'll need to compile this as a mex file for octave
% qpParseSum.c
%%%%%%%%%%%%%%%%%%%%%%%

#include "mex.h"
#include "stdio.h"
#include "stdlib.h"
#include "string.h"

// Error Codes
#define NOERROR 0
#define CELLNOTSTRING 1
#define ELEMENTNOTSTRING 2
#define UNBALANCEDINPUTOUTPUT 3
#define MISSINGOUTPUT 4
#define UNDEFINEDCOLUMN 5
#define FILENOTFOUND 6
#define COLUMNNOTFOUND 7
#define BADSUMMARYFILE 8

//  Files to open
FILE *sumFile, *typeFile;
long lSize; // Storeing size of file

// Different stages of File parsing
char *typeBuffer, *sumBuffer, *noTrash, *line, *token, *errBuffer;
char *columnName, *fileLocation, *columnNameNoEndl;
char *char13;
char **argv, **lines;

size_t result;
int cell_dims[2];
int nlhs1, nrhs1;
int matlabSuccesful;
// Details for columns

mxArray *rhs[1], *lhs[1];int nlhs1, nrhs1;
mxArray *cellArr, *charArr;

long double dataCell; // for converting to numbers

double *pr;
int *neededColumns;
double *columnTypes;
long double *output;

int argc, len;
int k, ncell; // For Input
int numberOfRows, numberOfColumns; // Of Entire File
int i,j; // Iteration vars
int inComment, firstLine, found; // Control Vars (are we in a comment) (is this the first line?);
int index; // index for cell arrays

int justHead, errorStatus;

/*************************************************************************************
//  mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
//
// Opens a summary file and parses columns into double arrays for OCTAVE if it is a number and cell strings the column is a string
//
// ARGUMENTS:
// nlhs - The number of left hand side arguments that will be returned to the calling Octave code
// plhs[] - An array of pointers to the data being returned should have a size of nlhs
// nrhs - The number of right hand sides passed in for use by the calling Octave code
// prhs[] - An array of pointers to the data being passed in, having a size of nrhs. The data being pointed to is read only.
//
// A note on mxCalloc & mxMalloc: Matlab & Octave free these vars automatically when return to the prompt
// occurs. Still free them though, when you are done with them.
***********************************************************************************/

void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) {
  matlabSuccesful = 0;
 
  errorStatus = NOERROR;
  argc = 0;
  j = 0;
  justHead = 0;
 
  /*****************************************************
       *
       * Verify all data passed in is string data
       *
       *****************************************************/
 
  for(k=0; k<nrhs; k++)
  {
    if( mxIsCell( prhs[k] ) )
    {
      argc += ncell = mxGetNumberOfElements( prhs[k] );
      for( i=0; i<ncell; i++ ) {
        if( !mxIsChar( mxGetCell( prhs[k], i ) ) ) {
          errorStatus = CELLNOTSTRING;
          goto out;
        }
      }
    }
    else
    {
      argc++;
      if( !mxIsChar( prhs[k] ) ) {
        errorStatus = ELEMENTNOTSTRING;
        goto out;
      }
    }
  }  

  if(nlhs != argc - 1) {
    if(nlhs == 1 && argc == 1) {
      justHead = 1;  
    } else {
      errorStatus = UNBALANCEDINPUTOUTPUT;
      goto out;      
    }
  }
 
  if(nlhs == 0) {
    errorStatus = MISSINGOUTPUT;
    goto out;
  }
 
  argv = (char **) mxCalloc( argc, sizeof(char *) );  

  /***************************************************
       *
       *  Convert data stored in cells to strings
       *
       ***************************************************/
 
  for( k=0; k<nrhs; k++ )
  {
    if( mxIsCell( prhs[k] ) )
    {
        ncell = mxGetNumberOfElements( prhs[k] );
        for( i=0; i<ncell; i++ )
            argv[j++] = mxArrayToString( mxGetCell( prhs[k], i ));
    }
    else
    {
        argv[j++] = mxArrayToString( prhs[k] );
    }
  }
 
  /**************************************************************************************************
       *
       *  Octave has array types of cellstr and double. If a column is numeric want to store the data as numeric data.
       *  Call summaryColumnLookUpType to get the type of each column for later conversion.
       *  String is represented by 1, Numeric by 0.
       *
       **************************************************************************************************/

  columnTypes = (double*) mxMalloc (sizeof(double)*(argc-1)); // 0 = Number, 1 = String
 
  char13 = (char*)mxCalloc(2,sizeof(char));
  k = sprintf(char13, "%c", 13);
 
  for(k = 0; k < argc - 1; k++) {
    columnName = (char*)mxCalloc(strlen(argv[k+1])+10,sizeof(char));
   
    strcpy(columnName, argv[k+1]);  
   
    columnNameNoEndl = strtok(columnName,char13);    
    strcpy(argv[k+1], columnNameNoEndl);
   
    nrhs1 = 1;
    nlhs1 = 1;
    rhs[0] = mxCreateString(columnNameNoEndl);    
   
    /* Note: this call is represented the same way as mexFunction.*/
   
    mexSetTrapFlag(1);
    matlabSuccesful = mexCallMATLAB(nlhs1, lhs, nrhs1, rhs, "summaryColumnLookUpType");
    mexSetTrapFlag(0);
    if(matlabSuccesful) {
      errorStatus = UNDEFINEDCOLUMN;
      goto out;
    }    
   
   
    pr = mxGetPr(lhs[0]);
    columnTypes[k] = *pr;
    mxFree(pr);
    mxFree(columnName);
  }
  mxFree(char13);
 
  /*******************************************************************************
      *
      *   Read in and parse the summary file. Remove any element known that it won't be needed
      *   to return. (linefeed, comments...)
      *
      ******************************************************************************/

  sumFile = fopen(argv[0], "rb");
 
  if (sumFile==NULL) { errorStatus = FILENOTFOUND; goto out; }
 
  fseek(sumFile, -4, SEEK_END);
  lSize = ftell(sumFile);
  rewind(sumFile);

  // sumBuffer holds data directly from file
  sumBuffer = (char*) mxCalloc (lSize+1,sizeof(char));
  // noTrash will not have comments or extra linebreaks
  noTrash = (char*) mxCalloc (lSize+1,sizeof(char));
  result = fread (sumBuffer,1,lSize,sumFile); // read in the data
 
  fclose(sumFile);
 
  j = 0;
 
  inComment = 0; // Have we found a comment?
  firstLine = 0; // Is this the first line?
  numberOfRows = 0;
  numberOfColumns = 0;
  for(i = 0; i < lSize; i++) {
    if(inComment == 0) {
      if(sumBuffer[i] == '#') {
        inComment = 1;
       continue;
      }
      if(sumBuffer[i] == '\n') {
        numberOfRows++;
        if(firstLine == 1) {        
          firstLine = 2; // Can't be first line anymore
        }
      }
      if(sumBuffer[i] == 13) continue; // Don't add char(13) to noTrash
     
      if(firstLine == 0 && sumBuffer[i] == ',') {
        numberOfColumns++;
        firstLine = 1; // This is the first comma, so this is the first line of data
      }
      if(firstLine == 1 && sumBuffer[i] == ',') {
        numberOfColumns++;
      }
      noTrash[j] = sumBuffer[i]; // this byte wasn't trash
     
      j++; // Current position in noTrash updated
    } else {
     
      if(sumBuffer[i] == '\n') inComment = 0; //Comment must be over, found a new line.
    }
  }

  /******************************************************************************
      *   Split the file first into lines and then into cols, thereby storing each piece of data in a different cell of a cell matrix.
      *   This makes it simple to just grab the appropriate column of data and place it in nlhs[]. This block of code also locates
      *   the columns that were requested in the original data.
      *******************************************************************************/
 
  lines = (char**) mxCalloc(numberOfRows,sizeof(char*));
  lines[0] = (char *)strtok(noTrash,"\n");
  for(i = 1; i < numberOfRows; i++) {
   lines[i] = (char *)strtok(NULL,"\n");
  }

  cell_dims[0] = numberOfRows;
  cell_dims[1] = numberOfColumns;

  neededColumns = (int*) mxMalloc (sizeof(int)*(argc-1));
 
  for(k = 0; k < argc-1; k++) {
    neededColumns[k] = -1;
  }
 
  cellArr = mxCreateCellArray(2,cell_dims);

  for(i = 0; i < numberOfRows; i++) {
    line = lines[i];
    token = (char*)strtok(line," ,");
    for(j = 0; j < numberOfColumns; j++) {
      if(token == NULL) {
        errorStatus = BADSUMMARYFILE;
        goto out;
      }
      charArr = mxCreateString(token);
      // Cell arrays are indexed based column major order
      index = calcIndex(numberOfRows,i,j);
      mxSetCell(cellArr,index,charArr);
     
      if(i == 0) {
        //Looking at first row. Find columns requested by user
        for(k = 0; k < argc - 1; k++) {
          if(strcmp(argv[k+1],token) == 0) {
            neededColumns[k] = j;
          }
        }
      }
      if(j != numberOfColumns) { // If still looking, keep tokening
        token = (char *)strtok(NULL," ,");
      }
    }
  }
 
  for(k = 0; k < argc-1; k++) {
    if(neededColumns[k] == -1) {
      errorStatus = COLUMNNOTFOUND;
      goto out;
    }
  }
 
  /***************************************************************************************
       * Place Data into PLHS[], the return array.
       *  - Convert each column to the type discovered earlier.
       *  - Place each column in PLHS
       *
       *  Numbers are stored as Double arrays. Strings are stored Cellstr arrays.
       *  For an unknown reason, must convert back to string to store in a new cell array, otherwise data is corrupted.
       *  Use long doubles to grab all of the data, in C you lose information with just %f,e,g
       ***************************************************************************************/
 
  if(justHead) {
    cell_dims[0] = 1;
    cell_dims[1] = numberOfColumns;
   
    plhs[0] = mxCreateCellArray(2,cell_dims);
   
    for(k = 0; k < numberOfColumns; k++) {      
      charArr = mxGetCell(cellArr, calcIndex(numberOfRows,0,k));
      token = (char*)mxCalloc(mxGetNumberOfElements(charArr)+1,sizeof(char));      
      mxGetString(charArr, token, mxGetNumberOfElements(charArr)+1);
      charArr = mxCreateString(token);
      mxSetCell(plhs[0],calcIndex(1,0,k),charArr);      
    }  
  } else {
    cell_dims[0] = numberOfRows-1;
    cell_dims[1] = 1;
    for(k = 0; k < argc-1; k++) {
      if(!(columnTypes[k] > 0)) { // 0 means numeric
        plhs[k] = mxCreateDoubleMatrix(numberOfRows-1,1,mxREAL);
        for(i = 1; i < numberOfRows; i++) {
          charArr = mxGetCell(cellArr, calcIndex(numberOfRows,i,neededColumns[k]));
          token = (char*)mxCalloc(mxGetNumberOfElements(charArr)+1,sizeof(char));
          mxGetString(charArr, token,mxGetNumberOfElements(charArr)+1);
          sscanf(token,"%Lf",&dataCell);
          output = (long double*) mxGetPr(plhs[k]);
          mxFree(token);
          output[i-1] = dataCell;
        }
      } else { // 1 meant string
        plhs[k] = mxCreateCellArray(2,cell_dims);
        for(i = 1; i < numberOfRows; i++) {
          charArr = mxGetCell(cellArr, calcIndex(numberOfRows,i,neededColumns[k]));
          token = (char*)mxCalloc(mxGetNumberOfElements(charArr)+1,sizeof(char));
          mxGetString(charArr, token,mxGetNumberOfElements(charArr)+1);
          charArr = mxCreateString(token);
          mxSetCell(plhs[k],i-1,charArr);
          mxFree(token);
        }
      }
    }
  }

  // CLEAN  UP

out:
  switch(errorStatus) {
    case COLUMNNOTFOUND:
      len = strlen(argv[0]) + 130;
      errBuffer =(char*) mxCalloc(len, sizeof(char));            
      sprintf(errBuffer, "qpParseSum was unable to find requested column, '%s', in '%s'  Verify all requested columns exist in the summary file.",argv[k+1],argv[0]);      
    case BADSUMMARYFILE:
      mxFree(lines);
      mxFree(sumBuffer);
      mxFree(noTrash);
      mxFree(neededColumns);      
    case FILENOTFOUND:
      if(errorStatus == FILENOTFOUND) {
        len = strlen(argv[0])*2 + 200;
        errBuffer =(char*) mxCalloc(len, sizeof(char));            
        sprintf(errBuffer, "qpParseSum expects a file that exists, but %s does not exist in the path, and can not be opened. Check that %s is spelled correctly and that it exists in your current directory.", argv[0],argv[0]);
      }
    case UNDEFINEDCOLUMN:
      mxFree(columnTypes);
      if( argc ) {
        for( j=argc-1; j>=0; j-- )
          mxFree( argv[j] );
        mxFree( argv );
      }      
    case MISSINGOUTPUT:
    case UNBALANCEDINPUTOUTPUT:
    case ELEMENTNOTSTRING:
    case CELLNOTSTRING:
      break;
    case NOERROR:
    default:
      mxFree(lines);
      mxFree(sumBuffer);
      mxFree(noTrash);
      mxFree(neededColumns);
      mxFree(columnTypes);  
  }  
 

  switch(errorStatus) {
    case NOERROR:
      return;    
    case CELLNOTSTRING:      
      mexErrMsgIdAndTxt("qpMEX:FunctionCall:WrongTypeOfInput", "qpParseSum expects all input to be string data but recieved a cell input element that was not a string. Check input values for numeric data.");
      break;    
    case ELEMENTNOTSTRING:
        mexErrMsgIdAndTxt("qpMEX:FunctionCall:WrongTypeOfInput", "qpParseSum expects all input to be string data but recieved an input element that was not a string. Check input values for numeric data.  ");
      break;    
    case UNBALANCEDINPUTOUTPUT:
        mexErrMsgIdAndTxt("qpMEX:FunctionCall:InvalidCall","qpParseSum expects the number of output variables(%d) to be equivalent to the number of columns requested(%d). Check the number of arguments being passed in.  ",nlhs,(argc-1));
      break;    
    case MISSINGOUTPUT:
        mexErrMsgIdAndTxt("qpMEX:FunctionCall:InvalidCall","qpParseSum expects at least one output variable, but recieved none.");
      break;    
    case UNDEFINEDCOLUMN:      
      mexErrMsgIdAndTxt("qpMEX:SummaryFileInteraction:DataNotFound","qpParseSum expected a defined column, but recieved '%s' which has not been defined in summaryColumnLookUpType.m",columnName);
      break;
    case FILENOTFOUND:
      mexErrMsgIdAndTxt("qpMEX:FileInteraction:FileNotFound",errBuffer);              
      break;    
    case COLUMNNOTFOUND:
      mexErrMsgIdAndTxt("qpMEX:SummaryFileInteraction:DataNotFound",errBuffer);
      break;
    case BADSUMMARYFILE:  
      mexErrMsgIdAndTxt("qpMEX:SummaryFileInteraction:EncounteredBadVersion","qpParseSum expected data to exist for each column, but one row was incomplete.");
      break;
    default:
      break;
  }
  return;
}

/*****************************************************************************
// calcIndex - Calculates the desired index for an array stored in column major order
//
//  ARGUMENTS:
//    rows - total number of rows in the table
//    row - row containing element e
//    col - column containing element e
//
//  RETURNS
//    An integer representing the index of a column major order entry of an array in memory
 *****************************************************************************/

int calcIndex(int rows, int row, int col) {
  return rows * col + row;
}

%%%%%%%%%%%%%%%%%%%%% testQpLoadAnalyzeData.m %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

function success = testQpLoadAnalyzeData(verbose,cleanUp)
  more('off');
  if(~exist('verbose')) verbose = false; end;
  if(~exist('cleanUp')) cleanUp = false; end;

  for i = 1:7
      files{i} = sprintf('00_GEN_99_00000%d.sum',i);
  end
  setup(files);
 
  success = false;
  try
    if(verbose)
      for i = 5:7
        t = sprintf('qpLoadAnalyzeData: Bad Calls %d',i);
        fail = true;
        try
          [a b c] = qpLoadAnalyzeData(files{i});
        catch
          fail = false;
        end    
        if(fail)
          qpFailTest(t);
        end
        qpEndTest(t);
      end
    end
   
    try
      for i = 1:4
        t = sprintf('qpLoadAnalyzeData: Perfect Calls %d',i);
        [a b c] = qpLoadAnalyzeData(files{i});
        qpEndTest(t);
      end
    catch
      qpFailTest(t);
    end  
    teardown(files,cleanUp);
    success = true;
  catch    
    teardown(files,cleanUp);
    qpHandleError(mfilename);
    rethrow(lasterror);
  end
end

function setup(files)
  a = (1:100)';
  try
    writeSummaryFile(files{1},{'rangeReference' 'rangeAvg' 'rangeStd'},a,a,a);
    writeSummaryFile(files{2},{'rangeReference' 'phaseAvg' 'phaseStd'},a,a,a);
    writeSummaryFile(files{3},{'rangeReference' 'phaseHFAvg' 'phaseHFStd'},a,a,a);
    writeSummaryFile(files{4},{'rangeReference' 'phaseLFAvg' 'phaseLFStd'},a,a,a);
   
    writeSummaryFile(files{5},{'undefined' 'rangeAvg' 'rangeStd'},a,a,a);
    writeSummaryFile(files{6},{'rangeReference' 'rangeAvg' 'undefined'},a,a,a);
    writeSummaryFile(files{7},{'rangeReference' 'undefined' 'rangeStd'},a,a,a);
  catch
    qpHandleError([mfilename 'setup']);
    teardown(files,cleanUp);
    rethrow(lasterror);
  end
end

function teardown(files,cleanUp)
  if(cleanUp)
    for i = 1:length(files)
      delete(files{i});
    end
  end
end

%%%%%%%%%%%%%%%%%%%%%%%% qpLoadAnalyzeData.m %%%%%%%%%%%%%%%%%%%%%%%%%


function [rangeReference mean standard] = qpLoadAnalyzeData(sumFile)
  try
    header = qpParseSum(char(sumFile));

    if(sum(strcmp(header, 'rangeAvg')) == 1)
      mnStr ='rangeAvg';
      stStr ='rangeStd';
    elseif(sum(strcmp(header, 'phaseAvg')) == 1)
      mnStr = 'phaseAvg';
      stStr = 'phaseStd';
    elseif(sum(strcmp(header, 'phaseLFAvg')) == 1)
      mnStr = 'phaseLFAvg';
      stStr = 'phaseLFStd';
    elseif(sum(strcmp(header, 'phaseHFAvg')) == 1)
      mnStr = 'phaseHFAvg';
      stStr = 'phaseHFStd';
    else
      char(header)
      disp('Unable to find range/phase Mean and Standard Columns\n');
      mnStr = input('Please enter the name of the Mean column: ','s');
      stStr = input('Please enter the name of the Standard column: ','s')

      disp('\n');
      disp('Checking for existance... ');
      if(sum(strcmp(header, mnStr)) == 1)
        disp(sprintf('%s found ... ', mnStr));
      else
        suggestion = ['Check to see if ' sumFile ' is the right file.'];
        error('qp:SummaryFileInteraction:DataNotFound',sprintf('qpLoadAnalyzeData expected %s, but could not find it.', mnStr));
      end

      if(sum(strcmp(header, stStr)) == 1)
        disp(sprintf('%s found ... ', stStr));
      else
        error('qp:SummaryFileInteraction:DataNotFound',sprintf('qpLoadAnalyzeData expected %s, but could not find it.', stStr));
      end
    end

    if(sum(strcmp(header, 'rangeReference')) == 1)
      refStr = 'rangeReference';
    else
      char(header)
      disp('Unable to find reference range data');
      refStr = input('Please enter the name of the Reference column: ','s');

      if(sum(strcmp(header, refStr)) == 1)
        disp(sprintf('%s found ... ', refStr));
      else
        error('qp:SummaryFileInteraction:DataNotFound',sprintf('qpLoadAnalyzeData expected %s, but could not find it.', refStr));
      end
    end

    cols{1} = refStr;
    cols{2} = mnStr;
    cols{3} = stStr;

    [rangeReference mean standard] = qpParseSum(sumFile, refStr, mnStr, stStr);

    return
  catch
    if(exist('suggestion'))
      qpHandleError(mfilename,suggestion);
    else
      qpHandleError(mfilename);
    end

    rethrow(lasterror);
  end
end


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% qpEndTest.m %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% qpEndTest
%   Call to end a test case. Prints to the terminal that the test is
%   over and that it passed. Prints the time it took to run the test.
%
% ARGUMENTS
%   NONE
%
% RETURNS
%   NONE
%
% DEPENDENCIES
%   Call qpSetCurrentTest before this function is called.
%
%   Original Author: Brian Kirklin
%   $Id:  $
%   (c) Copyright Quantapoint Inc. All rights reserved.
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

function qpEndTest(name)
  fprintf(['> ' name]);
  for i = 1:66 - length(name)
    fprintf('.');
  end  
  disp(['<<<[ P ]>>>']);
end

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% qpFailTest.m %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

function qpFailTest(name)

  lasterror
 
  fprintf(['> ' name]);
  for i = 1:66 - length(name)
    fprintf('.');
  end  
  disp(['<  ! F !  >']);
  error('qp:TestingSoftware:FailedTestCase',lasterror.message);  
end

%%%%%%%%%%%%%%%%%%%%%%%% writeSummaryFile.m %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

function [succeed] = writeSummaryFile(filename, header, varargin)

  try
    succeed = false;

    if(nargin < 3)
      error('qp:FunctionCall:InvalidCall',sprintf('writeSummaryFile expects at least 3 inputs and recieved %d', nargin));
    end

    if(nargout > 1)
      error('qp:FunctionCall:InvalidCall',sprintf('writeSummaryFile expects no more than one output variable and recieved %d', nargout));
    end

    if(~ischar(filename))
      suggestion = 'Check that first argument is a string. [ischar()]';
      error('qp:FunctionCall:WrongTypeOfInput','writeSummaryFile expects filename to be a string, but it was not a string');
    end

    % Check number of input arguments
    columns = max(size(header(1, :)));
    nvarargin = nargin-2;
    if (columns ~= nvarargin)
      error('qp:FunctionCall:InvalidCall',sprintf('writeSummaryFile expects the number of column names columns and number of columns of data to be the same, but recieved %d column name(s) and %d column(s) of data', columns, nvarargin));
    end


    % check vectors
    for it = 1:nvarargin
      sz = size(varargin{it});
      if(~isvector(varargin{it}) || sz(2) ~= 1)
        suggestion = 'Check that each array of data is a vector. [isvector()]';
        error('qp:FunctionCall:WrongTypeOfInput','writeSummaryFile expects each column of data to be a verticle vector, but at least one was not.')
      end
    end

    % combine data into 1 array
    data = varargin{1};
    data(:, 1) = 0;
    for col = 1:columns
      data(:, col+1) = varargin{col};
    end

    % Write file
    [fid msg] = fopen(filename, 'wb');

    if(fid == -1)
      if(isequal(msg,'Permision denied'))
        suggestion = ['Check that ' filename ' is not read only'];
        error('qp:FileInteraction:CouldNotWriteToFile',sprintf('writeSummaryFile expected %s to be writable, but it was protected.',filename));
      elseif(isequal(msg,'Invalid argument'))
        suggestion = ['Check that ' filename ' contains valid characters, is not protected, and is a legal name for a file on your operating sytem.'];
        error('qp:FileInteraction:UnableToOpen',sprintf('writeSummaryFile could not open %s because an argument was incorrect',filename));
      else
        suggestion = ['Check that ' filename ' contains valid characters, is not protected, and is a legal name for a file on your operating sytem.'];
        error('qp:FileInteraction:UnableToOpen',sprintf('writeSummaryFile expected to be able to open %s but could not for an unknown reason.',filename))
      end
    end

    % Write UTC and then rest of header
    fprintf(fid, '%s, ', 'UTC');
    for currentColumn = 1:columns
      if(currentColumn~=columns)
        fprintf(fid, '%s, ', char(header(1, currentColumn)));
      else
        fprintf(fid, '%s', char(header(1, currentColumn)));
      end
    end
    fprintf(fid, '%s\r\n', '');

    % Discover what format to write data
    format = '';
    for currentColumn = 1:columns+1
      if(isa(data(:, currentColumn), 'int32'))
        type = '%d';
      elseif(isa(data(:, currentColumn), 'float'))
        type = '%0.7e';
      elseif(isa(data(:, currentColumn), 'double'))
        type = '%0.7e';
      elseif(isa(data(:, curruntColumn), 'numeric'))
        type = '%0.7e';
      else
        type = '%s';
      end

      if(currentColumn~=columns+1)
        format =[format type ', '];
      else
        format =[format type '\r\n'];
      end
    end

    % Write the data
    fprintf(fid, format, data');

    % Close the file
    fclose(fid);

    % Does the file exist?
    if(exist(filename) ~= 2)
      return;
    end

    qpCrc32(filename,'w');
    if(~qpCrc32(filename,'c'))
      return;
    end

    succeed = true;
  catch

    if(exist('suggestion'))
      qpHandleError(mfilename, suggestion);
    else
      qpHandleError(mfilename);
    end

    if(exist('fid'))
      if(isequal(ferror(fid),''))
        fclose(fid);
      end
    end

    rethrow(lasterror);
  end
  return
end

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% qpStandardErrorSuggestions.m %%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%55%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% qpStandardErrorSuggestions(id)
%   Gets a standard suggestion on how to fix an error, for a standard
%   Quantapoint error identifier
%  
%   ARGUMENTS:
%     id - a standard Quantapoint identifier of format 'qp:task:error'.
%
%   RETURNS:
%     sug - a standard suggestion for a standard error identifier.
%
%   ADDING NEW SUGGESTIONS:
%     Add the related identifier to the identifier list 'identifiers'
%     Add the related suggestion to the suggestion list 'suggestions'
%
%   MATLAB/Octave portability unconfirmed
%
% Original Author: Brian Kirklin
% $Id: $
% (c) Copyright Quantapoint Inc. All rights reserved.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

function sug = qpStandardErrorSuggestions(id)
 
  n = nextNum;  
  identifiers{n} = 'qp:FileInteraction:UnableToOpen';
  suggestions{n} = 'Verify that the file is not protected, exists in the specified path, and contains no illigal characters in its name.';

  n = nextNum;  
  identifiers{n} = 'qp:FileInteraction:FileNotFound';
  suggestions{n} = 'Check your working directory for the file.';

  n = nextNum;  
  identifiers{n} = 'qp:FileInteractionCouldNotWriteToFile';
  suggestions{n} = 'Check file for protection, check working directory for file.';

  n = nextNum;  
  identifiers{n} = 'qp:FileInteraction:CouldNotReadFromFile';
  suggestions{n} = 'Check file for protection, check working directory for file.';  
 
  n = nextNum;  
  identifiers{n} = 'qp:FileInteraction:CouldNotCloseFile';
  suggestions{n} = 'Check for existance of file in working directory, free space on current device, etc.';  
 
  n = nextNum;  
  identifiers{n} = 'qp:SummaryFileInteraction:RequestedUndefinedColumn';
  suggestions{n} = 'Check that the right kind of file was passed in as the desired variable was not present.';
 
  n = nextNum;  
  identifiers{n} = 'qp:SummaryFileInteraction:RequestedAbsentColumn';
  suggestions{n} = 'Check that the right kind of file was passed in as the desired variable was not present.';
 
  n = nextNum;  
  identifiers{n} = 'qp:SummaryFileInteraction:EncounteredBlankLine';
  suggestions{n} = 'Check last opened summary file for a blank line. Blank lines are not allowed in summary files.';
 
  n = nextNum;  
  identifiers{n} = 'qp:SummaryFileInteraction:EncounteredBadVersion';
  suggestions{n} = 'Check the format of the last opened summary file to see if its the latest version of summary files.';  
 
  n = nextNum;  
  identifiers{n} = 'qp:SummaryFileInteraction:BadCRC';
  suggestions{n} = 'The data for the last opened summary file was corrupted and is unusable per quantapoint standards. Regenerate the data.';
 
  n = nextNum;  
  identifiers{n} = 'qp:LanguageSpecificTask:NeededOctave';
  suggestions{n} = 'Run this software under Octave.';  
 
  n = nextNum;  
  identifiers{n} = 'qp:LanguageSpecificTask:NeededMatlab';
  suggestions{n} = 'Run this software under Matlab.';
 
  n = nextNum;
  identifiers{n} = 'qp:LanguageSpecificTask:NeededGNUPlog';
  suggestions{n} = 'Turn off the jhandles package. Then, run this software again.';
 
  n = nextNum;
  identifiers{n} = 'qp:LanguageSpecificTask:NeededJhandles';
  suggestions{n} = 'Turn on the jhandles package. Then, run this software again.';
 
  n = nextNum;
  identifiers{n} = 'qp:TestingSoftware:TestCaseFailed';
  suggestions{n} = 'Check the code base for programmer errors.';
 
  clear('nextNum');
 
  idx = strmatch(id, identifiers);
  if(sum(idx) > 0)
    sug = suggestions{idx};
  else
    le = lasterror;
    if(sum(strfind(le.identifier,'qpMEX')))
      nm = strfind(le.message,':');
      sug = ['Review ''help ' le.message((nm(1)+2):(nm(2)-1)) ''''];
    else  
      sug = ['Review ''help ' le.stack.name ''''];
    end
  end
 
  return
end

function n = nextNum;
  persistent NUMCOUNTER;
  if(isempty(NUMCOUNTER))
    NUMCOUNTER = 1;
  else
    NUMCOUNTER = NUMCOUNTER + 1;
  end;
  n = NUMCOUNTER;
  return
end

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% qpCrc32.m %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% result = qpCrc32(file, action)
%
% Takes in a file name, then computes the CRC of the chosen file, using
% the polynomial 0x04c11db7. If the result of the CRC math is 0, then
% the file is assumed to be unaltered since the creation of the original
% CRC.
%
% ARGUMENTS:
%   file    - the name of the file on which to compute the CRC, place in
%            single quotes
%   action  - 'c' --> check crc
%             'w' --> write crc
%
% RETURNS:
%   result - Returns 1 for an ok file.
%
% MATLAB/Octave portability confirmed.
%
% Original Author:
% $Id: qpCrc32.m 8998 2008-06-06 20:28:13Z bkirklin $
% (c) Copyright Quantapoint Inc. All rights reserved.
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

function result = qpCrc32(file, action)
  try
    if(nargin < 2)
      error('qp:FunctionCall:InvalidCall','qpCrc32 expects at least 2 input values and recieved less');
    elseif(nargin > 2)
      error('qp:FunctionCall:InvalidCall','qpCrc32 expects no more than 2 input values and recieved more');
    end

    if(~(strcmp(action,'c') || strcmp(action,'w')))
      error('qp:FunctionCall:WrongTypeOfInput','qpCrc32 expects an action equivalent to ''c'' or ''w'', but recieved some other value.');
    end

    if(exist(file,'file') ~= 2)
      error('qp:FileInteraction:FileNotFound',['qpCrc32 expected to find ' file ' in the working directory, but could not.']);
    end    

    % Polynomial for CRC generation:
    KEY = uint32(79764919); % Integer equivalent of 0x04c11db7
    % Generate Look Up Table
    % Start crc at the decimial equivalent to 0xFFFFFFFF

    key = KEY;
    CRCTABLE = bitshift(uint32(0:255), 24);
    for m = 0:7
      b2 = bitand(CRCTABLE, bitshift(1, 31));
      fn0 = find(b2>0);
      f0 = find(b2==0);
      CRCTABLE(fn0) = bitxor(uint32(bitshift(CRCTABLE(fn0), 1)), key);
      CRCTABLE(f0) = uint32(bitshift(CRCTABLE(f0), 1));
    end

    CRC = uint32(4294967295);

    % Read in file, byte by byte and generate CRC Byte by Byte


    fFile = fopen(file,'rb');
    if(fFile == -1)
      if(isequal(msg,'Permision denied'))
        suggestion = ['Check whether ' file ' is read/write protected'];
        error('qp:FileInteraction:CouldNotReadFromFile',['qpCrc32 expects to be able to read from ' file ', but it was protected.']);
      else
        suggestion = ['Check to see if ' file ' exists.'];
        error('qp:FileInteraction:UnableToOpen',['qpCrc32 expected to be able to open ' file ' but could not.'])
      end
    end

    bytes = fread(fFile);
    fclose(fFile);

    len = length(bytes);
    bytes = uint8(bytes);
    CRCTABLE = uint32(CRCTABLE);

    for i = 1:len
      CRC = bitxor(bitshift(CRC, -8), CRCTABLE(uint32(bitxor(uint8(bitand(CRC, 255)), bytes(i)))+1));
    end

    if(strcmp(action,'c'))
      result = (~CRC);
    elseif(strcmp(action,'w'))
      fOutId = fopen(file, 'ab');

    if(fOutId == -1)
      if(isequal(msg,'Permision denied'))
        suggestion = ['Check whether ' file ' is read/write protected'];
        error('qp:FileInteraction:CouldNotReadFromFile',['qpOldSum2v14 expects to be able to read from ' file ', but it was protected.']);
      else
        suggestion = ['Check to see if ' file ' exists.'];
        error('qp:FileInteraction:UnableToOpen',['qpOldSum2v14 expected to be able to open ' file ' but could not.'])
      end
    end

      fwrite(fOutId, CRC, 'uint32');
      fclose(fOutId);
    end
    return;
  catch
    if(exist('suggestion'))
      qpHandleError(mfilename, suggestion);
    else
      qpHandleError(mfilename);
    end

    rethrow(lasterror);
  end
end

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% qpHandleError.m %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% qpHandleError(functionIdentifier)
%   called whenever an error is caught, this function is responsible for
% printing details to the terminal and controlling what information is printed
% based on what function called it.
%
% ARGUMENTS
%   functionIdentifier - An identifier for the function calling qpHandleError.
%                        If the function calling qpHandleError is a subfunction,
%                        or one of may functions inside of a single file, then
%                        functionIdentifier is equivalent to:
%                           [mfilename ':functionName']
%                        If the function calling qpHandleError is the first
%                        function in its file, then mfilename will suffice.
%
%   suggestions -        This field is optional and is used to define custom
%                        suggestions.
%
%                        A structure which contains all of the error
%                        identifiers and suggestions the calling script could
%                        have thrown. Suggestions should have the following
%                        fields:
%
%                         > ids - Stores all the identifiers
%                         > sug - Stores all of the suggestions
%
%                       ids and sug should be indexed so that matching pairs
%                       have the same index.
%
%   MATLAB/Octave portability unconfirmed.
%
%   Original Author: Brian Kirklin
%   $Id: $
%   (c) Copyright Quantapoint Inc. All rights reserved.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

function qpHandleError(functionIdentifier, suggestion)
  if(nargin <1)
    return
  end
  if(nargin > 2)
    return
  end  
 
  global TESTING_IS_ACTIVE;
  if(isempty(TESTING_IS_ACTIVE))
    TESTING_IS_ACTIVE = false;
    errFileId = 1;
  elseif(TESTING_IS_ACTIVE)
    errFileId = fopen('ErrorLog.txt','a');
    if(errFileId == -1)
      error('omg wtf?');      
    end
  else
    errFileId = 1;
  end
   
  le = lasterror;
  mex = false;
 
  if(~isempty(le.identifier)) % Octave errors do not recieve identifiers      
    if(isequal(le.stack.name,functionIdentifier))    
      if(sum(strfind(le.identifier,'qpMEX')))  
        mex = true;
        idx = strfind(le.message,':');        
        fprintf(errFileId,['>>> An error occured in ' le.message((idx(1)+2):(idx(2)-1)) sprintf('\n')]);
        fprintf(errFileId,['    (' le.identifier ')' sprintf('\n')]);
        edisp(['Error occured because ' le.message((idx(2)+1):(length(le.message)-1)) sprintf('\n')],errFileId,'      ');
      else
        fprintf(errFileId, ['>>> An error occured in ' functionIdentifier sprintf('\n')]);
        fprintf(errFileId, ['    (' le.identifier ')' sprintf('\n')]);
        edisp(['Error occured because ' le.message(8:(length(le.message)-1))],errFileId,'      ');
      end

      expFound = false;
      if(nargin == 2)
        edisp('Suggestion: ',errFileId,'    ');
        edisp(suggestion,errFileId,'      ');
        expFound = true;
      end

      if(~expFound)
        edisp('Suggestion: ',errFileId,'    ');
        edisp(qpStandardErrorSuggestions(le.identifier),errFileId,'      ');
      end

      edisp([le.identifier ' caused failure in:'],errFileId,'    ');
      if(mex)
        fprintf(errFileId, ['      + ' le.message((idx(1)+2):(idx(2)-1)) sprintf('\n')]);
      end
      fprintf(errFileId, ['      + ' functionIdentifier sprintf('\n')]);
    else
      fprintf(errFileId, ['      + ' functionIdentifier sprintf('\n')]);
    end
  end
  if(TESTING_IS_ACTIVE)
    fclose(errFileId);
  end
end

function edisp(txt,errFileId,prepend)
  a = qpSplit(txt, ' ');
  for j = 1:length(a)
    b(j) = length(a{j});
  end
  msg = [prepend a{1}];
  for j = 2:length(a)
    if(length(msg) + b(j) <= 70)
      msg = [msg ' ' a{j}];
    else
      fprintf(errFileId, [msg sprintf('\n')]);
      msg = [prepend a{j}];
    end
  end
  fprintf(errFileId, [msg sprintf('\n')]);
end

Configuration (please do not edit this section):
-----------------------------------------------

uname output:     $UN
configure opts:   $config_opts
Fortran compiler: $F77
FFLAGS:           $FFLAGS
F2C:              $F2C
F2CFLAGS:         $F2CFLAGS
FLIBS:            $FLIBS
CPPFLAGS:         $CPPFLAGS
INCFLAGS:         $INCFLAGS
C compiler:       $CC_AND_VERSION
CFLAGS:           $CFLAGS
CPICFLAG:         $CPICFLAG
C++ compiler:     $CXX_AND_VERSION
CXXFLAGS:         $CXXFLAGS
CXXPICFLAG:       $CXXPICFLAG
LD_CXX:           $LD_CXX
LDFLAGS:          $LDFLAGS
LIBFLAGS:         $LIBFLAGS
RLD_FLAG:         $RLD_FLAG
BLAS_LIBS:        $BLAS_LIBS
FFTW_LIBS:        $FFTW_LIBS
LIBS:             $LIBS
LEXLIB:           $LEXLIB
LIBGLOB:          $LIBGLOB
SED:              $SED
DEFS:

EOF

if $HAVE_FMT; then
  echo $DEFS | fmt | $SED 's/^/  /' >> $TEMP
else
  echo $DEFS >> $TEMP
fi

if test $# -gt 0; then
  if test -f "$1"; then
    cat >> $TEMP << EOF

User-preferences (please do not edit this section):
--------------------------------------------------

EOF
    cat $1 >> $TEMP
  fi
fi

chmod u+w $TEMP
cp $TEMP $TEMP.x

status=0

editing=true

while $editing; do
  if $EDITOR $TEMP; then
    while $editing; do
      echo $echo_n "(a)bort, (e)dit, (l)ist, (s)end? $echo_c"
      read ans
      case "$ans" in
        a* | A*)
          status=1
          editing=false
        ;;
        e* | E*)
          break;
        ;;
        l* | L*)
          $PAGER $TEMP
        ;;
        s* | S*)
          editing=false
        ;;
      esac
    done
  else
    echo "problems with edit -- no bug report submitted"
    status=1
    editing=false
  fi
done

if test $status -eq 0; then
  if cmp -s $TEMP $TEMP.x; then
    echo "file not changed -- no bug report submitted"
    status=1
  elif test `wc $TEMP | awk '{print $1}'` -eq 0; then
    echo "empty bug report file -- not submitted"
    status=1
  else

# Try to extract the recipient address, in case the To: line in the
# message template has been changed.  Also get cc: lines.

    TO_ADDR=`$SED -e '/^--------[ \t]*$/q' $TEMP | $SED -n -e 's/^[Tt][Oo]://p'`
    CC_ADDR=`$SED -e '/^--------[ \t]*$/q' $TEMP | $SED -n -e 's/^[Cc][Cc]://p'`

    if test -z "$TO_ADDR"; then
      echo "no valid \`To:' field found in header -- using $BUGADDR instead"
    else
      BUGADDR="$TO_ADDR"      
    fi

    BUGADDR="$BUGADDR $CC_ADDR"

    TMP_SUB=`$SED -e '/^--------[ \t]*$/q' $TEMP | $SED -n -e 's/^Subject://p'`

    if test -n "$TMP_SUB"; then
      SUBJECT="$TMP_SUB"
    fi

# Delete the `--------' separator in the message.

# Don't pretty-print this.  Odd whitespace kills Ultrix AWK!

    awk 'BEGIN{in_header=1;} /^--------[ \t]*$/ {
      if (in_header) { in_header=0; print ""; next; }
    } { print $0; }' $TEMP > $TEMP.x

# Now try to mail it.

    # indicate that we have not yet sent email successfully
    status=11

    if test $status -ne 0; then
      ( mailx -s "$SUBJECT" $BUGADDR < $TEMP.x ) > /dev/null 2>&1
      status=$?
      if test $status -ne 0; then
        ( Mail -s "$SUBJECT" $BUGADDR < $TEMP.x ) > /dev/null 2>&1
        status=$?
        if test $status -ne 0; then
          ( /usr/ucb/mail -s "$SUBJECT" $BUGADDR < $TEMP.x ) > /dev/null 2>&1
          status=$?
          # make /bin/mail our last resort -- it ignores the subject line
          if test $status -ne 0; then
            ( /bin/mail $BUGADDR < $TEMP.x ) > /dev/null 2>&1
            status=$?
            if test $status -ne 0; then
              echo "unable to send mail..."
            fi
          fi
        fi
      fi
    fi
  fi
fi

if test $status -ne 0; then
  dead_bug_file=$HOME/dead-octave-bug
  looking_for_file=true;
  n=1
  while $looking_for_file; do
    if test -f "$dead_bug_file-$n"; then
      n=`expr $n + 1`
    else
      looking_for_file=false
      dead_bug_file=$dead_bug_file-$n
    fi
  done
  echo "saving message in $dead_bug_file";
  cat $TEMP >> $dead_bug_file;
  exit 1
else
  echo "bug report sent to: $TO_ADDR"
  echo "             cc to: $CC_ADDR"
fi

exit $status

[gOS]

File might have helped... myFile.dat

gOS wrote:

stuff

[David Bateman]

gOS wrote:

> I've not sat through all the way, but on 2 seperate windows machines, one
> with a pentium 4 and one with a much older processor, I get similar results.
>
> I can confirm that in 3.0.0 the script took about 5-10 seconds to complete.
>
> In 3.0.1 I've sat for 10 minutes waiting for csvread to finish and it
> hasn't. Ctrl+c causes Octave to exit without warning and does not return
> control to the prompt.
>
> I'm reading in 3mb of text. I've attached said file.
>
> Command that hangs indefinately:
>
> data = csvread('myFile.dat');
>
> Help would be appreciated. I've confirmed that rolling back to 3.0.0 fixes
> the problem. I'm using the most up to date version from the Octave Forge
> site. Can anyone confirm that this is an Octave problem and not specific to
> windows? Can anyone one fix it? Thank you.
>
> I couldn't get a bug report to generate after calling csvread because it
> never finished. Here's my bug report for just starting up Octave though.
> Maybe it will help. :-/
>  

The change between the two versions was essentially to change the
C-string handling to C++ string handling. It also more correctly
duplicates the Matlab interface. This was done for this codes inclusion
in Octave itself.

Now the slow up appears impressive, and I'd like to see this situation
improved. However looking at the new dlmread.cc file used by csvread.m,
I can't see any obvious reason for the slow-up. Unfortuantely I don't
really have the time to look at this in the near future, so if the old
dlmread.cc was fine with your code I'd suggest getting it from

http://octave.svn.sourceforge.net/viewvc/*checkout*/octave/trunk/octave-forge/main/io/src/dlmread.cc?revision=4585

and using it instead. Alternatively, you might look at the new
dlmread.cc function and see how you might fix the slow-up issues.

D.

--
David Bateman                                David.Bateman@...
Motorola Labs - Paris                        [phone] (Ph)
Parc Les Algorithmes, Commune de St Aubin    [phone] (Mob)
91193 Gif-Sur-Yvette FRANCE                  [phone] (Fax)

The information contained in this communication has been classified as:

[x] General Business Information
[ ] Motorola Internal Use Only
[ ] Motorola Confidential Proprietary

_______________________________________________
Bug-octave mailing list
Bug-octave@...
https://www.cae.wisc.edu/mailman/listinfo/bug-octave

[John W. Eaton]

On 10-Jul-2008, David Bateman wrote:

| gOS wrote:
| > [...]
| Now the slow up appears impressive, and I'd like to see this situation
| improved. However looking at the new dlmread.cc file used by csvread.m,
| I can't see any obvious reason for the slow-up. Unfortuantely I don't
| really have the time to look at this in the near future, so if the old
| dlmread.cc was fine with your code I'd suggest getting it from
|
| http://octave.svn.sourceforge.net/viewvc/*checkout*/octave/trunk/octave-forge/main/io/src/dlmread.cc?revision=4585
|
| and using it instead. Alternatively, you might look at the new
| dlmread.cc function and see how you might fix the slow-up issues.

I might also be willing to look at this problem if you can create a
simple example that only requires a data file and calling dlmread to
read it.

jwe
_______________________________________________
Bug-octave mailing list
Bug-octave@...
https://www.cae.wisc.edu/mailman/listinfo/bug-octave

[gOS]

John W. Eaton wrote:

> On 10-Jul-2008, David Bateman wrote:
>
> | gOS wrote:
> | > [...]
> | Now the slow up appears impressive, and I'd like to see this situation
> | improved. However looking at the new dlmread.cc file used by csvread.m,
> | I can't see any obvious reason for the slow-up. Unfortuantely I don't
> | really have the time to look at this in the near future, so if the old
> | dlmread.cc was fine with your code I'd suggest getting it from
> |
> | http://octave.svn.sourceforge.net/viewvc/*checkout*/octave/trunk/octave-forge/main/io/src/dlmread.cc?revision=4585
> |
> | and using it instead. Alternatively, you might look at the new
> | dlmread.cc function and see how you might fix the slow-up issues.
>
> I might also be willing to look at this problem if you can create a
> simple example that only requires a data file and calling dlmread to
> read it.
>
> jwe
>
>  

I believe I included such a file in my second response.

It is uploaded to Nabble at this address:

http://www.nabble.com/file/p18384313/myFile.dat

Single line: data = csvread('myFile.dat')

_______________________________________________
Bug-octave mailing list
Bug-octave@...
https://www.cae.wisc.edu/mailman/listinfo/bug-octave

[gOS]

David Bateman wrote:
> http://octave.svn.sourceforge.net/viewvc/*checkout*/octave/trunk/octave-forge/main/io/src/dlmread.cc?revision=4585
>  
I'd be willing to try this out if I had an idea on how to compile an
Octave source file on a windows machine and just replace the file.

Would I just use a C++ compiler?
Would this involve using mkOctFile?

Is there a page in the documentation that I missed that would give me
all these instructions?

I believe I tried to compile Octave once before on a Windows machine and
received a very large headache.

Also, If I were to install Octave 3.0.0 somewhere, could I just grab the
old file out of its directories and replace the new one?

Thanks
_______________________________________________
Bug-octave mailing list
Bug-octave@...
https://www.cae.wisc.edu/mailman/listinfo/bug-octave

[John W. Eaton]

On 10-Jul-2008, Brian Kirklin wrote:

| I believe I included such a file in my second response.
|
| It is uploaded to Nabble at this address:
|
| http://www.nabble.com/file/p18384313/myFile.dat
|
| Single line: data = csvread('myFile.dat')

OK, you confused me with all that extra code at the end of your
message.  What was that for?

In any case, I can confirm the problem.

BTW, for a file this simple, you can read it with

  data = load ("myFile.dat");

jwe
_______________________________________________
Bug-octave mailing list
Bug-octave@...
https://www.cae.wisc.edu/mailman/listinfo/bug-octave

[John W. Eaton]

On 10-Jul-2008, David Bateman wrote:

| The change between the two versions was essentially to change the
| C-string handling to C++ string handling. It also more correctly
| duplicates the Matlab interface. This was done for this codes inclusion
| in Octave itself.

It looks like there was more than that in going from revision 4585 (a
version attributed to Kai Habel) to 4605 (a version attributed to
Jonathon Stickel).  In the current version, I think the problem is
that the output matrix is resized each time a new row is encountered
(or, if a line has more columns than are currently in the matrix).
For the example file provided, that means approximately 180000 calls
to resize, with most of these just adding three elements to the
matrix.  I think we need something better than that...  I don't have
time to work on it at the moment, but would consider a patch.

jwe
_______________________________________________
Bug-octave mailing list
Bug-octave@...
https://www.cae.wisc.edu/mailman/listinfo/bug-octave

[David Bateman]

John W. Eaton wrote:

> On 10-Jul-2008, David Bateman wrote:
>
> | The change between the two versions was essentially to change the
> | C-string handling to C++ string handling. It also more correctly
> | duplicates the Matlab interface. This was done for this codes inclusion
> | in Octave itself.
>
> It looks like there was more than that in going from revision 4585 (a
> version attributed to Kai Habel) to 4605 (a version attributed to
> Jonathon Stickel).  In the current version, I think the problem is
> that the output matrix is resized each time a new row is encountered
> (or, if a line has more columns than are currently in the matrix).
> For the example file provided, that means approximately 180000 calls
> to resize, with most of these just adding three elements to the
> matrix.  I think we need something better than that...  I don't have
> time to work on it at the moment, but would consider a patch.
>
> jwe
>
>  

No it should only resize if the size of the output matrix changes size
as there are additional columns added, at least that was the idea..

D.


--
David Bateman                                David.Bateman@...
Motorola Labs - Paris                        [phone] (Ph)
Parc Les Algorithmes, Commune de St Aubin    [phone] (Mob)
91193 Gif-Sur-Yvette FRANCE                  [phone] (Fax)

The information contained in this communication has been classified as:

[x] General Business Information
[ ] Motorola Internal Use Only
[ ] Motorola Confidential Proprietary

_______________________________________________
Bug-octave mailing list
Bug-octave@...
https://www.cae.wisc.edu/mailman/listinfo/bug-octave

[John W. Eaton]

On 10-Jul-2008, David Bateman wrote:

| No it should only resize if the size of the output matrix changes size
| as there are additional columns added, at least that was the idea..

It starts with rdata (and cdata) empty, then resizes anytime the
current row or column is larger than the size of rdata (or cdata).

I'd recommend starting with something like 100x100 and then only
resizing if those bounds are exceeded, and resizing by something like
another 100 rows or 100 columns (whichever is needed).  Then in the
end, resizing once more to the actual max number of rows and columns
found in the file.  That should improve performance somewhat, but
perhaps someone can come up with a better solution.

jwe

_______________________________________________
Bug-octave mailing list
Bug-octave@...
https://www.cae.wisc.edu/mailman/listinfo/bug-octave

[Dmitri A. Sergatskov]

On Thu, Jul 10, 2008 at 11:28 AM, John W. Eaton <jwe@...> wrote:

> On 10-Jul-2008, David Bateman wrote:
>
> | No it should only resize if the size of the output matrix changes size
> | as there are additional columns added, at least that was the idea..
>
> It starts with rdata (and cdata) empty, then resizes anytime the
> current row or column is larger than the size of rdata (or cdata).
>
> I'd recommend starting with something like 100x100 and then only
> resizing if those bounds are exceeded, and resizing by something like
> another 100 rows or 100 columns (whichever is needed).  Then in the
> end, resizing once more to the actual max number of rows and columns
> found in the file.  That should improve performance somewhat, but
> perhaps someone can come up with a better solution.
>

Double pass (first to figure out the size of the matrix, second to do
the actual parsing and filling in the array) on the file is out of
consideration?

> jwe
>

Dmitri.
--
_______________________________________________
Bug-octave mailing list
Bug-octave@...
https://www.cae.wisc.edu/mailman/listinfo/bug-octave

[Bill Denney-5]

Dmitri A. Sergatskov wrote:

> On Thu, Jul 10, 2008 at 11:28 AM, John W. Eaton <jwe@...> wrote:
>  
>> I'd recommend starting with something like 100x100 and then only
>> resizing if those bounds are exceeded, and resizing by something like
>> another 100 rows or 100 columns (whichever is needed).  Then in the
>> end, resizing once more to the actual max number of rows and columns
>> found in the file.  That should improve performance somewhat, but
>> perhaps someone can come up with a better solution.
>
> Double pass (first to figure out the size of the matrix, second to do
> the actual parsing and filling in the array) on the file is out of
> consideration?

My solution in similar situations before has been to stat the file
getting its size, read the first two rows (to allow for a possible
header), use the number of bytes in the second row as an estimate of the
number of bytes per row, and estimate the number of rows.  After that,
you just add a fudge factor to the estimate (say add 20%), let it run,
if you need to resize again, you now have a much better idea of the
amount of space you need, so you resize to that new value (using your
current file position and a smaller fudge factor), and adjust it at the end.

If you want to get fancy, you can always make the final estimator more
dynamic, but that worked very well for me.

A somewhat more concrete example that won't work but gets the idea
across is:

fid = fopen(filename);
## This next line doesn't work either, but you get the idea
filesize = stat(fid);
fileleft = filesize;
output = [];
row = 0;
maxrow = 2;
while (there is more of the file left)
    row++;
    ## In reality you would want to add something to verify that the number
    ## of columns has not changed, but changing the number of columns is
    ## relatively rare, so it doesn't need the fancier algorithm
    [output(row,:), bytes] = readcsvline(fid);
    fileleft = fileleft - bytes;
    if (row == 2)
        output = resize (output, ceil (filesize/bytes*1.2), size
(output, 2));
        maxrow = rows (output);
    elseif (row == maxrow)
        maxrows = max (100, ceil (1.1*fileleft/((filesize-fileleft)/rows
(output)));
        output = resize (output, maxrows, size (output, 2));
    endif
endwhile
output(row+1,:) = [];

function [x, bytes] = readcsvline(fid)
## Read the next line of the file, parse it and return the vector (x) for
## the row and the number of bytes in the line
## Assume that there's some code here that does that

Have a good day,

Bill
_______________________________________________
Bug-octave mailing list
Bug-octave@...
https://www.cae.wisc.edu/mailman/listinfo/bug-octave