Solving a large LP with lpsolve in Python

[traviserdman]

Been happily using lpsolve via Python for a few months now.  With each success, my working model has grown in size.  Now, I want to tackle a really large LP (really large in my context, maybe not yours).  The current matrix is 10985 x 2873.  Here is the problem:

>>> shape(a)
(10985, 2873)

>>> type(a)
<type 'numpy.ndarray'>

>>> lp = lp_maker(f,a.tolist(),b,e,vlb,vub,xint,scalemode=True,setminim=True)
alloc of 10096572 'REAL' failed

>>> ================================ RESTART ================================

After this last msg, the Python shell crashes and restarts itself.  Python ver 2.5.4, lpsolve v5.5.0.15, 1.7 GB of RAM.

Questions:

(1) Is there a way to get a model of this size loaded into lpsolve?

(2) Despite the reference guide language to the contrary, I cannot get lpsolve to accept numpy arrays or matrices via lp_maker nor via lpsolve('set_mat',lp,a) ... only seems to accept list of lists.  The "tolist()" construct may be a resource/memory hog, and my suspicion is that it's contributing to the system simply running out of memory ... is there a way to coerce lpsolve to accept the numpy.ndarray natively?

[William H. Patton]

The issue is trying to pass the dense array A; probably as you suspect the
.tolist runs out of memory.

 

It is time to consider using the sparse methods for filling the lp A matrix.

Set_columnex is best.

 

You could make a toSparse() function for columns and change

Lp_maker.py

  m = len(a)

  n = len(a[0])

  lp = lpsolve('make_lp', m, n)

  lpsolve('set_verbose', lp, IMPORTANT)

  lpsolve('set_mat', lp, a)

  lpsolve('set_rh_vec', lp, b)

  lpsolve('set_obj_fn', lp, f)

  lpsolve('set_maxim', lp) # default is solving minimum lp.

 

Replace the red line with a loop on col 1 to m

     a[col].tosparse(count, sparsecolumn, rowno)

   lpsolve('set_columnex', lp, col, count, sparsecolumn, rowno);

 

This will never consume more than the nonzeros of the A matrix new space
before running the solve.

You still might to release the python a columns as you pass them if you head
for even bigger problems.

You can also look for python sparse matrix methods to save initial space.

  _____  

From: lp_solve@... [mailto:lp_solve@...] On Behalf
Of traviserdman
Sent: Tuesday, September 29, 2009 11:38 PM
To: lp_solve@...
Subject: [lp_solve] Solving a large LP with lpsolve in Python

 

 

Been happily using lpsolve via Python for a few months now. With each
success, my working model has grown in size. Now, I want to tackle a really
large LP (really large in my context, maybe not yours). The current matrix
is 10985 x 2873. Here is the problem:

>>> shape(a)
(10985, 2873)

>>> type(a)
<type 'numpy.ndarray'>

>>> lp =
lp_maker(f,a.tolist(),b,e,vlb,vub,xint,scalemode=True,setminim=True)
alloc of 10096572 'REAL' failed

>>> ================================ RESTART
================================

After this last msg, the Python shell crashes and restarts itself. Python
ver 2.5.4, lpsolve v5.5.0.15, 1.7 GB of RAM.

Questions:

(1) Is there a way to get a model of this size loaded into lpsolve?

(2) Despite the reference guide language to the contrary, I cannot get
lpsolve to accept numpy arrays or matrices via lp_maker nor via
lpsolve('set_mat',lp,a) ... only seems to accept list of lists. The
"tolist()" construct may be a resource/memory hog, and my suspicion is that
it's contributing to the system simply running out of memory ... is there a
way to coerce lpsolve to accept the numpy.ndarray natively?

[William H. Patton]

It looks like the other possibility is to change the source of pythonmod.c
and rebuild the interface.

I think changing #if 1 to #if 0  is designed to get lpsolve to accept numpy
arrays or matrices

without the convert tolist.

 

#include "lpsolvecaller.h"

 

#define maxArgs 10

 

#if 1

  /* represent arrays via lists */

# define PyArray_Size PyList_Size

# define PyArray_New  PyList_New

# define PyArray_SET_ITEM PyList_SET_ITEM

# define PyArray_Resize(Array, Size) { /* Printf("PyList_Size(*Array) was
%d, resizing to %d\n", PyList_Size(*Array), Size); */ while
((PyList_Size(*Array) < Size) && (PyList_Append(*Array, Py_None) == 0)); /*
Printf("PyList_Size(*Array) becomes %d\n", PyList_Size(*Array)); */ }

#else

  /* represent arrays via tuples */

# define PyArray_Size PyTuple_Size

# define PyArray_New  PyTuple_New

# define PyArray_SET_ITEM PyTuple_SET_ITEM

# define PyArray_Resize _PyTuple_Resize

#endif

 

  _____  

From: lp_solve@... [mailto:lp_solve@...] On Behalf
Of William H. Patton
Sent: Friday, October 02, 2009 4:26 PM
To: lp_solve@...
Subject: RE: [lp_solve] Solving a large LP with lpsolve in Python

 

 

The issue is trying to pass the dense array A; probably as you suspect the
.tolist runs out of memory.

 

It is time to consider using the sparse methods for filling the lp A matrix.

Set_columnex is best.

 

You could make a toSparse() function for columns and change

Lp_maker.py

  m = len(a)

  n = len(a[0])

  lp = lpsolve('make_lp', m, n)

  lpsolve('set_verbose', lp, IMPORTANT)

  lpsolve('set_mat', lp, a)

  lpsolve('set_rh_vec', lp, b)

  lpsolve('set_obj_fn', lp, f)

  lpsolve('set_maxim', lp) # default is solving minimum lp.

 

Replace the red line with a loop on col 1 to m

     a[col].tosparse(count, sparsecolumn, rowno)

   lpsolve('set_columnex', lp, col, count, sparsecolumn, rowno);

 

This will never consume more than the nonzeros of the A matrix new space
before running the solve.

You still might to release the python a columns as you pass them if you head
for even bigger problems.

You can also look for python sparse matrix methods to save initial space.

  _____  

From: lp_solve@... [mailto:lp_solve@...] On Behalf
Of traviserdman
Sent: Tuesday, September 29, 2009 11:38 PM
To: lp_solve@...
Subject: [lp_solve] Solving a large LP with lpsolve in Python

 

 

Been happily using lpsolve via Python for a few months now. With each
success, my working model has grown in size. Now, I want to tackle a really
large LP (really large in my context, maybe not yours). The current matrix
is 10985 x 2873. Here is the problem:

>>> shape(a)
(10985, 2873)

>>> type(a)
<type 'numpy.ndarray'>

>>> lp =
lp_maker(f,a.tolist(),b,e,vlb,vub,xint,scalemode=True,setminim=True)
alloc of 10096572 'REAL' failed

>>> ================================ RESTART
================================

After this last msg, the Python shell crashes and restarts itself. Python
ver 2.5.4, lpsolve v5.5.0.15, 1.7 GB of RAM.

Questions:

(1) Is there a way to get a model of this size loaded into lpsolve?

(2) Despite the reference guide language to the contrary, I cannot get
lpsolve to accept numpy arrays or matrices via lp_maker nor via
lpsolve('set_mat',lp,a) ... only seems to accept list of lists. The
"tolist()" construct may be a resource/memory hog, and my suspicion is that
it's contributing to the system simply running out of memory ... is there a
way to coerce lpsolve to accept the numpy.ndarray natively?

[Peter Notebaert-2]

I don't remember the code so good and the descision to use lists instead of tuples, possibly I copied the code from somewhere, but I don't think it will change anything. Using tuples will not allow you to use numpy arrays I think. See http://www.sthurlow.com/python/lesson06/ for a difference between the two.

Peter


From: William H. Patton
Sent: Saturday, October 03, 2009 03:04
To: lp_solve@...
Subject: RE: [lp_solve] Solving a large LP with lpsolve in Python


 
It looks like the other possibility is to change the source of pythonmod.c and rebuild the interface.

I think changing #if 1 to #if 0  is designed to get lpsolve to accept numpy arrays or matrices

without the convert tolist.



#include "lpsolvecaller.h"



#define maxArgs 10



#if 1

  /* represent arrays via lists */

# define PyArray_Size PyList_Size

# define PyArray_New  PyList_New

# define PyArray_SET_ITEM PyList_SET_ITEM

# define PyArray_Resize(Array, Size) { /* Printf("PyList_Size(*Array) was %d, resizing to %d\n", PyList_Size(*Array), Size); */ while ((PyList_Size(*Array) < Size) && (PyList_Append(*Array, Py_None) == 0)); /* Printf("PyList_Size(*Array) becomes %d\n", PyList_Size(*Array)); */ }

#else

  /* represent arrays via tuples */

# define PyArray_Size PyTuple_Size

# define PyArray_New  PyTuple_New

# define PyArray_SET_ITEM PyTuple_SET_ITEM

# define PyArray_Resize _PyTuple_Resize

#endif




--------------------------------------------------------------------------------

From: lp_solve@... [mailto:lp_solve@...] On Behalf Of William H. Patton
Sent: Friday, October 02, 2009 4:26 PM
To: lp_solve@...
Subject: RE: [lp_solve] Solving a large LP with lpsolve in Python



 

The issue is trying to pass the dense array A; probably as you suspect the .tolist runs out of memory.



It is time to consider using the sparse methods for filling the lp A matrix.

Set_columnex is best.



You could make a toSparse() function for columns and change

Lp_maker.py

  m = len(a)

  n = len(a[0])

  lp = lpsolve('make_lp', m, n)

  lpsolve('set_verbose', lp, IMPORTANT)

  lpsolve('set_mat', lp, a)

  lpsolve('set_rh_vec', lp, b)

  lpsolve('set_obj_fn', lp, f)

  lpsolve('set_maxim', lp) # default is solving minimum lp.



Replace the red line with a loop on col 1 to m

     a[col].tosparse(count, sparsecolumn, rowno)

   lpsolve('set_columnex', lp, col, count, sparsecolumn, rowno);



This will never consume more than the nonzeros of the A matrix new space before running the solve.

You still might to release the python a columns as you pass them if you head for even bigger problems.

You can also look for python sparse matrix methods to save initial space.


--------------------------------------------------------------------------------

From: lp_solve@... [mailto:lp_solve@...] On Behalf Of traviserdman
Sent: Tuesday, September 29, 2009 11:38 PM
To: lp_solve@...
Subject: [lp_solve] Solving a large LP with lpsolve in Python



 

Been happily using lpsolve via Python for a few months now. With each success, my working model has grown in size. Now, I want to tackle a really large LP (really large in my context, maybe not yours). The current matrix is 10985 x 2873. Here is the problem:

>>> shape(a)
(10985, 2873)

>>> type(a)
<type 'numpy.ndarray'>

>>> lp = lp_maker(f,a.tolist(),b,e,vlb,vub,xint,scalemode=True,setminim=True)
alloc of 10096572 'REAL' failed

>>> ================================ RESTART ================================

After this last msg, the Python shell crashes and restarts itself. Python ver 2.5.4, lpsolve v5.5.0.15, 1.7 GB of RAM.

Questions:

(1) Is there a way to get a model of this size loaded into lpsolve?

(2) Despite the reference guide language to the contrary, I cannot get lpsolve to accept numpy arrays or matrices via lp_maker nor via lpsolve('set_mat',lp,a) ... only seems to accept list of lists. The "tolist()" construct may be a resource/memory hog, and my suspicion is that it's contributing to the system simply running out of memory ... is there a way to coerce lpsolve to accept the numpy.ndarray natively?

[Peter Notebaert-2]

It was a lot more complex than that to make it possible to provide numpy arrays to the lpsolve driver, but now it works. It will be available in the next version. If you want a preview, let me know. Especially also for which version and platform.

Peter


From: William H. Patton
Sent: Saturday, October 03, 2009 02:04
To: lp_solve@...
Subject: RE: [lp_solve] Solving a large LP with lpsolve in Python


 
It looks like the other possibility is to change the source of pythonmod.c and rebuild the interface.

I think changing #if 1 to #if 0  is designed to get lpsolve to accept numpy arrays or matrices

without the convert tolist.



#include "lpsolvecaller.h"



#define maxArgs 10



#if 1

  /* represent arrays via lists */

# define PyArray_Size PyList_Size

# define PyArray_New  PyList_New

# define PyArray_SET_ITEM PyList_SET_ITEM

# define PyArray_Resize(Array, Size) { /* Printf("PyList_Size(*Array) was %d, resizing to %d\n", PyList_Size(*Array), Size); */ while ((PyList_Size(*Array) < Size) && (PyList_Append(*Array, Py_None) == 0)); /* Printf("PyList_Size(*Array) becomes %d\n", PyList_Size(*Array)); */ }

#else

  /* represent arrays via tuples */

# define PyArray_Size PyTuple_Size

# define PyArray_New  PyTuple_New

# define PyArray_SET_ITEM PyTuple_SET_ITEM

# define PyArray_Resize _PyTuple_Resize

#endif




--------------------------------------------------------------------------------

From: lp_solve@... [mailto:lp_solve@...] On Behalf Of William H. Patton
Sent: Friday, October 02, 2009 4:26 PM
To: lp_solve@...
Subject: RE: [lp_solve] Solving a large LP with lpsolve in Python



 

The issue is trying to pass the dense array A; probably as you suspect the .tolist runs out of memory.



It is time to consider using the sparse methods for filling the lp A matrix.

Set_columnex is best.



You could make a toSparse() function for columns and change

Lp_maker.py

  m = len(a)

  n = len(a[0])

  lp = lpsolve('make_lp', m, n)

  lpsolve('set_verbose', lp, IMPORTANT)

  lpsolve('set_mat', lp, a)

  lpsolve('set_rh_vec', lp, b)

  lpsolve('set_obj_fn', lp, f)

  lpsolve('set_maxim', lp) # default is solving minimum lp.



Replace the red line with a loop on col 1 to m

     a[col].tosparse(count, sparsecolumn, rowno)

   lpsolve('set_columnex', lp, col, count, sparsecolumn, rowno);



This will never consume more than the nonzeros of the A matrix new space before running the solve.

You still might to release the python a columns as you pass them if you head for even bigger problems.

You can also look for python sparse matrix methods to save initial space.


--------------------------------------------------------------------------------

From: lp_solve@... [mailto:lp_solve@...] On Behalf Of traviserdman
Sent: Tuesday, September 29, 2009 11:38 PM
To: lp_solve@...
Subject: [lp_solve] Solving a large LP with lpsolve in Python



 

Been happily using lpsolve via Python for a few months now. With each success, my working model has grown in size. Now, I want to tackle a really large LP (really large in my context, maybe not yours). The current matrix is 10985 x 2873. Here is the problem:

>>> shape(a)
(10985, 2873)

>>> type(a)
<type 'numpy.ndarray'>

>>> lp = lp_maker(f,a.tolist(),b,e,vlb,vub,xint,scalemode=True,setminim=True)
alloc of 10096572 'REAL' failed

>>> ================================ RESTART ================================

After this last msg, the Python shell crashes and restarts itself. Python ver 2.5.4, lpsolve v5.5.0.15, 1.7 GB of RAM.

Questions:

(1) Is there a way to get a model of this size loaded into lpsolve?

(2) Despite the reference guide language to the contrary, I cannot get lpsolve to accept numpy arrays or matrices via lp_maker nor via lpsolve('set_mat',lp,a) ... only seems to accept list of lists. The "tolist()" construct may be a resource/memory hog, and my suspicion is that it's contributing to the system simply running out of memory ... is there a way to coerce lpsolve to accept the numpy.ndarray natively?

[traviserdman]

Mr. Patton, you were right.  I now need to save memory in the creation of the original matrix, even prior to uploading it to lpsolve.  I had mistakenly believed that the numpy array's were stored efficiently in memory as sparse matrices (they aren't).  Python's sparse matrices are in the Scipy package, which I am now using.

So now I am trying to upload my sparse matrix into lpsolve, via the set_columnex method along the lines of what Mr. Patton originally outlined.  However, I am unable to determine the proper syntax for set_columnex (or add_columnex).

I have tried it every which way.  Using the syntax indicated by Mr. Patton and outlined at http://web.mit.edu/lpsolve_v5507/doc/set_column.htm gets this response:

error: set_columnex requires 3 arguments.

This lpsolve/Python interface documentation page http://web.mit.edu/lpsolve_v5507/doc/Python.htm says:

#  set_column, set_columnex

    * return = lpsolve('set_column', lp, col_no, [column])
    * return = lpsolve('set_columnex', lp, col_no, [column])
    * Both have the same interface from set_column but act as set_columnex

This coincides with the demand for 3 arguments: lp, col_no, [column].

So, I tried that.  I used make_lp to create an m x n lprec structure.  Then, I try looping over the columns of my sparse matrix.  However, any lpsolve('set_columnex',lp,col,list) command where "col" is an integer between 0 and n-1 and "list" is an m-length list of real's (derived from my sparse matrix) is refuted with "error: invalid vector."

Help?

Also, a related question:  once uploaded to lpsolve, is the matrix stored efficiently in memory within lpsolve (ie as a sparse matrix data structure)?  if so, does it need to "densify" that matrix to solve the LP, or can lpsolve do its thing with the data in sparse format?


--- In lp_solve@..., "Peter Notebaert" <_peno_@...> wrote:

>
> It was a lot more complex than that to make it possible to provide numpy arrays to the lpsolve driver, but now it works. It will be available in the next version. If you want a preview, let me know. Especially also for which version and platform.
>
> Peter
>
>
> From: William H. Patton
> Sent: Saturday, October 03, 2009 02:04
> To: lp_solve@...
> Subject: RE: [lp_solve] Solving a large LP with lpsolve in Python
>
>
>  
> It looks like the other possibility is to change the source of pythonmod.c and rebuild the interface.
>
> I think changing #if 1 to #if 0  is designed to get lpsolve to accept numpy arrays or matrices
>
> without the convert tolist.
>
>
>
> #include "lpsolvecaller.h"
>
>
>
> #define maxArgs 10
>
>
>
> #if 1
>
>   /* represent arrays via lists */
>
> # define PyArray_Size PyList_Size
>
> # define PyArray_New  PyList_New
>
> # define PyArray_SET_ITEM PyList_SET_ITEM
>
> # define PyArray_Resize(Array, Size) { /* Printf("PyList_Size(*Array) was %d, resizing to %d\n", PyList_Size(*Array), Size); */ while ((PyList_Size(*Array) < Size) && (PyList_Append(*Array, Py_None) == 0)); /* Printf("PyList_Size(*Array) becomes %d\n", PyList_Size(*Array)); */ }
>
> #else
>
>   /* represent arrays via tuples */
>
> # define PyArray_Size PyTuple_Size
>
> # define PyArray_New  PyTuple_New
>
> # define PyArray_SET_ITEM PyTuple_SET_ITEM
>
> # define PyArray_Resize _PyTuple_Resize
>
> #endif
>
>
>
>
> --------------------------------------------------------------------------------
>
> From: lp_solve@... [mailto:lp_solve@...] On Behalf Of William H. Patton
> Sent: Friday, October 02, 2009 4:26 PM
> To: lp_solve@...
> Subject: RE: [lp_solve] Solving a large LP with lpsolve in Python
>
>
>
>  
>
> The issue is trying to pass the dense array A; probably as you suspect the .tolist runs out of memory.
>
>
>
> It is time to consider using the sparse methods for filling the lp A matrix.
>
> Set_columnex is best.
>
>
>
> You could make a toSparse() function for columns and change
>
> Lp_maker.py
>
>   m = len(a)
>
>   n = len(a[0])
>
>   lp = lpsolve('make_lp', m, n)
>
>   lpsolve('set_verbose', lp, IMPORTANT)
>
>   lpsolve('set_mat', lp, a)
>
>   lpsolve('set_rh_vec', lp, b)
>
>   lpsolve('set_obj_fn', lp, f)
>
>   lpsolve('set_maxim', lp) # default is solving minimum lp.
>
>
>
> Replace the red line with a loop on col 1 to m
>
>      a[col].tosparse(count, sparsecolumn, rowno)
>
>    lpsolve('set_columnex', lp, col, count, sparsecolumn, rowno);
>
>
>
> This will never consume more than the nonzeros of the A matrix new space before running the solve.
>
> You still might to release the python a columns as you pass them if you head for even bigger problems.
>
> You can also look for python sparse matrix methods to save initial space.
>
>
> --------------------------------------------------------------------------------
>
> From: lp_solve@... [mailto:lp_solve@...] On Behalf Of traviserdman
> Sent: Tuesday, September 29, 2009 11:38 PM
> To: lp_solve@...
> Subject: [lp_solve] Solving a large LP with lpsolve in Python
>
>
>
>  
>
> Been happily using lpsolve via Python for a few months now. With each success, my working model has grown in size. Now, I want to tackle a really large LP (really large in my context, maybe not yours). The current matrix is 10985 x 2873. Here is the problem:
>
> >>> shape(a)
> (10985, 2873)
>
> >>> type(a)
> <type 'numpy.ndarray'>
>
> >>> lp = lp_maker(f,a.tolist(),b,e,vlb,vub,xint,scalemode=True,setminim=True)
> alloc of 10096572 'REAL' failed
>
> >>> ================================ RESTART ================================
>
> After this last msg, the Python shell crashes and restarts itself. Python ver 2.5.4, lpsolve v5.5.0.15, 1.7 GB of RAM.
>
> Questions:
>
> (1) Is there a way to get a model of this size loaded into lpsolve?
>
> (2) Despite the reference guide language to the contrary, I cannot get lpsolve to accept numpy arrays or matrices via lp_maker nor via lpsolve('set_mat',lp,a) ... only seems to accept list of lists. The "tolist()" construct may be a resource/memory hog, and my suspicion is that it's contributing to the system simply running out of memory ... is there a way to coerce lpsolve to accept the numpy.ndarray natively?
>

[Peter Notebaert-2]

At this moment you cant provide Scipy data to lpsolve. Also the sparse call don't accept sparse matrices via Python.

lp_solve stores its matrices always in sparse structure, even if you can't provide it via Python in that way.

Peter


From: traviserdman
Sent: Friday, October 30, 2009 00:29
To: lp_solve@...
Subject: [lp_solve] Re: Solving a large LP with lpsolve in Python


 

Mr. Patton, you were right. I now need to save memory in the creation of the original matrix, even prior to uploading it to lpsolve. I had mistakenly believed that the numpy array's were stored efficiently in memory as sparse matrices (they aren't). Python's sparse matrices are in the Scipy package, which I am now using.

So now I am trying to upload my sparse matrix into lpsolve, via the set_columnex method along the lines of what Mr. Patton originally outlined. However, I am unable to determine the proper syntax for set_columnex (or add_columnex).

I have tried it every which way. Using the syntax indicated by Mr. Patton and outlined at http://web.mit.edu/lpsolve_v5507/doc/set_column.htm gets this response:

error: set_columnex requires 3 arguments.

This lpsolve/Python interface documentation page http://web.mit.edu/lpsolve_v5507/doc/Python.htm says:

# set_column, set_columnex

* return = lpsolve('set_column', lp, col_no, [column])
* return = lpsolve('set_columnex', lp, col_no, [column])
* Both have the same interface from set_column but act as set_columnex

This coincides with the demand for 3 arguments: lp, col_no, [column].

So, I tried that. I used make_lp to create an m x n lprec structure. Then, I try looping over the columns of my sparse matrix. However, any lpsolve('set_columnex',lp,col,list) command where "col" is an integer between 0 and n-1 and "list" is an m-length list of real's (derived from my sparse matrix) is refuted with "error: invalid vector."

Help?

Also, a related question: once uploaded to lpsolve, is the matrix stored efficiently in memory within lpsolve (ie as a sparse matrix data structure)? if so, does it need to "densify" that matrix to solve the LP, or can lpsolve do its thing with the data in sparse format?

--- In lp_solve@..., "Peter Notebaert" <_peno_@...> wrote:

>
> It was a lot more complex than that to make it possible to provide numpy arrays to the lpsolve driver, but now it works. It will be available in the next version. If you want a preview, let me know. Especially also for which version and platform.
>
> Peter
>
>
> From: William H. Patton
> Sent: Saturday, October 03, 2009 02:04
> To: lp_solve@...
> Subject: RE: [lp_solve] Solving a large LP with lpsolve in Python
>
>
>
> It looks like the other possibility is to change the source of pythonmod.c and rebuild the interface.
>
> I think changing #if 1 to #if 0 is designed to get lpsolve to accept numpy arrays or matrices
>
> without the convert tolist.
>
>
>
> #include "lpsolvecaller.h"
>
>
>
> #define maxArgs 10
>
>
>
> #if 1
>
> /* represent arrays via lists */
>
> # define PyArray_Size PyList_Size
>
> # define PyArray_New PyList_New
>
> # define PyArray_SET_ITEM PyList_SET_ITEM
>
> # define PyArray_Resize(Array, Size) { /* Printf("PyList_Size(*Array) was %d, resizing to %d\n", PyList_Size(*Array), Size); */ while ((PyList_Size(*Array) < Size) && (PyList_Append(*Array, Py_None) == 0)); /* Printf("PyList_Size(*Array) becomes %d\n", PyList_Size(*Array)); */ }
>
> #else
>
> /* represent arrays via tuples */
>
> # define PyArray_Size PyTuple_Size
>
> # define PyArray_New PyTuple_New
>
> # define PyArray_SET_ITEM PyTuple_SET_ITEM
>
> # define PyArray_Resize _PyTuple_Resize
>
> #endif
>
>
>
>
> ----------------------------------------------------------
>
> From: lp_solve@... [mailto:lp_solve@...] On Behalf Of William H. Patton
> Sent: Friday, October 02, 2009 4:26 PM
> To: lp_solve@...
> Subject: RE: [lp_solve] Solving a large LP with lpsolve in Python
>
>
>
>
>
> The issue is trying to pass the dense array A; probably as you suspect the .tolist runs out of memory.
>
>
>
> It is time to consider using the sparse methods for filling the lp A matrix.
>
> Set_columnex is best.
>
>
>
> You could make a toSparse() function for columns and change
>
> Lp_maker.py
>
> m = len(a)
>
> n = len(a[0])
>
> lp = lpsolve('make_lp', m, n)
>
> lpsolve('set_verbose', lp, IMPORTANT)
>
> lpsolve('set_mat', lp, a)
>
> lpsolve('set_rh_vec', lp, b)
>
> lpsolve('set_obj_fn', lp, f)
>
> lpsolve('set_maxim', lp) # default is solving minimum lp.
>
>
>
> Replace the red line with a loop on col 1 to m
>
> a[col].tosparse(count, sparsecolumn, rowno)
>
> lpsolve('set_columnex', lp, col, count, sparsecolumn, rowno);
>
>
>
> This will never consume more than the nonzeros of the A matrix new space before running the solve.
>
> You still might to release the python a columns as you pass them if you head for even bigger problems.
>
> You can also look for python sparse matrix methods to save initial space.
>
>
> ----------------------------------------------------------
>
> From: lp_solve@... [mailto:lp_solve@...] On Behalf Of traviserdman
> Sent: Tuesday, September 29, 2009 11:38 PM
> To: lp_solve@...
> Subject: [lp_solve] Solving a large LP with lpsolve in Python
>
>
>
>
>
> Been happily using lpsolve via Python for a few months now. With each success, my working model has grown in size. Now, I want to tackle a really large LP (really large in my context, maybe not yours). The current matrix is 10985 x 2873. Here is the problem:
>
> >>> shape(a)
> (10985, 2873)
>
> >>> type(a)
> <type 'numpy.ndarray'>
>
> >>> lp = lp_maker(f,a.tolist(),b,e,vlb,vub,xint,scalemode=True,setminim=True)
> alloc of 10096572 'REAL' failed
>
> >>> ================================ RESTART ================================
>
> After this last msg, the Python shell crashes and restarts itself. Python ver 2.5.4, lpsolve v5.5.0.15, 1.7 GB of RAM.
>
> Questions:
>
> (1) Is there a way to get a model of this size loaded into lpsolve?
>
> (2) Despite the reference guide language to the contrary, I cannot get lpsolve to accept numpy arrays or matrices via lp_maker nor via lpsolve('set_mat',lp,a) ... only seems to accept list of lists. The "tolist()" construct may be a resource/memory hog, and my suspicion is that it's contributing to the system simply running out of memory ... is there a way to coerce lpsolve to accept the numpy.ndarray natively?
>

[William H. Patton]

Peter is the authority.

 

http://lpsolve.sourceforge.net/5.5/Python.htm

http://sourceforge.net/projects/lpsolve/files/lpsolve/5.5.0.15/lp_solve_5.5.
0.15_Python_source.tar.gz/download

 

lpsolve('set_columnex',lp,col,list) command where "col" is an integer
between 0 and n-1 and "list" is an m-length list of real's (derived from my
sparse matrix)

You can see what is supposed to happen in the python interface source.

lp_solve_5.5\extra\Python

File lpsolve.c

static void impl_set_column(structlpsolve *lpsolve)

.

            result = set_columnex(lpsolve->lp, (int)
GetRealScalar(&lpsolve->lpsolvecaller, 2), count, vec, index);

is sent along to the c dll.

 

I am not sure if Python converts col_no to real  in the arg 2 slot for you
but it seems likely, so I do not know why it is complaining about wrong
number of arguments.

 

However, since you are probably still just transferring the A matrix column
by column, to

replace the set_mat( A ) in Lp_maker.py

Try using add column instead , it needs only the vector as list and the
interface does the sparsify.

It seems odd to take a nice scipy sparse column and make it a dense vector
just to have it made sparse again

Before passing it to the dll.  But that is life in the modern object
oriented wultilingual world.

 

/* return = xxlpsolve('add_columnex', lp, [column]) */

static void impl_add_column(structlpsolve *lpsolve)

 

So something easy like

For each column in A {

    lpsolve('add_columnex', lp, [column.tolist])

    free(column) // I have no idea about pthon garbage collection  or
blocking

}

should iterate the columns by expanding  then passing each sparse in the
lpsolve wrapper.

 

This still keeps your memory down to the size of a dense column at a time.

 

  _____  

From: lp_solve@... [mailto:lp_solve@...] On Behalf
Of Peter Notebaert
Sent: Thursday, October 29, 2009 6:39 PM
To: lp_solve@...
Subject: Re: [lp_solve] Re: Solving a large LP with lpsolve in Python

 

 

At this moment you cant provide Scipy data to lpsolve. Also the sparse call
don't accept sparse matrices via Python.

 

lp_solve stores its matrices always in sparse structure, even if you can't
provide it via Python in that way.

 

Peter

 

From: traviserdman <mailto:traviserdman@...>  

Sent: Friday, October 30, 2009 00:29

To: lp_solve@yahoogroup <mailto:lp_solve@...> s.com

Subject: [lp_solve] Re: Solving a large LP with lpsolve in Python

 

 



Mr. Patton, you were right. I now need to save memory in the creation of the
original matrix, even prior to uploading it to lpsolve. I had mistakenly
believed that the numpy array's were stored efficiently in memory as sparse
matrices (they aren't). Python's sparse matrices are in the Scipy package,
which I am now using.

So now I am trying to upload my sparse matrix into lpsolve, via the
set_columnex method along the lines of what Mr. Patton originally outlined.
However, I am unable to determine the proper syntax for set_columnex (or
add_columnex).

I have tried it every which way. Using the syntax indicated by Mr. Patton
and outlined at http://web.mit.
<http://web.mit.edu/lpsolve_v5507/doc/set_column.htm>
edu/lpsolve_v5507/doc/set_column.htm gets this response:

error: set_columnex requires 3 arguments.

This lpsolve/Python interface documentation page http://web.mit.
<http://web.mit.edu/lpsolve_v5507/doc/Python.htm>
edu/lpsolve_v5507/doc/Python.htm says:

# set_column, set_columnex

* return = lpsolve('set_column', lp, col_no, [column])
* return = lpsolve('set_columnex', lp, col_no, [column])
* Both have the same interface from set_column but act as set_columnex

This coincides with the demand for 3 arguments: lp, col_no, [column].

So, I tried that. I used make_lp to create an m x n lprec structure. Then, I
try looping over the columns of my sparse matrix. However, any
lpsolve('set_columnex',lp,col,list) command where "col" is an integer
between 0 and n-1 and "list" is an m-length list of real's (derived from my
sparse matrix) is refuted with "error: invalid vector."

Help?

Also, a related question: once uploaded to lpsolve, is the matrix stored
efficiently in memory within lpsolve (ie as a sparse matrix data structure)?
if so, does it need to "densify" that matrix to solve the LP, or can lpsolve
do its thing with the data in sparse format?

--- In lp_solve@yahoogroup <mailto:lp_solve%40yahoogroups.com> s.com, "Peter
Notebaert" <_peno_@...> wrote:
>
> It was a lot more complex than that to make it possible to provide numpy
arrays to the lpsolve driver, but now it works. It will be available in the
next version. If you want a preview, let me know. Especially also for which
version and platform.

>
> Peter
>
>
> From: William H. Patton
> Sent: Saturday, October 03, 2009 02:04
> To: lp_solve@yahoogroup <mailto:lp_solve%40yahoogroups.com> s.com
> Subject: RE: [lp_solve] Solving a large LP with lpsolve in Python
>
>
>
> It looks like the other possibility is to change the source of pythonmod.c

and rebuild the interface.
>
> I think changing #if 1 to #if 0 is designed to get lpsolve to accept numpy
arrays or matrices

>
> without the convert tolist.
>
>
>
> #include "lpsolvecaller.h"
>
>
>
> #define maxArgs 10
>
>
>
> #if 1
>
> /* represent arrays via lists */
>
> # define PyArray_Size PyList_Size
>
> # define PyArray_New PyList_New
>
> # define PyArray_SET_ITEM PyList_SET_ITEM
>
> # define PyArray_Resize(Array, Size) { /* Printf("PyList_Size(*Array) was

%d, resizing to %d\n", PyList_Size(*Array), Size); */ while
((PyList_Size(*Array) < Size) && (PyList_Append(*Array, Py_None) == 0)); /*
Printf("PyList_Size(*Array) becomes %d\n", PyList_Size(*Array)); */ }

>
> #else
>
> /* represent arrays via tuples */
>
> # define PyArray_Size PyTuple_Size
>
> # define PyArray_New PyTuple_New
>
> # define PyArray_SET_ITEM PyTuple_SET_ITEM
>
> # define PyArray_Resize _PyTuple_Resize
>
> #endif
>
>
>
>
> ----------------------------------------------------------
>
> From: lp_solve@yahoogroup <mailto:lp_solve%40yahoogroups.com> s.com

[mailto:lp_solve@yahoogroup <mailto:lp_solve%40yahoogroups.com> s.com] On
Behalf Of William H. Patton
> Sent: Friday, October 02, 2009 4:26 PM
> To: lp_solve@yahoogroup <mailto:lp_solve%40yahoogroups.com> s.com
> Subject: RE: [lp_solve] Solving a large LP with lpsolve in Python
>
>
>
>
>
> The issue is trying to pass the dense array A; probably as you suspect the
.tolist runs out of memory.
>
>
>
> It is time to consider using the sparse methods for filling the lp A
matrix.

>
> Set_columnex is best.
>
>
>
> You could make a toSparse() function for columns and change
>
> Lp_maker.py
>
> m = len(a)
>
> n = len(a[0])
>
> lp = lpsolve('make_lp', m, n)
>
> lpsolve('set_verbose', lp, IMPORTANT)
>
> lpsolve('set_mat', lp, a)
>
> lpsolve('set_rh_vec', lp, b)
>
> lpsolve('set_obj_fn', lp, f)
>
> lpsolve('set_maxim', lp) # default is solving minimum lp.
>
>
>
> Replace the red line with a loop on col 1 to m
>
> a[col].tosparse(count, sparsecolumn, rowno)
>
> lpsolve('set_columnex', lp, col, count, sparsecolumn, rowno);
>
>
>
> This will never consume more than the nonzeros of the A matrix new space

before running the solve.
>
> You still might to release the python a columns as you pass them if you
head for even bigger problems.
>
> You can also look for python sparse matrix methods to save initial space.
>
>
> ----------------------------------------------------------
>
> From: lp_solve@yahoogroup <mailto:lp_solve%40yahoogroups.com> s.com
[mailto:lp_solve@yahoogroup <mailto:lp_solve%40yahoogroups.com> s.com] On
Behalf Of traviserdman
> Sent: Tuesday, September 29, 2009 11:38 PM
> To: lp_solve@yahoogroup <mailto:lp_solve%40yahoogroups.com> s.com
> Subject: [lp_solve] Solving a large LP with lpsolve in Python
>
>
>
>
>
> Been happily using lpsolve via Python for a few months now. With each
success, my working model has grown in size. Now, I want to tackle a really
large LP (really large in my context, maybe not yours). The current matrix
is 10985 x 2873. Here is the problem:
>
> >>> shape(a)
> (10985, 2873)
>
> >>> type(a)
> <type 'numpy.ndarray'>
>
> >>> lp =
lp_maker(f,a.tolist(),b,e,vlb,vub,xint,scalemode=True,setminim=True)
> alloc of 10096572 'REAL' failed
>
> >>> ================================ RESTART
================================
>
> After this last msg, the Python shell crashes and restarts itself. Python
ver 2.5.4, lpsolve v5.5.0.15, 1.7 GB of RAM.
>
> Questions:
>
> (1) Is there a way to get a model of this size loaded into lpsolve?
>
> (2) Despite the reference guide language to the contrary, I cannot get
lpsolve to accept numpy arrays or matrices via lp_maker nor via
lpsolve('set_mat',lp,a) ... only seems to accept list of lists. The
"tolist()" construct may be a resource/memory hog, and my suspicion is that
it's contributing to the system simply running out of memory ... is there a
way to coerce lpsolve to accept the numpy.ndarray natively?
>

[traviserdman]

Ah, here it is.  Column needs to be a list of length m+1, not m.  I'll assume the extra item is the objective function:

>>> lp = lpsolve('make_lp',3,3)
>>> lpsolve('set_columnex',lp,0,[4,5,6])
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
error: invalid vector.
>>> lpsolve('set_columnex',lp,1,[4,5,6])
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
error: invalid vector.
>>> lpsolve('add_columnex',lp,[4,5,6])
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
error: invalid vector.
>>> lpsolve('set_columnex',lp,1,[4,5,6,7])
1L
>>> lpsolve('get_column',lp,1)
[[4.0, 5.0, 6.0, 7.0], 1L]

--- In lp_solve@..., "William H. Patton" <pattonwh@...> wrote:

>
> Peter is the authority.
>
>  
>
> http://lpsolve.sourceforge.net/5.5/Python.htm
>
> http://sourceforge.net/projects/lpsolve/files/lpsolve/5.5.0.15/lp_solve_5.5.
> 0.15_Python_source.tar.gz/download
>
>  
>
> lpsolve('set_columnex',lp,col,list) command where "col" is an integer
> between 0 and n-1 and "list" is an m-length list of real's (derived from my
> sparse matrix)
>
> You can see what is supposed to happen in the python interface source.
>
> lp_solve_5.5\extra\Python
>
> File lpsolve.c
>
> static void impl_set_column(structlpsolve *lpsolve)
>
> .
>
>             result = set_columnex(lpsolve->lp, (int)
> GetRealScalar(&lpsolve->lpsolvecaller, 2), count, vec, index);
>
> is sent along to the c dll.
>
>  
>
> I am not sure if Python converts col_no to real  in the arg 2 slot for you
> but it seems likely, so I do not know why it is complaining about wrong
> number of arguments.
>
>  
>
> However, since you are probably still just transferring the A matrix column
> by column, to
>
> replace the set_mat( A ) in Lp_maker.py
>
> Try using add column instead , it needs only the vector as list and the
> interface does the sparsify.
>
> It seems odd to take a nice scipy sparse column and make it a dense vector
> just to have it made sparse again
>
> Before passing it to the dll.  But that is life in the modern object
> oriented wultilingual world.
>
>  
>
> /* return = xxlpsolve('add_columnex', lp, [column]) */
>
> static void impl_add_column(structlpsolve *lpsolve)
>
>  
>
> So something easy like
>
> For each column in A {
>
>     lpsolve('add_columnex', lp, [column.tolist])
>
>     free(column) // I have no idea about pthon garbage collection  or
> blocking
>
> }
>
> should iterate the columns by expanding  then passing each sparse in the
> lpsolve wrapper.
>
>  
>
> This still keeps your memory down to the size of a dense column at a time.
>
>  
>
>   _____  
>
> From: lp_solve@... [mailto:lp_solve@...] On Behalf
> Of Peter Notebaert
> Sent: Thursday, October 29, 2009 6:39 PM
> To: lp_solve@...
> Subject: Re: [lp_solve] Re: Solving a large LP with lpsolve in Python
>
>  
>
>  
>
> At this moment you cant provide Scipy data to lpsolve. Also the sparse call
> don't accept sparse matrices via Python.
>
>  
>
> lp_solve stores its matrices always in sparse structure, even if you can't
> provide it via Python in that way.
>
>  
>
> Peter
>
>  
>
> From: traviserdman <mailto:traviserdman@...>  
>
> Sent: Friday, October 30, 2009 00:29
>
> To: lp_solve@yahoogroup <mailto:lp_solve@...> s.com
>
> Subject: [lp_solve] Re: Solving a large LP with lpsolve in Python
>
>  
>
>  
>
>
>
> Mr. Patton, you were right. I now need to save memory in the creation of the
> original matrix, even prior to uploading it to lpsolve. I had mistakenly
> believed that the numpy array's were stored efficiently in memory as sparse
> matrices (they aren't). Python's sparse matrices are in the Scipy package,
> which I am now using.
>
> So now I am trying to upload my sparse matrix into lpsolve, via the
> set_columnex method along the lines of what Mr. Patton originally outlined.
> However, I am unable to determine the proper syntax for set_columnex (or
> add_columnex).
>
> I have tried it every which way. Using the syntax indicated by Mr. Patton
> and outlined at http://web.mit.
> <http://web.mit.edu/lpsolve_v5507/doc/set_column.htm>
> edu/lpsolve_v5507/doc/set_column.htm gets this response:
>
> error: set_columnex requires 3 arguments.
>
> This lpsolve/Python interface documentation page http://web.mit.
> <http://web.mit.edu/lpsolve_v5507/doc/Python.htm>
> edu/lpsolve_v5507/doc/Python.htm says:
>
> # set_column, set_columnex
>
> * return = lpsolve('set_column', lp, col_no, [column])
> * return = lpsolve('set_columnex', lp, col_no, [column])
> * Both have the same interface from set_column but act as set_columnex
>
> This coincides with the demand for 3 arguments: lp, col_no, [column].
>
> So, I tried that. I used make_lp to create an m x n lprec structure. Then, I
> try looping over the columns of my sparse matrix. However, any
> lpsolve('set_columnex',lp,col,list) command where "col" is an integer
> between 0 and n-1 and "list" is an m-length list of real's (derived from my
> sparse matrix) is refuted with "error: invalid vector."
>
> Help?
>
> Also, a related question: once uploaded to lpsolve, is the matrix stored
> efficiently in memory within lpsolve (ie as a sparse matrix data structure)?
> if so, does it need to "densify" that matrix to solve the LP, or can lpsolve
> do its thing with the data in sparse format?
>
> --- In lp_solve@yahoogroup <mailto:lp_solve%40yahoogroups.com> s.com, "Peter
> Notebaert" <_peno_@> wrote:
> >
> > It was a lot more complex than that to make it possible to provide numpy
> arrays to the lpsolve driver, but now it works. It will be available in the
> next version. If you want a preview, let me know. Especially also for which
> version and platform.
> >
> > Peter
> >
> >
> > From: William H. Patton
> > Sent: Saturday, October 03, 2009 02:04
> > To: lp_solve@yahoogroup <mailto:lp_solve%40yahoogroups.com> s.com
> > Subject: RE: [lp_solve] Solving a large LP with lpsolve in Python
> >
> >
> >
> > It looks like the other possibility is to change the source of pythonmod.c
> and rebuild the interface.
> >
> > I think changing #if 1 to #if 0 is designed to get lpsolve to accept numpy
> arrays or matrices
> >
> > without the convert tolist.
> >
> >
> >
> > #include "lpsolvecaller.h"
> >
> >
> >
> > #define maxArgs 10
> >
> >
> >
> > #if 1
> >
> > /* represent arrays via lists */
> >
> > # define PyArray_Size PyList_Size
> >
> > # define PyArray_New PyList_New
> >
> > # define PyArray_SET_ITEM PyList_SET_ITEM
> >
> > # define PyArray_Resize(Array, Size) { /* Printf("PyList_Size(*Array) was
> %d, resizing to %d\n", PyList_Size(*Array), Size); */ while
> ((PyList_Size(*Array) < Size) && (PyList_Append(*Array, Py_None) == 0)); /*
> Printf("PyList_Size(*Array) becomes %d\n", PyList_Size(*Array)); */ }
> >
> > #else
> >
> > /* represent arrays via tuples */
> >
> > # define PyArray_Size PyTuple_Size
> >
> > # define PyArray_New PyTuple_New
> >
> > # define PyArray_SET_ITEM PyTuple_SET_ITEM
> >
> > # define PyArray_Resize _PyTuple_Resize
> >
> > #endif
> >
> >
> >
> >
> > ----------------------------------------------------------
> >
> > From: lp_solve@yahoogroup <mailto:lp_solve%40yahoogroups.com> s.com
> [mailto:lp_solve@yahoogroup <mailto:lp_solve%40yahoogroups.com> s.com] On
> Behalf Of William H. Patton
> > Sent: Friday, October 02, 2009 4:26 PM
> > To: lp_solve@yahoogroup <mailto:lp_solve%40yahoogroups.com> s.com
> > Subject: RE: [lp_solve] Solving a large LP with lpsolve in Python
> >
> >
> >
> >
> >
> > The issue is trying to pass the dense array A; probably as you suspect the
> .tolist runs out of memory.
> >
> >
> >
> > It is time to consider using the sparse methods for filling the lp A
> matrix.
> >
> > Set_columnex is best.
> >
> >
> >
> > You could make a toSparse() function for columns and change
> >
> > Lp_maker.py
> >
> > m = len(a)
> >
> > n = len(a[0])
> >
> > lp = lpsolve('make_lp', m, n)
> >
> > lpsolve('set_verbose', lp, IMPORTANT)
> >
> > lpsolve('set_mat', lp, a)
> >
> > lpsolve('set_rh_vec', lp, b)
> >
> > lpsolve('set_obj_fn', lp, f)
> >
> > lpsolve('set_maxim', lp) # default is solving minimum lp.
> >
> >
> >
> > Replace the red line with a loop on col 1 to m
> >
> > a[col].tosparse(count, sparsecolumn, rowno)
> >
> > lpsolve('set_columnex', lp, col, count, sparsecolumn, rowno);
> >
> >
> >
> > This will never consume more than the nonzeros of the A matrix new space
> before running the solve.
> >
> > You still might to release the python a columns as you pass them if you
> head for even bigger problems.
> >
> > You can also look for python sparse matrix methods to save initial space.
> >
> >
> > ----------------------------------------------------------
> >
> > From: lp_solve@yahoogroup <mailto:lp_solve%40yahoogroups.com> s.com
> [mailto:lp_solve@yahoogroup <mailto:lp_solve%40yahoogroups.com> s.com] On
> Behalf Of traviserdman
> > Sent: Tuesday, September 29, 2009 11:38 PM
> > To: lp_solve@yahoogroup <mailto:lp_solve%40yahoogroups.com> s.com
> > Subject: [lp_solve] Solving a large LP with lpsolve in Python
> >
> >
> >
> >
> >
> > Been happily using lpsolve via Python for a few months now. With each
> success, my working model has grown in size. Now, I want to tackle a really
> large LP (really large in my context, maybe not yours). The current matrix
> is 10985 x 2873. Here is the problem:
> >
> > >>> shape(a)
> > (10985, 2873)
> >
> > >>> type(a)
> > <type 'numpy.ndarray'>
> >
> > >>> lp =
> lp_maker(f,a.tolist(),b,e,vlb,vub,xint,scalemode=True,setminim=True)
> > alloc of 10096572 'REAL' failed
> >
> > >>> ================================ RESTART
> ================================
> >
> > After this last msg, the Python shell crashes and restarts itself. Python
> ver 2.5.4, lpsolve v5.5.0.15, 1.7 GB of RAM.
> >
> > Questions:
> >
> > (1) Is there a way to get a model of this size loaded into lpsolve?
> >
> > (2) Despite the reference guide language to the contrary, I cannot get
> lpsolve to accept numpy arrays or matrices via lp_maker nor via
> lpsolve('set_mat',lp,a) ... only seems to accept list of lists. The
> "tolist()" construct may be a resource/memory hog, and my suspicion is that
> it's contributing to the system simply running out of memory ... is there a
> way to coerce lpsolve to accept the numpy.ndarray natively?
> >
>