I have a script that is working fine with small values for MAX_SIZE;X and MAX_SIZE_Y.
But once I adjust them to higher values (321 and 273 in my case) and view “m” (by entering “m<RETURN>” in the python shell I see that the list stops someplace. Also len(m) does not show anything then. Usually it shows the number of “lists” in m.
I also tried to test if n.append(…): for nonzero and print some errormsg, but that did not help as it did not print anything.
def create_m():
global m
m = []
for y in range(MAX_SIZE_Y):
n = []
for x in range(MAX_SIZE_X):
n.append(str(x)+":"+str(y)) # for testing (usually a class)
m.append(n)
Any ideas what this could be? Memory problem? Other design for matrix better? What I need is some sort of 2 dimensional array: m[x][y]
I think, in a way, it is a memory problem: in order to print out the string representation of this fairly large array a huge string must be constructed and that may take a very long time. I constructed a 400x400 array and tried to print it with m<enter> after a minute i gave up. However: individual items printed out quite well:
Maybe the numpy package http://numpy.scipy.org/can help you out here (certainly for sparse matrices) but in general you took the right approach: you’ll just have to live with the limitations I guess…
what version, os …
can you check memory usage with tools like top (use M to sort for memory)
I changed your little routine to use params for calling
and i get:
>>> createm(500,500)
>>> print len(m)
500
>>> createm(800,500)
>>> print len(m)
800
>>> print len(m[0])
500
the createm def is
def createm(a=10, b=10):
and uses a and b in the ranges
so its easy to call it with different values and check the python-memory footprint
i tried it with bigger values and the only thing the generation of the array is noticable, you can count …20, 21,22, 23 … till its done
I tried your function and it works for me, I tested up to 500. I used a script like this:
def create_m():
global m
m = []
for y in range(MAX_SIZE_Y):
n = []
for x in range(MAX_SIZE_X):
n.append(str(x)+":"+str(y)) # for testing (usually a class)
m.append(n)
print m
if __name__ == "__main__":
m = []
MAX_SIZE_X = 500
MAX_SIZE_Y = 500
create_m()
print m, len(m)
Btw, I think I read that using range in a loop is not very efficient, better use xrange instead. Yup, I did, in this guide.
i am working on windows vista right now. at the office we use xp. one thing i noticed is that under vista i dont have a working IDLE after installing python. thats why i am working with the free ActivePython IDE.
i have tried to run your version sanne, in the ActivePython WinEditor, but although it looks good visually the program seem to loop forevery in the taskmanager and does not return to the prompt.
i have tried to run the script in the standard python shell, but maybe i am doing it wrong. i was trying it with just import test.py and then i end up at the prompt with no access to m or the function.
One idea might be to use a sparse matrix. In a sparse matrix you assume that all values that have not been modified are zero. This way you can save a lot of memory. It’s easy to implement one as well. I coded one just for fun to see how easy it is. Feel free to use it and alter as you need if you want. It needs more rigorous tests for all the corner cases (bad inits) and some operations of course.
As a sidenote the implementation of MatrixRow could be optimized further by creating rows on demand, not on init. This might not give a big gain though as the object itself is relatively light.
# -*- coding: utf-8 -*-
from unittest import main, TestCase
class TestSparseMatrix(TestCase):
def setUp(self):
self.matrix = SparseMatrix(10, 10)
def testAccessItem(self):
self.assertEquals(self.matrix[5][5], 0)
# a list in this case is not callable so its not possible to use assertRaises :(
try:
value = self.matrix[5][15]
except IndexError:
print "exception1 ok"
else:
print 'duh'
try:
value = self.matrix[15][5]
except IndexError:
print "exception2 ok"
def testModifyItem(self):
self.matrix[4][4] = 15
self.assertEquals(self.matrix[4][4], 15)
self.matrix[4][4] = 20
self.assertEquals(self.matrix[4][4], 20)
try:
self.matrix[4][20] = 5
except IndexError:
print "exception3 ok"
class SparseMatrix():
def __init__(self, rows, columns):
self.rows = []
for row in range(rows):
self.rows.append(MatrixRow(columns))
def __getitem__(self, x):
return self.rows[x]
class MatrixRow():
def __init__(self, columns):
self.columns = columns
self.values = {}
def _check_column(self, column):
if column >= self.columns:
raise IndexError
def __getitem__(self, column):
self._check_column(column)
try:
return self.values[column]
except KeyError:
return 0
def __setitem__(self, column, value):
self._check_column(column)
self.values[column] = value
if __name__ == '__main__':
main()
also thanks sanne, i found that i could simply click the file in the windows explorer and run it that way. just had to insert a wait for press key thing because otherwise it just closes the window so fast again that i cant see anything
at about noon today i have decided that i have reached the rather dangerous stage where i know enough to make lots of problems but not enough to solve them learning from little tutorials here and there can produce that … so i went to the bookstore and bought myself a copy of o’reilly’s introduction to python and i start to understand the concepts much better i even was able to install python so that i can use the idle under vista!
so thanks for all who helped me so far and i hope i will be of more use in python in the future
The cool thing about O’Reilly books (at least the ones I have) is that they are Safari enabled. You can use Safari (http://safari.oreilly.com/) to read your books electronically for free. Furthermore the service allows you to read any book available for minor cost. Besides that they give you discounts at their store if you use Safari.
yes, free e-books like diveintobython or byteofpython helped me alot in the beginning. but after staring at a screen all day long i prefer to have a physical manifestation of a book in my hands for a change
and the sparse matrix thing is exactely the right way for me!
