XNA help - too much memory used?

I'm writing code in XNA 4.0, and part of my game entails drawing a textured grid of cubes. It is working, but the problem is the memory usage is growing way too rapidly. Each cube has 8 points, and each point is a VertexPositionTexture. A 16x9 grid takes 50MB memory, and that is just the grid and lines, no textures or anything else. Doubling the grid to 32x18 makes the memory jump to 150MB....I don't understand what is taking up so much room. I draw the grid using DrawUserIndexedPrimitives. The cubes can move around and be different sizes so I can't consolidate them to one set of vertices. What is the proper way to render a few thousand vertices without using so much memory?

These are all the members for my Cube class so far:

GraphicsDevice graphicsDevice; // for drawing this cube
Tileset tileset; // reference to parent object
VertexPositionTexture[] pointList; // the list of 8 vertices
BasicEffect basicEffect; // effect to draw using a shader
Vector3 localTranslation; // just 3 floats
VertexBuffer vertexBuffer; // for sending the vertices to the GPU

There are also a few functions that allocate memory for the pointList...but nothing out of the ordinary...