worst time taken by Mapnik while rendering map

[Balram]

Hi friends,

Hope you find this mail in good health.

I have created a map rendering application using Mapnik & Python. I have implemented basic operation on map rendering too.

The major problem is large time taken by Mapnik to render a map.

I am rendering a map by firing queries on postgreSql database & assigning this data to various layers of map & finally rendering map by calling function render_to_file().
I have defined various styles & rules function similar to osm.xml style sheet with filter, maxscale & minscale parameters to filter our data at various zoom level.

When I am rendering map using Europe data & it is taking  time about 5 to 6 minutes for rendering map image.
Again whenever I perform some operation like zoom in, zoom out, pan etc ; it takes same amount of time.

Most people recommended tile server as solution to this problem but  I can’t use tile server.
Because I have to provide “route planning & management” as one of the feature of my project, and I guess this will not be possible using tile server.

I guessed using estimate_extent parameter in PostGIS function could help me out of this.
But when I tried it skipped most of the data from query result and rendered map with very low details.

Could you please help me out of this problem?
Please do provide me some concrete solution for this issue.

Expecting your helping hands toward me.

Thanks in Advance
BalRam

[Dane Springmeyer]

First make sure you have built indexes for any large shapefiles you  
might be pulling from.

For large postgis datasets you should set estimate_extent=false and  
then provide the 'extent' parameter, essentially hardcoding the right  
extent for each layer.

Either way, the problem you describe in terms of 5-6 minute renders  
sounds more like either a messed up Postgres database with maybe  
indexes that are corrupted or you are running on a system without  
enough memory for these types of queries. I've never seen a basic  
select query take that long even with osm size databases as I would  
assume it is the postgraduate query that is taking a long time(even  
hanging) not actually the mapnik rendering.

I would advise putting effort toward debugging your data and Postgres  
before playing with mapnik settings. For example, test basic select  
queries against your data inside the psql interperer.

Also you should subscribe to the list.

Dane



On Oct 5, 2009, at 1:18 AM, Balram <balram.chavan@...> wrote:

>
> Hi friends,
>
> Hope you find this mail in good health.
>
> I have created a map rendering application using Mapnik & Python. I  
> have
> implemented basic operation on map rendering too.
>
> The major problem is large time taken by Mapnik to render a map.
>
> I am rendering a map by firing queries on postgreSql database &  
> assigning
> this data to various layers of map & finally rendering map by calling
> function render_to_file().
> I have defined various styles & rules function similar to osm.xml  
> style
> sheet with filter, maxscale & minscale parameters to filter our data  
> at
> various zoom level.
>
> When I am rendering map using Europe data & it is taking  time about  
> 5 to 6
> minutes for rendering map image.
> Again whenever I perform some operation like zoom in, zoom out, pan  
> etc ; it
> takes same amount of time.
>
> Most people recommended tile server as solution to this problem but  
> I can’t
> use tile server.
> Because I have to provide “route planning & management” as one of  
> the
> feature of my project, and I guess this will not be possible using  
> tile
> server.
>
> I guessed using estimate_extent parameter in PostGIS function could  
> help me
> out of this.
> But when I tried it skipped most of the data from query result and  
> rendered
> map with very low details.
>
> Could you please help me out of this problem?
> Please do provide me some concrete solution for this issue.
>
> Expecting your helping hands toward me.
>
> Thanks in Advance
> BalRam
> --
> View this message in context: http://www.nabble.com/worst-time-taken-by-Mapnik-while-rendering-map-tp25746800p25746800.html
> Sent from the Mapnik - Users mailing list archive at Nabble.com.
>
> _______________________________________________
> Mapnik-users mailing list
> Mapnik-users@...
> https://lists.berlios.de/mailman/listinfo/mapnik-users

_______________________________________________
Mapnik-users mailing list
Mapnik-users@...
https://lists.berlios.de/mailman/listinfo/mapnik-users

[Jon Burgess-2]

On Mon, 2009-10-05 at 01:18 -0700, Balram wrote:
> When I am rendering map using Europe data & it is taking  time about 5
> to 6
> minutes for rendering map image.
> Again whenever I perform some operation like zoom in, zoom out, pan
> etc ; it
> takes same amount of time.

How detailed are the maps you are rendering? If you attempt to generate
europe-wide maps with every single residential road included then it
will take a long time. Reducing the number of features rendered should
speed things up.

> Most people recommended tile server as solution to this problem but  I
> can’t
> use tile server.
> Because I have to provide “route planning & management” as one of the
> feature of my project, and I guess this will not be possible using
> tile
> server.

The normal solution employed by lots of web sites is to render the maps
as a set of static background tiles and then use something else to
overlay the dynamic routing data over the top.

        Jon


_______________________________________________
Mapnik-users mailing list
Mapnik-users@...
https://lists.berlios.de/mailman/listinfo/mapnik-users

[Balram]

Hi Jon,
Thanks very much for your reply.

Yes, I am desired to render europe data with high level details along with very low level details. But the thing which makes me surprise is when i am at higher zoom level my output is just various country border & names of countries. Ideally this should take less amount of time because it has to process less amount of data (I guess so). But it is taking more time to render at high zoom level as compared to low zoom level. As I am increasing my zoom level from low to high, the rendering time increase in proportion.

Again, could you please explain me in more details about how to merge static map image tiles with dynamic map image containing route planning information?

Please help me, if possible.

Thanks a lot !!!
Balram

Jon Burgess-2 wrote:

On Mon, 2009-10-05 at 01:18 -0700, Balram wrote:
> When I am rendering map using Europe data & it is taking  time about 5
> to 6
> minutes for rendering map image.
> Again whenever I perform some operation like zoom in, zoom out, pan
> etc ; it
> takes same amount of time.

How detailed are the maps you are rendering? If you attempt to generate
europe-wide maps with every single residential road included then it
will take a long time. Reducing the number of features rendered should
speed things up.

> Most people recommended tile server as solution to this problem but  I
> can’t
> use tile server.
> Because I have to provide “route planning & management” as one of the
> feature of my project, and I guess this will not be possible using
> tile
> server.

The normal solution employed by lots of web sites is to render the maps
as a set of static background tiles and then use something else to
overlay the dynamic routing data over the top.

        Jon


_______________________________________________
Mapnik-users mailing list
[e-mail]
https://lists.berlios.de/mailman/listinfo/mapnik-users

[Balram]

Hi Dane,
Thanks for your reply.

I have tried with estimate_extent & extent parameter.

Now my average rendering time is decreased to 20 sec. At very low zoom level time decreases to 6 sec even and at higher zoom level it tends to 25 to 30 sec.

For all layers in my python script I am using following stuff:

Power_lyr.datasource = PostGIS(host='localhost', dbname='postgis_europe', user='balram', password='decos', table="(select way from planet_osm_line where power='line') as power",estimate_extent='false',extent='-20037508,-19929239,20037508,19929239')

Power_lyr.srs = "+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +no_defs +over"

and after that I am doing

m.zoom_to_box(Envelope(3.06293548889,51.1743208093,6.5670669221,52.4263865259))

to zoom to netherlands map.

Being honest, I don't know exact meaning of value passed to "extent" parameter. I have supplied this value which I found on mapnik forum.

Could you please explain me in more detail "extent" parameter? What extent value shall I specify for europe data for each layer?

Please find attached complete python script file, which render europe map that I am using.

Please help me, if possible.

Thanks
Balram

Dane Springmeyer wrote:

First make sure you have built indexes for any large shapefiles you  
might be pulling from.

For large postgis datasets you should set estimate_extent=false and  
then provide the 'extent' parameter, essentially hardcoding the right  
extent for each layer.

Either way, the problem you describe in terms of 5-6 minute renders  
sounds more like either a messed up Postgres database with maybe  
indexes that are corrupted or you are running on a system without  
enough memory for these types of queries. I've never seen a basic  
select query take that long even with osm size databases as I would  
assume it is the postgraduate query that is taking a long time(even  
hanging) not actually the mapnik rendering.

I would advise putting effort toward debugging your data and Postgres  
before playing with mapnik settings. For example, test basic select  
queries against your data inside the psql interperer.

Also you should subscribe to the list.

Dane



On Oct 5, 2009, at 1:18 AM, Balram <[e-mail]> wrote:

>
> Hi friends,
>
> Hope you find this mail in good health.
>
> I have created a map rendering application using Mapnik & Python. I  
> have
> implemented basic operation on map rendering too.
>
> The major problem is large time taken by Mapnik to render a map.
>
> I am rendering a map by firing queries on postgreSql database &  
> assigning
> this data to various layers of map & finally rendering map by calling
> function render_to_file().
> I have defined various styles & rules function similar to osm.xml  
> style
> sheet with filter, maxscale & minscale parameters to filter our data  
> at
> various zoom level.
>
> When I am rendering map using Europe data & it is taking  time about  
> 5 to 6
> minutes for rendering map image.
> Again whenever I perform some operation like zoom in, zoom out, pan  
> etc ; it
> takes same amount of time.
>
> Most people recommended tile server as solution to this problem but  
> I can’t
> use tile server.
> Because I have to provide “route planning & management” as one of  
> the
> feature of my project, and I guess this will not be possible using  
> tile
> server.
>
> I guessed using estimate_extent parameter in PostGIS function could  
> help me
> out of this.
> But when I tried it skipped most of the data from query result and  
> rendered
> map with very low details.
>
> Could you please help me out of this problem?
> Please do provide me some concrete solution for this issue.
>
> Expecting your helping hands toward me.
>
> Thanks in Advance
> BalRam
> --
> View this message in context: http://www.nabble.com/worst-time-taken-by-Mapnik-while-rendering-map-tp25746800p25746800.html
> Sent from the Mapnik - Users mailing list archive at Nabble.com.
>
> _______________________________________________
> Mapnik-users mailing list
> [e-mail]
> https://lists.berlios.de/mailman/listinfo/mapnik-users
_______________________________________________
Mapnik-users mailing list
[e-mail]
https://lists.berlios.de/mailman/listinfo/mapnik-users

europe_map.py

[Jon Burgess-2]

On Wed, 2009-10-07 at 21:01 -0700, Balram wrote:
> Hi Jon,
> Thanks very much for your reply.
>
> Yes, I am desired to render europe data with high level details along with
> very low level details.

If you are rendering maps with lots of details then ~20 seconds seems a
reasonable rendering time.

>  But the thing which makes me surprise is when i am
> at higher zoom level my output is just various country border & names of
> countries. Ideally this should take less amount of time because it has to
> process less amount of data (I guess so). But it is taking more time to
> render at high zoom level as compared to low zoom level. As I am increasing
> my zoom level from low to high, the rendering time increase in proportion.

What you might find is that the country borders themselves are very slow
to render. They are often highly detailed and contain lots of data.
Rendering more of them may slow things down. You can speed things up by
simplifying the data. This can be done by using the postgis simplify()
function to create a new set of boundaries which are used for high zoom
levels. Then you can switch to the original more detailed boundaries at
low zooms.


> Again, could you please explain me in more details about how to merge static
> map image tiles with dynamic map image containing route planning
> information?

That depends entirely on your application. If the tiles are displayed in
a web browser then people often use javascript in the client side
browser to add the overlay. Several good examples of this can be found
on the OpenLayers website http://gallery.openlayers.org/

If you are using another application framework then you will probably
have APIs letting you load a bitmap into a window and then draw points,
icons & lines over the top.

        Jon


_______________________________________________
Mapnik-users mailing list
Mapnik-users@...
https://lists.berlios.de/mailman/listinfo/mapnik-users

[Alberto Valverde]

Balram wrote:
> (...) But the thing which makes me surprise is when i am
> at higher zoom level my output is just various country border & names of
> countries. Ideally this should take less amount of time because it has to
> process less amount of data (I guess so). But it is taking more time to
> render at high zoom level as compared to low zoom level. As I am increasing
> my zoom level from low to high, the rendering time increase in proportion.
(snip)

This reminds me of a performance problem I had yesterday with a big
(>4Gb) vector dataset I stored in postgres. The problem turned out to be
that the spatial index on the geometry column was not being used because
the query planner thought that a sequential scan would be faster (due to
the relatively small number of total rows but very large geometries).
Since the spatial predicate had to be evaluated on *every* geometry the
database was doing much more work than it needed.

This problem is described in here [1] (section 5.1) and one of the
proposed workarounds fixed it for me: Create a another geometry column
to store the pre-calculated envelope of each geometry, create an index
on that and, optionally, CLUSTER the table on that index. You should
then first filter out all non-overlapping rows using this index (which
is very fast) before operating on the real geometry column (on a,
hopefully, much reduced recordset)

I'm not sure if Mapnik can be configured to filter on a geometry column
different than the one that it is rendering (haven't got there yet) but
you could "split up" the table into several views (one for each region)
using this index internally and then configure each one as an individual
datasource using the "real" geometry column. Alternatively you could
configure each Mapnik layer using a sub-select that does the filtering,
explained here [2].

Hope it helps,

Alberto

[1] http://postgis.refractions.net/documentation/manual-1.3/ch05.html
[2] http://trac.mapnik.org/wiki/PostGIS

_______________________________________________
Mapnik-users mailing list
Mapnik-users@...
https://lists.berlios.de/mailman/listinfo/mapnik-users

[Dane Springmeyer]

On Oct 7, 2009, at 11:19 PM, Balram wrote:

>
> Hi Dane,
> Thanks for your reply.
>
> I have tried with estimate_extent & extent parameter.
>
> Now my average rendering time is decreased to 20 sec.

Great, thats better than 5-6 minutes :)

> For all layers in my python script I am using following stuff:
>
> Power_lyr.datasource = PostGIS(host='localhost',  
> dbname='postgis_europe',
> user='balram', password='decos', table="(select way from  
> planet_osm_line
> where power='line') as
> power
> ",estimate_extent
> ='false',extent='-20037508,-19929239,20037508,19929239')
>
> Power_lyr.srs = "+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0  
> +lon_0=0.0
> +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +no_defs +over"
>
> and after that I am doing
>
> m
> .zoom_to_box
> (Envelope(3.06293548889,51.1743208093,6.5670669221,52.4263865259))
>

Interesting. So your data is stored in mercator and your map is long/
lat? That will require reprojection of the geometries during rendering  
and will be much slower that if you have your Map projection the same  
as your input data.

> to zoom to netherlands map.
>
> Being honest, I don't know exact meaning of value passed to "extent"
> parameter. I have supplied this value which I found on mapnik forum.
>

That is the extent of your layer's features in the coordinates of the  
input geometry. Those ones you pulled from the mapnik forum are the  
extent of (nearly) the whole world in spherical mercator  
(+proj=merc....).

To understand more about this try: http://weait.com/content/map-tiles-and-bounding-boxes

> Could you please explain me in more detail "extent" parameter? What  
> extent
> value shall I specify for europe data for each layer?
>

Also, try seeing the extent of the planet_osm_line by doing:

$ psql postgis_europe
# select ST_Extent(way) from planet_osm_line;

... which is essentially what PostGIS will do in the background if you  
have 'estimate_extent'=False and do not provide the 'extent' parameter.

Dane

_______________________________________________
Mapnik-users mailing list
Mapnik-users@...
https://lists.berlios.de/mailman/listinfo/mapnik-users