Benchmarking Mapping Toolkits in Tile Seeding


1. Introduction

The unprecedented growth of different cartographic web services (Google Maps, Here Maps, Bing Maps, Navteq, Mapbox, CartoDB etc.) and the availability of a huge variety of user-defined maps have rapidly expanded the distribution of maps. The most common way of distributing maps through internet is using a web map tile service (WMTS). WMTS is a standard protocol for serving a set of pre-rendered georeferenced map images, called tiles, over the Internet. The most obvious benefit of a tile based approach is that map tiles can be prepared (rendered) beforehand at any location. Use of tile caches increases the performance and scalability of map services significantly. Nowadays, a great variety of software packages for rendering maps exist. For example, these packages are ArcGIS Server, QGIS Server, MapServer, GeoServer, Mapnik, Cadcorp GeognoSIS, Thinkgeo's Map Suite, and many others. There are two basic factors that show how powerful a certain software package is. These factors are: the capability to produce a cartographically plausible product and the performance of the software. In the context of WMS Shootout (see also this post), there have been many attempts to compare the performance of different tools in rendering spatial data. According to the latest available benchmarks, conducted in 2010-2011, the winners were MapServer and Mapnik, as they showed the best performance in the tests and left far behind other competitors.

This article concentrates on comparing performance of different existing GIS applications and libraries in map generation, namely in tile seeding. Generally, the performance of such programs depends on different factors such as data fetching, feature rasterization (rendering), image quantization, disk writing, memory re-allocation etc. All these factors influence the performance of tile seeding as a whole.

Note that the given benchmarks is only roughly presents the actual performance of the tested toolkits, as their performance characteristics are platform dependent and we conducted our experiments only on a Windows machine. Furthermore, it is rather difficult to properly configure the toolkits as they have diverse set of parameters. Nevertheless, we tried to do our best to make the comparison of the software packages under equal conditions.

2.Software packages

The goal of this article is to provide performance comparisons between different free, open source and commercial applications and toolkits for rendering maps. They are:

A full description of each tested toolkit is given in the table below.

ToolkitLanguageRuntime EnvironmentArchitectureVersionBuild Date
GeoServerJavaJRE 7/8x642.7.020.03.2015
MapServerC/C++, C#unmanaged codex866.4.102.01. 2014
Map SuiteC#.NET 4.0x648.0.0.022.12.2014
MapSurfer.NETC#/C++.NET 4.5x642.0.302.06.2015
Mapnik/TileMillC++/Node.jsunmanaged codex862.2/0.10.110.10.2012

As it can be seen, the libraries and toolkits are written in different programming languages such as C/C++, C# and Java. Furthermore, they are either compiled into so called unmanaged code (C/C++) or executed in runtime environments such as Java Runtime Environment (JRE) and Microsoft’s .NET framework. GeoServer has been tested using two versions of the runtime environment. Note that according to the official information JRE 8 is not supported.

3. Testing Environment, Datasets and Settings

Our experiments have been performed on a machine with an Intel® Core™ i5-2500 CPU @ 3.30 GHz running Windows 7 Professional x64 with 8GB installed memory. The runtime execution environments of our test application were JRE 7/8 (x64) and .NET Framework 4.5 (x64) respectively.

We ran our tests on datasets derived from OpenStreetMap data. More precisely, we took shapefiles of land areas (continents and islands) from OpenStreetMapData web site. Two shapefiles (split versions) with data in WGS84 and Mercator projection, with about 400Mb in size each, were used. Each toolkit has been configured using the following parameters:

ToolkitShapefile IndexMetatileImage Format# of ThreadsCache Format
GeoServer (JRE 7/8)yes8x8png84disk directories
MapServeryes8x8png84disk directories
Map Suiteyes1x1png244disk directories
MapSurfer.NETyes8x8png84mbtiles
Mapnik/TileMillyes8x8png84mbtiles

It is important to make a note of some differences in configuration and custom changes in tile seeding procedures. First, Map Suite Tile Cache Generator does not have capabilities to work with metatiles or save tiles in png8 format. Nevertheless, we decided to include this library in our benchmarking. Second, we make use MapServer and C# mapscript to render and slice metatiles into smaller tiles. Third, it can be seen in the table that TileMill and MapSurfer.NET use an mbtiles storage. According to our prior tests, the difference in performance between “disk directories” and mbtiles is not significant (less than 1%).

The shapefiles and index files were stored on Samsung SSD P810 128GB. Western Digital WD1002FBYS Caviar RE3 SATA 1TB hard drive was used to store tile caches. For Test #3, we use the same Western Digital SATA disk to store data in a PostgreSQL database.

In the experiments, every toolkit was run two times and the best shown time was wrote down.

3.1 Map Styles

In the experiments, we use two maps styles. These styles are:

Style 1

CartoCSS

 #landpolygons {
   polygon-fill:#000;
   line-color:#ffffff;
   line-width:1;  
 }

SLD

<UserStyle>
  <FeatureTypeStyle>
    <Rule>
      <Name>Rule1</Name>
      <Title>Polygon with Outline</Title>
      <PolygonSymbolizer>
        <Fill>
          <CssParameter name="fill">#AAAAAA</CssParameter>
        </Fill>
      </PolygonSymbolizer>
      <LineSymbolizer>
        <Stroke>
          <CssParameter name="stroke">#FF0000</CssParameter>
          <CssParameter name="stroke-width">1</CssParameter>
        </Stroke>   
      </LineSymbolizer>
    </Rule>
  </FeatureTypeStyle>
</UserStyle>

#### Style 2

In Style 2, we have added a TextStymbolizer to label polygons.

CartoCSS

 #landpolygons {
   polygon-fill:#000;
   line-color:#ffffff;
   line-width:1;  

   ::label{
     text-face-name:"DejaVu Sans Book";
     text-name:"[FID]";
     text-fill:#444;
     text-halo-fill:rgba(255,255,255,1.0);
     text-halo-radius:0;
     text-size:9; 
  }
 }

SLD

<UserStyle>
  <FeatureTypeStyle>
    <Rule>
      <Name>Rule1</Name>
      <Title>Polygon with Outline</Title>
      <PolygonSymbolizer>
        <Fill>
          <CssParameter name="fill">#AAAAAA</CssParameter>
        </Fill>
      </PolygonSymbolizer>
      <LineSymbolizer>
        <Stroke>
          <CssParameter name="stroke">#FF0000</CssParameter>
          <CssParameter name="stroke-width">1</CssParameter>
        </Stroke>   
      </LineSymbolizer>
    </Rule>
  </FeatureTypeStyle>
  <FeatureTypeStyle>
    <Rule>
      <Name>Rule 2</Name>
      <Title>Text</Title>
      <TextSymbolizer>
        <Geometry>
          <ogc:Function name="centroid">
            <ogc:PropertyName>the_geom</ogc:PropertyName>
          </ogc:Function>
        </Geometry>
        <Label>
           <ogc:PropertyName>FID</ogc:PropertyName>
        </Label>
        <Font>
          <CssParameter name="font-family">DejaVu Sans Book</CssParameter>
          <CssParameter name="font-size">9</CssParameter>
          <CssParameter name="font-style">normal</CssParameter>
         </Font>
        <LabelPlacement>
          <PointPlacement>
            <AnchorPoint>
              <AnchorPointX>0.5</AnchorPointX>
              <AnchorPointY>0.0</AnchorPointY>
            </AnchorPoint>
          </PointPlacement>
        </LabelPlacement>
        <Fill>
          <CssParameter name="fill">#990099</CssParameter>
        </Fill>
      </TextSymbolizer>
    </Rule>
  </FeatureTypeStyle>
</UserStyle>

4. Comparisons

The following sections compare mapping toolkits using different map styles and storages for spatial data.

4.1 Test #1: Shapefile without Re-projection

In this section, we present the results of our benchmarks evaluating time needed to render a large shapefile (approx. 400 MB) without re-projection and to seed tiles for small scales (see Figure 1), to be more exact for zoom levels z0 to z9. We used Style 1 in this scenario.

Figure 1

A bar chart in Figure 2 shows the throughput comparison of mapping toolkits.

Figure 2

Figure 3 illustrates the results of the second benchmark, which has been carried out using Style 2.

Figure 3

4.2 Test #2: Shapefile with Re-projection

In this section, the mapping toolkits are tested on how fast they can re-project vector data. The task was to re-project the coordinates of geometries from WGS84 (EPSG:4326) to Spherical Mercator projection (EPSG:3857). Style 1 was used in this benchmark.

Figure 4

4.3 Test #3: PostgreSQL (PostGIS) without re-projection

In this section, the results of our experiments using PostgreSQL 9.3 (PostGIS 2.0) as data storage are given. The shapefile used in Test #1 has been imported into a PostgreSQL table using the same EPSG:3857 projection.

Figure 5

4.4 Tests Summary

This section summarizes the results of our benchmarks given in Sections 4.1-4.3. As can be seen from the charts (see Figure 1-5) given above, in most tests, especially when the number of tiles is very large (i.e. at z8 or z9), the best performance has been shown by MapSurfer.NET framework. In bar graphs (Figure 6-7) below, the ratio number (x-axis) indicates how much is slower or faster a tested software package in comparison to MapSurfer.NET.

Figure 6

Figure 7

5. Conclusions

In general, the worst performance has been shown by Map Suite. It can be explained by the fact that Map Suite does not support metatiling and is not able to save 8-bit png files. Nevertheless, at relatively larger scales (z8, z9) this library is even faster than GeoServer using JRE 8. At the same time, GeoServer using JRE 7 is faster than GeoServer running JRE 8. GeoServer has impressive benchmarking results at small scales (z0-z4) in Test #1. This is most likely because it stores the whole shapefile in memory. Next, MapServer was ultra fast in all tests in generating tiles for zoom levels z0-z7. At larger scales (z8-z9), MapServer has been overcome by two other competitors such as Mapnik/TileMill and MapSurfer.NET. As a whole, TileMill powered by Mapnik demonstrated really good performance in all tests except Test #2 (see Figure 7), where it failed and performed quite poorly. It was 1.85 times slower than MapServer. Probably, it is a result of a well-known issue in Mapnik when it deals with a re-projection using proj4 library. In our benchmarking tests, MapSurfer.NET had a role of a black horse. It has not demonstrated any noticeable results in making tiles at small scales. But suddenly MapSurfer.NET shot ahead by demonstrating amazing performance (see Figures 6-7) in generating thousands of tiles with lightning speed. For examples, it reaches throughput of 1285 tiles per second (see Figure 2).

To summarize the results, Mapnik/TileMill and MapSurfer.NET are two fastest renderers and tile seeders in our tests conducted on a Windows machine. In tests #2 and #3 these toolkits are about 400% and 700% faster than GeoServer and Map Suite (see Figures 6-7).

Furthermore, we observed that GeoServer (both JREs) and Map Suite require much more memory to perform the tasks than other tested toolkits. Their memory consumption was about 700-770 MB against 250-300 MB by others respectively.

It is important to note that we used quite old versions of Mapnik/TileMill and MapServer. It is most likely that recently released versions (RC) of Mapnik (v3.0.0 rc3) and MapServer (v7.0 beta), which have a bunch of new features and improvements, can show even better results with greater performance gains. Unfortunately, it is unlikely that we will be able to test TileMill using latest Mapnik, as TileMill is no longer in active development. Instead, we could use a new open-source mapping framework kosmtik that internally uses Mapnik to render maps.

See also


Share this:

Comments

comments powered by Disqus