Jan 18

Political boundaries with open-source licenses

This is a post to reference some useful websites to download shapefiles of countries political boundaries. It happens regularly to me that I need to include borders in my web applications and I also want to be cautious about licensing. So here are the best places to find them :

The original  : http://thematicmapping.org/downloads/world_borders.php. There are two datasets (a full version and a simplified version). The data is provided under CC-BY-SA which means Share-Alike. So you need to publish any change to the dataset in the same way. This dataset is not maintained ans for example South Sudan is missing from this dataset. Nevertheless, it is still pretty useful and I have created a version in GeoJSON here.

If you need just one country in a specific format, a fast option is to go to http://gadm.org/country where you can quickly download exactly what you want in the format that you need. It is more precise than the previous site and seems to be maintenaind with a list of “known problems”. Nevertheless, the main issue is the license as it is only “freely available for academic and other non-commercial use. Redistribution, or commercial use, is not allowed without prior permission.”

And finally, the best : http://www.naturalearthdata.com/downloads/. This website has a load of useful datasets to download and the license is very good too as it is a Public Domaine licence.  You can use this data freely for any personal, educational, and commercial purposes. Borders are provided at three different scale : 1/10m, 1/50m and 1/110m. I have converted all three datasets in GeoJSON and then created a version that mix that various resolution so that any single GeoJSON file per country is never heavier than 150KB. You can find it here.

Posted in English, News | Comments Off on Political boundaries with open-source licenses
Jan 02

Multi-threaded upload to Amazon S3 in Python

I recently wanted to upload almost 9 millions of imagery files to Amazon S3. This was a TileCache of a whole country up to level 17. Each JPEG or PNG file was relatively small with an average size of 6 KB. Globally, it summed up to 53 GB which is a mere 1.3 US$/month of hosting on Amazon with reduced redundancy (only 99.99% instead of 99.999999999% durability).

But the biggest challenge was the huge number of single upload requests to make to S3. This was a good opportunity to test multi-thread python scripts. Here is the script that you can use for your own needs. You just need to create a configuration file called main.cfg file with your Amazon AWS id key and secret key or pass them in the command line.

Over a standard ADSL line, it took this script 2 days to upload the full TileCache to Amazon S3. The script was running 40 threads in parallel and the upload bandwidth was capping at roughly 10 Mb/s. The next step is to improve my Internet connection !

Posted in English, News | Comments Off on Multi-threaded upload to Amazon S3 in Python
Oct 13

PHP Proxy from Esri

A JavaScript application cannot access a resource or an API located on another website through Ajax due to security restriction known as “Same Origin Policy“. Most public API have implemented the CORS mechanism that enables the browser to discuss with the server before allowing a cross-origin request.

But some API, like the Airbus Catalog API, have not implemented the CORS mechanism. In this case, it is necessary to use a proxy on the server that will redirect requests to the API. The JavaScript application will only access to its origin server.

Nevertheless, setting up a proxy is tricky because it must me configured properly in order not to be a security breach. Esri has made available a proxy that can be easily configured and used. A .NET, Java and PHP version are available : https://github.com/Esri/resource-proxy/

Posted in News | Comments Off on PHP Proxy from Esri
Sep 15

Get satellite imagery metadata

Let’s assume that you have a list of identifiers from Airbus DS satellite imagery and that you would like to get a list of all associated metadata to sort them and organize them.

All you need is an Airbus DS Catalog API key and this little python script !
https://gist.github.com/jeffaudi/9fcac727cf6251287ecc

It is useful to note that since this week, the Airbus DS Catalog API provides some extra angles to characterize an image acquisition :

  • incidence angle : this is the angle between the viewing direction of the satellite and the vertical to the ground (at the middle of the image)
  • along-the-track incidence angle : this is the angle between the forward or backward viewing direction of the satellite and the vertical to the ground.
  • across-the-track incidence angle : this  is the angle between the side-viewing direction of the satellite and the vertical to the ground.
  • combined viewing angle : this is the angle between the viewing angle of the satellite and the vertical to the satellite
  • roll : this is the angle that the satellite does to take pictures on the side
  • pitch : this is the angle that the satellite does to take pictures forward or backward (negative)

Analyzing these angles with the date of acquisition enables to find stereo pairs in the archive. Stereo pairs are useful to create Digital Elevation Models.

Posted in English, News | Comments Off on Get satellite imagery metadata
Jul 30

TMS and GoogleMaps Tiles

Even for something as simple as Web Mercator tiles for web mapping applications like Google Maps, there can be different standards ! Google actually numbers tiles from North to South while the open source TMS format defines the numbering from South the North. OpenStreetMap uses the same scheme as Google probably to make it more easy to switch from one to the other. Here is the link to the documentation for Google Maps coordinates.

As gdal2tiles.py follows the TMS standard, it numbers the tiles from South to North. If you want to use the script to produce GoogleMaps compatible tiles, you should follow this article to modify the script. Here is a link to the modified script. Basically, it revert the Y-axis by doing ynew = 2^zoom – yold. The number of tiles at each level being 2 at the power of current zoom level. See here for zoom level relation to ground resolution.

The example in my previous post can be reprocessed to with a GoogleMaps tiling : http://wonthaggi2.s3-website-eu-west-1.amazonaws.com/airbusmaps.html

Then the standard googlemaps.html file generated by gdal2tiles.py needs to be changed to support the Google Maps tile naming. You can find a new version here (also updated with Google Maps API V3).

Posted in English, News | Comments Off on TMS and GoogleMaps Tiles
Jul 24

Airbus DS @ Esri UC 2014

Here is the extract of the video presenting the Airbus Archive & Tasking App now accessible from ArcGIS Online MarketPlace. A great moment during the Esri User Conference 2014 plenary session !


Extracted from a video posted on Esri Video website.
Posted in English, News | Comments Off on Airbus DS @ Esri UC 2014
Jul 12

Un serveur web local grâce à Node.JS

Le temps est fini où il fallait installer Apache sur son portable pour faire du développement web en local. Grâce à Node.JS, on peut mettre cela en place en quelques secondes sur toutes les systèmes d’exploitation (présentation ici sous Mac)

La première étape est d’installer Node.JS depuis la page nodejs.org. Vous pouvez trouver des informations utiles sur ce tutorial (en anglais).

Ensuite dans le répertoire choisi, taper les commandes suivantes :

npm init
npm install express —save

Puis créer un fichier server.js pour y placer le code JavaScript suivant :

var express = require('express');
 
var server = express();
server.use(express.static(__dirname + '/www'));
 
var port = 10001;
server.listen(port, function() {
    console.log('server listening on port ' + port);
});

Et enfin dans un terminal, lancer votre server web :

node server.js

Il reste à ouvrir http://localhost:10001/index.htm dans votre navigateur.
Ta dam !

Posted in Français, News | Comments Off on Un serveur web local grâce à Node.JS
Jul 09

New Map Journal Template for Esri Story Maps

I have tweeted recently about the great Story Map that the Esri team has put together about Brazil and its World Cup Stadiums.

MapJournal

Display on a large laptop screen

I really like this app because it mixes maps, photos and satellite imagery in a very nice and seamless way. It is also impressive as a responsive app that adapts to the size of the current screen :

BrazilStadiums2

Display on an iPhone

And as the World Cup was drawing to an end, Esri has released this as a Configurable App to be used with your ArcGIS Online content: Map Journal. And the source code is also available on GitHub also for full customization ! Great job !

Posted in English, News | Comments Off on New Map Journal Template for Esri Story Maps
Jun 16

Streaming Satellite Imagery from Amazon Web Services

Following my previous post, this is another one using GDAL and Amazon.

Under Windows, install GDAL from the OSGeo4W installer.

Then, use the gdal2tiles script to create the tiles :

gdal2tiles [imagery-file]

You can add the option –srcnodata=0 if your imagery contains NODATA values. After the folders and tilesare created, we need to upload everything to Amazon S3 (Simple Storage Service). Then, we need to make the content public and enable website hosting. After a few minutes, the image is available online :

http://s3-eu-west-1.amazonaws.com/wonthaggi/index.html

Based on the official Amazon pricing for web services (@ $0.03 / GB / month), hosting these 50MB of imagery will cost 1.8 cent per year. Using JPEG compression for the tiles would further reduce that. You just need to make a provision for bandwidth consumption @ $0.004 per 10 000 requests depending on the usage.

The next step is just to add a Content Delivery Network (CDN) like CloudFront on top of that service to provide an efficient experience worldwide :

http://dpzosckqhekmg.cloudfront.net/index.html

Posted in News | Comments Off on Streaming Satellite Imagery from Amazon Web Services