EU Forest
Visualizing more than 500,000 trees using Google Maps and Deck.gl
Sometimes we have some projects that challenge us to prove the limits of Google Maps, one of the biggest difficulties that we have as developers when we're usingthis tool, is the need and capacity to display a thousand, or a hundred of thousand points inside a custom map. Is this possible?
Google Maps, and their API have been evolved across the last couple of years, and with this, they've been bring us concepts like `Custom Overlays` that allow us to extendthe natural capacity of what we generated, also created a flexible environment to let projects like deck.gl do a smooth integration.
In this article I would like to tell you my experience to achieve representing 588,983 trees on a regular Google Maps, so I'm encourage you to bring your favorite drink tobe your partner to read this post.(☕|🥤|🍺|🍷)
Table of Contents
- The Inspiration
- Selecting the Problem
- Coordinate Systems, more than latitude and longitude
- Getting trees colors
- And, What is deck.gl?
- Integrating deck.gl and Google Maps
- Conclusion
- References and links
The Inspiration
The past year I had the opportunity to see in action to Alex Muramoto
giving a talk named "Awesome web data viz with Google Maps Platform and deck.gl", if you didn't have the chance to attend one of the multiple
DevFest where Alex was, you can watch the talk on Youtube following the next
link: https://bit.ly/alexsgmapdeckgl.
During the talk, Alex drives us through different examples that allow us to extend Google Maps capacity using Deck.gl, he shows us the basis to create
something beyond add a simple marker, now, we can add
multiple points, arcs and make real time visualizations in determinante map regions, so after a while, I encouraged myself to do an experiment
with this two technologies.
Selecting the Problem
Doing a small research on Internet, I could find a variety of complete datasets through different websites, some of them are the
Paris - Open Data Site or the European Portal Data,
also I retrieved differente papers related with geospatial visualization, one of the catched my eye: "EU-Forest, a high-resolution tree occurrence dataset for Europe"1,
the paper addresses the way and methodology about how they do a information curation related with the European tree species
to obtain an unique dataset that can give a good distribution through different countries, one of the best points is
that the dataset is open and give some information valuable to be used, like coordinates, species and visualization examples
that can be followed as clues to determinate if we're on the right way.
Spatial distribution of all occurrences in EU forest.With all that in place, it will be really exciting reproduce the results of the experiment with Google Maps and Deck.gl.
Coordinate Systems, more than latitude and longitude
In the webpage of the publication you will see different documents, the one that we're going to use as referent2
and make sense for our goal of visualzation is curated respect the tree species, this file is in CSV format and as we can read in the metadata file, the coordinates X and Y are regarding to the ETRS89-LAEA reference ... 🤨, ok, that the fun begins!
Typically the libraries or map services that we integrate in our code, teach us to reference the position as latitude and longitude,
so the first challenge that I had processing the dataset is understand and make the transformation
from the coordinate values present in the file to something that can make sense on Deck.gl or Google Maps.
Is time to get the Google Search help!, I discover that the ETRS89-LAEA has an equivalent: EPSG:3035...
so, with this in mind , now we can try to solve this first part.
EPSG
EPSG is the abbreviation that correspond to "European Petroleum Survey Group", this group made different datasets
with the purpose to have good spatial references that could be applied in a global, regional, national and local way,
so in theory, one of this datasets correspond to the use of latitude or longitude values.
One of the most useful resources that I found and that allows you use all this references simplily, is the webpage epsg.io,
inside it, you can use a service where you can transform the value between different systems.
Doing a quick inspection on the code, I can realize that they are using a library: pro4JS, Bingo! That is the part I need to start the data conversion 😁, and thanks to they a full list of all EPSG codes, I can get the equivalent of the traditional way that we use coordinates: EPSG:4326.
proj4JS
pro4JS, is an Open Source project that take inspiration in other: proJ, this library allow the conversion between coordinate systems,being a very specific niche in the Software World, sometimes the documentation could be not enough or could be hard to understand from where and what calculation is done.
You can install proj4 via npm or yarn to start using it, and according to the documentation the usage of the function is the next:
proj4(fromProjection[, toProjection, coordinates])
The values that are required make reference to projections3, a projection is the operation required to get the equivalent between systems,
under the proJ project, you can learn how to use this project, for now we'll be focused in the value, understand how they form the formula could another great post.
So to define the projection for EPSG:3035 you need to write the next Array:
[
'EPSG:3035',
'+proj=laea +lat_0=52 +lon_0=10 +x_0=4321000 +y_0=3210000 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs ',
];
As I mentioned, the value for the projection could be complicated at the beginning, and many mistakes happen for a misspelling in the string, so that's why I created a repository with a list of values that can be used to obtain the value of a EPSG code through the use of an index. The list contains 6137 values that we can use in the conversions, you can
read the full list and get the names in the README file of the project, to use it on proj4, you need to write a code as follows:
const proj4 = require("proj4")
const proj4list = require("./list.min");
proj4.defs([
proj4list["EPSG:3035"],
proj4list["EPSG:4326"]
);
proj4("EPSG:3035", "EPSG:4326", [ 4305500, 2674500 ]);
So now we have the way to convert the X and Y original values present in the dataset, now we can have them on latitude and longitude!,right now our script to convert the CSV file to a JSON format will follow the next logic:
- 📃 Read each row in the CSV document.
- 🗺 Convert X and Y to latitude y longitude.
- 🌲 Replace the tree specie with an number, that ca be mapped to manage the colors.
The content of our dataset will be something like this:
[
{
"specie": 70,
"lat": 35.01966518543305,
"lng": 32.6269824790667
}, ...
]
If you wouli like access to the final dataset, you can find the information classified by country in the next repository:
Datasets:EU-Forest
Getting trees colors
The visualization that we want to do could fine really well just having with the position of the trees, and the result will be something similar as the presented in the paper, but as users that are constantly consuming information, is important represent the data in the right way and make it a great experience and especially understable.
So with this premise, let's start working on this part 🤓.
The main idea is get a couple of images from Google Search Images, merge them and obtain the main color of it.
To help in this purpose, I found the node.js library that use Puppeteer, images-scraper,
that allow us get an Array of image paths from a query.
const scraper = require('images-scraper');
const google = new scraper({ puppeteer: { headless: true } });
function searchAndAnalize(term, limit) {
/*
results [
{
url: 'https://...',
source: 'https://...',
description: '...'
},
...
]
*/
return new Promise(async (resolve, reject) => {
const results = await google.scrape(term, limit);
});
}
For each element in the Array, we're going to download the file, so now, we need to write a function that allow us save the files into a directory, and at the same time, let us save the path directory inside other Array to be used later, so the function to make the download is the next:
const fs = require('fs');
const request = require('request');
async function downloadImage(url) {
try {
return new Promise((resolve, reject) => {
request.head(url, function (error, res, body) {
if (error) {
reject(error);
}
let extension =
res.headers['content-type'] === 'image/jpeg' ? 'jpg' : 'png';
let file_name = `${collectionPath}/${new Date().getTime()}.${extension}`;
request(url)
.pipe(fs.createWriteStream(file_name))
.on('close', () => {
resolve(file_name);
});
});
});
} catch (error) {
console.log('error in', url);
}
}
Now that we have the downloadImage function, we can proceed to get the images and merge them using merge-img,
since we don't know the image dimensions we're going to save them in PNG, to make a better color analysis.
function searchAndAnalize(term, limit) {
return new Promise(async (resolve, reject) => {
const results = await google.scrape(term, limit);
let images = await Promise.all(
results.map(async (result) => {
return await downloadImage(result.url);
})
);
let img = await mergeImg(images);
const output = `${collectionPath}/result_${new Date().getTime()}.png`;
img.write(`${output}`, async () => {
// HERE WLL BE THE IMAGE ANALYSIS
});
});
}
The resulting image will be something like this:
Now that we have a image to process, we're going to use Color Thief,a popular library that get the main color,
the only parameter that we need to set is the merged image.
img.write(`${output}`, async () => {
resolve(await colorThief.getColor(output));
});
Again you can find the full code in one repository that I created, you can access using via this link.
If you follow the install instructions you'll be able to run the next command in your terminal;
node example.js palette "Batman" 10
The values corresponding to the main colors that this script produces, can be found in the next
file,
they are the colors related with the 242 tree species processed.
And, What is deck.gl?
Deck.gl is one Open Source tool that is changing the way we can do map data visualization as developers, and at the end impact
to our product customers, created by the Uber Visualization team, through WebGL2
can do a better use of our hardware, and let explore and visualize BIG AMOUNT of data using superposed layers in the browser,
also give to the developers a way to personalize each layer, update or remove them if necessary.
First steps using deck.gl
Ok! now that we understand that deck.gl is using layers, is time to use them and visualice something.
One of the characteristics that I like from deck.gl, is the independence of a map system, so you can use it with
techs like Google Maps, Mapbox, Arcgis and either without a map framework.
To have a first approach to deck.gl, we will use two datasets created by geojson.xyz.
We can make use of deck.gl in different ways, one of them and is helpful if you're new in code, is directly loading the script into our HTML.
<script
type="text/javascript"
src="https://unpkg.com/deck.gl@latest/dist.min.js"
></script>
When we add the Javascript without webpack or a module handler, we need to initialize the methods that we're going to use in some variables or constants:
const { DeckGL, GeoJsonLayer, ArcLayer } = deck;
We will also initialize three constants, two of them for the dataset management, and the other will be the central point
corresponding to the Mexico City Airport.
const COUNTRIES =
'https://d2ad6b4ur7yvpq.cloudfront.net/naturalearth-3.3.0/ne_50m_admin_0_scale_rank.geojson';
const AIR_PORTS =
'https://d2ad6b4ur7yvpq.cloudfront.net/naturalearth-3.3.0/ne_10m_airports.geojson';
const INITIAL_VIEW_STATE = {
latitude: 19.43,
longitude: -99.08,
zoom: 4,
bearing: 0,
pitch: 30,
};
To start deck.gl,now we need to create a new instance of the main Class, adding as parameters
an initial state and an Array corresponding to the layers where we will show the information.
new DeckGL({
initialViewState: INITIAL_VIEW_STATE,
controller: true,
layers: [
// ... HERE WILL BE THE DATA LAYERS
],
});
Given that our datasets are already in JSON format, we can use GeoJsonLayer to
create them quickly, in this case we're going to start adding the Country layer.
...
layers : [
new GeoJsonLayer({
id: 'country-layer', //Layer Id
data: COUNTRIES, //Dataset
stroked: true, //Stroke or Border
filled: true, //Fill
lineWidthMinPixels: 1, //Pixel Width
opacity: 0.4, //Opacity
getLineColor: [154, 154, 154], //RGB color for stroke
getFillColor: [242, 242, 242] //RGB color for fill
}),
]
...
On screen, we're going to have a result as shown in the following image:
Deck.gl's ability to manage layers allows us to create our own maps.With the map created, we can now add a layer corresponding to the existing airports:
...
layers : [
...,
new GeoJsonLayer({
id: 'airports',
data: AIR_PORTS,
filled: true,
pointRadiusMinPixels: 2,
pointRadiusScale: 2000,
getRadius: f => 11 - f.properties.scalerank,
getFillColor:[21, 192, 25],
pickable: true,
autoHighlight: true,
onClick: info =>
info.object && alert(`${info.object.properties.name}`)
})
]
...
Now we have two overlays, one corresponding to the map and the other to airports.Now we will add a new layer to display some connectivity, for that we're going to use
ArcLayer, the idea is to have a single point as origin, from it the ars will be displayed as airport connections.
new ArcLayer({
id: 'arcs',
data: AIR_PORTS,
dataTransform: (d) => d.features.filter((f) => true),
getSourcePosition: (f) => [-99.08, 19.43],
getTargetPosition: (f) => f.geometry.coordinates,
getSourceColor: [238, 157, 30],
getTargetColor: [21, 192, 25],
getWidth: 1,
});
The result of the 3 layers would be the following:
With the property dataTransform, we can also modify the behavior of the data,
También podemos modificar el comportamiento de los datos relacionados con la propiedad de dataTransform,
let's imagine that we need to display only the connections between Mexico and the
existing airports in the strip shown in the following image.
To do this, we will modify the attribute as follows:
const MAX = -93, MIN = -90;
...
new ArcLayer({
...
dataTransform: d => d.features.filter(f => f.geometry.coordinates[0] < MIN && f.geometry.coordinates[0] > MAX),
...
})
...
Being one of the most popular map systems to represent information, Google Maps can be used together with deck.gl
for layer management, reading the source code
we can see that it uses Custom Overlays to achieve integration.
...
export default class GoogleMapsOverlay {
constructor(props) {
this.props = {};
this._map = null;
const overlay = new google.maps.OverlayView();
overlay.onAdd = this._onAdd.bind(this);
...
Understanding Custom Overlays
We cand define Custom Overlays4as objects that will be on the map and that are linked to
coordinates given by a latitude and longitude. One of the main features is that they will move when you zoom or drag the map to another position.
Using Custom Overlays mainly requires 5 steps:
1.-Set your custom overlay object's prototype to a new instance of google.maps.OverlayView()
USGSOverlay.prototype = new google.maps.OverlayView();
2.- Create a constructor for your custom overlay, and set any initialization parameters.
function USGSOverlay(bounds, image, map) {
this.bounds_ = bounds;
this.image_ = image;
this.map_ = map;
this.div_ = null;
this.setMap(map);
}
3.- Implement an onAdd() method within your prototype, and attach the overlay to the map.
USGSOverlay.prototype.onAdd = function () {
var div = document.createElement('div');
div.style.borderStyle = 'none';
div.style.borderWidth = '0px';
div.style.position = 'absolute';
var img = document.createElement('img');
img.src = this.image_;
img.style.width = '100%';
img.style.height = '100%';
img.style.position = 'absolute';
div.appendChild(img);
this.div_ = div;
var panes = this.getPanes();
panes.overlayLayer.appendChild(div);
};
4.- Implement a draw() method within your prototype, and handle the visual display of your object.
USGSOverlay.prototype.draw = function () {
var overlayProjection = this.getProjection();
var sw = overlayProjection.fromLatLngToDivPixel(this.bounds_.getSouthWest());
var ne = overlayProjection.fromLatLngToDivPixel(this.bounds_.getNorthEast());
var div = this.div_;
div.style.left = sw.x + 'px';
div.style.top = ne.y + 'px';
div.style.width = ne.x - sw.x + 'px';
div.style.height = sw.y - ne.y + 'px';
};
5.- And finally you should also implement an onRemove().
USGSOverlay.prototype.onRemove = function () {
this.div_.parentNode.removeChild(this.div_);
this.div_ = null;
};
Once we have implemented our Custom Overlay we can call it inside the function we used to start
our map.
In our method we need to define the area that our image will occupy,
for this we will use the LatLngBounds method, which receives 2 parameters: LatLng southwest and LatLng northeast, this wants
say that to correctly indicate the position of our image, we first have to indicate the corresponding coordinate
to the lower left side in which it will be positioned, and as second the upper right coordinate.
function initMap() {
//Nuestro mapa principal
var map = new google.maps.Map(document.getElementById('map'), {
zoom: 11,
center: { lat: 19.487711, lng: -99.008554 },
});
//Indicamos el área que ocupará la imagen
var bounds = new google.maps.LatLngBounds(
//Suroeste o inferior izquierdo
new google.maps.LatLng(19.389876, -99.1009),
//Noreste o superior derecho
new google.maps.LatLng(19.599925, -98.858176)
);
//Imagen que agregaremos al mapa
var srcImage = './map.png';
//Nuestro overlay recive el área, la imagen y el mapa
overlay = new USGSOverlay(bounds, srcImage, map);
}
In the following example, we can see the result of superimposing an image corresponding to the area of
Texcoco Lake5 in Mexico City.
Google Maps Overlay and our trees dataset
All right! The time has come to apply everything we have learned, we already have our cured dataset of tree species,and the main color for each one, also we already have a better understanding about how deck.gl and Custom Overlay work.
As we mentioned, deck.gl has its own method to make use of the Custom Overlays calledGoogleMapsOverlay, and with it, we can save lot of the work and time as we saw in the previous section.
import { colors } from './colors'; //Tree colors
async function init() {
await loadScript();
const GMAP = new google.maps.Map(MAP, MAP_PROPS);
const deckGL_overlay = new GoogleMapsOverlay();
//Here we indicate the map that we want to use
deckGL_overlay.setMap(GMAP);
//Adding the layers as Array,
deckGL_overlay.setProps({ layers: [await getLayer()] });
}
As we can see in the implementation, we need to send the layers into an Array as property of the GoogleMapsOverlay variable.
Now, we can do the request to get the tree positions that processed previously, we're going to implement a function named getLayer, an in this, we'll get the data directly from the JSON file that contains them,
we will also indicate the color to display respect to the specie index.
async function getLayer(layer = 'all') {
//Realizamos la petición de información
let request = await fetch(
`https://dataset-euforest.storage.googleapis.com/country_${layer}.json`
);
let data = await request.json();
return await new ScatterplotLayer({
id: 'scatterplot-layer',
data: data, //Esta es la inforamción de los árboles
opacity: 1,
stroked: false,
filled: true,
radiusScale: 20,
radiusMinPixels: 1,
radiusMaxPixels: 100,
lineWidthMinPixels: 1,
//Obtenemos la latitu y la longitud
getPosition: (d) => [d.lng, d.lat],
getRadius: (d) => 50,
//A través del índice de la especie, retornamos el color
getFillColor: (d) => colors[d.specie],
getLineColor: (d) => [30, 30, 30],
});
}
The result can be seen in the following iframe or by visiting the URL: https://eu-forest.mapsviz.com .
If you want, you can visit the repository with the full code,
and take a look about how the web was built.
Conclusion
There is no doubt, that the combination of deck.gl and Google Maps allow us generate experiences to visualize data in a
very simple way and with relatively little computing power. Nowadays, we have the tools that make possible create
Geospatial analysis newly way, different what we used to, also on a relative low cost.
Although exist tasks that are already solved by specialized software, we can give them a different perspective and "democratize" it thanks to the web technologies.
In these times when the information flow from different points and is constant, allow that other people understand the magnitude of it,
is one of the duties that we have as Software Engineers.
For me make this experiment was really fun, I learned about different ways that we have developed to measure and represent
our location, including things that surround us closely or globally. I hope this article is a first step to inspire you, feed your
curiosity or be a reference to create an interesting new visualization.
References and links
4.- Custom Overlays Google Maps Documentation
5.- All Things living, all things dead - Cartografías del Lago de Texcoco
