Commit d060ce41 authored by David Youssefi's avatar David Youssefi
Browse files
parents 37718945 3f15f95f
# OTB guided tour
1. Install VirtualBox https://www.virtualbox.org/
2. Download the Ubuntu Budgies iso https://ubuntubudgie.org/downloads
3. Download the OTB standalone package https://www.orfeo-toolbox.org/download/
4. Download the Miniconda installer https://docs.conda.io/en/latest/miniconda.html
4. Create Virtual Machine (2048Mo, VMDK, 15Gio) and install VirtualBox Guest Additions
5. Install librairies and softwares
- Miniconda : ./Miniconda3-latest-Linux-x86_64.sh
- Create otb env : conda create -n otb python=3.5 numpy rasterio jupyter matplotlib gdal
- Install ipyleaflet : pip install --user ipyleaflet +
- OTB : ./OTB-6.6.1-Linux64.run --target /home/otb/OTB
- Jupyter
6. Configure network settings (proxy & certificates) for CNES environment.
Instructions from : https://gitlab.cnes.fr/hpc/wikiHPC/wikis/connexion-proxy
and https://gitlab.cnes.fr/hpc/wikiHPC/wikis/Acc%C3%A8s-%C3%A0-un-site-https-hors-master-CNES
7. Add dataset
**Note:** Python3 workaround (https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/issues/1540)
\ No newline at end of file
# OTB guided tour
\ No newline at end of file
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img style=\"float: left; margin:0px 15px 15px 0px; width:120px\" src=\"https://www.orfeo-toolbox.org/wp-content/uploads/2016/03/logo-orfeo-toolbox.png\">\n",
"\n",
"# OTB Guided Tour - FOSS4G 2019 Bucharest \n",
"## Yannick TANGUY and David YOUSSEFI (CNES, French Space Agency)\n",
"\n",
"<br>\n",
"\n",
"<b> Press <span style=\"color:black;background:yellow\">SHIFT+ENTER</span> to execute the notebook interactively cell by cell </b></div>\n",
"\n",
"## Let's play with OTB classification framework to produce a land cover classification"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import otbApplication\n",
"import os\n",
"import utils\n",
"import numpy as np\n",
"\n",
"# Data directory\n",
"DATA_DIR = \"data\"\n",
"\n",
"# Our dataset contains polygons, of 4 different land cover classes : urban, cultures, forest, sea/water\n",
"DB = \"GroundTruth_4_classes.sqlite\"\n",
"\n",
"# Associates a RGB color to each class\n",
"CMAP = \"4_classes_colormap.txt\"\n",
"\n",
"# Output directory\n",
"OUTPUT_DIR = \"output\"\n",
"\n",
"# Input / Output filenames\n",
"images_list = utils.list_images(DATA_DIR,\"xt_SENTINEL2B\",\".tif\")\n",
"training_set = DATA_DIR+\"/\"+DB\n",
"colormap = DATA_DIR+\"/\"+CMAP\n",
"image_stack = OUTPUT_DIR+\"/\"+\"image_stack.tif\"\n",
"stats = OUTPUT_DIR+\"/\"+\"stats.xml\"\n",
"samples = OUTPUT_DIR+\"/\"+\"samples.sqlite\"\n",
"rf_model = OUTPUT_DIR+\"/\"+\"rf_model.txt\"\n",
"classif = OUTPUT_DIR+\"/\"+\"classif_4_classes.tif\"\n",
"confmatrix = OUTPUT_DIR+\"/\"+\"conf_matrix.tif\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### First, make an image stack with all our images\n",
"We will work on a (small..) time serie of Sentinel2 images"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"app = otbApplication.Registry.CreateApplication(\"ConcatenateImages\")\n",
"app.SetParameterStringList(\"il\",images_list)\n",
"app.SetParameterString(\"out\",image_stack)\n",
"app.ExecuteAndWriteOutput()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Then, we compute stats on our ground-truth polygons\n",
"The aim is to observe how the different classes are represented over the input region..."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"app = otbApplication.Registry.CreateApplication(\"PolygonClassStatistics\")\n",
"app.SetParameterString(\"in\",image_stack)\n",
"app.SetParameterString(\"vec\",training_set)\n",
"# OTB Python API trick : we need to call \"UpdateParameters\" so OTB opens the training set\n",
"# and list the possible field names...\n",
"app.UpdateParameters()\n",
"app.SetParameterString(\"field\",\"code\")\n",
"app.SetParameterString(\"out\",stats)\n",
"app.ExecuteAndWriteOutput()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Display stats file"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"stats_file = open(stats,\"r\") \n",
"for cpt in range(8):\n",
" print(stats_file.readline())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### We observe that the class \"3\" (forest) is less represented\n",
"* we will limit the number of samples per class to this value (or lower)\n",
"* a lot of other strategies exist !!\n",
"\n",
"Let's choose 80 000 samples by class"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"app = otbApplication.Registry.CreateApplication(\"SampleSelection\")\n",
"app.SetParameterString(\"in\",image_stack)\n",
"app.SetParameterString(\"vec\",training_set)\n",
"# OTB Python API trick : we need to call \"UpdateParameters\" so OTB opens the training set\n",
"# and list the possible field names...\n",
"app.UpdateParameters()\n",
"app.SetParameterString(\"field\",\"code\")\n",
"app.SetParameterString(\"instats\",stats)\n",
"app.SetParameterString(\"strategy\",\"constant\")\n",
"app.SetParameterInt(\"strategy.constant.nb\",80000)\n",
"app.SetParameterString(\"out\",samples)\n",
"app.ExecuteAndWriteOutput()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Now we have to extract values for each sample, from our image stack\n",
"\n",
"Each sample (a pixel of 10m by 10m) will be updated with the time serie values (Blue band for image 1, Green band for image 1, ..., Near Infrared band for image 3)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"app = otbApplication.Registry.CreateApplication(\"SampleExtraction\")\n",
"app.SetParameterString(\"in\",image_stack)\n",
"app.SetParameterString(\"vec\",samples)\n",
"# OTB Python API trick : we need to call \"UpdateParameters\" so OTB opens the training set\n",
"# and list the possible field names...\n",
"app.UpdateParameters()\n",
"app.SetParameterString(\"field\",\"code\")\n",
"app.SetParameterString(\"outfield\",\"prefix\")\n",
"app.SetParameterString(\"outfield.prefix.name\",\"band_\")\n",
"app.ExecuteAndWriteOutput()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### We can now train our classifier !\n",
"* Let's choose a Random Forest classifier\n",
"* it will learn from our sample set \n",
"* it needs to know :\n",
" * the name of the field representing the classes\n",
" * the names of the parameters to learn from"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"app = otbApplication.Registry.CreateApplication(\"TrainVectorClassifier\")\n",
"app.SetParameterStringList(\"io.vd\",[samples])\n",
"app.SetParameterString(\"io.out\",rf_model)\n",
"# OTB Python API trick : we need to call \"UpdateParameters\" so OTB opens the training set\n",
"# and list the possible field names...\n",
"app.UpdateParameters()\n",
"app.SetParameterString(\"cfield\",\"code\")\n",
"app.SetParameterStringList(\"feat\",['band_0', 'band_1', 'band_2', 'band_3', \n",
" 'band_4', 'band_5', 'band_6', 'band_7', \n",
" 'band_8', 'band_9', 'band_10', 'band_11'])\n",
"app.SetParameterString(\"classifier\",\"rf\")\n",
"app.SetParameterInt(\"classifier.rf.cat\",4)\n",
"app.SetParameterString(\"io.confmatout\",confmatrix)\n",
"app.ExecuteAndWriteOutput()\n",
"\n",
"# workaround because the UpdateParameters() did not seem to work fine for this app !\n",
"#import subprocess\n",
"#args = ['otbcli_TrainVectorClassifier', '-io.vd', samples, \n",
"# '-cfield', 'code', '-io.out', rf_model, '-classifier', 'rf', #\n",
"# '-classifier.rf.cat', '4', '-feat', 'band_0', 'band_1', 'band_#2', \n",
"# 'band_3', 'band_4', 'band_5', 'band_6', 'band_7', 'band_8',# \n",
"# 'band_9', 'band_10', 'band_11','-io.confmatout',confmatrix]#\n",
"#subprocess.call(args)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The confusion matrix give us some stats on the learning step"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"stats_learn = open(confmatrix,\"r\") \n",
"#for line in stats_learn:\n",
"# print(line)\n",
" \n",
"tab_val = []\n",
"for i in range(6):\n",
" line = stats_learn.readline()\n",
" if (i>1):\n",
" datas = np.array(line.split(',')).astype(int)\n",
" print(\"Recall class\",i-2,\" \\t:\",datas[i-2]/datas.sum())\n",
" tab_val.append(datas)\n",
"print(\"---------------------------------------------\")\n",
"for i in range(4):\n",
" precision = tab_val[i][i]/(tab_val[0][i]+tab_val[1][i]+tab_val[2][i]+tab_val[3][i])\n",
" print(\"Precision class\",i, \" \\t:\",precision)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"app = otbApplication.Registry.CreateApplication(\"ImageClassifier\")\n",
"app.SetParameterString(\"in\",image_stack)\n",
"app.SetParameterString(\"model\",rf_model)\n",
"app.SetParameterString(\"out\",classif)\n",
"app.Execute()\n",
"\n",
"app2 = otbApplication.Registry.CreateApplication(\"ColorMapping\")\n",
"app2.SetParameterInputImage(\"in\",app.GetParameterOutputImage(\"out\"))\n",
"app2.SetParameterString(\"method\",\"custom\")\n",
"app2.SetParameterString(\"method.custom.lut\",colormap)\n",
"app2.SetParameterString(\"out\",classif)\n",
"app2.ExecuteAndWriteOutput()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import rasterio\n",
"import display_api\n",
"\n",
"raster = rasterio.open(classif)\n",
"\n",
"m, dc = display_api.rasters_on_map([raster, rasterio.open(images_list[0])], OUTPUT_DIR, [\"Classification 4 classes\",\"Image S2B\"])\n",
"m"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
......@@ -18,3 +18,19 @@ def list_images(data_dir, suffix):
print(len(list_im)," images will be used")
return list_im
def list_images(data_dir, prefix, suffix):
"""
returns a list of file names, contained in a directory, with a name that matches a suffix
:param data_dir: directory to browse for files
:param prefix: prefix of the files to search for
:param suffix: suffix of the files to search for
"""
list_im = []
for root, dirs, files in os.walk(data_dir):
for name in files:
if name.endswith(suffix) and name.startswith(prefix):
list_im.append(str(root)+"/"+str(name))
print(len(list_im)," images will be used")
return list_im
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment