NEWS.md
"gcloud" as the default SAFE server.s2_order(), safe_is_online(), *_scihub_login(), "scihub" methods in s2_list() and s2_download()."s2cogs" (not yet implemented) value as source server.https://r-spatial.org/r/2022/12/14/evolution2.html).s2_download() if not needed.extent_as_mask behaviour in sen2r() in case of GEOMETRYCOLLECTION type (threat as POLYGON, allowing masking).full_tests is set to FALSE by default (in this case, only fast / strategic tests are run). Set to TRUE to run all tests.s2_angles() to do it;s2_translate();sen2r();naming_convention default value in sen2r_getElements().master to main according to Git and Branch Naming
sen2r_getElements() (currently used only for internal usage).tmpdir in s2_list().s2_mask() to manage inputs with sen2r_new naming convention.smooth_mask().s2_list() and s2_download();check_gcloud(), is_gcloud_configured(), check_gcloud_connection() to interface Google Cloud SDK;sen2r();sen2r() main functions were restored on Actions thanks to the implementation of Google Cloud download.s2_list(): argument availability has default value "ignore" unless server = c("scihub", "gcloud"), in which case default is "check".is_scihub_configured(): accessory function which allows easily checking if SciHub credentials ca be read. It is used to selectively run tests and examples basing on that.abort in s2_download() (see documentation).https://apihub.copernicus.eu/apihub instead of https://scihub.copernicus.eu/apihub as required after changes at SciHub side (https://scihub.copernicus.eu/news/News00868).dhus (#381)..s2_translate() (#368).dhus (resend APIs more times in case of too many requests).gdal_warp() was partially rewritten: now reshaped rasters conserve square pixels also after warping operations. This change also affects sen2r() products if reshaping features (reprojection or custom resolution) are required. This update was necessary due to changes performed in GDAL since version 3.2.0 (see OSGeo/gdal♯3294), which did not grant homogeneity between outputs produced with different GDAL versions. Note that, in case the user wants to update an existing sen2r archive produced with sen2r < 1.4.0, the old existing grid is maintained so that product grids are equal (in order to produce outputs with the new grid, users must create a new archive).s2_download() is no more able to use existing products equivalent to found ones.s2_download() no more uses existing SAFE products instead than downloading new equivalent ones (this in order to manage images split in two SAFE archives).service to s2_list() and s2_download() for using "dhus" API service instead of default "apihub" (this could be useful in case of apihub downtimes).kill_errored to sen2cor() (see documentation).server to s2_list() to be used for future implementations.build_example_param_file() now generates more recent images, 2020-08-01 instead than 2019-07-23 (this because the previous ones required two SAFE which are now on LTA); the same was done for some tests.CLD, SNW, AOT, WVP (see documentation for description). The GUI and internal functions were modified to support their selection and generation.Rcc, Gcc, Bcc) and Excess Green (ExG) (#330).paths.json).leaflet, leafpm, mapedit, shiny, shinyFiles, shinydashboard, shinyjs, shinyWidgets) are now suggested dependencies. In the case they are missing and the user tries to run the GUI, an error is returned with the command for installing them.gdalUtil(): function used to perform GDAL operations, calling C-based GDAL utilities using sf::gdal_utils(), and Python-based ones through system calls (a standalone GDAL environment is request to do it, as it was in previous versions).rgdal is now an explicit dependency (this because it is used by raster but it is not a mandatory dependency).sf (see gdalUtil()). This allows reducing the use of external runtime dependencies.sys is installed (#257).*_ignorelist.txt and *_cloudlist.txt) were created in the directory of the parameter file (if existing) and used, during subsequent sen2r() executions, in order not to try reprocessing images cloud covered or failed for some reason. Now these files were replaced with an hidden TOML file .ignorelist.txt containing the list of base names of non-produced files and the dates of cloud covered images. This file is placed in the output folder, so to be linked with the produced archive. The usefulness of this edit is twofold:
req_res to safe_isvalid() and rm_invalid_safe(): now the presence of the JP2 images required by the processing chain is checked by default. L2A products without all the three resolutions will not be used."res" to the default safe_getMetadata() output, with the list of the available resolutions (“all” is returned in case of L1C products).timeout to sen2cor() to limit the maximum Sen2cor execution time (after that time, Sen2Cor is stopped). This is useful to avoid blocking processing chains due to internal Sen2Cor errors.sys (it is used to call Sen2Cor using sys::exec_wait() instead than system(), allowing exporting Sen2Cor output to log files).rgdal (since it is implicitly called by internal functions), as requested by CRAN policy.units and tools to suggested dependencies.shiny* and leaflet* dependencies to suggested (this will be done in a future release).dhus or apihub (experimental).GET() calls with RETRY(), so to avoid errors in case of temporary unavailability of services.init_python() (this urgent bug fixing was the reason of the urgent release of the current version).sen2r(..., rm_safe = "yes") and "all".sen2r() defaults.max_mask - mask_type coherence (if max_mask < 100 and mask_type is not specified, a warning is returned).st_crs2() accepting also WKT strings / EPSG codes in the form "EPSG:xxxx"
sen2r() a short report is returned to output summarising the status of the required processing (see new internal function sen2r_process_report()). This should facilitate applying sen2r in a cycle until all “expected dates” are processed.sen2cor() and sen2r() (see below) and a new option was added in the GUI.read_gipp() and set_gipp() to read / create GIPP XML files with the parameters passed to Sen2Cor.st_as_text_2() which returns WKT or WKT2 depending on PROJ versionsen2r_process_report() to manage processing reports returned by sen2r().init_python() was deprecated (Python is no longer managed by reticulate).bigtiff for BigTIFF management in functions s2_translate(), s2_merge(), s2_mask(), s2_rgb() and s2_calcindices().use_dem / gipp (function sen2cor()) and sen2cor_use_dem / sen2cor_gipp (sen2r()) to manage GIPP parameters in Sen2Cor.use_python argument in sen2r() was deprecated (no longer needed).reticulate and magrittr were removed.safe_getMetadata() (from existing offline SAFE archives) and s2_list() (from SciHub online metadata).safelist to sf (using footprint).st_crs2() using class methods (see the function help for newly accepted inputs).safe_getMetadata() now directly read XML SAFE files instead than calling GDAL through reticulate (this avoids errors in particular OS conditions).gdal-config instead than whereis gdalinfo.proj.db file in rgdal.safelist class for lists of SAFE Sentinel-2 archives (see safelist-class);sen2r() execution now causes saving the used parameters in a json file located in ~/.sen2r/proc_par (this can be used to take trace of the executed processing chains);safe_getMetadata() function was rewritten: now files are scanned only if the user requires metadata which must be retrieved from file content, otherwise only file names are analysed. Inputs can be also safelist objects. The support for oldname SAFE products is deprecated.link_sen2cor() can be used to link an existing Sen2Cor installation to sen2r;as() methods can be used to convert safelist from/to character, data.frame and data.table (as well as as.character()',as.data.frame()andas.data.table()` functions).safe_getMetadata(): new arguments format, simplify and allow_oldnames (see the function reference for details);s2_list(): argument output_type was deprecated (use as.data.table to obtain a data.table instead than the new default safelist object);safe_shortname(): arguments tiles, force_tiles, set.seed and multiple_names were deprecated, since they are not used with SAFE compact names (old names are no more supported);Sys.setenv() effects now do not affect the R environment after exiting from sen2r() execution;safe_shortname() does no more support oldname SAFE products;Starting from this version sen2r supports ordering products from Long Term Archive (LTA) (see news at https://inthub.copernicus.eu/userguide/LongTermArchive). Now the user can automatically order SAFE products which are not available for direct download, and use them when made available. Some internal functions can be exploited to manually manage orders.
Here above the related changes:
safe_is_valid() to check if an order was processed;s2_order() to order products from LTA.sen2r() and s2_download() have a new argument order_lta (default: TRUE) to order SAFE archives not available for direct download;s2_gui() has a new checkbox to set the previous argument.build_example_param_file() does no more compute TOA and RGB432T (this to avoid downloading 2 SAFE archives).
## sen2r CRAN release _________________________ sen2r was accepted on CRAN (2019-10-21, version 1.1.0). From now, it is possible to install the CRAN version with the following command:
install.packages("sen2r")The development version – which may contain updated features – can still be installed with the following command:
remotes::install_packages_github("ranghetti/sen2r")"list_prods" argument in sen2r() is NULL instead of 'BOA'.'no' value for argument "step_atmcorr" in sen2r() (use 'l2a' without selecting any L2A products instead)."smoothing" and "buffer" arguments in s2_mask() are now 0 (like in sen2r()) instead than 10/20 m.parallel = FALSE as default in sen2r() (parallelisation must be explicitly required).sen2r package directory is no more used to store files, but a subfolder .sen2r of the user Home directory is created and used. In this way, reinstalling sen2r will not require anymore to reinstall runtime dependencies / to reconfigure the package. The permission to write on this folder is asked to the user when the package is loaded for the first time.s2_list() and s2_download() now use internal methods instead of calling python function derived from s2download scripts by Hagolle (this substantially improve the speed of SAFE search), which were removed.sen2r is now supported also on MacOS.sp and rgdal from explicit dependencies.stringr and gdalUtils dependencies.stars dependency.str_pad2(), equivalent to str_pad() without the needing to load stringr.raster_metadata(), to get raster metadata, without rgdal dependency.check_scihub_login() to check if SciHub credentials are right.check_scihub_connection() to check for internet connectivity.build_example_param_file() (not exported) to build an example parameter file.testthat.check_gdal() was rewritten without using gdalUtils, in order to be simpler and faster."by groups" instead than "step by step": this ensures best computing speed and improves the management of disk usage, although it is not possible to see all the output messages in standard output (to see them, it is necessary to redirect them in a log file)."cloud_low_proba" is not yet available in the GUI (if is still a valid argument value for compatibility), and "cloud_and_shadow" now includes class “unclassified”. This in order to include “unclassified” only in “conservative” mask values, and to exclude it in non-conservative ones.httr: a change in Wget available version for Windows (1.20.3 instead than 1.19.4) caused the old link to break; moreover, downloading Wget using the new link was not possible with download.file() due to a redirect. So, httr dependency was introduced, and download of Wget, aria2 and sen2cor is now performed with GET function.dplyr: the internal function tiles_intersects() now perform dissolve operations over polygons of S2 tiles using dplyr commands (group_by() and summarise()).s2_calcindices(): now internal raster routines are used by default instead than gdal_calc.py, allowing reducing RAM usage and computation time.sen2r_getElements() was rewritten in order to:
sen2r();s2_calcindices() uses internal routines; the legacy mode can be used with the argument proc_mode = "gdal_calc" (default value is "raster").list_indices() now returns only checked indices by default (specify all = TRUE to return also non-checked ones).sen2r_getElements() is now a data.table instead than a list (use format = "list" to reproduce the old default).First stable release of package sen2r! See the announcement here.
path_out to path_rgb / path_indices is not yet necessary (if empty, path_out is used)tiles_intersects(): when an extent is loaded, now only required tiles are automatically used, instead than all the overlapping ones.v8 instead than to v8-3.14
sen2r() documentation. The 4 different modes allow to privilege stability, velocity or disk space. In this version, the legacy mode (“by_step”) is maintained as the default.s2_rgb() to produce RGB images, integrating it in sen2r() and in the GUI.s2_mask()
s2_gui() was improved by adding a modaldialog which explains how to apply buffer and smoothing to cloud masks)check_gdal()
sen2r()
safe_isvalid() to check for valid SAFE archive names, and use it to filter existing SAFE folders (#104)s2_merge() or s2_calcindices()
This is an improvement of version 0.3.2, with several fixes and improvements.
log of function sen2r())s2_dop() to know the Dates Of Passage over orbitssen2r() and the GUInames_missing into names_cloudcovered and names_missing
s2_getMetadata -> safe_getMetadata; s2_shortname -> safe_shortname; fs2nc_getElements -> sen2r_getElements)This is an improvement of version 0.3.1, with several fixes and improvements.
check_sen2r_deps(): a new GUI to help checking dependencies.
Add function s2_perc_masked() to compute the percentage of cloud-masked surface.
Major GUI improvements in the definition of the extent: now the main map is used only to show the extent and the tiles, while the definition of the extent (drawn, specified with a bounding box or taken from a loaded polygon) is done in separate maps (this allows to avoid many errors).
Add radiobutton for index datatype, to choose the datatype when an index is generated.
Add maximum cloud cover threshold.
Implement seasonal time period (#74).
Implement the download with aria2c.
Major commit: retrieve also existing filenames. Before this commit, if some products were partially retrieved (e.g. output BOA) and some depending on them are missing (e.g. an index) but SAFE were not required, the corresponding expected vectors were not retrieved (missing SAFE tile names nothing was generated). Now, existing names matching the parameters are attached to each “_exp” vector.
Manage parallelisation:
Improvement in integer / byte data types for generation of indices: Int16 is now the default data type; choosing an integer data type the output values are clipped on the valid format range, and the scaling factor can be chosen by the user; choosing Byte, interval -1 to 1 is coerced to 0-200 (with nodata=255).
Use standard nodata values to compute indices.
Return the list of created files after processing.
Accept custom mask combinations: now it is possible to define masks with a custom combination of SCL classes to be masked.
Allow argument “timewindow” in sen2r() to be a single integer value, which is interpreted as the period between today and n days ago.
Uniform temporary directories (#73). Now the principal functions have the parameters tmpdir and rmtmp to set the temporary directory and the choice to delete /not to delete temporary files. Only two exceptions: create_indices_db() uses a fixed temporary directory (since it is a function intended not to be run by end users); sen2cor() launched from sen2r() uses a default temporary directory if tmpdir is in a SAMBA mountpoint over Windows.
Update to sen2cor 2.5.5, and manage S2A L2A operational products (https://scihub.copernicus.eu/news/News00305).
Add a welcome message and a logo.
Change package name from SALTO to sen2r, and rename the main function consequently to sen2r().
Remove s2download as dependence (now it is integrated within the package).
Remove sprawl dependencies.
Avoid blurs and artifices in thumbnails. stack2rgb() now works with GDAL routines; gdalwarp is used instead than gdal_translate for reducing the size of input images (to avoid averaging NA values, which produced blurs on borders of BOA thumbnails); intermediate GTiff files are used to bypass the presence of artifices in SCL PNG thumbnails and to produce JPEG multiband TOA/BOA thumbnails.
Load python modules correctly. Now on Windows all the modules are imported using import_from_path() in init_python(), which is called anywhere modules are needed.
Fix unrecognised osgeo in Windows. When running gdal_calc in Windows with multiple Python installations, the wrong one (i.e. Anaconda) is used, and “no module names osgeo” error is returned.
Other minor fixes.
This is an improvement of version 0.3.1, with several fixes and secondary improvements:
Changes: - Allows output directories to be created by fidolasen_s2 (#54) - Convert NA arguments to NULL - Add check that the mask polygons are not empty - Add parameter extent_name (#58) - Add exception for Hub unavailable - Add support for multiple tiles and extent names - Allow to generate thumbnails also on maps with wrong naming convention … - Allow to generate thumbnails also on maps with wrong naming convention … - Add “tiles” argument to consider only specified tiles - Add the possibility to download products over Windows (#49) - Add extent_name in the GUI (#58) - Add automatic installation of Wget on Windows - Add support for TCI thumbnail
Some fixes: - Fix recognising gdalUtils after R restarts - Fix error applying sen2cor on oldname SAFE - Delete also L1C product when applying sen2cor on a copy of the SAFE - Fix to recognise all SAFE paths - Do not ask for overwriting dependences in non interactive mode - Fix the check on dstnodata length - Fix the number of polygons required to mask - Combine multiple features in mask / extent - Accept non-sf extents or sf with NAs within fields - Change TCI mask values (use 0 instead of NA in order not to create problems with 8-bit format) - Various fixes (see solved issues)
This version allows to perform all basic operations with Sentinel-2 data: finding and downloading the required tiles (on Linux), correcting them with sen2cor, processing them (merging adjacent tiles, clipping, reprojecting, rescaling, applying atmospheric mask) to create output multiband files (in a format managed by GDAL) and spectral indices.
The version follows the issued scheduled for milestone 0.3.0; future improvements with Sentinel-2 data are scheduled for release 0.4.0.
This is an intermediate version, which does not allow yet to perform a complete workflow, but it allows to manually find (s2_list()), download (s2_download()), correct (s2_sen2cor()), translate in a format managed by GDAL (s2_translate()) and merge by orbit number (s2_merge()) Sentinel-2 data.
An example of possible manual workflow is shown in the temporary script WIP/tutorial.R. For a complete automatic workflow on Sentinel-2 data, wait version 0.3.0.