{targets} daily pipeline
| {meteospain} ->
| {meteoland} ->
| {medfate} ->
| {arrow} + {geoarrow}
|
|- geoparquet files in custom S3 (No AWS)
|
|- {shiny}
| {duckdb} +
| {mapdeck} +
| {echarts4r}
|
|- Interactive visualizations of daily modelled forest water balance in Spain at 500m2 resolution.
Interesting podcast if you are interested in learning more about the geoparquet file format: https://open.spotify.com/episode/0ugH6HO5Szz1cD1WgV5aAx?si=9044a9e0222c4ae5
Listen to this episode from The MapScaping Podcast - GIS, Geospatial, Remote Sensing, earth observation and digital geography on Spotify. Cloud-native geospatial, range requests, chucks, COGs and COPCs ... [ insert confusing acronym here ] Sometimes It feels like we need to learn a whole new vocabulary and if you have been doing #geo for a while you might be wondering how much of this is actually going to impact me. What bits of this are the ones that I need to know about? I don’t think that anyone is going to be talking about cloud native in 10 years, in the same way, no one talks about digital cartography or computer analysis because where else would you do your cartography? And how else would you do your analysis? Maybe the names won’t be as important but the concepts will and while this episode is focused on Geoparquet it does so within the context of cloud-native geospatial - and this concept is not going away! You can connect with Kyle Barron here https://x.com/kylebarron2 or here https://kylebarron.dev/ If you want to learn more about cloud-native geospatial I can highly recommend these episodes https://mapscaping.com/podcast/cloud-optimized-point-clouds/ https://mapscaping.com/podcast/introduction-to-cloud-native-geospatial/ https://mapscaping.com/podcast/planet-scale-tiled-maps-without-a-server/ https://mapscaping.com/podcast/what-is-modern-gis/ https://mapscaping.com/podcast/the-planetary-computer/ I am working on a new project called https://quickmaptools.com/ like the name suggests it is a bunch of browser-based map tools. So far we have created several different conversation tools and will continue to add more to the list. Check it and let me know what you think!
Since I've recently quit my job, I've enabled github sponsors now. Without any expectations, let's see if I can make a living as an #rstats #rspatial #whateverspatial opensource developer 🤔
https://github.com/sponsors/tim-salabim
Thanks to inspiration by @kylebarron I am currently deep diving into #deckgl and #geoarrow to enhance the geo-spatial visualisation toolkit available in #rstats #rspatial...
Finally, well-designed modules will let you extend them beyond their initial use case to fit your specific workflows or data formats. For instance, the Arrow format has already been extended to represent PyTorch data frame (with #TorchArrow) and geospatial data with #GeoArrow.
That's it for today! To read the full chapter of the #ComposableCodex "Standards over Silos", it's at https://voltrondata.com/codex/standards-over-silos
I will continue soon with the next chapter on UIs for data systems.
🧵 17/...
I am using some database in #Azure that is sold by #Microsoft. How do I find the flippin' name of the product and the documentation for that product? I found at least two ways to identify the name from a SQL query, and (spoiler alert 1) it gave me two different names. And (spoiler alert 2) neither of those product names are listed in their support pages. And (spoiler alert 3) any suggestion on any of the similar-sounding-product-name help pages did not actually help. And (spoiler alert 4) most of those pages contradicted themselves on the same page or a linked page.
All I'm trying to do is create a table with a `geography` column.
With #FOSS it's dead simple, free, portable, and runs fast--just use #rstats and #geoarrow (https://github.com/paleolimbot/geoarrow).
But I'm not allowed to use FOSS for the most part, since it's "not supported". Never mind that I have never once seen any actual support from the company that is being paid millions--only problems. And the folks who create FOSS are always supporting it (THANK YOU!!!), for free mostly.
For the life of me, I can't figure out why anyone wants to pay that company in Redmond a single dime, and yet their market cap is $2.5T. I guess it's #monopoly money.
@martinfleis @geopandas thank you, Martin. That's good to know.
Btw, #GeoArrow and #GeoParquet work like a charm 👍
Another building block for more efficient data pipelines
@wnd I ended up mamba installing:
- libgdal-arrow-parquet
- pyarrow
The speedup for writing and the disk space saved is pretty neat. I've tested with point and line geometries so far
Víctor Granda
Philippe Massicotte
TimSalabim
François Michonneau
Dan
Anita Graser 🇪🇺🇺🇦🇬🇪