Comments (3)
Half done (b81f726). Slightly problematic because the data files come from a different server than the index page (the data source's source_url
), which means that the downloaded zip files don't get automatically unzipped. WIP.
from bowerbird.
OK, have implemented a minimal wget-like crawler in R, which means that the sync process now knows all about which files it's downloading and where they are being saved (rather than leaving that to wget). So this should now work (small example):
remotes::install_github("ropensci/bowerbird", ref = "usbuildings")
library(bowerbird)
datadir <- tempdir()
cf <- bb_config(datadir) %>% bb_add(bb_source_us_buildings(states = "District of Columbia"))
res <- bb_sync(cf, verbose = TRUE, confirm_downloads_larger_than = NULL)
With the downloaded files unpacked locally:
dir(file.path(datadir, dirname(res$files[[1]]$file)))
[1] "DistrictofColumbia.json" "DistrictofColumbia.zip"
from bowerbird.
Awesome!
from bowerbird.
Related Issues (20)
- Mike's postrev notes HOT 1
- Add guidance on writing new handler functions
- possible issue, found on RStudio cloud HOT 1
- structure of list-col for method HOT 6
- bb and GADM HOT 5
- Interrupting a download does not necessarily delete the partially-downloaded file
- rewrite rget to use curl directly, not httr HOT 1
- Resolution available for satellite data
- reusing curl handle has odd behaviour
- Unexpected timeout error in syncing from password-protected ftp server? HOT 2
- Oceandata downloader broken HOT 1
- Consider an extensible metadata model? HOT 4
- Support for 'local' or unpublished data? HOT 1
- Use R.utils in place of archive? HOT 3
- how to add datasource with source_url without filename in url HOT 2
- Adjustments to zenodo handler
- File counter HOT 1
- CRAN
- getting OISST HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bowerbird.