Occurrences download broken about ala4r HOT 16 CLOSED

text: is working today
http://biocache.ala.org.au/ws/occurrences/search?q=text:%22Abrophyllum%20ornans%22
http://biocache.ala.org.au/ws/occurrences/search?q=%22Abrophyllum%20ornans%22

from ala4r.

Thanks for opening the issue with ALA. Was using occurrences() today and getting mixed results - some taxa return results as expected and some don't. See below; Abrotanella nivigena works fine, Abrophyllum ornans returns no occurrence records, though Abrophyllum alone works OK and in fact includes records for A. ornans.

> x <- ALA4R::occurrences("Abrotanella nivigena", download_reason_id = 4)$data
> dim(x)
[1] 124  60
> x <- ALA4R::occurrences(taxon = "Abrophyllum ornans", download_reason_id = 4)$data
> dim(x)
[1]  0 44
> x <- ALA4R::occurrences(taxon = "Abrophyllum", download_reason_id = 4)$data
> dim(x)
[1] 719  65

Also noted that meta is NULL in cases where no records are returned:

> x <- ALA4R::occurrences(taxon = "Abrophyllum ornans", download_reason_id = 4)$data
> x$meta
NULL
> 

from ala4r.

Hi, I'm getting similar issues for Ramsayornis modestus and Platycercus adscitus.

Any idea when this will get fixed?

from ala4r.

Hi @jeffreyhanson : no estimate yet, sorry. A couple of things have happened: (1) the headers of the CSV file coming back from the ALA servers have changed. This is simple in principle to fix but the old header names have been hard-coded and it will take some time to track them all down and change them. (2) searching seems to be hit and miss, as per @snubian's report above. I don't know what's causing this, yet.

from ala4r.

Thanks @raymondben - when I noticed this the other day I'd just run occurrences() for several thousand taxa. About half of these came back with no records, the rest worked fine, and I couldn't see any obvious pattern.

from ala4r.

Ok - thanks for letting me know @raymondben

from ala4r.

Minor update: a fix for the indexing issue is expected (at the ALA server end) early next week. Assuming no major change is needed to ALA4R itself we should be back to normal shortly after that.

from ala4r.

There seems to be a workaround for this, at least for single species, which is to set the taxon parameter to the genus, then use the taxon_name field in the fq parameter to filter to the desired species. E.g.:

> x <- ALA4R::occurrences("Abrophyllum ornans", download_reason_id = 4)$data
> dim(x)
[1]  0 44
> x <- ALA4R::occurrences("Abrophyllum", download_reason_id = 4, fq = "taxon_name:Abrophyllum ornans")$data
> dim(x)
[1] 599  65

Actually taxon can be kingdom:Plantae or anything that includes the species.

from ala4r.

Still having problems with some taxa, even using the above workaround. Noticed that occurrence data for some taxa were giving very weird results, but the offline search (via the website) works OK. So a better workaround is to use the taxonConceptLsid (as per offline search) rather than taxon name. E.g. the occurrence results for "Acacia mariae" currently look totally spurious, but using the taxon GUID (in this case 2920084) seems to work:

> x <- occurrences("Acacia mariae", download_reason_id = 4)$data
> dim(x)
[1] 43 46
> x <- occurrences("taxon_concept_lsid:http://id.biodiversity.org.au/node/apni/2920084", download_reason_id = 4)$data
> dim(x)
[1] 718  59

from ala4r.

Sorry about the delay in fixing this issue, it has slipped way past when we intended. The bug is with the text: (default) field and was due to a schema change. Fingers crossed, it will be fixed by this week.

Just some clarification about the searching... if no field name is specified (e.g. just Abrophyllum), then it falls back to a full-text (which is equivalent to text:Abrophyllum). I'd recommend always using a specific field if you can. Fields are marked as indexed: true here: http://biocache.ala.org.au/ws/index/fields.

In the example above, it should be:

ALA4R::occurrences("taxon_name:%22Abrophyllum ornans%22", download_reason_id = 4)

http://biocache.ala.org.au/ws/occurrences/search?q=taxon_name:%22Abrophyllum%20ornans%22

taxon_name is copied_into text, as are a handful of other commonly used fields, which is why many users simply use brophyllum ornans, as it normally just works. But it may provide some unexpected results, as you will get hits where the text appears in other fields (comments for example).

Note, its wise to surround multi-term clauses with quotes so that the field is applied to both terms, otherwise it can be interpreted as taxon_name:Abrophyllum AND text:ornans.

The fq param provides a performance benefit (over using boolean AND) for when a user is refining a search using faceting, by providing a caching benefit. For users of ALA4R performing one-off searches, this caching won't help much.

from ala4r.

Thanks @nickdos - that info is very much appreciated :)

from ala4r.

Dittto - thanks @nickdos. I'll echo this advice into the ALA4R docs, and also see if we can tighten up how we are doing the searching.

from ala4r.

Thanks @adam-collins. I think we are largely back on track now: the github version of ALA4R seems to be working. Install with devtools::install_github("AtlasOfLivingAustralia/ALA4R"). Nick's advice above is now paraphrased in the docs for the occurrences and specieslist functions.

I haven't fully tested the handling of field names - in fact mostly these seem to have reverted to their original values and so didn't need any code changes. But I am assuming that these are liable to change so will keep an eye on them.
@snubian @jeffreyhanson let me know if there are still issues.

from ala4r.

Awesome - thank you very much for fixing this! I'll check it out and let you how it goes.

from ala4r.

@snubian - Many thanks once again!

from ala4r.

Just tidying things up - I think this issue has been resolved, and I haven't seen any further problems with field name changes, so closing it now.

from ala4r.