kbase / search_api2 Goto Github PK

View Code? Open in Web Editor NEW

2.0 2.0 2.0 544 KB

Elasticsearch API for KBase

Home Page: https://kbase.us

License: MIT License

Dockerfile 1.09% Shell 1.08% Python 97.58% Makefile 0.25%

kbase

search_api2's Introduction

KBase Developer Documentation

KBase SDK Documentation

Core Service Docs

Workspace: release, develop
- Release KIDL spec, Develop KIDL spec
Narrative UI
KBase UI (non-narrative UI): release, develop
Relation Engine: API, Specifications
Search Engine: API, Indexer, Config
Auth server
File Caching Service
ID mapping API
Assembly homology API
Service Wizard: KIDL spec
Catalog: KIDL spec

Relation Engine and Search codebases

The Relation Engine (using ArangoDB) can be found here: https://github.com/kbase/relation_engine
The Search API (using Elasticsearch) is found here: github.com/kbase/search_api2/
The Index Runner (which listens to Kafka events and imports data into Elasticsearch and ArangoDB) can be found here: https://github.com/kbase/index_runner

User help

search_api2's People

Contributors

Stargazers

Watchers

Forkers

slebras eapearson lgtm-migrator

search_api2's Issues

Remove temporary narratives from legacy results

Requested by Erik

Probably will make a PR on index runner to stop indexing temporary narratives, and then remove all those from elastic

[legacy api] Narrative object names should show up in search result highlights

As seen in spec/config.yaml in index_runner, data_objects.name is copied into agg_fields, so should be searched by legacy_api, so theoretically should show up in highlights. There may be other unrelated issues with the highlighting.

Offset not working for /rpc "search_workspace" method

As reported by Dakota

Without auth token, access_groups_info and access_group_narrative_info are empty

When calling search_objects with add_access_group_info and add_narrative_info set to 1, and without an Authorization header, search results are returned, but the access_groups_info and access_groups_narrative_info are empty. With authentication, they are correctly populated.

Inconsistent counts for searchapi2's search_objects with and without sort applied

Reported by Erik on Jira: https://kbase-jira.atlassian.net/browse/PTV-1583

Queries I can look at in CI

{
	"jsonrpc": "2.0",
	"id": "afdd6432-d66f-44a3-a491-e713f3a67835",
	"method": "search_objects",
	"params": {
		"query": {
			"bool": {
				"must": [{
					"term": {
						"tags": "refdata"
					}
				}, {
					"term": {
						"source": "RefSeq"
					}
				}]
			}
		},
		"only_public": true,
		"indexes": ["genome"],
		"size": 0,
		"from": 0,
		"track_total_hits": true
	}
}

And with the sort but no search term:

{
	"jsonrpc": "2.0",
	"id": "197034cd-cee7-48ea-a6fe-3994dcf20443",
	"method": "search_objects",
	"params": {
		"query": {
			"bool": {
				"must": [{
					"term": {
						"tags": "refdata"
					}
				}, {
					"term": {
						"source": "RefSeq"
					}
				}]
			}
		},
		"sort": [{
			"scientific_name.raw": {
				"order": "asc"
			}
		}, "_score"],
		"only_public": true,
		"indexes": ["genome"],
		"size": 20,
		"from": 0,
		"track_total_hits": true
	}
}

with sort and user search for “prochlorococcus”:

{
	"jsonrpc": "2.0",
	"id": "fd697975-bbd5-473b-b24b-da3bf43169d9",
	"method": "search_objects",
	"params": {
		"query": {
			"bool": {
				"must": [{
					"term": {
						"tags": "refdata"
					}
				}, {
					"term": {
						"source": "RefSeq"
					}
				}, {
					"match": {
						"scientific_name": {
							"query": "prochlorococcus"
						}
					}
				}]
			}
		},
		"sort": [{
			"scientific_name.raw": {
				"order": "asc"
			}
		}, "_score"],
		"only_public": true,
		"indexes": ["genome"],
		"size": 20,
		"from": 0,
		"track_total_hits": true
	}
}

scientific_name field for genome index cannot be searched using a match query

It looks like genome scientific_name is no longer indexed for text and keyword (via multifield). The result is that partial matches on scientific name don't work.
This is for the narrative refdata refseq doodad, which searches against scientific name based on free-form user input. Users are free to enter any part of a scientific name and expect corresponding matched genomes to appear.
I can get exact term searches to work (using an exact scientific name), but not match queries.

For example, the following query provides no results

{
	"jsonrpc": "2.0",
	"id": "123",
	"method": "search_objects",
	"params": {
		"query": {
			"bool": {
				"must": [
					{
						"term": {
							"tags": "refdata"
						}
					},
					{
					    "term": {
					        "source": "RefSeq"
					    }
					},
					{
					    "match": {
					        "scientific_name": {
					            "query": "marina"
				            }
				        }
					}
				]
			}
		},
		"sort": [{
				"scientific_name": {
					"order": "asc"
				}
			},
			"_score"
		],
		"only_public": true,
		"indexes": ["genome"],
		"size": 20,
		"from": 0
	}
}

{
    "id": "123",
    "jsonrpc": "2.0",
    "result": {
        "count": 0,
        "hits": [],
        "search_time": 10,
        "aggregations": {}
    }
}

whereas the following does

{
	"jsonrpc": "2.0",
	"id": "123",
	"method": "search_objects",
	"params": {
		"query": {
			"bool": {
				"must": [
					{
						"term": {
							"tags": "refdata"
						}
					},
					{
					    "term": {
					        "source": "RefSeq"
					    }
					},
					{
					    "term": {
					        "scientific_name": {
					            "value": "Acaryochloris marina MBIC11017"
				            }
				        }
					}
				]
			}
		},
		"sort": [{
				"scientific_name": {
					"order": "asc"
				}
			},
			"_score"
		],
		"only_public": true,
		"indexes": ["genome"],
		"size": 20,
		"from": 0
	}
}

{
    "id": "123",
    "jsonrpc": "2.0",
    "result": {
        "count": 1,
        "hits": [
            {
                "index": "genome_1",
                "id": "WS::15792:235363",
                "doc": {
                    "genome_id": "GCF_000018105.1",
                    "scientific_name": "Acaryochloris marina MBIC11017",
                    "publication_titles": [
                        "Direct Submission",
                        "Niche adaptation and genome expansion in the chlorophyll d-producing cyanobacterium Acaryochloris marina",
                        "Direct Submission"
                    ],
                    "publication_authors": [
                        "Touchman,J.",
                        "Swingley,W.D., Chen,M., Cheung,P.C., Conrad,A.L., Dejesa,L.C., Hao,J., Honchak,B.M., Karbach,L.E., Kurdoglu,A., Lahiri,S., Mastrian,S.D., Miyashita,H., Page,L., Ramakrishna,P., Satoh,S., Sattley,W.M., Shimada,Y., Taylor,H.L., Tomo,T., Tsuchiya,T., Wang,Z.T., Raymond,J., Mimuro,M., Blankenship,R.E. and Touchman,J.W.",
                        "Touchman,J.W."
                    ],
                    "size": 8361599,
                    "num_contigs": 10,
                    "genome_type": null,
                    "gc_content": 0.46956,
                    "taxonomy": "Bacteria; Terrabacteria group; Cyanobacteria/Melainabacteria group; Cyanobacteria; Synechococcales; Acaryochloridaceae; Acaryochloris; Acaryochloris marina",
                    "mean_contig_length": 836159.9,
                    "external_origination_date": "25-Aug-2019",
                    "original_source_file_name": "GCF_000018105.1_ASM1810v1_genomic.gbff",
                    "cds_count": 7495,
                    "feature_count": 7495,
                    "mrna_count": 0,
                    "non_coding_feature_count": 178,
                    "assembly_ref": "15792:235362:1",
                    "source_id": "NC_009925",
                    "feature_counts": {
                        "CDS": 7495,
                        "gene": 7579,
                        "misc_feature": 1,
                        "ncRNA": 3,
                        "non_coding_features": 178,
                        "non_coding_genes": 84,
                        "protein_encoding_gene": 7495,
                        "rRNA": 6,
                        "regulatory": 8,
                        "repeat_region": 1,
                        "tRNA": 74,
                        "tmRNA": 1
                    },
                    "source": "RefSeq",
                    "warnings": [],
                    "creator": "jayrbolton",
                    "access_group": 15792,
                    "obj_name": "GCF_000018105.1",
                    "shared_users": [
                        "qzhang",
                        "jayrbolton",
                        "scanonadmin",
                        "kbasedata",
                        "kbaseindexer"
                    ],
                    "timestamp": 1580358520592,
                    "creation_date": "2020-01-30T04:28:45+0000",
                    "is_public": true,
                    "version": 1,
                    "obj_id": 235363,
                    "copied": null,
                    "tags": [
                        "refdata"
                    ],
                    "obj_type_version": "17.0",
                    "obj_type_module": "KBaseGenomes",
                    "obj_type_name": "Genome"
                }
            }
        ],
        "search_time": 751,
        "aggregations": {}
    }
}

Make queries into a documented, abstracted subset of the elasticsearch query API

Try to retain much of the power of the elasticsearch query API but extract only an abstracted subset, which we should document in the readme.

Document how to (rapidly) include types using the default indexer into search2 api results.

Clean up error responses

Make sure all error are handled well

Update to use Elasticsearch 7

Link to the search index type specs in the docs

So users can see the document format that will get returned for indexes and aliases.

Abstract the return format and return JSON RPC 2.0

Abstract the Elasticsearch outer return format so we're not coupled to it.

Make sure we are returning valid JSON RPC 2.0 in the response

Workspace admin should see any private documents

Currently you need explicit permissions on a workspace to see private docs for that access group, regardless of whether you are a workspace admin.

We should add a special check to see if the user is a workspace admin, which would then bypass the ws_auth workspace ID check for the user and allow them to see any workspace data.

[legacy methods] Use AND operator for full_text_in_all query

Requested by Erik

Investigate and fix timeout issues on CI

It may be an application layer problem, an elasticsearch problem, or an nginx problem.

search_objects error: User X may not read workspace Y

(Reported by Erik on Jira)

When searching for “coli” in narrative-dev, I’m receiving the error “Searchapi2/legacy search_objects error: User x may not read workspace y”

I’ve replicated with the reduced api call:

{
   "params": [{
   	"match_filter": {
   		"full_text_in_all": "coli",
   		"exclude_subobjects": 1,
   		"source_tags": ["narrative"]
   	},
   	"pagination": {
   		"start": 12,
   		"count": 1
   	},
   	"sorting_rules": [{
   		"is_object_property": 0,
   		"property": "access_group_id",
   		"ascending": 0
   	}]
   }],
   "method": "KBaseSearchEngine.search_objects",
   "version": "1.1",
   "id": "6774851485289732"
}

Which returns:

{
   "jsonrpc": "2.0",
   "id": "6774851485289732",
   "error": {
   	"code": -32001,
   	"message": "Server error",
   	"data": {
   		"method": "KBaseSearchEngine.search_objects",
   		"details": "Object 27 cannot be accessed: User eapearson may not read workspace 65372"
   	}
   }
}

I attempted to disable the workspace and object lookup behavior by setting add_narrative_info and add_access_group_info to 0, but no change in behavior.

In addition to the error itself, it is returned in jsonrpc 2.0 format, so the ui actually reports:

An error was encountered running an rpc method

Because the ui expects a KBase jsonrpc 1.1 error format.

This could have any number of causes, so I won’t even try to guess.

If it is an unavoidable condition, e.g. caused by chasing a reference and hitting a copy of copy bug, it should be handled gracefully by returning the expected results but indicating the error or ignoring it.