Usability would be seriously improved if you didn't require an API key for requests under maybe 1,000/day per IP. As it stands, I can't do anything entirely client side right now which sucks. Plus I don't want to have to sign up just to see if I would want to use it.

Currently, there is no error callback on the request to the API.

http://fbopen.gsa.gov/?q=office&show_closed=on&data_source=&naics=&parent_only=&p=

Scroll down to see a title with "(O&ampM)"

Tested on iPhone and in iOS Simulator (iPhone and iPad). Tapping the "x" does nothing.

Evaluate and fix any 508 non-compliance

When loading a search at fbopen.gsa.gov, the query performs:

• OPTIONS HTTP request to url
• GET HTTP request to url

These two occur in series and the UI doesn't render the search results until the GET request is fulfilled.

In testing (from the west coast), the OPTIONS request can take a long time (anecdotally, 650ms).

Two questions:

• Why does api.data.gov take such a long time to respond to the OPTIONS request?
• Is the OPTIONS request necessary given that the API documentation states that the GET request is supported?

The original attachment scraper had a facility to provide a cookie and clickthrough to attachments located at www.dibbs.bsm.dla.mil.

Cookie should look like:

'DIBBSDoDWarning=AGREE; path=/;'

Add this functionality to the new attachments code.

Highlighting is returning just a truncated snippet of the field where there are matches. This can be configured to return longer or more snippets.

We might also be able to return snippets from matching attachments...? That would be cool, to help people figure out why a given result is in the set.

On iPhones, the API documentation runs in on itself.

Via https://github.com/MoonScript/jQuery-ajaxTransport-XDomainRequest

Is there a reference document for all the returned records? There are some strange things that I get, like:

FBO_CLASSCOD
FBO_NAICS
solnbr
solnbr_ci
FBO_ARCHDATE_dt
FBO_SETASIDE
_version_
score
highlights

What do these mean?

Using the following JSON calls in Python, you get a lot of strangely escaped text data:

data = requests.get("https://api.data.gov/gsa/fbopen/v0/opps?q=software+development&api_key=JOE_API_KEY")
a = json.loads(data.text, encoding="utf-8")
print json.dumps(a, indent=4)

Newline characters and HTML-escaped characters are still present in the JSON text, despite adjusting for the proper encoding.

Outdated browser warning currently states: "Sorry, but FBOpen is not supported on Internet Explorer 9 and older. Please use another browser, or upgrade to Internet Explorer 10 or greater." Maybe we could link to something so people can ugrade or learn more abou it, like: http://whatbrowser.org/

So this query:
http://api.data.gov/gsa/fbopen/v0/opps?q=software+development&fq=notice_type:PRESOL

returns data but this one
http://api.data.gov/gsa/fbopen/v0/opps?q=software+development&fq=notice_type:presol

doesn't. I feel like it should be cases insensitive? Also, is there a list of the valid notice types? I know some of the ones from the fbo docs, but I'm not sure what's in fbopen. I could just be overlooking it.

We'll need to figure out how to do this, as there doesn't currently exist a scraper (@aaronsnow am I missing one?). But it should be fairly easy to fit into the new attachment retrieval framework (esearch branch).

The esearch branch does not yet support all of the query parameters that are supported in master. We need to finish converting them all to Elasticsearch-ese.

To display on listings.

This might seem nitpicky, but I think FBOpen is a good opportunity to get away from the crazily abbreviated fields of fbo.gov (i.e. CLASSCOD, solnbr, offadd). Renaming these to "class_code", "solicitation_number" and "office_address" would just make it more readable to people who haven't already used fbo.gov a bunch. It seems like someone had this same thought when they used "Opportunity Number" on the FBOpen frontend site instead of "solnbr".

https://acquisition.army.mil
https://www.neco.navy.mil

[more to come]

Trying to do a search with Lucene query syntax, eg:

https://api.data.gov/gsa/fbopen/v0/opps?q=data_type:opp+AND+agency:NASA&api_key=

Gets me a super strange facet list result with what looks like an undocumented return.

Not sure how I feel about this, but there could be value in stripping HTML from the results, especially since I've noticed some inline CSS that could mess things up as well as tags like <highlight> which I guess someone thought was HTML at some point?

When there are network or certificate failures, we should be able to go back and run just the failed ones.

(esearch branch)

Or get rid of solr-files, or whatever makes sense. There's probably a little bit of ES config that could be included here.

I can't seem to find how far back the fbo data goes. This should exist on the frontend somewhere, and also in the api as a metadata object.

We still need to figure out how we are going to treat awards. My initial thought is to have them be a different doc type in Elasticsearch.

But this will likely allow us to do some cool things, such as link awards to usaspending.gov. (See #16.)

Adding fq=agency:air+force returns zero docs. q=agency:air+force, however, works properly.

On the esearch branch: we should separate scraping (pulling links) from downloading from loading.

These processes should be easily resumable.

Not sure if this is the right place to request/suggest new features but linking out to usaspending.gov and/or itdashboard.gov or even better showing basic stats/figures from those sites on this new site would seem beneficial.

I'm a developer so let me know if this sounds like a good idea and point me in the right direction of where to begin work in this repo if possible.

This has the side-effect that the -f (force) flag has no effect.

We have a graphic Mollie prepared for us, let's deploy it!

Let's get some proper logging going. (esearch branch) Leaning towards Bunyan over Winston due to this write-up, and also what looks like its general simplicity vs winston.

http://www.joyent.com/blog/node-js-in-production-runtime-log-snooping

NAICS filtering is definitely not working right.

via @cjoh:

https://fbopen.gsa.gov/?q=web+design&data_source=&naics=541430&parent_only=&p=

Definitely not the results I'd hoped for. I added 541430 to an empty NAICS parameter up top. I think one thing you may want to do with NAICS is search for the original NAICS code the client asked for first. If that returns no results, I'd search within the 3 digit realm of that NAICS code (in this case 541). And if that returns no results, then don't return any results. If a NAICS code parameter is sent in, it really should limit scope.

One thing I learned from working at Jeeves is that it's better to return no results than it is bad results. No results means, to a user, change your query. Bad results means "slog through this stuff."

I modified this query to have a wildcard query string and return everything from this NAICS code (Graphic Design Services: http://www.naics.com/naics-code-description/?code=541430).

https://fbopen.gsa.gov/?q=*:*&data_source=&naics=541430&parent_only=&p=

This returns some super obvious false positives, such as "ACCUMULATOR, HYDRAUL (GAMBLE)", with a NAICS code of:

NAICS Code:
336 -- Transportation Equipment Manufacturing/336413 -- Other Aircraft Parts and Auxiliary Equipment Manufacturing

Obviously, this needs to be fixed or, if it can't be fixed, disabled. I think our first line of attack in the short term should just be to not add that parameter to the query string.

"client libraries are a true force multiplier" - @adelevie

Per @cjoh:

expose search data so you can say to COs "more people are searching for 'Web Design' than they are 'Information Discovery Portal System Artistic Services'" in a scalable way. At the same time, it may help some younger companies like ours learn the language of government, and fine-tune our searches.

Love the little calendar icon in the search results but I wanna click it so bad...
https://fbopen.gsa.gov/?q=enterprise+software+development&show_closed=on&data_source=&naics=&parent_only=&p=

Do people still use iCal? I dunno, just a suggestion.

What are all the possible values I could pass in to data_source?

Even on fast WiFi, an iPhone 5s doesn't appear to be loading search results; there is continuous network activity but no results are loaded/displayed. (iOS Simulator on a MBP does not suffer this problem.)

The documentation implies that POST functionality is coming, which means that API tokens will soon give upgraded privileges. API tokens should not be passed as GET parameters in the URL.

Recommend passing the API token in the header only in an authentication call that returns a session token, then subsequently passing only the session token.

As an API user, I would like the ability to specify a limit on the amount of search results. For example, I'd like to get 200 results rather than just the default amount. Would probably cap the upper limit to 400 or 500 results.

Example API Call: Search for software opportunities, return up to 200 results

GET HTTP/1.1 http://api.data.gov/gsa/fbopen/v0/opps?q=software&limit=200&api_key=YOUR_KEY

Create a simple way to get a working dev environment up using vagrant with puppet provisioning over virtualbox. Will use 32-bit virtualbox image (Ubuntu 12.04 LTS) for universal support.

api.data.gov supports SSL, would be awesome if your docs presented that as the default.

Consider adding example queries somewhere, particularly some that work out quite a bit better than on FBO.

via @adamjacobbecker
https://twitter.com/AdamJacobBecker/status/443890598840643585

If you're on a bandwidth constrained or latency constrained device or network, it can take a while for the results to come back from api.data.gov.

Propose adding a Loading message and spinner in the results area that is shown until the search results return and can be rendered.

When a user navigates to a higher page result then runs another another search. The p=# does not reset. So, if the new search results has fewer pages the results page is blank.

Example

First Search

Go to page 8 of result set for term "data"
https://fbopen.gsa.gov/?q=data&show_noncompeted=on&data_source=grants.gov&naics=&parent_only=&p=8

Second Search

Do a new search for "statisitcs"
https://fbopen.gsa.gov/?q=statistics&show_noncompeted=on&data_source=grants.gov&naics=&parent_only=&p=8

Solution

remove the # for p=

https://fbopen.gsa.gov/?q=statistics&show_noncompeted=on&data_source=grants.gov&naics=&parent_only=&p=

Is this within the near-term realm of possibility for this project by @18F or is it better suited for a fork? What I am thinking is a place to query not just federal opps and grants but also local, state, etc. ones.

See https://github.com/dobtco/openrfps-scrapers/wiki/List-of-Procurement-Websites#scrapers-in-progress for a list of scrapers ("loaders") in progress/completed.

This is really two features:

1. Filter by full NAICS code or by first two or three digits of NAICS code (e.g., 541_)
2. When filtering by a full 6-digit NAICS code (e.g., naics=541430), if the query returns no results, fall back to searching for results that match the first three digits (e.g., 541_).

Suggested by @cjoh

Note that this is already supported by the API, just not the website. E.g.:

Exact NAICS match on 541712:
http://api.data.gov/gsa/fbopen/v0/opps?q=software+development&fq=FBO_NAICS:541712&api_key=DEMO_KEY

Three-digit NAICS match on 541_:
http://api.data.gov/gsa/fbopen/v0/opps?q=software+development&fq=FBO_NAICS:541_&api_key=DEMO_KEY

Match any NAICS code in the range 541500-541800:
http://api.data.gov/gsa/fbopen/v0/opps?q=software+development&fq=FBO_NAICS:{541500%20TO%20541800}&api_key=DEMO_KEY

Etc.

I want the ability to limit query results from the API to specific agencies (e.g. Department of the Air Force).

Right now, I specify the query string in quotes

GET http://api.data.gov/gsa/fbopen/v0/opps?q=%22department+of+the+air+force%22&api_key=SOME_KEY

Maybe instead query with all or part of the agency name. Example below queries for solicitations put out by the air force containing the word database

GET http://api.data.gov/gsa/fbopen/v0/opps?q=database&agency=air+force&api_key=SOME_KEY

Solr was doing this already... We need to change the ES analyzer and reindex the data, probably to use the snowball analyzer.