Similar to the S3Bucket import, we need a way for admins to upload zip files or pdf files to /tmp and have that create an ImageUpload. The UI for this should probably be an optional file upload control on the ImageUpload#new action's form.

This probably requires either paperclip or carrierwave. See the Attachments section of this blog post, this example for CarrierWave, and this example for PaperClip.

Currently the filter displays a start date field and an end date field.

This needs to be a single "date" field, and the conditions need to check for

input_date BETWEEN asset_collection.start_date AND asset_collection.end_date

(Or rather the MongoDB version of that logic)

Currently importing an S3 bucket fires off a background task which exports files and then creates an ImageUpload. This works great, but the user experience is that they see an "Importing" message, then wait an hour, then see the new thing appear under uploads.

Here's what I'd like to see happen:
The user is directed to the ImageUpload list and sees their new bucket in a "uploading" state.
If they click on the upload they see a count of the files to be exported and the files currently exported.
Once the export finishes, the ImageUpload status changes and they see the regular image upload screen.

Here's how I'd do it:

On export start:

• Create new ImageUpload record with
  • a new attribute for the number of files expected (to be pulled from the S3Bucket::ls
    results -- it'd be fine to store the ls contents in an array instead, though)
  • a status of "Importing"
• Launch the export

During the export, when the admin clicks on an ImageUpload in the 'Importing' status, either redirect to a new view/page or conditionally render a page displaying the normal ImageUpload attributes but also including

• The total count of files to be exported, as recorded above, and
• The count of files already exported, as generated by listing the current contents of the export directory
  (It would be fine if this included filenames and sizes exported already)

On export completion

1. Mark the ImageUpload status as 'New'

Kirk writes:

1. Include the Register Type in the fields displayed in the Results List so that the reason for apparently identical results will be clear. (Likely a good idea)

(Previous discussion:
EricB:
I just did a search that contained records form a transcription by others, that as been incorporated, as well as 'normal] Parish Register records transcribed by myself.

So, now the results are listed in date order, one get 2 lines in most cases for each person - a TR line and a normal PR line. -quite correct.

But when displayed in a big like, I fear this could bane the researcher curios at lease as to why double records.
Can we display Register Type [PR and TR etc] on the listing page, so to easily show the researcher why 2 entires for what appears to be the same event?

Then, that brings me to the detail of the entry:

The example is Baptism of Martha Fuddergill 21 Apr 1716 at Dewsbury WRY. or [FUDDORGILL]

The TR record is correctly identified from Register Type.
but the record transcribed by me is from the PR - but PR is not shown anywhere on the record in F1.

So don't we need to map PR onto all those records that don't come from a different known source?

Or perhaps those that don't have an image number?

-needs something on the F2 record to show it is PR. I think.

Kirk:
An interesting observation.

Reg1 will also display both records so in this regard Reg2 and Reg1 are doing the same.

I write:
I'm reviewing the email threads to add to the issues database, and am wondering where this stands.

For the existing database of 24M records, do we have any way of knowing a record's provenance well enough to display the source to the researcher as 'donation', 'PR' or something similar? I gather from the other thread that we 1) don't track this, and 2) may not have enough information to reconstruct a likely provenance for a record that's not explicitly marked as e.g. "St. Mary's-PR" or something.

I also gather that moving forward, record provenance is something we would like to track. Is that correct?

Kirk responds:

I believe that EricB is noting that REG1 and REG2 both, correctly, display records that are transcribed from different documents. One is from a TR and the other is from an unspecified source i.e. it will have a register type of "Blank"

He makes 2 requests.

1. Include the Register Type in the fields displayed in the Results List so that the reason for apparently identical results will be clear. (Likely a good idea)

2. Have "Blank" default to "PR". In a separate set of emails EricD agreed with me that this was not a good idea as we would never clean up the "Blank" nature of the field if it was automatically converted to "PR"

Everyone is used to surname at the top -- this needs to be changed to reflect that expectation.

Eric B writes:
BenB

Another thing that is not picked up...

Actual record says:
Marriages 30 apr 1715 Norfolk.
James Che*rin + Mary Ricks.

Where in the protocol the * stands for 1 or more chars. are not readable

If I search for a surname say CHEERIN within the year of 1715 and tick fuzzy, it is not included. als though the following list is returned:-

This reminds me that we should display the person's role alongside
their name in the result list -- so rather than
Participants:
Esther NEEDHAM
John BRIMBLE

Others:
Charles BRIMBLE
Joseph NEEDHAM
Joseph NEEDHAM
Gertrude Lavinia CANNARD

We'd have
Participants:
Esther NEEDHAM (Bride)
John BRIMBLE (Groom)

Others:
Charles BRIMBLE (Groom Father)
Joseph NEEDHAM (Bride Father)
Joseph NEEDHAM (Witness 1)
Gertrude Lavinia CANNARD (Witness 2)

As a researcher I want to see counties grouped by region on the search form so that I can find the county in which I want to search quickly.

ACCEPTANCE CRITERIA

• I can identify counties quickly in the search form.

IMPLEMENTATION NOTES

re: the County drop down table. -your Chapman code field in the Search panel.

All the UK counties are there I now see. But appear to be sorted English in alpha order, then Scottish in its alpha order followed by Welsh in its Alpha order and finally Channel Islands.

I would find this highly frustrating if I were doing a lot of searching of Welsh or CI parishes because the list is very long!

Could you do it as 2 drop down lists first REGION = E,S,W,CI then those present COUNTY in the chosen region then, when you have added them, the place and parish?

Eric Booker reports:

The Search Results page shows which records have been viewed. -great. BUt it does not always work (as below)

Searched on: First name = blank
last name = GREENFIELD
fuzzy not ticked.
record type = baptism
Chapman code : Yorkshire West Riding(WRY)
inclusive = blank.

RESULT gave me about 2 pages or so.

I skipped record no 1 just to see what happened.
Looked at DETAIL of record no 2 then navigated back to results page. The flag (ie Viewed) was shown against the right record.
so I looked at the detail of record 3 and navigated back again to search results page. The flag was not set.
Selected the same record again (no 3) looked at it, then back to Search results page. This time the flag was set.

Tried with record 4. same result -flag not set until i went back and forth a second time.
Tried with record 5. same result - not set. I went right back to the search spec page, then using Safari (i'm on Mac) screen navigation arrows this time, I went to search results. and the flag was set for no 5.

So, only the first record chosen to vied behaved properly.
All the rest of my searches needed something to jog it along.

It Would be a bit confusing.

Creating a Vagrant file would make installing MyopicVicar much easier.

Vagrantpress is an example of how powerful Vagrant can be. You can get a full install of WordPress in a matter of minutes after typing just one command.

Andy suggests:

Entry
CountySOM Should be name not code
PlaceChew Magna
Register typeBlank "Should, in this case be Marriages?"
Groom forename_e-ian
Groom surnameKINGSTON
Bride forenameRachell
Bride surname_IOTONS
File line number30 Not needed
Line idAndyPhillips.SOMCHMMA.CSV.30 Not needed
File Not needed
CountySOM Not needed
PlaceChew Magna Not needed
Register typeBlank Not needed
Record typema Not needed
File nameSOMCHMMA.csv Not needed
Transcriber nameAndy Phillips Not wanted
Transcriber syndicateSomerset Not needed
Credit nameAndy Phillips and Brenda Paternoster Not wanted
Record Why repeat
Primary namesfirst_name: rachell Why repeat
last_name: *iotons Why repeat
origin: transcript Why repeat
first_name: *e-ian Why repeat
last_name: kingston Why repeat
origin: transcript Why repeat

  Other family namesfirst_name: rachell Why repeat          
  last_name: *iotons    Why repeat          
  origin: transcript    Why repeat          
  first_name: *e-ian    Why repeat          
  last_name: kingston   Why repeat          
  origin: transcript    Why repeat          

  Record typema             

Display the total count of search results on the result list page and conditionally display next/prev buttons.

Show the user which terms they searched on above the header.

Eric writes:
Would it be too much processing time to allow the researcher to select which order to present the results? eg date or church name/place??

Andy writes:

After viewing the details screen (in this case Burials but I suspect all events) and selecting "Back to Results" you are taken back to the top of the Results. Most infuriating if you are looking at a long list!

You should go back to the entry from whence you came.

Make sure that the headings of the displays have all of the information on what has been selected

We need to support place and church in a multi-select format.

It's hard to tell exactly which name your search matched.

Running bundle exec rake db:migrate gives me this error

rake aborted!

Problem:
  Invalid option :in provided for field :record_type.
Summary:
  Mongoid requires that you only provide valid options on each field definition in order to prevent unexpected behaviour later on.
Resolution:
  When defining the field :record_type on 'Freereg1CsvFile', please provide valid options for the field. These are currently: as, default, identity, label, localize, metadata, pre_processed, subtype, type, versioned. If you meant to define a custom field option, please do so first like so:

   Mongoid::Fields.option :in do |model, field, value|
     # Your logic here...
   end
   class Freereg1CsvFile
     include Mongoid::Document
     field :record_type, in: true
   end

Record Type Blank is a concern for some
Andy says
Re Blank - happy to go with your alternaive wording and leaving Blank displayed. Let's keep things simple rather than have different versions depending on who is viweing.

The current church code changes church names and propagates to Freereg!_csv_files. It will not work correctly when there are multiple churches in a place and one attempts to rename to one of the exiting church names. In reality we need a merge rather than a name change. i.e. we need to merge the registers of two churches under one name and eliminate the other church name.

Andy suggests:

I am assuming the Search input screen is just a dummy. If not then I would make the following comments.

1. First name - should be "name(s)
2. Fuzzy - a bit jargonistic. Can we not use the more commonly term "Soundex"?
   3 Record type - allow facility to choose more than one
   4 Chapman code. Not sure many researchers will know what this is. Should be called "County" (and the Chapman Code not displayed in the drop down menu
3. Parish needs to be displayed for selection to narrow the search
   6 Year (ranges) must be included
   7 Not sure what "Inclusive" means

Consider improvements to minimize/improve scrolling.

Eric reports:
there are 100 lines shown, then NextPage. and so on. but if less than 100 records, there should be no NextPage -but box to navigate forward is still there.

On my machine [iMac 24"], I see only the first 10 records before needing to scroll down. then having scrolled down, one looses the column headers. -odd but sort of understandable -and get 15 records. and so on.

As one might not have looked at the details of the last record on the page, before scrolling down, its a bit hard remembering just which was the last one, and where to scroll to.
So, could we not just show more records per page by reducing the line height per record, and then page down with a side button? This would necessitate in baptisms, having separate columns for Father and Mother. so you know who is whom.

I ask:
So do we want a three-way search (bride/groom, +fathers of
bride/groom, +witnesses) or a two-way search (bride/groom,
+fathers+witnesses)?

Kirk responds:
Personally I would go 3 way

The data set can have a range from x-y where x<1530 and y>2019.

Do we change the range to reflect the data? Suggestions being solicited.

EricD votes for bins at either end

Eric writes:
headers - too generalised so I would find it easier to have 3 separate headers with the right/ specific column headers rather than Participants/ Other Names.

Marriage records with "Other Names" include a line-break at the beginning of the page: http://testweek1.aspengrovefarm.com/search_queries/519a299d8ebe611200000021

Commenting on navigation, Eric B writes

Also "Back to results" takes me back to the Search definition page when there are no results.

Andy reports:

Where there are more than two witnesses to a marriage the instruction is to repeat the entry and change the witness names. This currently results in more than one result eg

 # Record Type Marriage Date Surname Forename County Place Name Found As
 1 Marriages 03 Mar 1853 ROYNON Thomas Somerset Churchill Groom Father Surname, Groom Surname
 2 Marriages 03 Mar 1853 ROYNON Thomas Somerset Churchill Groom Father Surname, Groom Surname
 3 Marriages 03 Mar 1853 ROYNON Thomas Somerset Churchill Groom Father Surname, Groom Surname

The details on each are the same except for thw witnesses and Notes eg

Search Record Details
County Somerset
Place Churchill
Church St John the Baptist
RegisterNumber 52
MarriageDate 03 Mar 1853
GroomForename Thomas
GroomSurname ROYNON
GroomAge 36
GroomParish
GroomCondition Bachelor
GroomOccupation Farmer
GroomAbode Churchill
BrideForename Mary
BrideSurname SHORLAND
BrideAge 30
BrideParish
BrideCondition Spinster
BrideOccupation Servant
BrideAbode Churchill
GroomFatherForename John
GroomFatherSurname ROYNON
GroomFatherOccupation Farmer
BrideFatherForename John
BrideFatherSurname SHORLAND
BrideFatherOccupation Labourer
WitnessOneForename Ann
WitnessOneSurname JENKINS
WitnessTwoForename James
WitnessTwoSurname CLEGGE?
Notes Duplicate entry, 6 witnesses. Bride and Groom sign X
FileNumber 12946

In such situations could the additional entries be combined so that just one record is displayed with all the witnesses shown?

Currently the search query for provides no validation at all on search date order. As a result, researchers can search for impossible ranges.

We're displaying some random number instead of the actual notes on the record.

Eric compares FreeREG1 and FreeREG2 to demonstrate:

Eric B writes:

Hi, BenB.

I have just noticed that the Parish name does not appear on the details page

Eric Booker suggests:

It would also be most useful to show the date and sort in date order -so one can skip forward or backwards as needed.

Andy writes:
A search for Joan Emma KING will return the correct record.

A search with just Joan or Emma KING does not display the record.

bundle install
Fetching git://github.com/elia/activeadmin-mongoid.git
remote: Counting objects: 837, done.
remote: Compressing objects: 100% (573/573), done.
remote: Total 837 (delta 305), reused 648 (delta 207)
Receiving objects: 100% (837/837), 227.66 KiB | 219 KiB/s, done.
Resolving deltas: 100% (305/305), done.
Fetching gem metadata from https://rubygems.org/......
Fetching gem metadata from https://rubygems.org/..
Resolving dependencies...
You have requested:
  mongoid ~> 3.0

The bundle currently has mongoid locked at 2.7.1.
Try running `bundle update mongoid`

Running bundle update mongoid works but I shouldn't have to do that

Eric B writes:

When one has just done one test, and from the Search Results page selects: New Search, it just returns to the last search definition page, -surely New Search should return to a clear Search page.?

Returning to the last search parameters page is where one should go from a button: Revise Search

We need to transform the display of record types to a user-friendly string.

Consider switching from 'should display' to 'should suppress' to pull this in.

Julie writes (about the result list):

Not easy to cut and paste into any other useful format - would take a long time to work through a lot of entries.

Ben L suggests a graphical display
http://dygraphs.com/ seems pretty nice for that.
EricB suggests bar graph

Andy points out that on the search results screen,

Last Name is in lower case in the search details criteria but upper case in the details line. For consistency upper case should be used for both.

EricB suggests adding previously viewed

You shouldn't need to re-type everything on a new search

As a researcher I want to navigate from the record I am viewing to either the next or the previous record with a single click so that I can read through a set of records in order

ACCEPTANCE CRITERIA

• When the first record of a set is displayed I can see a "next" button at the top and bottom of the record.
• When the first record of a set is displayed I can see a "previous" button at the top and bottom of the record.
• When a record other than the first or last in a set is displayed I can see both a "next" and a "previous" button at the top and bottom of the record.
• When both a "next" and a "previous" button are displayed, the "previous" button appears to the left of the "next" button.
• I can click on a "next" button. The next record in the set is displayed.
• I can click on a "previous" button. The previous record in the set is displayed.

If a user reports a problem with an image, the admin should be able to go to the image file screen and upload a new replacement image.

Researchers will not know what a chapman code is.

Come up with a canonical list of indices for all queries performed during this test period.

Eric B writes:

In fact I have discovered that it accepts a search of a fully EMPTY input panel!

I terminated the connection, to stop it after about 10 minutes.
Surely, this one must be prevented?

We should only allow combinations that we wish to allow, and actively reject any that don't have the requisite elements completed logically methinks.

Previous discussions have talked about stopwords for "de la", and other particles when transforming complex surnames, but at minimum we need to deal with the surnames themselves. Andy writes:

A search on LLEWELLYN gives no results but a search on LLEWELLYN-LLEWELLYN finds the record:

Rupert Eryron Douglas Tudor LLEWELLYN-LLEWELLYN (Baptizee)  William Tudor LLEWELLYN-LLEWELLYN (Father)
Alice Gertrude LLEWELLYN-LLEWELLYN (Mother)     Baptism     21 Sep 1892     SOM