@JuliaRegistrator register

Find a table that uses a scalingfactor and test/build compatibility. There's a table attribute for this but I've never seen/used a table that has one. Would like to scale all rates as appropriate when scaling is necessary.

Some tables "belong together", they partition the population and these table collections can be used for experience studies.

• Create a mapping that specifies the tables that are logically grouped.
• Provide an API for users to retrieve multiple tables.

Currently, I am thinking about some sort of JSON config that specifies the table groupings. Then allow users to retrieve a tidy data set of all the tables in the collection.

To make table loading faster, could pre-process each table at precompile time and save to scratchspace? Would help with the UX of #74

Needed for commutation functions in ActuarialScience

E.g. if you want your mortality to be x% of a base table.

use https://github.com/JuliaPackaging/Requires.jl to lazy load XMLDict and HTTP

Allow for iteration, e.g.

for q in mortalitytable
    '# do something
end

This will also make the logic for LifeContingencies a lot better because we don't have to worry so much about omegas because the iteration interface will handle the termination. Also will make more compatible with Transducers

e.g. 2017 Loaded CSO Smoker Distinct Nonsmoker Female ALB (has extra trailing whitespace)

Motivation:

• It's almost never needed to contiguously access mortality rates across issues ages, almost always attained age or duration.
• It's been a real pain to try and make a 2D array format because the Issue Age dimension naturally starts at 0. Trying to use OffsetArrays hasn't been pleasant because of the verbose types, lack of support across the whole ecosystem (e.g. DataFrames compatibility, and the begin syntax for array indexing is brand new in Julia 1.4
• Storing everything in Dictionaries also feels like the wrong answer because it's awkward to try and handle knowing when the table begins and ends

Therefore:

• Restructure around 1D arrays which have pre-computed select/ultimate vectors for a given issue age:
  MortalityVector[Duration]
• Store these arrays in a Select and Ultimate DefaultDict indexed by issue age, which is natural for the times when early issue ages aren't defined:
  SelectTable[IssueAge] or UltimateTable[IssueAge]`
• Enforce that all calls need to specify an issue age and duration - trying to allow indexing by attained age for ultimate rates is complicated:
  • For recursive calcs that depend on the start of the table (e.g. lx), but ultimate rates not defined for early issue ages, it's awkward to handle.

Benefits:

• The most basic building block for most calculations is very simple - a simple 1D, 1-indexed array
• Should be performant
• Easier to rationalize calcs that are recursive or depend on knowing when the table would start or end such as lx

Add the remaining parametric models. A great reference for the formulas is in https://github.com/mpascariu/MortalityLaws and would require adding them to src/parameterized_models.jl and adding some tests.

• gompertz0
• makeham0
• carriere1
• carriere2

The tag name "v.0.0.6" is not of the appropriate SemVer form (vX.Y.Z).
cc: @alecloudenback

See discussion here: actuarialopensource/pymort#2

Run Blue linter/formatter on code

E.g. with a slightly customized (e.g. improved) mortality rate, let users create a mort object like:

mortality(vec,start_age=age)

instead of

using OffsetArrays
OffsetArray(vec,age)

Can currently do this with UltimateMortality but it's not clear to user that this is semantically the same as above.

julia> m2000 = get_SOA_table(887)
ERROR: XMLError: Start tag expected, '<' not found from XML parser (code: 4, line: 1)
Stacktrace:
 [1] throw_xml_error() at C:\Users\AlecLoudenback\.julia\packages\EzXML\ZNwhK\src\error.jl:87
 [2] macro expansion at C:\Users\AlecLoudenback\.julia\packages\EzXML\ZNwhK\src\error.jl:52 [inlined]
 [3] parsexml(::String) at C:\Users\AlecLoudenback\.julia\packages\EzXML\ZNwhK\src\document.jl:80
 [4] parse_xml at C:\Users\AlecLoudenback\.julia\packages\XMLDict\vlQGP\src\XMLDict.jl:60 [inlined]
 [5] xml_dict(::String, ::Type{T} where T; options::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at C:\Users\AlecLoudenback\.julia\packages\XMLDict\vlQGP\src\XMLDict.jl:127
 [6] xml_dict(::String, ::Type{T} where T) at C:\Users\AlecLoudenback\.julia\packages\XMLDict\vlQGP\src\XMLDict.jl:127 (repeats 2 times)
 [7] getXML at C:\Users\AlecLoudenback\.julia\packages\MortalityTables\kGPlQ\src\XTbML.jl:11 [inlined]
 [8] get_SOA_table(::Int64) at C:\Users\AlecLoudenback\.julia\packages\MortalityTables\kGPlQ\src\get_SOA_table.jl:20
 [9] top-level scope at REPL[20]:1

Annuity 2012 works:

julia> m2012 = get_SOA_table(2585)
MortalityTable (Annuitant Mortality):
   Name:
       2012 IAM Period Table – Male, ANB
   Fields:
       (:ultimate, :metadata)
   Provider:
       Susie Lee
   mort.SOA.org ID:
       2585
   mort.SOA.org link:
       https://mort.soa.org/ViewTable.aspx?&TableIdentity=2585
   Description:
       2012 Individual Annuity Mortality Period Table – Male. Basis: Age Nearest Birthday. Minimum Age: 0. Maximum Age: 120

Instead of manually updating markdown file.

From a vector of mortality rates, create a traditional summary table: http://demographer.com/dsitl/10-d101-life-table-basics/

I think we'd want it to return a vector of named tuples that could then be passed into a DataFrame! or used with PrettyTables (don't want to add those as direct dependencies though).

when passing a range of timepoints/ages, the cumulative part of cumulative_decrement is redundant. Remove the redundant portion for the 1.0 release.

@JuliaRegistrator register

including covariance

The tag name "0.0.1" is not of the appropriate SemVer form (vX.Y.Z).
cc: @alecloudenback

e.g. "2001 VBT Select and Ultimate - Male Nonsmoker, ANB " - the trailing whitespace in the source XML makes it harder to refer to the respective table.

The table should terminate at attained age 120 instead of having the two trailing missing values and omega = 120

julia>  cso_2001 = tables["2001 CSO Super Preferred Select and Ultimate - Male Nonsmoker, ANB"]
julia> cso_2001.select[98]
25-element OffsetArray(::Array{Union{Missing, Float64},1}, 98:122) with eltype Union{Missing, Float64} with indices 98:122:
 0.31637
 0.33726
 0.35952
 0.37749
 0.39632
 0.41631
 0.43748
 0.45913
 0.48215
 0.50662
 0.53263
 0.56026
 0.58959
 0.62074
 0.6538
 0.68891
 0.72615
 0.76567
 0.80759
 0.85205
 0.89922
 0.94922
 1.0
  missing
  missing

julia> omega(cso_2001.select[98])
122

I would like to press ctrl+space after writing the following code:

vbt2001 = MortalityTables.table("2001 VBT Residual Standard Select and Ultimate - Male Nonsmoker, ANB")
vbt2001. # Press ctrl + space here

and see that there are select and ultimate rates available to me. The guidance on declaring return types is confusing me. This seems to work fine -

But this seems to be causing problems -

Inspecting the return type of a function is not a thing in Julia?

A few related things:

• survival(table,from,to) isn't AD'able because that method only operates on integer arguments. Results in a stackoverflow error.
• At the transition point between partial ages, the deriv goes to zero:

Is there a way to generalize decrement and survival to match the integer-age approach at integral ages but avoid the discreteness in the AD? Does it require making the survival curve parametric? An alternative to Balducci/Constant/Uniform?

I have a few problems with your package;

1. No help:
   MortalityTables.jl has no docstrings, no help nor methods registered.

2. Althought the tables are accessible with MortalityTables.tables(), the method table() is not found:

using MortalityTables

table = MortalityTables.table("2001 VBT Residual Standard Select and Ultimate - Male Nonsmoker, ANB")

>> ERROR: UndefVarError: table not defined

I installed the library as follow;

>> julia:  ]
>> julia (pkg): add MortalityTables

End goal of #109

Preliminary work completed in example here: https://juliaactuary.org/tutorials/mortalitytablesdataframe/

"""
	long(m::MortalityTable)

Return an array of tuples containing `issue_age`,`attained_age`,`duration`,`select` rate, and `ultimate` rate.

"""
function long(m::MortalityTables.SelectUltimateTable)
	earliest_age = min(firstindex(m.select),firstindex(m.ultimate))
	last_age = max(lastindex(m.select),lastindex(m.ultimate))
	
	table = map(earliest_age:last_age) do issue_age
		map(issue_age:last_age) do attained_age
			# use `get` to provide default missing value
			ultimate = get(m.ultimate,attained_age,missing)
			if issue_age <= lastindex(m.select)
				select = get(m.select[issue_age],attained_age,missing)
			else
				select = missing
			end
			duration = attained_age - issue_age + 1
			(;issue_age,attained_age, duration, select,ultimate)
		end
	end
	
	vcat(table...)
		
	
end

"""
	long(m::MortalityTable)

Return an array of tuples containing `issue_age`,`attained_age`,`duration`,`select` rate, and `ultimate` rate.

"""
function long(m::MortalityTables.UltimateTable)
	earliest_age = firstindex(m.ultimate)
	last_age = lastindex(m.ultimate)
	
	table = map(earliest_age:last_age) do issue_age
		map(issue_age:last_age) do attained_age
			# use `get` to provide default missing value
			ultimate = get(m.ultimate,attained_age,missing)
			select = missing
			duration = attained_age - issue_age + 1
			(;issue_age,attained_age, duration, select,ultimate)
		end
	end
	
	vcat(table...)
		
	
end

To-dos

• tests
• is long the right function name?
• what about irregular tables (#107)

I've considered several different implementations for how to store the underlying data of a Mortality Table.

Some desired characteristics

• Performant
• Supports missing values (e.g. table 1118, which is missing young ages/durations) or tables without child rates
• Tractable error messages
• Quick calculation of metadata, e.g. omega (ie the last age defined)
• Pleasant indexing syntax (e.g. be able to return durations 1:30 in one call)
• Indexed from 0 for age, and from 1 for duration

Implementations considered

OffsetArrays.jl

• Performant
• Supports missing values (e.g. table 1118, which is missing young ages/durations) or tables without child rates
• Tractable error messages
• Quick calculation of metadata, e.g. omega (ie the last age defined)
• Pleasant indexing syntax (e.g. be able to return durations 1:30 in one call)

DimensionalData.jl

DefaultDicts (from DataStructures.jl)

Built-in arrays

Built-in Dicts

notes are WIP

MortalityTables.table("Hello World")

Yields an error like this for me:

ArgumentError: ArgumentError: invalid index: nothing of type Nothing

Stacktrace:
  [1] to_index(i::Nothing)
...

Can this error be handled in a more informative way?

Errors on some tables

using MortalityTables
MortalityTables.table("2012 IDEC Select Termination Rates - Male, Occ Cl 1, Acc and Sick, 14 day EP")
MortalityTables.table("Guatemala Abridged Life Tables 1980-85 Males")

• Distinguish between tables that will have a custom mortality-table specific API and those that should not. Tables that cannot be parsed into the existing API will inform the user that they are trying to load a table that is not a supported table.
• Provide a general API for all tables. When parsing the XTbML the only assumption we are making is that it is a valid XTbML file.

The REQUIRE file could not be found.
cc: @alecloudenback

With 250+ tables, it takes up to 10 seconds to load all of the tables into memory.

I haven't profiled the call, but I would speculate that most of the time is spent parsing the whole XML into a dict. The current process:

This happens in the file XTbML.jl, starting with the tables function:

1. Loop through and open each file
2. Use XMLDict package to parse the XML into a Dict (I did this so it would be easier to work with than the XML tree)
3. Populate three other DefaultDicts (with the default being missing) by looping throug the values in the the three DefaultDicts :
   • One for the metadata
   • One for the select values
   • One for the ultimate values
4. Loop through and create OffsetArrays (so that ages can start at 0) for both the select and ultimate tables, ultimately returning a MortalityTable type for each source file.

Given a rate, list the table and ages for which the rate occurs.

This seems slightly odd and like a bug:

using MortalityTables, Plots

tables = MortalityTables.tables()
tab_now = tables["2001 CSO Preferred Select and Ultimate - Female Nonsmoker, ALB"]

p = plot()
for a ∈ 95:0.6:98
    plot!(p, [survivorship(tab_now.ultimate, a, x, MortalityTables.Balducci()) 
        for x ∈ a:0.1:120], label = "$a", xlim = (0,24), xlabel = "Months", 
            ylabel = "Survivorship")
end
p

Produces:

I've confirmed that this problem exists for different ages, different mortality tables, and all methods of splitting up the year (Uniform, Constant, and Balducci).

Is this a bug or am I doing something I shouldn't be doing?

Let me start off by saying really nice package, I'm not an actuary but this is really easy to work with!

Are there any thoughts as to the implementation of mortality multiples in the package?

A bit of background: I've got a set of insured lifes with current age and a given life expectancy. This life expectancy is lower than what would be implied by any table (which I do by finding the first survivorship < 0.5).

Googling around a bit it appears that people are actually just multiplying mortality by some constant and capping things at 1.0, which seems a little less than optimal?

Not sure whether this kind of functionality would be a good fit for this package, but thought I'd throw it out there.

As a side note:
One thing I've been doing (not sure how great an idea this is though) is to basically find the age at which life expectancy equals the given life expectancy, and then using the survival probabilities from that age onwards, so basically:

# Objective function
given_LE = [5.9]
obj(x; given_LE = given_LE) = abs(0.5 - survivorship(male_table.ultimate, x, x+given_LE[1], 
                                                    MortalityTables.Uniform()))

# Find age between 60 and 110 at which LE equals the given LE
implied_age = optimize(obj, 60.0, 110.0).minimizer

• Worked examples of typical workflow
• More inline examples
• Organize pages:
  • Parametric Models on one page
    • Document formulas with LateX, e.g.: https://www.mortality.org/Public/HMD_4th_Symposium/Pascariu_poster.pdf
  • Standard API on another (decrement, survivorship,etc.)
  • Link to JuliaActuary.org
• Review how docstrings come together for the same function with different methods (e.g. survivorship for SOA table vs parametric)

Balducci, Uniform, Constant

@JuliaRegistrator register

The REQUIRE file could not be found.
cc: @alecloudenback

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!