As a Herd-MDL admin, I want to be able to pick up a new Herd version

Currently this would involve building a whole stack. This story is to upgrade Herd in place.

Acceptance Criteria

• New Herd version (App, DB) is active with one click
• All Herd data is unchanged including metadata catalog and Herd configuration
• Negotiable - works with zero downtime (assumption = Herd release has been verified as zero-downtime compatible by Herd team)
  • NOTE - Herd assumption requires that the new DB release is applied before the new App release

Technical note

• Could use CodeDeploy - but need to either refactor CodeDeploy to remove any steps that should only be run one time e.g. Namespace creation, set RDS password, etc. -- OR -- write a new CodeDeploy and CFT just for the purpose of this story.

As Metastore developer, I want to oibtain Metastore artifacts from Sonatype

Currently artifacts are only available in GitHub

Acceptance Criteria

• Build/deploy Docker image is published to Docker Hub
• After a release, Metastore artifacts are present in GitHub and Sonatype

As OSS Herd-MDL user I want clean, understandable CFTs

These items were taken from the engineering wish list

Acceptance Criteria

• All parameters in all CFTs include indication of required or optional in the description
• All CFTs and scripts are proof-read for grammatical errors/typos.
• There are no default values for any CFT other than the parent 'wrapper' template

As a Herd-MDL OSS contributor I want to build an MDL stack from Herd-MDL code I forked and modified so I can dev/build/test contributions

NOTE - Only applies to Herd-MDL code (CFTs and scripts) -- does not need to work with customized Metastor and BDSQL code. These are handled in other stories

Acceptance Criteria

• User is able to trigger MDL build and deploy process that uses their customized MDL code
• Takes inputs: target AWS account, GitHub branch, build bucket location
  • Build bucket can be any S3 bucket
• Any pre-requisites are documented
  • see Technical Notes for AMI vs Docker

Technical Notes

• Team should determine if AMI or Docker is a better approach

As Herd Admin I want to run MDLT on any MDL environment to ensure that MDL stack is fully functional

Acceptance Criteria

• MDLT can be configured to point to any MDL environment with minimal effort. Inputs are: stack name, instance name, S3 bucket for results.
  • MDLT does not require stack name and instance name to be the same
• MDLT pre-requisites are documented including: IAM Roles, KeyPair, any network/VPC requirements
• MDLT results are in a single location and easy to retrieve

Technical Notes

• Can run from an EC2 in any account

This is for Demo users and MDLT functional tests.

Acceptance Criteria

• Defined scenarios work for Herd and BDSQL authorization testing

As a Data Publisher I want to indicate the SME for a Data Entity so users know who to contact with questions

Acceptance Criteria

• Data Publisher can register SME by calling BDef SME POST endpoint in Herd
• SME contact information displays on Herd-UI

Technical notes

• Herd must be configured to point to an LDAP Directory
• LDAP directory must have user first name, last name, email, phone number

Describe the bug
While following the steps for "Basic Install" I get the following error.
Embedded stack arn:aws:cloudformation:us-east-1:############:stack/herd-mdl-MdlStack-XOBVK3U2F2DE/########-####-####-####-############ was not successfully created: The following resource(s) failed to create: [PrerequisitesSecondaryStack].
I am using a personal account as root access.

To Reproduce
Steps to reproduce the behavior:

1. Download release artefact: installMDL.yml (installMDL_20190201.yml.txt) to your local file system, this will install version 1.4.0.
2. Login to AWS console and navigate to the Cloudformation service.
3. Create a new stack using the option: "Upload a template to Amazon S3" - Refer to AWS documentation on how to select a local template.
4. Select the same installMDL.yml file from your local file system (which was downloaded in step 1).
5. On the next page

• Enter a unique value for the 'stack name' parameter.
• Changed DeployComponents to Herd
  -- Note: A stack name can contain only alphanumeric characters (case-sensitive) and hyphens. It must start with an alphabetic character and can't be longer than 128 characters. Further reading: Specifying Stack Name and Parameters
• Leave all other parameters to their default values and click 'Next'.

6. On the next page, click 'Next'. Further reading on tags: AWS documentation
7. Review the information on the next page and click on the 'Create' button, this will initiate stack creation.
   ERROR HERE
   Embedded stack arn:aws:cloudformation:us-east-1:############:stack/herd-mdl-MdlStack-XOBVK3U2F2DE/########-####-####-####-############ was not successfully created: The following resource(s) failed to create: [PrerequisitesSecondaryStack].
8. Wait for 'CREATE_COMPLETE' on the stack and all its nested stacks. (never reached)

Expected behavior
I expect the CloudFormation template to complete successfully with a 'CREATE_COMPLETE' message.

Details and/or logs

Screenshots

Additional context
Just realized that I changed the DeployComponent to Herd. I modified the steps above.

As a Herd-MDL developer I want to use SecureString in CFT instead of using SSM from scripts

Currently all of our secrets have to be handled in scripts by using SSM services. Now we can create and retrieve secrets from directly from CFT. This simplifies the code significantly.

Acceptance Criteria

• Use of secrets is reviewed in all scripts and CFT and determine which should use SecureString directly from CFT
• Where applicable, based on review above, replace SSM usage with use of SecureString in CFT

Describe the bug
It appears that in the mdl.yml file a duplicate line exists. When trying to open the YAML file in cloudformation designer I get the following error.
4/8/2019, 1:15:38 PM - Cannot render the template because of an error.: YAMLException: duplicated mapping key at line 308, column 119: ... Environment, /S3/URL/Shepherd]] ^

Lines 302 and 308 look to be identical. Once line 302 is removed the error no longer appears.

To Reproduce
Steps to reproduce the behavior:

1. Login to AWS -> Cloudformation -> Design Template
2. Open a local file and point to mdl.yml from herd-mdl 0.81.0
3. See error

Expected behavior
The designer should open w/o any error. Should also display the diagram.

Details and/or logs

Screenshots

Additional context
Using https://github.com/FINRAOS/herd-mdl/releases/tag/mdl-v1.4.0

Describe the bug
Running uploader jar on freshly created 1.4 no-auth stack. Uploader fails when trying to get temp credetials from STS. See exact error below

To Reproduce
Steps to reproduce the behavior:

1. Build no-auth stack with release 1.4
2. Create Namespace, BDef, BData as described in registration demo section of User Guide
3. Run uploader as described in registration demo section of User Guide

Expected behavior
Uploader succeeds, registers BData in Herd and places file in S3

Details and/or logs
Exact failure
Dec-22-2018 23:01:49.765 [main] WARN herd.tools.common.databridge.AutoRefreshCr edentialProvider.getAwsCredential - Error retrieving new credentials. Message: F ailed to get business object data upload credential, Status Code: 500, Status De scription: Internal Server Error, Response Message: 1 validation error detected: Value '{{UPLOAD_ARN}}' at 'roleArn' failed to satisfy constraint: Member must h ave length greater than or equal to 20 (Service: AWSSecurityTokenService; Status Code: 400; Error Code: ValidationError;

Observed in 1.2

Old MySQL version for Metastore caused failure during stack build

Observed in 1.2 stack

Steps to reproduce

1. Register BData
2. Call BData Delete including optional file cleanup

Defective behavior

• BData Delete fails with S3 permissions error

<?xml version="1.1" encoding="UTF-8" standalone="yes"?> <errorInformation> <statusCode>500</statusCode> <statusDescription>Internal Server Error</statusDescription> <message>Failed to delete keys/key versions with prefix "reg-demo/exchange/source/txt/demo-data/schm-v0/data-v0/transaction-date=2018-09-20/" from bucket "REDACTED". Reason: One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: 4B4BD720C0A1E854; S3 Extended Request ID: JqmMvj+wp4HpmWnXzMoUQbY3ymeEkGOCGZSWJQmxhZvv8uz9bwE5kadhIJyi7n4j4qx7silZ53o=)</message> <messageDetails> <message>One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: 4B4BD720C0A1E854; S3 Extended Request ID: JqmMvj+wp4HpmWnXzMoUQbY3ymeEkGOCGZSWJQmxhZvv8uz9bwE5kadhIJyi7n4j4qx7silZ53o=)</message> </messageDetails> </errorInformation>

Desired behavior

• BData Delete successfully cleans up files in S3

Description

CloudFormation installation does not complete when enabling 'EnableSSLAndAuth'. Error 500 occurs during configureAndStartHerd.sh in the 'enableSSLAndAuth' section (line 182~189). The logs error output indicate Please update "herd.notification.user.namespace.authorization.change.message.definitions" configuration entry which contradicts what is specified in the code configuration (ConfigurationValue.java) for that field "There is no default value which will cause no messages to be sent."

Steps to reproduce

1. Upload installMDL.yaml to CloudFormation
2. Modify parameters with the following (I've modified my domain/ssl parameters with generic ones:
   • DeployComponents = Herd
   • CertificateInfo = certificate info (ex: "CN=*.mydomain.com")
   • HerdDBClass = db.m5.large (without this being modified herd fails at RDS DB creation)
   • CreateOpenLDAP = true
   • CertificateArn = certificate arn (ex: arn:aws:acm:us-east-1:1234567890:certificate/mycertificate)
   • HostedZoneName = ex: mydomain.com.
   • DomainNameSuffix = ex: mydomain.com
   • EnableSSLAndAuth = true
3. CloudFormation stack will fail, logs are in CloudWatch

Relevant logs

CloudFormation output - HerdEC2Stack

Herd codedeploy

[2019-07-29 16:05:20.869] [d-19OME41WN][stdout]07/29/2019 16:05:20 *** ERROR *** curl --request POST --header 'Content-Type: application/json' --data '
{
    "userNamespaceAuthorizationKey": {
        "userId": "mdl_user",
        "namespace": "MDL"
    },
    "namespacePermissions": [
        "READ",
        "WRITE"
    ]
}
' https://AWSALBevalHerdeval-1761683017.us-east-1.elb.amazonaws.com/herd-app/rest/userNamespaceAuthorizations --insecure has failed with error 500

Herd application logs

Jul-29-2019 16:05:20.805 [ajp-nio-8009-exec-2] DEBUG org.finra.herd.ui.RequestLoggingFilter.logRequest userId=admin_user - HTTP Request [uri=/herd-app/rest/userNamespaceAuthorizations;method=POST;client=10.0.10.189;session=E1E4FB1B9FEEF0BCE3E0A27064AEE840;payload=
{
    "userNamespaceAuthorizationKey": {
        "userId": "mdl_user",
        "namespace": "MDL"
    },
    "namespacePermissions": [
        "READ",
        "WRITE"
    ]
}
]
Jul-29-2019 16:05:20.843 [ajp-nio-8009-exec-2] ERROR finra.herd.service.helper.HerdErrorInformationExceptionHandler.logError userId=admin_user - A general error occurred.
java.lang.IllegalStateException: Notification message destination must be specified. Please update "herd.notification.user.namespace.authorization.change.message.definitions" configuration entry.
at org.finra.herd.service.helper.notification.AbstractNotificationMessageBuilder.buildNotificationMessages(AbstractNotificationMessageBuilder.java:146) ~[herd-service-0.81.0.jar:?]
at org.finra.herd.service.helper.notification.NotificationMessageManager.buildNotificationMessages(NotificationMessageManager.java:100) ~[herd-service-0.81.0.jar:?]
at org.finra.herd.service.impl.MessageNotificationEventServiceImpl.processUserNamespaceAuthorizationChangeNotificationEvent(MessageNotificationEventServiceImpl.java:98) ~[herd-service-0.81.0.jar:?]
at org.finra.herd.service.impl.MessageNotificationEventServiceImpl$$FastClassBySpringCGLIB$$bb1fb8f9.invoke(<generated>) ~[herd-service-0.81.0.jar:?]
a

Describe the bug
When I execute a CFT, I get an s3 conflicing conditional operation error. This has started to happen recently.

To Reproduce
Steps to reproduce the behavior:

1. Login to AWS
2. Navigate to CloudFormation
3. Upload installMDL.yml
4. Enter a stack name
5. Change DeployComponents to Herd or All
6. Click Next
7. Click Next
8. Check the “I acknowledge…” boxes
9. Click Create
10. Error: for CFT [MDL - S3 buckets - Installation Template]
    A conflicting conditional operation is currently in progress against this resource. Please try again. (Service: Amazon S3; Status Code: 409; Error Code: OperationAborted; Request ID: F8AD1BA818366C8B; S3 Extended Request ID: bKyi7H62jzvIuYj38LPMm7FwBVBAxyGo5+jcBkT8ca/sKUSK9jsic+KsaaUfcgNDsUo46P5RFd8=)

Expected behavior
A CloudFormation template to spin up w/o any error

Details and/or logs
I have spun this up in my account many many times. Lately it has changed to always get this error.

Screenshots

Additional context
Add any other context about the problem here.

Hi Team,

I am Facing other issue, while creating the Web server with basic installation cloud-formation template. Please find the attached screenshot.
I have modified as per our earlier discussion, as mentioned in #84

I have set MetastorDBClass to - db.m5.large
Image Id - ami-0ad42f4f66f6c1cc9 - (ap-south-1 region)
Any suggestions please. Thanks in advance.

Regards,
Bhargav

Hi team,

We are trying install the Finra Herd MDL using "Basin install" yaml file (installMDL.yaml) as per the instructions mentioned in https://finraos.github.io/herd-mdl/docs.html .
But facing the issue mentioned in attached screen shot. We have Poweruser privileges too.

Any suggestions would be greatly appreciated. If it is not the right forum, the could you please let me know the right forum.

Thanks in advance.

Regards,
Bhargav

Observed in MDL:
Herd fails to launch EMR clusters in regions other than the default (us-east-1)

Steps to reproduce
Install MDL in us-west-2 (or any other region)

Defective Behavior

Metastor (nested) stack fails to create because cluster creation fails. The underlying issue is that while trying to launch an EMR cluster to process Herd objects it looks for subnets in the us-east-1 region which are not available.

Desired Behavior

EMR cluster-create 'action' looks for subnets in the region where the stack is deployed.

As Basic User I only want to access publicly available services and data

This is currently working but we need to add negative cases to testing

Acceptance Criteria

• Basic User is allowed to log in to UDC and view Data Entities
• Basic User is not allowed to drill down in UDC -- they should not see 'Data Object Details'
• Basic User is allowed to call public Herd services like BDef Get
• Basic User is not allowed to call secured Herd services like BData Get
• Basic User is not allowed to log into BDSQL (or can log in but not see any tables)
• Basic User is not allowed to log into MLiy

Trying to bring up the HERD-MDL Stack via the cloud formation script.
The stack fails to launch root cause is -- BdsqlWaitCondition times out. It appears that it never gets a signal.

I can successfully launch:
PreRequisites, Herd, and MetaStore. When I attempt to launch BDSQL it appears to successfully create all the components as shown on the following screen shot.

I have traced this to line 330 of /herd-mdl/mdl/src/main/cft/mdlBdsql.yml where it appears that the handle is passed to the config step.

It's not clear what this is trying to achieve, however.
It is an argument to the script: /bootstrap/configurePresto.sh

Any suggestions?

Hey there! I noticed some possible problems in some code in this repo. A quick summary of a few of them is below, but let me know if you're interested in seeing a full report or talking about cloud security in general.

severity: serious

filename: ./mdl/src/main/cft/mdlCreateIAMRoles.yml

line number(s): [58, 116]

resource(s):

IAM role should not allow * resource with PassRole action on its permissions policy

severity: warning

filename: ./mdl/src/main/cft/mdlCreateIAMRoles.yml

line number(s): [27, 58, 116]

resource(s):

IAM role should not allow * resource on its permissions policy

severity: warning

filename: ./mdl/src/main/cft/mdlCreateIAMRoles.yml

line number(s): [27, 58, 116]

resource(s):

Resource found with an explicit name, this disallows updates that require replacement of this resource

severity: warning

filename: ./mdl/src/main/cft/mdlHerdRds.yml

line number(s): [153]

resource(s):

Resource found with an explicit name, this disallows updates that require replacement of this resource

severity: warning

filename: ./mdl/src/main/cft/mdlCreateNsAuthSyncUtil.yml

line number(s): [80]

resource(s):

Resource found with an explicit name, this disallows updates that require replacement of this resource

severity: warning

filename: ./mdl/src/main/cft/mdlMetastor.yml

line number(s): [141]

resource(s):

Resource found with an explicit name, this disallows updates that require replacement of this resource

severity: warning

filename: ./mdl/src/main/cft/mdlCreateKeyPair.yml

line number(s): [58]

resource(s):

IAM role should not allow * resource on its permissions policy

As Herd PO I want to reduce AWS costs without making functional compromise

Later we might need to take additional cost-saving steps but initially let's see what we can do without compromising anything.

Acceptance Criteria

• Time-box analysis has been performed to identify low-hanging fruit cost savings opportunities
• Cost per stack is reduced by >= 25%
• Cost per hour per stack is reduced by >= 25%

Technical notes:

• Eliminate RDS snapshot when RDS is created and prior to RDS deletion
• Remove RDS snapshots from all previous Herd-MDL stacks
• Reduce instance size for EC2, RDS, and EMR -- but ensure MDLT still works and there is some room to scale the environment for POC or lightweight operations

As a Herd-MDL Administrator I want to view clean, organized logs in the CloudWatch console.

In #17 we introduced lambdas that clean up certain AWS resources when a stack is deleted. Each of these Lambdas writes to its own log groups which looks noisy in the CloudWatch console.

Acceptance Criteria

• Cleanup Lambdas write to their own log stream within the MDL stack parent log group

...during/after Authorization re-factor

Need to have one user that is read-only in both Herd and BDSQL

As Herd Product Owner I want Herd-MDL to use the AWS ElasticSearch service for Herd indexed search filters

Currently Herd-MDL stands up EC2s, installs ElasticSearch on these EC2s, and sets up security groups that allow Herd to talk to ElasticSearch. Instead we want Herd-MDL to stand up an AWS ElasticSearch domain.

Acceptance Criteria

• Herd 0.81.0 is used as it has code that is compatible with AWS ElasticSearch service
• Herd-MDL creates AWS ElasticSearch domain
• ElasticSearch domain has comparable redundency and capacity as previous EC2-based infrastructure
• ElasticSearch domain is only accessible from Herd, controlled by Security Group and IAM Role

As a Herd-UI user I want to be able to link directly to a Herd-UI page so I can bookmark and/or send links.

Currently both scenarios listed in the acceptance criteria return a 404. We need them to work as documented here:

Acceptance Criteria

• User can enter full URL like herd-ui.com/data-entities/SEC_MARKET_DATA/SecurityData in a new browser session and go directly to that page
• User can be on any Herd-UI page and reload their browser and it will reload and display the same page

Technical Notes:

• Options appear to be:
  • Put Herd-UI on Herd's Apache
  • Put rewrite rules in S3 bucket under CloudFront like https://blog.goguardian.com/nerds/redirects-in-aws-cloudfront
  • Use cloudfront + lambda like https://linuxacademy.com/howtoguides/posts/show/topic/19955-url-rewriting-in-aws-cloudfront
• Takes apache config like:

<IfModule mod_rewrite.c>

  RewriteEngine on

  # -- send bots/spiders/crawlers 404 page not found
  RewriteCond %{HTTP_USER_AGENT} (bot|spider|crawler|search|find|walker) [NC]
  RewriteRule .* - [R=404,L]

  # -- HTTP to HTTPS
  RewriteCond %{HTTP:X-Forwarded-Proto} ^http$
  RewriteRule .* https://%{HTTP_HOST}%{REQUEST_URI} [R,L]

  ## custom rule for herdui
  # Rewrite routes to index.html unless they are for specific files
  # Don't rewrite files or directories
  RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f [OR]
  RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -d
  RewriteRule ^ - [L]

  # If the requested resource doesn't exist, use index.html
  RewriteRule ^ /index.html


</IfModule>

As a Herd OSS data publisher I want to upload a sample data file and make it available for users

See https://github.com/FINRAOS/herd/wiki/Publishing-Sample-Data for some details

Acceptance Criteria

• Data Publisher can pre-register Sample Data file by calling Sample Data Upload Initiate endpoint in Herd
• Data Publisher can place file in S3 location specified by Sample Data Upload Initiate
• Herd completes registration of Sample Data
• User can view and download Sample Data in Herd-UI
• Sample Data Upload Initiate and S3 upload steps are performed during sample data population in MDL CFT

As an MDL admin I want to perform an upgrade to Herd in an existing MDL stack

This is part of a series of stories that will introduce and incrementally build out the capability to maintain and manage MDL instances.

Acceptance Criteria

• Upgrade capability exists that will replace existing Herd version with new version

  • Herd executable is replaced with new version. Moving down versions or rolling back is not supported.
  • Herd DB upgrade scripts run
  • Any newer version of Herd can be applied. Does not require sequential version, can upgrade several versions at once
  • (needs decision) - Herd runtime is updated (EC2 instance)

• Persistent state information in Herd is untouched - Herd registration data in RDS, Herd configuration data in RDS and app server configs

• Herd smoke test runs successfully

Note - the following aspects are not included. These will be addressed in future stories:

• upgrade without downtime
• upgrade of other components (Metastore, BDSQL)
• upgrade all MDL components together

As an MDL Developer I want to make sure all non-CFT resources are cleaned up when stack is deleted

By default the stack will tear down anything explicitly created in CFT but it will not clean up things that are created by other scripting that runs. This is okay if the resources created by the other scripting go away with CFT resources (eg stuff on EC2s) – but not okay if the resources are in a persistent location like Parameter Store or Credstash

Acceptance Criteria

• All entries in Parameter Store or Credstash that are related to a specific stack are cleaned up when the stack is deleted.
• All self-signed certs that are related to a specific stack are cleaned from ACM when the stack is deleted
• All S3 buckets and contents that are related to a specific stack are cleaned up when the stack is deleted

As Herd-MDL User I want all endpoints to use a certificate from a trusted authority so I can reduce the likelihood of any challenges with end users connecting to endpoints with self-signed certificates

Pre-requisite - user specifies they want to create stack with certificate and authentication; user supplies appropriately wildcarded certificate from Trusted Authority; user supplies certificate private key

Acceptance Criteria

• Herd endpoint uses certificate from trusted authority
• Shepherd endpoint uses certificate from trusted authority
• Browser access to Herd and Shepherd endpoints do not result in any certificate warnings
• BDSQL endpoint uses certificate from trusted authority - no shady combination of trusted cert + self-signed added to keystore.
• OpenLDAP endpoint has ALB and uses certificate from trusted authority. Herd SME endpoint is tested against OpenLDAP with trusted certificate.

Note - MLiy team to verify in integrated demo environment

Not integrated with Herd-MDL code
Not tested
Just experimental, working

Steps

1. Read authorization for a given User+Namespace combination from Herd User Namespace Authorization service. This might require some specific Role/Function mapping for the Herd user that performs the read.
2. Populate SQL grants in Metastore to match response from Herd service
   1. include creation of user's schema in Metastore and write permissions to their schema
3. Scope = use existing set of grants in MDLT and verify with existing authorization functional tests
   1. existing functional tests need to point to Herd APIs instead of LDAP as source of desired authorization state

• register metastore singleton workflow
• configure improved herd logging and stream to cloudwatch
• memberOf overlay

Observed in MDL 1.2 release

Steps to reproduce

• Pre-requisite = Namespace, Bdef, Format created. User has Namespace permissions and required Security Role
• Attempt to upload file using uploader.jar

Defective Behavior

• Uploader receives error when calling Herd 'Storage Unit Upload Credentials GET' endpoint.
  • Tech note - error indicates that Herd did not have permissions to obtain credentials from AWS STS service.
• Uploader fails to place data in S3

Desired Behavior

• Uploader receives credentials and places data in S3