Cover vim shortcut integration for tagbar plugin.

Today, the roadctl explain commands reads a text file that documents the datamgr definitions file format. As we add new template types, the documentation needs to be specific to the given template.

This change affects two repositories:

pavedroadio/templates
pavedroadio/roadctl

In the templates repo we want to move

docs/templates.txt to microservices/ga/datamgr/docs/datamanger.txt

Then in roadctl, change it to read from .templates/ga/datamgr/docs/datamager.txt

The change needs is the cmd/templates.go file

// tplExplain
// is stored in docs/explain.txt
// docs is defined by eTLD
// explain is tplResourceName
func tplExplain(tplListOption string, rn string) tplExplainResponse {
var response tplExplainResponse

// Load explaination
fn := defaultTemplateDir + "/" + eTLD + "/" + tplResourceName + ".txt"
if _, err := os.Stat(fn); os.IsNotExist(err) {
return response
}

Implement a core logging library supporting local and Kafka logging capabilities.

Include a README.md on usage

To be tracking in our core repository, https://github.com/pavedroad-io/core/tree/master/go/logger

This broken when we moved from dep to mods

GitGutter is a vim plugin that provides a visual diff between the current code and the last git commit. It also allows to issue git commands but the preferred method is to use fugitive. We need to explain why and our key mappings

The code expects date time stamps for each row. Currently, these are inserted in the YAML instead of being generated automatically. If an end-user forgets them, the code breaks when executed.

We want to modify cmd/templates.go to include them automatically

92 // structUUID
93 // name of table, type of data json|yaml
94 const structUUID = "\t%sUUID string %s:%suuid\n"

... const structCreated = ...
const struct Updated = ...

Note, the JSON field string is also not proper

const structUUID = "\t%sUUID string %s:%suuid\n"

should include quotes like this

const structUUID = "\t%sUUID string %s:**"**%suuid**"**\n"

Needs to cover using Skaffold profiles to create a debug session and attaching using command line dlv to connect to a remote headless instance of dlv.

Currently with checkout templates using the GitHub API. This requires using a PAT to avoid rate limiting. Creating a PAT is a barrier to end-users who want to try roadctl. We need the ability to clone and sync the repository on a publicly accessible server via simple HTTPS.

S3 looks like a possible solution.

The generated test cases for updated and created timestamps fail because we Unmarshall into an interface. This means Created and Updated use the JSON field identifiers which are lowercase.

Switch both to use lowercase.

var m map[string]interface{}
//var md map[string]interface{}
json.Unmarshal(response.Body.Bytes(), &m)

//Test we can decode the data
cs, ok := m["Created"].(string)

Video demonstrating using Delve via Vim and covering our key mappings.

The code for tracking metrics is getting duplicated across micro-services. Create a core library and then use it in templates.

Modify command completion to follow normal conventions

[ ] write to stdout
[ ] look at arg1 instead of using flags

Issuing template commands when not in the parent directory causes backtraces.

Three changes are required.

• Search for a template directory
• Allows template directory to be specified in an environment variable
• Allows template directory to be specified as a command-line option
• Return error message if the template directory can't be located

Sample error

$ roadctl describe templates datamgr 
panic: runtime error: slice bounds out of range [:-4]

goroutine 1 [running]:
github.com/pavedroad-io/roadctl/cmd.tplDescribeResponse.RespondWithText(0x0, 0x0, 0x0, 0xc0000953e0, 0xc0000953d0)
	/home/jscharber/eng/src/github.com/pavedroad-io/roadctl/cmd/templates.go:573 +0x2be
github.com/pavedroad-io/roadctl/cmd.runDescribe(0xe7a5a0, 0xc0000a28a0, 0x2, 0x2)
	/home/jscharber/eng/src/github.com/pavedroad-io/roadctl/cmd/describe.go:74 +0x1a2
github.com/pavedroad-io/roadctl/cmd.glob..func8(0xe7a5a0, 0xc0000a28a0, 0x2, 0x2)
	/home/jscharber/eng/src/github.com/pavedroad-io/roadctl/cmd/describe.go:43 +0x49
github.com/spf13/cobra.(*Command).execute(0xe7a5a0, 0xc0000a2820, 0x2, 0x2, 0xe7a5a0, 0xc0000a2820)
	/home/jscharber/eng/src/github.com/pavedroad-io/roadctl/vendor/github.com/spf13/cobra/command.go:830 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0xe79e20, 0xe7ad20, 0xc000113f20, 0x43d03a)
	/home/jscharber/eng/src/github.com/pavedroad-io/roadctl/vendor/github.com/spf13/cobra/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
	/home/jscharber/eng/src/github.com/pavedroad-io/roadctl/vendor/github.com/spf13/cobra/command.go:864
github.com/pavedroad-io/roadctl/cmd.Execute()
	/home/jscharber/eng/src/github.com/pavedroad-io/roadctl/cmd/root.go:56 +0x31
main.main()
	/home/jscharber/eng/src/github.com/pavedroad-io/roadctl/roadctl.go:35 +0x93

An API gateway is a common design pattern. We need a template that covers this use case.

These are two keystroke vim mappings for the most commonly executed commands used in development and testing.

Today we put quality badges for MD / HTML into the definitions file making it and end-users problem. We need to move this into the integration logic and generate them. That means the end-user can just specify which badges they want.

If the entire integration is disabled, no badges should be generated and any specified in the definitions file can be ignored.

This is a large change involving

• Extend github.com/pavedroad-io/integrations to generate all required badges
• Modify definitions structure to take a list of enabled badges instead of the current enabled, URL, name
• Modify describe command to use the new syntax
• Modify existing example code samples
• Update documentation
• Update UI structs in github.com/pavedroad-io/frontend

Add Kafka manifest generation for logging support along with Skaffold Kustomization support.

Remove docs from .gitignore
Make sure all cheat sheets, and template docs are stored in samples and templates
Remove dangling docs

Ferret provides a multi-file search and replaces via vim. The video needs to demonstrate its usage and explain our key mappings.

https://opentracing.io/ using https://www.jaegertracing.io/

If microk8s is not running, apply kustomize filter to stop deployment to k8s

If it is running, update .kube/config using new dev script describe in issue19

see pavedroad-io/pavedroad#19

microk8s.config > ~jscharber/.kube/config

Today, we include community files in the YAMLs definitions but we do not create them.

Add them to the template creation process. Only files listed are to be included in templates that are generated.

community:
  community-files:
  - {name: CODE_OF_CONDUCT.md, path: ., src: templatedir}
  - {name: CONTRIBUTING.md, path: ., src: templatedir}
  - {name: COVERNANCE.md, path: ., src: templatedir}
  - {name: MEETING.md, path: ., src: templatedir}
  - {name: OWNERS.md, path: ., src: templatedir}
  - {name: SECURITY.md, path: ., src: templatedir}
  - {md5: '123123123123123123231', name: SUPPORT.md, path: ., src: 'git-api://templates.pavedroad.io/templates/oos/support.md'}
  - {name: ROADMAP.md, path: ., src: 'https://w.pavedroad.io/templates/roadmap.md'}
  - {name: SECURITY.md, path: ., src: templatedir}
  - {name: ROADMAP.md, path: ., src: 'https://w.pavedroad.io/templates/roadmap.md'}
  description: list of community files to include

This will need a design document, at a high level we want:
Implement an event emitter abstraction layer
Create an instance of a kafka emitter
Create a consumer that logs to syslog
Create a consumer that logs to Elastic search
Update logging type with more data, https://docs.microsoft.com/en-us/windows/win32/http/w3c-logging

https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
https://godoc.org/github.com/Shopify/sarama#example-AsyncProducer--Goroutines
https://godoc.org/github.com/rcrowley/go-metrics#example-GetOrRegisterGauge

[] Need to pick which client to use

Shopify sarama

Large community with no dependencies
MIT license

Confluent, inc

Fast, support automatic partitioning but requires librdkafka
Apache 2 for go lib
librdkafka has restrictions re notifications
License: https://github.com/edenhill/librdkafka/blob/master/LICENSE
godoc

performance benchmarks: https://gist.github.com/savaki/a19dcc1e72cb5d621118fbee1db4e61f

go-vim integrates go development support into vim. We need a series of videos covering the most commonly used features and our vim key mappings.

Today, we look for files that contain the work hook, a convention we use for identifying end-user code. However, end-users may also use our sample code like httpJob.go and httpScheduler.

This task will create an equivalent of the .gititnore file that takes regex patterns for files to not modify during template generation.

A worker pool is a common design pattern in Go. It implements a series of workers as goroutines that receive and execute jobs.

More details can be found here: https://github.com/pavedroad-io/templates/blob/master/microservices/ga/workerPool/README.md

Ultisnips provides support for inserting code and text into projects. This includes pre-defined snippets and user-created snippets. We need a video explaining vim key mappings we've created and pointers to existing tutorials on using Ultisnip. The video needs to cover using fzf to quickly find snippets.

Delve is the standard debugger for Go and it can be invoked in three different ways.

Command-line
Vim
Or attached to a kubernetes cluster when using Skaffold

We need a video for each of these use cases

We provide a dev/preflight.sh to help end-users set up their environment. Testing the token environment variables are set correctly and populating the templates directory needs to move inside roadctl as an init method.

Currently, we have

roadctl get templates --init

We move the preflight logic here.

NOTE: GitHub is deprecating support for basic authentication, so start using PAT as the default method.

Add an istio manifest for code being deployed to k8s cluster. This can use Skaffold profiles to enable istio support.

This should include HTTPs support

Create a manifest and Skaffold profile for reading log information from our core logging interface via Kafka streams.

Add docker-compose and kubernetes manifest for supporting docker

The docker version will use, wurstmeister/kafka-docker and zookeeper

This requires setting the environment variable KAFKA_ADVERTISED_HOST_NAME to the IP address of the docker interface.

This work is needed for @MarkGreenPR core log integration.

We can also pre-configure a list of topics for k8s and docker-compose

Add it to docs directory of all templates and on the website
vim-ide-cheatsheet.pdf

While setting up a DevKit only takes pasting a curl command, there is nothing showing the user what to expect. Creating a short video to help them understand what will happen.

Vim fugitive add git support to Vim, create a video showing the most common use cases and explain our vim key mappings

The work pool template provides a management interface for setting configuration variables and controlling the service. There are two data structures that define commands and fields.

The managementOptions structure contains mgtCommand and Fields. The latter being a list of strings. The managementInit() method initializes the managementOptions in the dispatcher configuration. Management commands are executed by the ProcessManagementRequest() methods.

Several changes need to happen for this work.

The templateScheduler.go and templateJob.go need to add interfaces for:

[ ] Adding commands
[ ] Adding fields
[ ] Processing fields
[ ] The httpScheduler.go and httpJob need to implement those methods

The templateDispatcher.go needs

[ ] Extend management, mgtCommand structure, with an invocation type: internal, scheduler, job
[ ] Call the commands and fields interfaces during initialization adding responses to managementOptions
[ ] Extend ProcessManagement Request to call the correct interface or handle internal commands

Docker images must be in lower case. Making that the default for tplData.Name is the simplest fix.

Add documentation in the README.md file
Add proper community files

Today we have the required sonar cloud configuration files but we don't call the rest endpoints to provision them. Allow this to be handled via the integrations resource type.

roadctl get integrations sonarcloud
roadclt create integrations sonarcloud -f sonarcreate.yaml

don't pass sync or mutex as value
remove unreachable code

https://github.com/pavedroad-io/roadctl/blob/create-cmd/src/roadctl/cmd/definitions.go

The roadctl create template read a definition.yaml file. In that file, a data object is specified using the Table structure. Each table contains a list/slice of columns. Columns consist of a name, mapped name, modifiers, and constraints.

Today, we have no validation of the types/contents of these fields.

Validation should be added as follows:

type should be a valid JSON value
modifiers should allow "omitempty", or "string" https://godoc.org/encoding/json
the mapped name should be the name as it will appear in JSON or YAML or;

• "-," for a name represented as "-" in JSON

The TableType should be limited to JSONB

json: field should be json: "field"

Review and correct

Time to automate building roadctl releases.

Implement Travis CI builds that push to GitHub releases.

Add FOSSA license scanning and include badge in README.md

The management interface provided by templates in the current release is not extensible. Add go interfaces that enable clients to extend the management API.

Currently, we use the go log library, modify the code to use our core logging services.

Implement test harness for worker pool template and httpJob samples