Calling a script from stdin, while nifty, is no longer needed and ugly. Revert to standard exec w/ args.

The only thing that prevents one external task from impersonating another, possibly accessing private / privileged memories, is the difficulty of guessing the GOPHER_CALLER_ID of another external task that may still be running. Should be simple enough to make this large enough and random enough for it to be sufficiently improbable.

As encryption becomes more heavily used, the current chat-only "encrypt" and template-only "decrypt" are proving inadequate. Need something like:

gopherbot encrypt "somestring"
gopherbot encrypt (-b) -f path/to/file # -b encode with base64
gopherbot decrypt "string"
etc.

Encrypt and decrypt file should print to stdout, which can be redirected.

I want to add a callback endpoint on Bot. How can i achieve this ?

When a job starts, GOPHER_CHANNEL should be set to the job channel, so all Say's go there

At issue is that e.g. the parameters for weather.rb can't be overridden / replaced, since parameters are an array. Also, items in a configured array can't be removed; for instance, a user may wish to remove a command matcher to avoid interfering with a locally defined command, or update the help for a plugin.

A potentially useful variant of Prompt*Reply, allowing any user listed in Task.Users to reply; mainly for pipelines, to allow overriding defaults for a pipeline - e.g. "Tests passed, deploy? (y/n)" or "Skip deploying? (y/n)"

The simplest way to email the log you want is for the build job to remember build refs from the short commit hash, and say them in the output, so for instance:

; mail joblog #c564ab

Make a 3-argument version of localbuild for repo, branch, pipeline; let localbuild do the work of initializing ssh and cloning a repository, then run a custom pipeline script for the job.

• Add support for Rocket.Chat
• Script should run the robot, then run it again, allowing the robot to write out the administrator
• Remove builtin_admin config once admin added

Gopherbot needs a TLS, authenticated restful API to support triggering jobs with webhooks, pinging another instance in an HA pair, and other stuff. The design still needs to be worked out, but generally:

• The central bot library should use configuration in gopherbot.yaml to set up the URL and TLS
• Loadable Go modules should be able to register individual handlers with unique, configurable tokens
• The library authenticates the request by checking for ?token=XYZZY, then passes the token and request to the registered handler for the provided token
• The TLS key file should be a #GOPHERBOT-ENCRYPTED-BASE64 file in custom
• Non-TLS endpoints should be allowed when e.g. the webhook may sit behind an SSL proxy

mdbook? https://github.com/rust-lang-nursery/mdBook

"tell me about floyd" or "ping floyd" shouldn't make the robot think it's being spoken to; instead "ping, floyd"

It would be very useful to add attachment. In my case, bot generates a config file for the user and i want to reply with file as a attachment.

When "meme" plugin was renamed, robot panic'ed trying to load config that didn't exist.

Unprivileged plugin/job scripts, running as e.g. nobody, should be able to run without having full path access to the running script in $GOPHER_CONFIGDIR.

Add new Homed boolean to tasks, interpreted as follows:

• When a new pipeline starts, if the initial job/plugin is Homed, the basePath should be ".", and environment vars should be set:

  • GOPHER_WORKSPACE should be set to the location of the workspace, likely a path relative to cwd
  • GOPHER_CONFIGDIR should be set to the configuration directory

• When a task has Homed = true, that task should always start in $(pwd), with GOPHER_WORKSPACE and GOPHER_CONFIGDIR set, and also GOPHER_WORKDIR set to c.workingDirectory

• Update cleanup task to run Homed, and to clean up GOPHER_WORKDIR; be sure to check and fail on absolute paths

• Update robot/* interface to add SetWorkingDirectory

• Update pipeline to set basePath to "." when Homed = true, or c.cfg,workSpace otherwise

• Check ExtendNamespace for correct operation

• Check calltask for correct operation

• Check SetWorkingDirectory for correct operation

• Check task environment to eliminate absolute paths when Homed = false

• Re-futz Start() to deal with relative paths; bootstrapping should work with relative config path, but not absolute - "clean" will need to clean configDir during bootstrap, which will fail for abs. paths

• When Quiet: true, job events including namespace should emit no messages
• gopherci should log stuff, like "skipping foo, not listed" when nothing happens silently
• gopherci or localbuild should emit a single job start message with URL link if available
• gopherci should add fail task(s) to spit out messages on fail
• gopherci should take an "event" argument; currently one of "tag" / "commit"; the branch argument is either the branch committed to or the tag
• individual services such as github, gitea, etc. should have small "connector" jobs that match messages from the service and call gopherci <branch/tag>

-> Robot should only emit messages for e.g. user jobs that don't use the library and don't have quiet: true
-> gopherci should be responsible for messaging when a job fails

Updates to Makefile and pipeline to build and publish small static builds for Windows and Mac, disabling "run" mode but allowing e.g. memory and secret management with the CLI.

The only thing that would prevent a build on Windows ATM is privsep.go. This could be re-split in to privsep_unix and privsep_win with empty function definitions.

The Mac and Win builds should probably include the terminal connector and statically link all brains, but leave off e.g. duo and totp plugins.

• RegisterTask, RegisterJob for static compiles (less likely, but might as well)
• GetJobs, GetTasks for loadable modules (more likely)

The install archive should include markdown sources for book in doc/.

the SimpleBrain interface should include a list function available to the innards and the CLI, to list all the keys. The CLI should be enhanced with a "list" command for listing memories, and a "store" command for storing a file to a given memory. This would allow easy and flexible migrations between different brain types, or to/from encrypted brains.

Per @pasali, API docs are high prio; especially for Go plugins (tasks, jobs)

• Update external script API docs
• API docs for Go plugins, tasks, jobs
  • Compiled-in Go modules
  • Building loadable modules
• Document deployment
  • Deploying a container
  • Running from the CLI
  • Manual systemd setup
  • Deploying with Ansible
• Documentation for workflows and admin robot operations
• Jobs, pipelines, scheduled jobs

Administrators should be able to e.g. set the PATH with a stanza like:

ExternalEnvironment:
- Name: PATH
  Value: /usr/bin:/usr/bin:/opt/gopherbot

Sending error logs is kinda painful; make a simple to use task for this (which really just adds a command).

There's really no good reason to fix the local listening port at e.g. 8080, since it's passed to scripts in GOPHER_HTTP_POST anyway. Using a value of "0" should select one dynamically.

There is a lot of information in slack.Msg like file, attachment, comment etc. so passing it in commands would be very nice. Just passing slack.Msg.Text won't be enough in every case.
For ex:

It would be very nice to get userId information from bot.
Ex: r.userId or maybe r.User.Id

Usability for "set task parameter" is pretty lousy. Storing it in per-repository Parameters is more transparent. Replace "set namespace parameter" with a new "NameSpaces" section in gopherbot.yaml with shared parameters for a given namespace. Task/job/plugin-specific parameters should override anything from the namespace. Remove "set task parameter", evaluate whether "get/get * secret" is needed.

• The GetRepoData() call should have all Parameters blanked out.
• When ExtendNamespace() is called, parameters for type:repo should be set in the pipeline, e.g. localbuild:github.com/lnxjedi/gopherbot

Support for loadable modules, especially connectors and user plugins.
Compile options need to support:

• Module support with connectors / plugins either built in or loadable
• Fully static builds w/ no support for modules

Standard build archive is a modular build including loadable modules for most connectors.

I have a one specific channel for jira slackbot. It posts jira events to its channel. On the other hand my own bot listens jira channels messages and does some action. That does not work currently with gopherbot. So i debugged the code and realize that on the below line:
https://github.com/uva-its/gopherbot/blob/master/connectors/slack/util.go#L160
msg.Msg.Text is empty when its a bot message. And userID is also empty. So if a message comes from another bot, gopherbot should extract text from msg.Attachments[0].Fallback. What do you think about this ?

.Include'd templates should be expanded before being returned, to allow for encrypted secrets - duh! Don't forget to update the doc for .Include.

Needs more thinking and work, for instance:

• list <something> is too general, needs to be show list <something> or show (the) <something> list

When a plugin is given the same name as a task, the plugin should be disabled to highlight the error to the admin.

• The robot remembers duo answers in short-term memory, but doesn't say anything before activating Duo, so the user thinks the robot isn't responding
• The plugin should have commands for permanently saving or changing settings in long-term memory

If the .env file exists, it must only be readable by the privileged user, or fail fatally.

Goal: simplified portable installation for server and container based deployment, well documented. Standard install based on github & slack.

Initial Installation

You should only ever need to run the gopherbot binary, and the setup plugin, bootstrap & terminal connectors should do the rest. Installing an instance of gopherbot means creating the custom and optional private git repositories such that a particular robot can be run in any environment with just a small set of environment variables.

• Setting up a robot in a local docker container should just mean running the container in the foreground initially, redirecting the log to a file: docker run gopherbot 2>setup.log
• Setting up a robot on a Linux host should just mean running e.g. /opt/gopherbot/gopherbot 2>setup.log

Post Install Setup

Once the robot is running and linked to it's configuration repository, the administrator will need to configure users and administrators using the globally available whoami and identify <user> commands. This information can be used to populate the user roster and administrator configuration items, which can then be made active with reload <setupkey>.

Optional post-install setup would include github integration for automated git pull and reload when the custom repository updates.

Default and Custom Configuration

Most of the default configuration should be moved out in to robot.skel for populating the starting configuration for a freshly-created robot.

Super-powered setup plugin

• When the robot is completely unconfigured:
  • the "setup" plugin goes in to action during "init"
  • Robot automatically starts with the "terminal" connector, and prompts for:
    • brain encryption key
    • custom configuration repository
    • optional private memories repository
  • ... these values will be written out to ./.env
• When the robot starts with only a set of environment vars:
  • minimal env:
    • GOPHER_ENCRYPTION_KEY - min. 32 char key for decrypting binary brain key
    • GOPHER_CUSTOM_REPOSITORY - publicly available or password-accessible repo
    • GOPHER_PROTOCOL - connector protocol should default to terminal unless this is set; not strictly required, but default for new installs
    • NOTE: the binary brain key will need to be in custom repo, but it's only useful w/ the encryption key
  • optional additional env:
    • GOPHER_GIT_USER - username for access to GOPHER_CUSTOM_REPOSITORY
    • GOPHER_GIT_PASS - password for git access
    • GOPHER_PROTOCOL - connector protocol to use; normally for setting to "term" to interact with a specific robot on the command line, for e.g. encrypting passwords
  • If the custom repo is empty, the robot should start with the bootstrap (null) connector, and the setup plugin init should try to clone it
  • If the clone returns an empty repo, the robot should restart with the terminal connector running the setup plugin which will prompt for protocol (if not already set), protocol credentials, and a setup token needed to issue the reload <token> command.

New canonical directory structure

To support the simplified installation and bootstrapping, the following canonical directory structure will be adopted:

• clu/ - top-level directory, not a repository root, optionally persistent in containerized installs
  • gopherbot -> /path/to/gopherbot - optional symlink to binary for cd path/to/clu; ./gopherbot
  • .env - optional environment file w/ brain unlock key (most secure)
  • custom/ - custom configuration directory, possibly with custom jobs/plugins and potentially with encrypted secrets (ala {{ decrypt "encryptedstring" }} a public (or possibly private) github repo in the standard install that the robot monitors and performs a pull/reload on change
    • ssh/ - robot's ssh config file and keys
    • binary-encrypted-key - internal binary brain encryption key, base64 encoded, used for all brains
  • private/ - location for files that should rarely or never be shared
    • environment - backup location for loadable environment; useful for dev robots
    • brain/ - directory for file-based brain; a private github repo in the standard install that the robot can automatically back up to with it's ssh key
    • NOTE: the default .gitignore should exclude noisy memories like history logs
  • history/ - location for file-based history logs, considered disposable but might want to back up (not to git, it would be too noisy); if backing up history is desired, the associated memories will also need to be backed up
  • workspace/ - location where local builds are done, definitely disposable

Let say i defined a property like this:

Github:
  Token: "XXXxXXXXXX"

How can i read this property from gopherbot.yaml ? Or is there any other way to do this ?

To support running in a docker container, the robot should be able to start with an empty config directory (in docker), and pull the config repo. Make sure admin can be stored in environment variable.

Plugins should run with dropped privileges by default, jobs should run privileged. AddJob should fail if the plugin isn't running privileged.

The robot should be executed by the appropriate privileged user with access to e.g. read and update configuration, and setuid to an unprivileged user, e.g. daemon. The privsep init func should setresuid to priv,priv,unpriv, then droppriv should setresuid to unpriv,unpriv,unpriv. The privcheck function should be renamed to raisepriv, and called for job pipelines and privileged plugins.

Re-purpose outdated "protected" flag as a "privileged" flag, to mark plugins as privileged, and this flag must be set in gopherbot.yaml, and ignored in the plugin external config file. Plugins that were formerly "protected" now need only set working directory to $GOPHER_CONFIGDIR (which should be in the environment).

The current means of scanning hosts doesn't work well with Debian-based systems where the hostname is hashed; explore alternative means using e.g. ssh -oStrictHostKeyChecking=no user@host exit, or

ssh <<EOF | ssh -oStrictHostKeyChecking=no user@host
EOF

Many calls to Say, Reply, etc. end up importing fmt and using fmt.Sprintf; it's easy enough to convert methods from string to strting, ...interface{}, avoiding the need to import fmt for future go plugin loadble modules.

The current v2 snapshot implementation of shared namespaces is not very transparent and hard to use. Implement a separate "NameSpaces" that just takes shared parameters; require NameSpace parameter for tasks, plugins and jobs to refer to one.

Individual gopherbot.yaml files are getting long and hard to read.

There are still lots of places in the code where the global lock on currentCfg is taken, even though there's a valid cfg attached to the current context. Audit and remove all those extra locks in favor of using the non-changing value attached to the context.

This is an issue with the Slack connector. When users get a command syntax wrong, it's normal to try and edit the last message - but the robot isn't reading it.

Is there any way to mention a user other than who calls bot ? For example hubot's encourage command, you can pass a bot something like this hubot encourage name then it responds with mentioning target user.

For increased reliability and availability, Gopherbot should support simple HA pairs (not clusters).

A few design notes:

• HA should rely on webhooks between instances on the same subnet, in case one or both instances are unable to reach the chat service - but are still running automated jobs; optionally the standby could check chat availability of the active instance, and complain but not take over if the active instance doesn't respond
• gopherbot.yaml will need a new stanza for HA config, and the standby instance shouldn't start until it can actually reach the primary
• A dotfile in $GOPHER_HOME should indicate to each instance whether it is 'active' or 'standby'
• The active node should have a scheduled (late-night) job to initiate a "swap", with positive acknowledgement, where the instances swap active/passive roles on a regular basis, to insure that both robots are capable of running performing all tasks (they can write out new dot-files indicating active/standby)
• The standby robot shouldn't run scheduled jobs, but should expect to hear from the active node when a scheduled job completes, and complain otherwise
• gopherbot.yaml will need to have IP/port for contacting both instances, with a per-robot unique ID that each robot can locally determine (perhaps a unique dot-file in $GOPHER_HOME); logic to determine how to contact the other instance should be look at both endpoints and contact the one that's "not me"