autometrics-dev / am Goto Github PK
View Code? Open in Web Editor NEWAutometrics Companion CLI app
License: Apache License 2.0
Autometrics Companion CLI app
License: Apache License 2.0
When using am start
we should also add the rules for the SLO's.
These alerting rules can be used in the following way:
am
Explorer can then connect to this ws connection and then show something in the UI once it happens.
Alternatively we can add the monitoring of these metrics directly to am
but that can still expose the event through a ws connection.
Currently we just use the tracing
crate with the standard tracing_subscriber
printing every log line to stderr
.
This does come with quite some overhead such as the time and where in the code the log originated from. This can be useful when debugging the application itself, but it can be a bit overwhelming.
Do we go with the approach that we used in fp
and create a different format which removes a bunch of things, such as the timestamp, call site, and fields associated with the log message, and then allow the user to pass in the --verbose
flag to get the default tracing_subscriber
?
This allow am
to automatically download and hook it up to the selected Prometheus instances.
It should also automatically setup the autometrics dashboard, and allow for a user to supply their own custom dashboard to be loaded automatically (this could possibly be a separate ticket).
Before am
starts Prometheus, it will write the config to disk. Currently the same location is used all the time. This will be a problem if multiple am
instances are started, since the configuration will be overwritten.
am
should write the Prometheus config to a semi-random file in .autometrics
and should be cleanup once am start
finishes. If the ephemeral flag is given, then we should not store it in .autometrics
but in a temporary file.
When a users visits the root of the webserver, they should be redirected to /explorer/
.
Related: When a user does not specify any path with /explorer/
then we should serve index.html
.
We currently download the archive but we do not verify it. We can download the checksum file from the GitHub release and verify it against that value. In this case it mostly protects against corrupt downloads, since we will retrieve the checksum file in the same way that we will download the archive itself.
If we wanted to protect against malicious archives being downloaded (ie. maybe a man in the middle attack), then we would need to get this checksum through some other means, such as letting the user provide it through a cli argument.
Switch to a different version of hermit-abi
to resolve this issue.
The idea is that people can set the prometheus URL to http://localhost:6789
and then the explorer UI would parse the query parameters there and redirect the user to the relevant functions/graph page.
The CLI should generate a markdown file with a reference for all documented features in it. The markdown file will be loaded in the project's official documentation site: docs.autometrics.dev
Stretch goal: pass on the markdown file to the documentation repo and trigger rebuild automatically
Currently the flag --enable-pushgateway
only says that it will start off a
pushgateway without explaining what it is and why you'd need it. We should add
more documentation explaining it.
It would be nice if am
detects if a Prometheus instance is already running (for instance via quickmetrics/docker) and won't download/run a local binary.
An alternative would be to run am
with a --no-prometheus
option that assumes that the user is already running, or connecting to a different Prometheus instance.
am
should periodically check if there is a new version available and guide the user into updating the binary. For installs through homebrew we should advise the user to use the homebrew command.
The flag --listen-address
is ambiguous as to what exactly will be listening on
that port. We should consider renaming it to --explorer-listen-address
or
--explorer-port
.
The Pushgateway is meant for one-off batch jobs, not for long-running processes that keep updating the metrics. Metrics scraped from the Pushgateway will be incorrect when it's used with Autometrics.
We should use the prom-aggregation-gateway or the prometheus-gravel-gateway instead, both of which are built for aggregating metrics like we need for Autometrics.
Currently the Prometheus and Pushgateway process inherit the working directory of the am
binary. Prometheus will create some files there for its storage. It would be better if we move that into $PWD/.autometrics/{prometheus,pushgateway}
(not sure if it needs to be scoped to version as well). That way we can also create command to clean it up.
Another things related to this would be a --ephemeral
flag which will delete the data after am
shutsdown.
Currently when the prometheus install/run task fails it will not print this to the output and it looks like the application is running fine. Instead am
should exit and display the error message.
This command would generate a am.toml
file with some sensible defaults and ideally show a wizard that will ask the relevant questions;
etc
When using am start
we start a webserver which can be scraped by other processes.
Optionally we should let the Prometheus instance scrape that endpoint. The reason why we probably don't want this by default is that it can be distracting and probably isn't useful when a user is working on their own application.
users should get a notification if a new version is released and maybe even have the ability to upgrade with ease with just one command
This should download and start the push gateway. It's api should also be exposed through the webserver so that the user can manipulate their running push gateway.
If the user is using the Prometheus that am
provides, then it should also add the gateway as a target (ie. the user should not have to setup any targets in this situation).
Do we enable the push gateway by default or should we make the user opt-in to be able to use it? We could enable the push gateway, if the user has not supplied any endpoints (ie. am start
) and in other situations they have to use --push-gateway
.
Currently we are running prometheus and pushgateway on a specific port, and we are also forwarding the content using the listen address of am + /prometheus
. This requires us to specify the web-address as a config to either components. The downside is that the origin then uses the new path in the prometheus web server. Which can be confusing for a user since they need to forward their metrics to localhost:9091/pushgateway/metrics
which can be confusing.
Pro's of proxying:
Con's:
Add a README.md
which display the following:
This command would be used by a user to open the explorer in their browser. It should also display the URL and allow for an option to not open the browser.
By default the URL that is opened should point to the local explorer giving its defaults.
The first argument to explore
can be a different Prometheus endpoint. This will still use the same explorer instance but it will pass this Prometheus endpoint to explorer to use. Example: am explore https://prometheus.example.com
.
We should also be able to use a different explorer endpoint, so this should be another argument. Example: am explore --explorer-endpoint https://explorer.autometrics.dev
.
The defaults for the Prometheus endpoint and explorer endpoint should also come from a am configuration file for the user.
Currently am start
set the Prometheus URL as field in the html. In the scenario where we use a different explorer endpoint we have no control over the html and so we need a different way of passing the Prometheus URL to explorer. This would probably be a query-string parameter.
We should first only push to the Docker registry. The Docker image should be a multi-arch image, containing both x86 and arm architectures.
I another PR we can support different registries such as:
Currently we allow a user to submit endpoints using various short hand forms, such as :3000
. Currently this is mentioned in the docs that this is also allowed in the [[endpoint]].url
property in a am.toml
. Currently this is not supported as it is parsed as a regular URL.
We should only display the details once the download and unpack is complete for both prometheus and pushgateway.
This is not an issue when the users already has the applications cached. But it does give a poor experience for a first time user.
am start :3000
Explorer endpoint: http://127.0.0.1:6789
Prometheus endpoint: http://127.0.0.1:9090
Now sampling your application at http://localhost:3000/metrics for metrics
When using am
to monitor your local application it makes sense to get a quicker feedback loop. By changing the default scrape duration to be lower than the default we can get data from your application into Prometheus faster and you'll be able to see a better resolution.
We should also allow the user to set a default value as a command line argument, or as a value in the am.toml. Any endpoint configured in the am.toml should also allow a override from the default.
The call to /metrics
on the am
web server should proxy this call to the pushgateway /pushgateway/metrics
. This payload will contain the metrics from a component that is not able to be scraped, such as a function, batch job, or a client.
Note that we cannot redirect the request since it is a POST request. Even if we were to do a 307
I'm not sure how well that is implemented in all the different languages/packages. So it is easier just to proxy it.
Related discussion: #60 (comment)
PR #42 introduced a bunch of duplication ๐ We should be able to get rid of some
This should include the following (either automatic or manual):
Crate.toml
https://github.com/autometrics-dev/autometrics-rs/tree/main/autometrics-cli
And update docs to point to new location:
It would be nice if the CLI could serve an endpoint that would expose metrics that mock traffic on your service, using the actual function names from your code. It would probably want to use the same static analysis that we've discussed elsewhere to see what the function names are.
Ideally, it would also figure out the caller
label, but that's harder.
Also, it might be nice if you had a visual dashboard with a bunch of knobs you could use to adjust the traffic and potentially errors for specific functions.
This, or something like it, would be one way of testing out your alerts or doing manual integration tests with them.
We should make it as easy for a user to supply the endpoints that they want to monitor. The only thing that is required is the host, all other components have some defaults:
http
and https
, where http
is the default.80
for http
and 443
for https
./metrics
if the path is empty. It should not be appended if a path is already there.This should result in the following examples:
am start 127.0.0.1
: would result in http://127.0.0.1:80/metrics
am start https://127.0.0.1
: would result in https://127.0.0.1:443/metrics
am start localhost:3030
: would result in http://localhost:3030/metrics
am start localhost:3030/api/metrics
: would result in http://localhost:3030/api/metrics
am start localhost:3030/api/observability
: would result in http://localhost:3030/api/observability
/metrics
to it)A user can also mix and match these: am start 127.0.0.1 https://localhost:3030/api/observability
NOTE: since we are adding the /metrics
path when a path is not present in a endpoint it is not possible to have Prometheus scrape the root of a webserver. I think this is an acceptable limitation, since it makes the "normal" usage way more easy to use and users will not likely have their metrics in the /
path.
Displaying this information is quite verbose and not super useful, since it is in a different format which makes reading it quite confusing.
We should store this data into a buffer (maybe only collecting the last x bytes) and then expose this to the user.
The main way of retrieving this information would be through the am
's api. The explorer can display this information. We can even create a live view of these logs through web-sockets or server side events.
Another way of using this data would be if an error occurred, that am
would dump these logs into a file. These can then be used by the user when creating a bug report.
Finally a cli argument can be added that will make the Prometheus stdout/stderr go to into am
's stdout/stderr.
The default prometheus config has this set to false
, but it is more useful to have it set to true
when scraping from pushgateway:
honor_labels controls how Prometheus handles conflicts between labels that are
already present in scraped data and labels that Prometheus would attach
server-side ("job" and "instance" labels, manually configured target
labels, and labels generated by service discovery implementations).If honor_labels is set to "true", label conflicts are resolved by keeping label
values from the scraped data and ignoring the conflicting server-side labels.If honor_labels is set to "false", label conflicts are resolved by renaming
conflicting labels in the scraped data to "exported_" (for
example "exported_instance", "exported_job") and then attaching server-side
labels.Setting honor_labels to "true" is useful for use cases such as federation and
scraping the Pushgateway, where all labels specified in the target should be
preserved.Note that any globally configured "external_labels" are unaffected by this
setting. In communication with external systems, they are always applied only
when a time series does not have a given label yet and are ignored otherwise.
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config
This can already be a indicator that the user has used an incorrect configuration.
When invoking am start [endpoint]
we can already make a request to the provided endpoints and see if they work and contain metrics data. If they do not work we can provide a warning, but we should not stop execution of am
, since the user might not have started their application.
Depending on the application to be up and running, is a bad experience. Perhaps the workflow of the user is to just turn on am
at the start of their development cycle and then just leave it running.
Some of these options would only be shown depending on arguments that are passed into the application. So if the pushgateway is not running, then it shouldn't show this endpoint.
Some potential details that could be interesting:
--verbose
flag)There is the rust-maintainers
group that should be assigned by default.
Passing in :
plus a number would result in using http
and localhost
as the defaults. It is still possible to use a path after this.
So am start :3000
would become http://localhost:3000/metrics
.
See #9 for more details on the previous work.
This will allow us to define all the endpoints, whether to enable pushgateway and others in the file which is going to be stored in the repository itself. This makes it easier for users that work on the same codebase to quickly get up and running with am
with their local development flow.
Possible locations (note we could try them all):
.autometrics/am.toml
am.toml
Edit: Changed to toml since that is probably easier to work with for our users.
Each target can have a name. This will be exposed as the job
label in the metrics. It would be useful for the user to specify these names.
Currently we just have app_{}
where the parenthesis will contain a incrementing number (to avoid collisions).
With a config file we can simply add another property, something like name. For the terminal approach, we might want to include a prefix or postfix. So something like api|:3000
. The first part would be the name of the job, in this case api
, and the part after the |
will just be parsed as it currently is.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.