cloudfoundry / bosh-acceptance-tests Goto Github PK
View Code? Open in Web Editor NEWBATs: BOSH Acceptance Tests
License: Apache License 2.0
BATs: BOSH Acceptance Tests
License: Apache License 2.0
This example will try to check the running services on a deployment, but the output of ps
command with bosh ssh doesn't have a proper format, that cause the check cannot pass.
For more details, please see:
http://paste.openstack.org/show/722865/
Hi
We are running the tests from branch gocli-bats. We are using bosch cli 2 and we have a bosh running in aws.
After running some of then successfully, all the others starts failing with message:
Could you help us on that? What are we doing wrong?
This is my configuration:
export BAT_STEMCELL_URL="https://bosh.io/d/stemcells/bosh-aws-xen-hvm-ubuntu-trusty-go_agent?v=3421.11"
export BAT_STEMCELL_VERSION=$(echo $BAT_STEMCELL_URL | sed ''s/.*v=//)
export BAT_STEMCELL=bosh-aws-xen-hvm-ubuntu-trusty-go_agent
export BAT_DEPLOYMENT_SPEC=/bats/bats.yml
export BAT_BOSH_CLI=bosh
export BAT_DNS_HOST=8.8.8.8
export BAT_INFRASTRUCTURE=aws
export BAT_NETWORKING=manual
export BAT_PRIVATE_KEY=bosh/bosh.pem
export BOSH_OS_BATS=true
export BOSH_ENVIRONMENT=${BOSH_IP}
export BOSH_CLIENT=admin
export BOSH_CLIENT_SECRET=`bosh int creds.yml --path /admin_password`
export BOSH_CA_CERT=`bosh int creds.yml --path /director_ssl/ca`
echo "Importing stemcell ---------------------------"
curl -O -sL $BAT_STEMCELL_URL
mv bosh-aws-xen-hvm-ubuntu-* $BAT_STEMCELL
echo "Create config file ---------------------------"
cat <<< '---
cpi: aws
properties:
stemcell:
name: bosh-aws-xen-ubuntu-trusty-go_agent
version: 3421.11
pool_size: 1
instances: 1
vip: 54.54.54.54
second_static_ip: 10.***.101
networks:
- name: default
static_ip: 10.***.100
cidr: 10.***.0/24
reserved: ["10.***.2 - 10.***.25"]
static: ["10.***.100 - 10.***.110"]
gateway: 10.***.1
subnet: subnet-***
security_groups: sg-***
key_name: ***
' > bats.yml
echo "Running tests --------------------------------"
bundle exec rspec spec
The error logs:
BATs example started 'auditd, sshd, cron, rsyslogd should be running' ./spec/system/auditd_spec.rb:15
BATs example failed 'auditd, sshd, cron, rsyslogd should be running' ./spec/system/auditd_spec.rb:15
1) auditd, sshd, cron, rsyslogd should be running
Failure/Error: @requirements.requirement(deployment, @spec)
expected command to exit with 0 but was 1. output was
{
"Tables": null,
"Blocks": [
"\n13:38:43 | ",
"Deprecation: Ignoring cloud config. Manifest contains 'networks' section.",
"\n13:38:43 | ",
"Preparing deployment: Preparing deployment",
" (00:00:00)",
"\n L Error: Stemcell 'bosh-aws-xen-ubuntu-trusty-go_agent/3421.11' doesn't exist",
"\n",
"\n13:38:43 | ",
"Error: Stemcell 'bosh-aws-xen-ubuntu-trusty-go_agent/3421.11' doesn't exist"
],
When I check from my console the stemcell is there at the beginning. After some tests the stemcell is not any more there.
bosh -e 10.*** stemcells
I found the test case of persistent disk migration, we create the file by root but read it by bosh_ssh_user.
https://github.com/cloudfoundry/bosh-acceptance-tests/blob/master/spec/system/with_release_stemcell_spec.rb#L88
https://github.com/cloudfoundry/bosh-acceptance-tests/blob/master/spec/system/with_release_stemcell_spec.rb#L105
Why did we think the process by two different users?
When we built and tested stemcell-3422.7(for umask case), the case would be failed due to permission.
cat: /var/vcap/store/batarang/save: Permission denied
So I dug into the command execution process and found the file is 600 because of umask '0077'.
So how about to run write/read in the same user, because this case it to test persistent disk migration.
The test is green with or without let(:ca_cert)
statement.
Is this a leftover from ruby cli times?
Otherwise the deployment manifest shown by the tests is of very limited use. Consider e.g.
+ properties:
+ openstack:
+ connection_options:
+ idempotent: "<redacted>"
+ batlight:
+ fail: "<redacted>"
+ missing: "<redacted>"
Will it fail or not? Wdyt? :D
Comparing templates/cloud_config_vsphere.yml.erb#L17-L24 to the subnet section of templates/cloud_config_google.yml.erb, you will see that there is no way to specify a reserved list of ip ranges for google. This means for networks that are shared with other services, we cannot run these tests.
While BATs are a good test bucket that has served us well to help verify and allow developers and users of BOSH to know that any change(s) to stemcell, agent, and/or director are working together well, there is a need to understand at a high-level the various commands and workflows that BATs is covering.
The current tests do not seem to have a test plan associated with it, and the only way to determine the actual plan for BATs is to run the tests with --dry-run
which simply shows the sequence of tests' descriptions.
Ideally some type of test plan (automatically generated with code or in a document) would allow others who are interested in adding or improving BATs a means to see what is missing and where to improve the coverage and or workflow that BATs is doing.
When I running bats based on ubuntu bionic release and got the following error:
Failures:
1) network configuration when using manual networking deploys multiple manual networks
Failure/Error: expect(bosh_ssh('batlight', 0, 'PATH=/sbin:/usr/sbin:$PATH; ifconfig', deployment: deployment.name).output).to include(static_ips[1])
TypeError:
no implicit conversion of nil into String
# ./spec/system/network_configuration_spec.rb:79:in `block (3 levels) in <top (required)>'
Finished in 35 minutes 39 seconds (files took 0.41484 seconds to load)
72 examples, 1 failure, 12 pending
Failed examples:
rspec ./spec/system/network_configuration_spec.rb:73 # network configuration when using manual networking deploys multiple manual networks
This error did not happen in the ubuntu xenial release.
The test should survive agent dying
[1] is flaky in our build pipeline. It fails with:
Failure/Error: expect(bosh_safe("logs batlight 0 --agent --dir #{tmpdir}")).to succeed
expected command to exit with 0 but was 1. output was
RSA 1024 bit CA certificates are loaded due to old openssl compatibility
Acting as user 'admin' on deployment 'bat' on 'micro'
Director task 11
Started fetching logs for batlight/1b0362eb-6deb-41a1-a837-99d0511e8c8b (0) > Finding and packing log files. Failed: Timed out sending `fetch_logs' to 39f5c7ff-89ce-42b3-9e6f-30e66cd7849a after 45 seconds (00:00:45)
Error 450002: Timed out sending `fetch_logs' to 39f5c7ff-89ce-42b3-9e6f-30e66cd7849a after 45 seconds
To us it seems that the test scenario is not correct. First the agent is killed, after that wait_for_vm
[2] is called which doesn't check the vm state just searches for this vm existence. At the end the test tries to fetch the agent logs but agent could be still in unresponsive state. Is the scenario correct or should the wait_for_vm
wait for state running before returning?
We can provide a fix if need.
[1]
bosh-acceptance-tests/lib/bat/bosh_helper.rb
Line 113 in 6c62618
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.