Comments (2)
Just to clarify what happens in each stage:
stage1
- DRAC network configurations are applied.
- Temporary, predefined user name ("stageone") and password are configured.
stage2
- The final user name ("admin") and random password are applied. The random password is stored in a new GCD entity for the machine.
from epoxy-images.
tl;dr: at this point this appears to be a firmware issue, which was likely resolved in some newer version of the firmware.
I have a bit more data on this based on my experiences and testing with bringing up BOG04. All machines at BOG04 were stuck in stage1 because of a known bug in the epoxy_client. However, since we can login as root to stage1 boots, I was able to login and experiment with ipmitool manually. What I found is that more often than not, calls to ipmitool to modify network settings would yield something like the following:
root@mlab1-bog04:~# ipmitool lan set 1 ipaddr 200.189.196.132
Setting LAN IP Address to 200.189.196.132
LAN Parameter Data does not match! Write may have failed.
The command would hang for right about 30s, and then dump that last message and exit. In some of the cases, I found that the value had actually been modified, despite the warning. In other cases, the value was not modified. I found that this problem existed in stage1, as well as when booted to stage3 being part of the cluster.
Searches yielded little, but a number of results indicated that a firmware issue was likely the root cause, so on mlab2-bog04 I upgraded the iDRAC with Lifecycle Controller firmware to 4.20.20.20, the latest version. After the upgrade, calls to ipmitool to modify network settings returned nearly instantly, and worked.
The workaround suggested in the first comment of this issue is likely the easiest fix, for now, and can't hurt in any case. I discovered that despite the warning/error, after a couple of tries the setting eventually took.
Upgrading the firmware could be nice and there are several possible options:
- Give instructions to site installers on how to upgrade the firmware.
- Upgrade the firmware manually on each node ourselves as part of bringing a site up.
- Figure out some way to automate the upgrade of the firmware.
The latter option sounds the best, but at this moment I have no idea how it could be accomplished, thought I am sure it can be done.
from epoxy-images.
Related Issues (20)
- Nodes need to know whether itβs 10g or 1g uplink
- TCP pacing maxrate is currently set to the max uplink rate: revisit this setting HOT 3
- Need separate builds for stage1 images (longer) and generic images (faster) HOT 4
- stage1 images should have a fall-back login mechanism HOT 2
- Servers won't boot if an Internet connection isn't immediately available
- epoxy-images builds epoxy_client from HEAD HOT 2
- epoxy-images deployment for images should preserve older versions HOT 1
- fix-hung-shim.sh must recognize multiple shim processes HOT 1
- Enforce that epoxy-images are not tagged before k8s master updates HOT 1
- stage1 builds are failing due to python missing from the image HOT 1
- configure_tc_fq.sh for CoreOS does not work with v2 names
- Write the USB equivalent of mlx_update HOT 1
- Enable unattended upgrades for security updates in Ubuntu
- Disable "dynamic MOTD" when logging in via SSH HOT 4
- R640 CPU frequency set for "performance" HOT 2
- Create systemd unit to trigger node reboot when uptime is too long
- Create versioned stage3 images and URLs HOT 2
- IPv6 SLAAC and RAs appear to take down container IPv6 HOT 11
- Update packages on each build of stage3 Ubuntu image
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from epoxy-images.