Windows Stemcell Size too large about bosh-windows-stemcell-builder HOT 12 CLOSED

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/145582705

The labels on this github issue will be updated when the story is started.

from bosh-windows-stemcell-builder.

Are you executing the vSphere instructions? Just before the sysprep step, run these commands in order:

• Optimize-Disk
• Compress-Disk

These will take some time. Then follow through with sysprep and creating the stemcell and tell us the size. We're considering putting these into the canonical instructions.

from bosh-windows-stemcell-builder.

@awmartin
Yes executing the vSphere instruction as we are working on vsphere env.
Created a Windows 2012R2 VM with 60 gb disk on vsphere 6.5
Ran the rake commands, Install CF-features, Windows update and dotnet framework.
Ran dism and disk fragmentation to

• optimize disk
• compress disk
  created ovf from vcenter
  created ova using the ovf tool
  then converted the ova to stemcell using rake command and we got 12 gb stemcell

from bosh-windows-stemcell-builder.

Did you start with a fresh MSDN ISO?

Another step that could help would be to use a disk analysis tool to find unusually large folders. We found recently in a stemcell build that WinSXS was leaving behind unneeded data. We cleared this with:

Dism.exe /online /Cleanup-Image /StartComponentCleanup /ResetBase

from bosh-windows-stemcell-builder.

@awmartin
ok will try with that..

by the way wht max size of stemcell you have uploaded or you have successfully uploaded ?

from bosh-windows-stemcell-builder.

The nginx process in the bosh director has a limit of 10GB for uploads, and unfortunately, Windows is large. So we've successfully uploaded 9GB stemcells and ones of similar size, but the ones we produce from a raw MSDN ISO are around 6GB.

This is why I ask if you started with an MSDN ISO, or if you used a Windows image with additional software installed. If the former, we'd like to understand more about why the size is ballooning.

from bosh-windows-stemcell-builder.

Thank you for your fast replies @awmartin

We have used a windows 2012 R2 pre-deployed template which we use in our organization.
Its a data center edition after that updates have been installed and the .Net framework has been installed as given in the doc.
But we surely try a new installation from raw MSDN ISO.

Just found a way to log into the director VM.

The Windows VM we used to create the stemcell was under a domain controller. Do you think this will cause any problems?

from bosh-windows-stemcell-builder.

Thanks. Yeah, try it from the ISO as a comparison.

And yes, having a bosh-managed VM under a domain controller will cause problems. Bosh and the domain controller may conflict in trying to manage the OS, and local policies that we provide to ensure the bosh agent (and other components) work may be overridden by the domain controller. Also, if the domain controller does something to cause the OS to restart, bosh will recreate the VM in its original state, potentially causing downtime and other unexpected behaviors.

from bosh-windows-stemcell-builder.

so thanks again @awmartin
we logged into the director VM and increased the limit size of the stemcell to be uploaded in nginx.conf
restarted the services and it starts to upload the windows stemcell.

After around 4 hours it throws following error. Vsphere account has admin credentials and enough space as well. I am not getting why its getting failed. In the logs it is trying to upload the image to the esxi but in the end its getting failed.

[root@ bosh-2]# bosh -e bosh-2 upload-stemcell /stemcell/bosh-stemcell-1-vsphere-esxi-windows2012R2-go_agent.tgz
Using environment 'x.x.2.76' as client 'admin'

###################################################### 100.00% 10.68 MB/s 18m57s
Task 6

06:11:31 | Update stemcell: Extracting stemcell archive (00:04:31)
06:16:02 | Update stemcell: Verifying stemcell manifest (00:00:00)
06:16:02 | Update stemcell: Checking if this stemcell already exists (00:00:00)
06:16:02 | Update stemcell: Uploading stemcell bosh-vsphere-esxi-windows2012R2-go_agent/1 to the cloud (04:05:37)
L Error: Unknown CPI error 'Unknown' with message 'execution expired' in 'create_stemcell' CPI method

10:21:40 | Error: Unknown CPI error 'Unknown' with message 'execution expired' in 'create_stemcell' CPI method

Started Thu May 18 06:11:31 UTC 2017
Finished Thu May 18 10:21:40 UTC 2017
Duration 04:10:09

Task 6 error

Uploading stemcell file:
Expected task '6' to succeed but was state is 'error'

Exit code 1

In task debug->

I, [2017-05-18T06:16:11.659538 #2067] INFO -- : Extracting stemcell to: /var/vcap/data/tmp/director/d20170518-2067-769our
D, [2017-05-18T06:17:11.665326 #2067] DEBUG -- : Running method 'CurrentTime'...
D, [2017-05-18T06:18:11.694871 #2067] DEBUG -- : Running method 'CurrentTime'...
D, [2017-05-18T06:19:11.746684 #2067] DEBUG -- : Running method 'CurrentTime'...
D, [2017-05-18T06:20:11.759619 #2067] DEBUG -- : Running method 'CurrentTime'...
D, [2017-05-18T06:21:11.774532 #2067] DEBUG -- : Running method 'CurrentTime'...
I, [2017-05-18T06:21:32.845282 #2067] INFO -- : Generated name: sc-bfa566b0-6075-4022-93ac-1c2b5527cd27
D, [2017-05-18T06:21:32.847070 #2067] DEBUG -- : All clusters provided: {"AP-Cluster"=>#<VSphereCloud::ClusterConfig:0x0055f9e1516e00 @name="AP-Cluster", @config={}>}
D, [2017-05-18T06:21:32.847257 #2067] DEBUG -- : Running method 'FindByInventoryPath'...
D, [2017-05-18T06:21:32.878423 #2067] DEBUG -- : Running method 'RetrievePropertiesEx'...
D, [2017-05-18T06:21:32.906334 #2067] DEBUG -- : Running method 'RetrievePropertiesEx'...
D, [2017-05-18T06:21:32.949841 #2067] DEBUG -- : Running method 'RetrievePropertiesEx'...
D, [2017-05-18T06:21:33.089121 #2067] DEBUG -- : Running method 'RetrievePropertiesEx'...
D, [2017-05-18T06:21:33.160411 #2067] DEBUG -- : Running method 'QueryAvailablePerfMetric'...
D, [2017-05-18T06:21:33.313114 #2067] DEBUG -- : Running method 'QueryPerfCounter'...
D, [2017-05-18T06:21:33.395389 #2067] DEBUG -- : Running method 'QueryPerf'...
D, [2017-05-18T06:21:33.446397 #2067] DEBUG -- : Running method 'RetrievePropertiesEx'...
D, [2017-05-18T06:21:33.466168 #2067] DEBUG -- : All clusters provided: {"AP-Cluster"=>#<VSphereCloud::ClusterConfig:0x0055f9e1516e00 @name="AP-Cluster", @config={}>}
D, [2017-05-18T06:21:33.468850 #2067] DEBUG -- : Running method 'RetrievePropertiesEx'...
D, [2017-05-18T06:21:33.497639 #2067] DEBUG -- : Running method 'RetrievePropertiesEx'...
I, [2017-05-18T06:21:33.582900 #2067] INFO -- : Deploying to: <[Vim.ClusterComputeResource] domain-c52> / <[Vim.Datastore] datastore-34>
D, [2017-05-18T06:21:33.584401 #2067] DEBUG -- : Running method 'CreateImportSpec'...
D, [2017-05-18T06:21:33.648621 #2067] DEBUG -- : Running method 'FindByInventoryPath'...
I, [2017-05-18T06:21:33.653994 #2067] INFO -- : Importing VApp
D, [2017-05-18T06:21:33.654162 #2067] DEBUG -- : Running method 'ImportVApp'...
I, [2017-05-18T06:21:33.677605 #2067] INFO -- : Waiting for NFC lease to become ready
D, [2017-05-18T06:21:33.679317 #2067] DEBUG -- : Running method 'RetrievePropertiesEx'...
D, [2017-05-18T06:21:34.689993 #2067] DEBUG -- : Running method 'RetrievePropertiesEx'...
D, [2017-05-18T06:21:35.699883 #2067] DEBUG -- : Running method 'RetrievePropertiesEx'...
I, [2017-05-18T06:21:35.781872 #2067] INFO -- : Uploading
D, [2017-05-18T06:21:35.783267 #2067] DEBUG -- : Running method 'RetrievePropertiesEx'...
D, [2017-05-18T06:21:35.796786 #2067] DEBUG -- : Running method 'HttpNfcLeaseProgress'...
I, [2017-05-18T06:21:35.799004 #2067] INFO -- : Uploading disk to: https://xx.xx.34.2xx/nfc/5295db30-0c6a-7426-b1b4-356557fcad18/disk-0.vmdk
I, [2017-05-18T06:21:35.799270 #2067] INFO -- : Uploading file to https://xx.xx.34.2xx/nfc/5295db30-0c6a-7426-b1b4-356557fcad18/disk-0.vmdk...
D, [2017-05-18T06:21:36.818591 #2067] DEBUG -- : Running method 'HttpNfcLeaseProgress'...
D, [2017-05-18T06:21:37.828390 #2067] DEBUG -- : Running method 'HttpNfcLeaseProgress'...
............................
D, [2017-05-18T10:21:35.075549 #2067] DEBUG -- : Running method 'HttpNfcLeaseProgress'...
D, [2017-05-18T10:21:36.739474 #2067] DEBUG -- : Running method 'HttpNfcLeaseProgress'...
D, [2017-05-18T10:21:36.874255 #2067] DEBUG -- : Running method 'Logout'...
, exit_status: pid 2067 exit 0
E, [2017-05-18 10:21:40 #2012] [task:6] ERROR -- DirectorJobRunner: Unknown CPI error 'Unknown' with message 'execution expired' in 'create_stemcell' CPI method
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-261.4.0/lib/cloud/external_cpi.rb:144:in handle_error' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-261.4.0/lib/cloud/external_cpi.rb:93:in invoke_cpi_method'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-261.4.0/lib/cloud/external_cpi.rb:50:in create_stemcell' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-261.4.0/lib/bosh/director/jobs/update_stemcell.rb:102:in block (2 levels) in perform'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-261.4.0/lib/bosh/director/jobs/base_job.rb:56:in block in track_and_log' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-261.4.0/lib/bosh/director/event_log.rb:99:in advance_and_track'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-261.4.0/lib/bosh/director/jobs/base_job.rb:54:in track_and_log' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-261.4.0/lib/bosh/director/jobs/update_stemcell.rb:100:in block in perform'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-261.4.0/lib/bosh/director/jobs/update_stemcell.rb:82:in each' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-261.4.0/lib/bosh/director/jobs/update_stemcell.rb:82:in perform'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-261.4.0/lib/bosh/director/job_runner.rb:106:in perform_job' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-261.4.0/lib/bosh/director/job_runner.rb:32:in block in run'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh_common-261.4.0/lib/common/thread_formatter.rb:49:in with_thread_name' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-261.4.0/lib/bosh/director/job_runner.rb:32:in run'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-261.4.0/lib/bosh/director/jobs/base_job.rb:10:in perform' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-261.4.0/lib/bosh/director/jobs/db_job.rb:32:in block in perform'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-261.4.0/lib/bosh/director/jobs/db_job.rb:80:in block (3 levels) in run' /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/eventmachine-1.0.4/lib/eventmachine.rb:1046:in block in spawn_threadpool'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
D, [2017-05-18 10:21:40 #2012] [task:6] DEBUG -- DirectorJobRunner: (0.000494s) SELECT NULL
D, [2017-05-18 10:21:40 #2012] [task:6] DEBUG -- DirectorJobRunner: (0.000150s) BEGIN
D, [2017-05-18 10:21:40 #2012] [task:6] DEBUG -- DirectorJobRunner: (0.001168s) UPDATE "tasks" SET "state" = 'error', "timestamp" = '2017-05-18 10:21:40.194835+0000', "description" = 'create stemcell', "result" = 'Unknown CPI error ''Unknown'' with message ''execution expired'' in ''create_stemcell'' CPI method', "output" = '/var/vcap/store/director/tasks/6', "checkpoint_time" = '2017-05-18 06:11:31.432083+0000', "type" = 'update_stemcell', "username" = 'admin', "deployment_name" = NULL, "started_at" = '2017-05-18 06:11:31.431986+0000', "context_id" = '' WHERE ("id" = 6)
D, [2017-05-18 10:21:40 #2012] [task:6] DEBUG -- DirectorJobRunner: (0.001062s) COMMIT
I, [2017-05-18 10:21:40 #2012] [] INFO -- DirectorJobRunner: Task took 4 hours 10 minutes 8.777990391999992 seconds to process.

from bosh-windows-stemcell-builder.

@mayankkaushik2 were you able to resolve this issue? We haven't encountered this in our testing, but this link suggests that deleting and recreating the datastore could help.

from bosh-windows-stemcell-builder.

@mayankkaushik2 Are you still having this issue?

from bosh-windows-stemcell-builder.

@mayankkaushik2 Closing due to no response. Please ping back if you still want our help.

from bosh-windows-stemcell-builder.