Revert "Extract systemvm.iso using bsdtar (#3536)" by anuragaw · Pull Request #3800 · apache/cloudstack

anuragaw · 2020-01-10T09:06:10Z

This reverts commit 8a55c93.

Description

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

This reverts commit 8a55c93.

anuragaw · 2020-01-10T09:06:59Z

@blueorangutan package

blueorangutan · 2020-01-10T09:07:56Z

@anuragaw a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

yadvr · 2020-01-10T09:13:38Z

Why revert it @anuragaw - instead can we fix it to make it work?

anuragaw · 2020-01-10T09:14:58Z

@rhtyd - This is for testing purposes for @borisstoyanov only we will close it after testing without merge once he is done.

blueorangutan · 2020-01-10T09:31:24Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-566

borisstoyanov · 2020-01-10T09:32:08Z

@blueorangutan test

blueorangutan · 2020-01-10T09:32:56Z

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

weizhouapache · 2020-01-10T12:53:03Z

@rhtyd @andrijapanicsb @anuragaw @borisstoyanov
fyi, I faced the following issue when restarted mgt server

2020-01-06 14:13:46,536 DEBUG [c.c.s.ConfigurationServerImpl] (main:null) (logid:) Executing: /bin/bash /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh /var/lib/cloudstack/management/.ssh/id_rsa.pub /var/lib/cloudstack/management/.ssh/id_rsa /usr/share/cloudstack-common/vms/systemvm.iso
2020-01-06 14:13:46,543 DEBUG [c.c.s.ConfigurationServerImpl] (main:null) (logid:) Executing while with timeout : 3600000
2020-01-06 14:13:46,706 DEBUG [c.c.s.ConfigurationServerImpl] (main:null) (logid:) Exit value is 1
2020-01-06 14:13:46,707 DEBUG [c.c.s.ConfigurationServerImpl] (main:null) (logid:) .: Can't restore timeagent.zip: Can't create 'agent.zip'cloud-scripts.tgz: Can't create 'cloud-scripts.tgz'bsdtar: Error exit delayed from previous errors.
2020-01-06 14:13:46,708 INFO  [c.c.s.ConfigurationServerImpl] (main:null) (logid:) Injected public and private keys into systemvm iso with result : .: Can't restore timeagent.zip: Can't create 'agent.zip'cloud-scripts.tgz: Can't create 'cloud-scripts.tgz'bsdtar: Error exit delayed from previous errors.
2020-01-06 14:13:46,709 WARN  [c.c.s.ConfigurationServerImpl] (main:null) (logid:) Failed to inject generated public key into systemvm iso .: Can't restore timeagent.zip: Can't create 'agent.zip'cloud-scripts.tgz: Can't create 'cloud-scripts.tgz'bsdtar: Error exit delayed from previous errors.

I do not knw what caused it. The issue is not happening now but I have no idea how it was fixed.

anuragaw · 2020-01-10T12:55:47Z

@weizhouapache - are you observing this issue on master on this branch with reverted commit?

weizhouapache · 2020-01-10T13:45:10Z

@weizhouapache - are you observing this issue on master on this branch with reverted commit?

@anuragaw this issue never happened before 2019 Dec. it only happened with latest master

blueorangutan · 2020-01-10T22:06:47Z

Trillian test result (tid-744)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 42796 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3800-t744-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_host_maintenance.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 78 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

borisstoyanov · 2020-01-13T09:15:46Z

I don't see this and other errors with master and with this PR, the only case that I can think of this appearing is when cloudstack-common is upgraded and management restarted. let me try to find a way to test it and will get back.

borisstoyanov · 2020-01-13T09:58:11Z

Confirmed, on upgrade the following error appears:

z: Can't create 'cloud-scripts.tgz'bsdtar: Error exit delayed from previous errors.
2020-01-13 09:53:21,769 WARN  [o.e.j.w.WebAppContext] (main:null) (logid:) Failed startup of context o.e.j.w.WebAppContext@27c6e487{/client,file:///usr/share/cloudstack-management/webapp/,UNAVAILABLE}{/usr/share/cloudstack-management/webapp}
org.springframework.context.ApplicationContextException: Failed to start bean 'cloudStackLifeCycle'; nested exception is com.cloud.utils.exception.CloudRuntimeException: Failed to inject generated public key into systemvm iso .: Can't restore timeagent.zip: Can't create 'agent.zip'cloud-scripts.tgz: Can't create 'cloud-scripts.tgz'bsdtar: Error exit delayed from previous errors.
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:186)
	at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:52)
	at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:358)
	at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:159)
	at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123)
	at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:884)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:552)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:145)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:122)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:245)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:233)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:117)
	at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:79)
	at org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37)
	at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:71)
	at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:58)
	at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:62)
	at org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:52)
	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:890)
	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:532)
	at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:853)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344)
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1515)
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1477)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:785)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:133)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:107)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:273)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:133)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:115)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:133)
	at org.eclipse.jetty.server.Server.start(Server.java:418)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:107)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
	at org.eclipse.jetty.server.Server.doStart(Server.java:385)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:189)
	at org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:105)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to inject generated public key into systemvm iso .: Can't restore timeagent.zip: Can't create 'agent.zip'cloud-scripts.tgz: Can't create 'cloud-scripts.tgz'bsdtar: Error exit delayed from previous errors.
	at com.cloud.server.ConfigurationServerImpl.updateKeyPairs(ConfigurationServerImpl.java:675)
	at com.cloud.server.ConfigurationServerImpl.persistDefaultValues(ConfigurationServerImpl.java:303)
	at com.cloud.server.ConfigurationServerImpl.configure(ConfigurationServerImpl.java:162)
	... 51 more

Perhaps we'll need to fix this in a separate PR? @anuragaw @andrijapanicsb

andrijapanicsb · 2020-01-13T11:22:57Z

@cl-k-takahashi we will have to revert the original PR #3536 , since it causes failures in the during upgrades.

Please test by having i.e. 4.13 installation, and then upgrading to the version built from your PR.

The problem seems to be in the existance (from the previous version) of "systemvm_mnt" folder, owned by root:root, so the script can't write it's files:

ls -lah /var/cloudstack/management/
drwxrwx---. 4 cloud cloud 36 Jan 10 09:12 .
drwxr-xr-x. 4 root root 33 Jan 13 09:32 ..
drwx------. 2 cloud cloud 36 Jan 13 09:34 .ssh
drwxr-xr-x. 2 root root 6 Jan 13 09:34 systemvm_mnt

See below more details on the failure please:

[root@trl-163-k-M7-bstoyanov-mgmt1 ~]# runuser -l cloud -c "/bin/bash /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh /var/cloudstack/management/.ssh/id_rsa.pub /var/cloudstack/management/.ssh/id_rsa /usr/share/cloudstack-common/vms/systemvm.iso"

TMP=/tmp
MOUNTPATH=/var/cloudstack/management/systemvm_mnt
TMPDIR=/tmp/cloud/systemvm
umask 022
[[ 995 -ne 0 ]]
SUDO='sudo -n '
mkdir -p /var/cloudstack/management/systemvm_mnt
'[' 3 -ne 3 ']'
newpubkey=/var/cloudstack/management/.ssh/id_rsa.pub
newprivkey=/var/cloudstack/management/.ssh/id_rsa
systemvmpath=/usr/share/cloudstack-common/vms/systemvm.iso
'[' '!' -f /var/cloudstack/management/.ssh/id_rsa.pub ']'
'[' '!' -f /var/cloudstack/management/.ssh/id_rsa ']'
command -v mkisofs
++ command -v bsdtar
'[' '!' -x /bin/bsdtar ']'
inject_into_iso systemvm.iso /var/cloudstack/management/.ssh/id_rsa.pub
local isofile=/usr/share/cloudstack-common/vms/systemvm.iso
local newpubkey=/var/cloudstack/management/.ssh/id_rsa.pub
local tmpiso=/tmp/systemvm.iso
mkdir -p /var/cloudstack/management/systemvm_mnt
'[' '!' -f /usr/share/cloudstack-common/vms/systemvm.iso ']'
bsdtar -C /var/cloudstack/management/systemvm_mnt -xf /usr/share/cloudstack-common/vms/systemvm.iso
.: Can't restore time
agent.zip: Can't create 'agent.zip'
cloud-scripts.tgz: Can't create 'cloud-scripts.tgz'
bsdtar: Error exit delayed from previous errors.
'[' 1 -ne 0 ']'
++ basename /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
echo 'injectkeys.sh: Failed to extract original iso /usr/share/cloudstack-common/vms/systemvm.iso'
injectkeys.sh: Failed to extract original iso /usr/share/cloudstack-common/vms/systemvm.iso
clean_up
rm -rf --preserve-root /var/cloudstack/management/systemvm_mnt
return 1
'[' 1 -ne 0 ']'
exit 5

andrijapanicsb · 2020-01-13T11:24:31Z

@cl-k-takahashi kindly fix this and submit a new PR, we can make it for 4.14 again. Thank you!

cl-k-takahashi · 2020-01-13T12:08:29Z

@anuragaw @andrijapanicsb Thank you for finding out this problem and reverting the commit.
I will fix the problem.

andrijapanicsb · 2020-01-13T12:09:21Z

The further testing show, that this error will happen only once, and on next run of the script it would be fine (since the same folder /var/cloudstack/management/systemvm_mnt is removed at the end of the script)

This reverts commit 8a55c93.

Revert "Extract systemvm.iso using bsdtar (apache#3536)"

0a3fdc5

This reverts commit 8a55c93.

anuragaw changed the title ~~[WIP DO NOT MERGE] Revert "Extract systemvm.iso using bsdtar (#3536)"~~ [FOR TESTING ONLY - DO NOT MERGE] Revert "Extract systemvm.iso using bsdtar (#3536)" Jan 10, 2020

svenvogel added type:bug component:virtual-router labels Jan 12, 2020

andrijapanicsb changed the title ~~[FOR TESTING ONLY - DO NOT MERGE] Revert "Extract systemvm.iso using bsdtar (#3536)"~~ Revert "Extract systemvm.iso using bsdtar (#3536)" Jan 13, 2020

andrijapanicsb merged commit 1771727 into apache:master Jan 13, 2020

DaanHoogland mentioned this pull request Jan 13, 2020

Extract systemvm.iso using bsdtar #3536

Merged

5 tasks

Spaceman1984 pushed a commit to shapeblue/cloudstack that referenced this pull request Jan 17, 2020

Revert "Extract systemvm.iso using bsdtar (apache#3536)" (apache#3800)

da02f16

This reverts commit 8a55c93.

ustcweizhou pushed a commit to ustcweizhou/cloudstack that referenced this pull request Feb 28, 2020

Revert "Extract systemvm.iso using bsdtar (apache#3536)" (apache#3800)

1c33b2b

This reverts commit 8a55c93.

cl-k-takahashi mentioned this pull request Apr 6, 2020

Inject keys into systemvm.iso using bsdtar (related to #3536) #4015

Closed

5 tasks

Uh oh!

Conversation

anuragaw commented Jan 10, 2020

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

anuragaw commented Jan 10, 2020

Uh oh!

blueorangutan commented Jan 10, 2020

Uh oh!

yadvr commented Jan 10, 2020

Uh oh!

anuragaw commented Jan 10, 2020

Uh oh!

blueorangutan commented Jan 10, 2020

Uh oh!

borisstoyanov commented Jan 10, 2020

Uh oh!

blueorangutan commented Jan 10, 2020

Uh oh!

weizhouapache commented Jan 10, 2020

Uh oh!

anuragaw commented Jan 10, 2020

Uh oh!

weizhouapache commented Jan 10, 2020

Uh oh!

blueorangutan commented Jan 10, 2020

Uh oh!

borisstoyanov commented Jan 13, 2020

Uh oh!

borisstoyanov commented Jan 13, 2020

Uh oh!

andrijapanicsb commented Jan 13, 2020

Uh oh!

andrijapanicsb commented Jan 13, 2020

Uh oh!

cl-k-takahashi commented Jan 13, 2020

Uh oh!

andrijapanicsb commented Jan 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

andrijapanicsb commented Jan 13, 2020 •

edited

Loading