Weekly Meeting 2017 01 26

Agenda

• Development Update
• Opens

##Minutes

#ciao-project: weekly meeting

Meeting started by kristenc at 17:00:33 UTC. The full logs are available at ciao-project/2017/ciao-project.2017-01-26-17.00.log.html .

Meeting summary

• role call (kristenc, 17:00:40)

• development update (kristenc, 17:07:58)

  • ACTION: obedmr to investigate travis alternatives that can launch vms so we can do CI on ciao-down and singlevm and also run bat tests. (kristenc, 17:22:22)

• Opens (kristenc, 17:24:20)

  • ACTION: everyone review the configuration update proposal on the mailing list. (kristenc, 17:28:21)
  • albertom is going to make a POC to move singlevm setup to ansible. (kristenc, 17:36:22)
  • we need to move all the functionality from verify.sh to BAT tests. (kristenc, 17:36:38)
  • ACTION: mcastelino to create issues to track development work for moving tests out of verify.sh and into BAT. (kristenc, 17:43:55)

Meeting ended at 17:45:46 UTC.

Action Items

• obedmr to investigate travis alternatives that can launch vms so we can do CI on ciao-down and singlevm and also run bat tests.
• everyone review the configuration update proposal on the mailing list.
• mcastelino to create issues to track development work for moving tests out of verify.sh and into BAT.

Action Items, by person

• mcastelino
  • mcastelino to create issues to track development work for moving tests out of verify.sh and into BAT.
• obedmr
  • obedmr to investigate travis alternatives that can launch vms so we can do CI on ciao-down and singlevm and also run bat tests.
• UNASSIGNED
  • everyone review the configuration update proposal on the mailing list.

People Present (lines said)

• kristenc (54)
• mcastelino (50)
• albertom (25)
• markusry (18)
• mrkz (8)
• tcpepper (8)
• obedmr (8)
• sameo (4)
• rbradford (3)
• ciaomtgbot (3)
• btwarden (1)
• jvillalo_mobl (1)
• _erick0zcr (1)

Generated by MeetBot_ 0.1.4

.. _MeetBot: http://wiki.debian.org/MeetBot

###Full IRC Log

17:00:33 <kristenc> #startmeeting weekly meeting
17:00:33 <ciaomtgbot> Meeting started Thu Jan 26 17:00:33 2017 UTC.  The chair is kristenc. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:33 <ciaomtgbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
17:00:33 <ciaomtgbot> The meeting name has been set to 'weekly_meeting'
17:00:40 <kristenc> #topic role call
17:00:44 <kristenc> o/
17:00:47 <rbradford> o
17:00:48 <mrkz> o/
17:01:55 <btwarden> o/
17:02:28 <markusry> o/
17:02:33 <obedmr> o/
17:02:45 <tcpepper> o/
17:04:50 <mcastelino> o/
17:05:33 <_erick0zcr> o/
17:05:35 <albertom> sup
17:06:20 <jvillalo_mobl> 0/
17:07:51 <kristenc> sorry. Let's go to our first agenda item.
17:07:58 <kristenc> #topic development update
17:08:20 <kristenc> This was week 1 of our development sprint.
17:08:24 <kristenc> It’s been a slow week, only 4 PRs merged this week, mostly still random bug fixes.
17:08:34 <kristenc> We merged a PR which allows the CNCI to retain network state across reboots (#1039). This is the first PR related to our Migration project.
17:08:46 <kristenc> We finished specifying the API definition for the quotas and limits support. This will be a ciao API and not OpenStack compatible
17:09:04 <kristenc> Mark closed the P1 bug we had last week related to webui, so it should now be possible to use the webui with our development environment within ciao-down.
17:09:18 <kristenc> Unfortunately, we have 2 more P1 bugs that opened up this week #1064 and #1065. Manohar is working on #1064 and reports that he’s root caused the issue.
17:09:30 <kristenc> mcastelino, do you have any additional info?
17:09:40 <kristenc> on #1064?
17:10:00 <mcastelino> -Sort of
17:10:15 <mcastelino> - 1. I can make it go away by turning of brigde filters
17:10:22 <mcastelino> but I have no idea what caused this change
17:10:27 <markusry> It's really weird
17:10:31 <markusry> Neither do I
17:10:36 <mcastelino> as markusry observed on some systems ciao-down it works
17:10:40 <mcastelino> older ciao-down VM
17:10:41 <markusry> I spent some time looking at it today and am no the wiser
17:10:47 <markusry> none the wiser
17:10:58 <markusry> I even downgraded my kernel and it did not fix the issue
17:11:04 <mcastelino> So same code works on older ciao-down VM
17:11:08 <markusry> Exactly
17:11:12 <kristenc> yesterday I tried a lot of older kernels and it made no difference.
17:11:17 <mcastelino> and based on feedback I have seen it is not a kernel issue
17:11:26 <markusry> I also reverted my controller fix to make sure this wasn't the issue, but it wasn't
17:11:42 <kristenc> yes - latest ciao master works perfectly fine on older ciao-down vms.
17:12:06 <mcastelino> qq, how many people have older working ciao-down VMs
17:12:09 <mcastelino> I do not
17:12:36 <markusry> I have one
17:12:58 <tcpepper> I have an older working singlevm, not ciao-down managed.
17:13:25 <mcastelino> tcpepper, please clone and keep that VM around
17:13:27 <kristenc> unfortunately, I stupidly blew mine away yesterday.
17:13:44 <mcastelino> work case I will pull it from you and go down each setting and configuration
17:13:48 <mcastelino> s/work/worst
17:14:05 <markusry> I don't understand what is failing.
17:14:11 <markusry> It seems that the routes are messed up
17:14:53 <mcastelino> markusry, it is not routes.. it is bridging that is not working
17:14:58 <mcastelino> the traffic gets into the bridge
17:15:06 <mcastelino> but does not get out of it
17:15:14 <kristenc> mcastelino, are netfilter rules being applied to the bridge which are eating the packets?
17:15:25 <mcastelino> I checked that the bridge has discovered the macs of the VM's and containers
17:15:32 <mcastelino> so basic ARP is working
17:15:44 <mcastelino> but ICMP or IP packets do not get through
17:15:51 <mcastelino> that is where I ended up yesterday
17:16:11 <mcastelino> Let us not derail this meeting on this. I will keep debugging and reporting on IRC
17:16:22 <mcastelino> and may need help from people with working systems
17:16:32 <mcastelino> to provide me with settings etc
17:16:47 <kristenc> ok - I'm available to help if you need it, I'm a bit stuck now anyway since I can no longer test with ciao-down.
17:17:05 <markusry> ciao-down is still working isn't it.
17:17:13 <markusry> It's only verify.sh that doesn't work
17:17:17 <markusry> You can still create instances
17:17:25 <mcastelino> ciao-down is working if you disable the bridge filters
17:17:27 <markusry> They just can't talk to each other I guess
17:17:30 <mcastelino> as I mentioned in the email
17:17:31 <kristenc> markusry, but without networking I can't verify my workload creation stuff worked.
17:17:42 <markusry> Okay.
17:17:47 <mcastelino> kristenc, for now disable the rules.. Iahev sent a note howto
17:17:53 <mcastelino> so that you are no longer blocked
17:17:54 <kristenc> mcastelino, ok, thanks - I'll do that.
17:17:54 <markusry> I'm able to create instances without disabling the rules
17:18:06 <kristenc> #1064 is a certificate problem when running singlevm in “singlemachine” mode, i.e. without ciao-down. Are we going to keep supporting singlemachine mode, or should we just stop supporting it? I’d be in favor of not officially supporting it since it complicates things.
17:18:09 <mcastelino> So keep your ciao-down VM :)
17:18:31 <mcastelino> kristenc, we should continue to support it.. specially for people with less powerful systems
17:18:35 <mcastelino> I will fix that issue
17:18:38 <mcastelino> I fixed it once
17:18:42 <mcastelino> it was DNS
17:18:49 <mcastelino> as far as I remember
17:19:04 <kristenc> ok. I also noted that we really don't have any CI to confirm that ciao-down and singlevm don't get broken.
17:19:09 <kristenc> is there a solution for this?
17:20:03 <obedmr> we'd need a CI that support vms launching
17:20:10 <rbradford> and that same solution would also be able to run BAT tests too..
17:20:19 <obedmr> I think semaphore do that
17:20:25 <kristenc> are there travis alternatives that do this?
17:20:40 <tcpepper> is sameo around?
17:20:41 <mcastelino> obedmr, do you have semaphore working?
17:20:51 <sameo> tcpepper: Yep
17:20:52 <mcastelino> it will be good to try that
17:20:56 <obedmr> I tried it some months ago
17:21:04 <obedmr> I can re-try it
17:21:24 <obedmr> as far I remember, sameo mentioned it has vms launching support
17:21:26 <tcpepper> sameo: is clear containers using anything for a public ci system that would be relevant to us ciao folks?
17:21:27 <kristenc> obedmr, it would be great to see if there was something we could use in addition to travis that integrated with github somehow.
17:21:51 <obedmr> kristenc: sure, I can investigate it and let you know
17:22:22 <kristenc> #action obedmr to investigate travis alternatives that can launch vms so we can do CI on ciao-down and singlevm and also run bat tests.
17:22:56 <obedmr> sure
17:23:00 <kristenc> thanks.
17:24:13 <kristenc> well, that was all I have for development update.
17:24:16 <kristenc> #opens
17:24:20 <kristenc> #topic Opens
17:24:25 <mrkz> o/
17:24:30 <albertom> o/
17:24:43 <kristenc> mrkz wins.
17:24:48 <mrkz> o/
17:25:26 <kristenc> whenever you are ready mrkz
17:25:40 <mrkz> I'll be quick; so Yesterday I sent a proposal for the configuration live updates to the mailing list  (ciao-devel@lists.clearlinux.org)
17:26:12 <mrkz> I'd like to get feedback about that from the component owners
17:26:47 <mrkz> in order to check that proposes make sense, In the meantime I'll continue familiarizing with ciao code across components :)
17:27:18 <kristenc> mrkz, the controller stuff regarding identity makes sense to me.
17:27:27 <mrkz> so that's all from my side :)
17:28:21 <kristenc> #action everyone review the configuration update proposal on the mailing list.
17:28:31 <mrkz> thanks kristenc
17:28:35 <kristenc> ok albertom
17:28:47 <albertom> So i have been thinking on ways to converge development and prod deployments
17:28:48 <albertom> and realized that singlevm is like our guide to follow for deployments since every
17:28:48 <albertom> commit that affects deployment, has to include changes to singlevm before being merged
17:29:07 <kristenc> true.
17:29:16 <albertom> but is is bash and is not prod ready. thats why i have been working on rewriting the ansible scripts
17:29:26 <albertom> in a way that they could deploy on a single vm and on multiplen odes
17:29:32 <albertom> using the same tasks
17:29:38 <albertom> just changing the hosts file
17:29:58 <albertom> ie. for single vm, oyu write the same hostname under [controller] [compute] and [network]
17:30:04 <sameo> tcpepper: Sorry, I was on the phone...
17:30:27 <sameo> tcpepper: We've been investigating semaphoreCI as they offer VT-x access.
17:30:29 <albertom> ciao-down could call singlevm.yml instead of singlevm/setup.sh transparently
17:30:36 <kristenc> albertom, so we could change setup.sh to just populate the hostname and call the ansible scripts.
17:30:54 <sameo> tcpepper: CoreOS uses it for that purpose.
17:31:43 <albertom> or even ship the default hosts file for singlevm, users who want to deploy on multiple nodes have to modify it anyways
17:32:04 <albertom> and remove comlexity on running the ansible scripts
17:32:22 <albertom> right now we require users to get docker, run our ciao-deploy container, mapping /dev/ and te repositorory
17:32:24 <albertom> thats ugly
17:32:35 <albertom> i need to rewrite them in a way that the only requirement is ansible itself
17:32:44 <albertom> and ansible get the other requirements automagically
17:33:00 <tcpepper> starts sounding a lot like ciao-down
17:33:10 <tcpepper> I like the idea of convergence
17:33:13 <kristenc> sounds like a good idea. i was just going to say that is what ciao-down does.
17:33:18 <albertom> well, ciao-down gets you a vm with proxies configured where you can run singlevm.yml
17:33:25 <albertom> but if you have a nuc somewhere
17:33:35 <tcpepper> but say ciao-down deployed to a machine list (like ansible does)
17:33:36 <albertom> youy ucan run singlevm.yml to have an all in one ciao cluster in baremetal
17:33:50 <tcpepper> by default it might be a machine list of 1) the VM 2) end of list ... :)
17:34:08 <mcastelino> albertom, so we can use it on something like Semaphore then to setup the env?
17:34:16 <albertom> that woul be great
17:34:18 <albertom> mcastelino:
17:34:38 <albertom> basically, i want to make singlevm.sh easier (you know i hate bash and love ansible)
17:34:38 <albertom> :P
17:35:24 <mcastelino> kristenc, one more open
17:35:36 <albertom> I hope to have a POC for tomorrow
17:35:39 <kristenc> seems like it would definitely bring us close to convergence which everyone has already agreed is a good thing.
17:35:40 <mcastelino> we should move as much of the verify.sh to BAT tests
17:35:49 <mcastelino> the external IP one may be hard
17:35:52 <kristenc> albertom, thanks.
17:35:58 <mcastelino> all the rest of the tests should just port over
17:36:01 <kristenc> mcastelino, is that your open?
17:36:04 <mcastelino> yes
17:36:16 <markusry> cloudinit can pull the init file from the network I think, so you probably could use something like ciao-down to configure a cluster of nodes installed with ubuntu-server
17:36:22 <kristenc> #info albertom is going to make a POC to move singlevm setup to ansible.
17:36:38 <kristenc> #info we need to move all the functionality from verify.sh to BAT tests.
17:37:27 <kristenc> mcastelino, I agree we should do this. What's hard about the external IP tests?
17:38:10 <mcastelino> kristenc, they need the cluster level routing to be setup so that the external IP traffic can make it to the CNCI
17:38:32 <kristenc> but you can do that from inside BAT, right?
17:38:56 <mcastelino> depends on how the cluster is setup.. on an isolated cluster yes
17:39:08 <mcastelino> if the cluster is on a network like a corp network
17:39:26 <mcastelino> they will not like to see traffic to an IP they do not recognize
17:39:29 <kristenc> ok - well, step one I think it to modify verify.sh to run BAT. Then start moving bits out of verify into BAT that are duplicates.
17:39:46 <mcastelino> kristenc, agree
17:40:05 <kristenc> mcastelino, can you make an issue or a project (if you think this is multiple issues) for this work?
17:40:20 <kristenc> then we can just get to pieces of it when we have time.
17:41:42 <mcastelino> kristenc, I will add two issues
17:41:52 <mcastelino> 1. To add BAT to verify.sh
17:42:07 <mcastelino> 2. Remove duplicate tests from verify.sh
17:42:27 <kristenc> ok - there's a 3. Migrate non-duplicated tests into BAT.
17:42:38 <mcastelino> yes
17:42:43 <kristenc> hopefully we can do that - if external IP is too hard we can leave it in verify.sh
17:43:55 <kristenc> #action mcastelino to create issues to track development work for moving tests out of verify.sh and into BAT.
17:44:10 <kristenc> were there any other opens?
17:45:38 <kristenc> ok, I'm going to call the meeting then.
17:45:43 <rbradford> sgtm