| *** qwazix has left #maemo-meeting | 00:42 | |
| chem|st | o/ | 01:30 |
|---|---|---|
| DocScrutinizer05 | hi chem|st | 01:34 |
| DocScrutinizer05 | chem|st: you noticed my request to postpone tmo migration which is scheduled for tomorrow? | 01:34 |
| chem|st | DocScrutinizer05: that is not really an option | 01:37 |
| chem|st | and there is no reason as is... | 01:37 |
| DocScrutinizer05 | errr, why? | 01:38 |
| chem|st | the problem is not in maemo.org but in other infra so may the forums be down... | 01:38 |
| chem|st | may they be down! | 01:38 |
| DocScrutinizer05 | sorry? | 01:38 |
| chem|st | migration has nothing to do with server downtime | 01:39 |
| DocScrutinizer05 | we're not talking about server downtime. there are bandwidth/packet-loss issues with access to *.maemo.org vservers | 01:39 |
| chem|st | the server might be down but there is nothing like a head crash of new tmo server or something... | 01:40 |
| DocScrutinizer05 | to put it plain: moving tmo to the new server tomorrow will probably render it "down" | 01:40 |
| chem|st | and that is what? a networking problem | 01:40 |
| chem|st | nothing to worry | 01:40 |
| DocScrutinizer05 | aaah of course | 01:40 |
| chem|st | a fire and no copies of tmo left would be a problem | 01:41 |
| DocScrutinizer05 | tmo being down too is nothing to worry about | 01:41 |
| chem|st | if reggie has to shutdown the other server, so it be, does not prevent hime from pulling the data from it beforehands | 01:42 |
| DocScrutinizer05 | sorry I guess you lost me with your rationale | 01:42 |
| chem|st | when it comes to servers I lost my "chicken" abilities years ago | 01:42 |
| DocScrutinizer05 | [2013-01-15 21:31:13] <DocScrutinizer05> given the current situation I'd like to suggest postponing of tmo migration planned for tomorrow | 01:43 |
| DocScrutinizer05 | [2013-01-15 21:31:42] <DocScrutinizer05> chem|st: ^^^^ | 01:43 |
| DocScrutinizer05 | [2013-01-15 21:32:01] <MentalistTraceur> I would agree, but we don't know how soon it'll be resolved and reggie's limit is the 20th, right? | 01:43 |
| DocScrutinizer05 | [2013-01-15 21:32:06] <MentalistTraceur> Buys us at most four days? | 01:43 |
| DocScrutinizer05 | [2013-01-15 21:32:17] <MentalistTraceur> Better than nothing I suppose. | 01:43 |
| DocScrutinizer05 | [2013-01-15 21:32:29] <DocScrutinizer05> I'd hope 4 days are sufficient for Nemein to sort stuff | 01:43 |
| chem|st | first question 'do we have a backup?' if the answer is 'yes' my next question is 'why do you even bother' | 01:44 |
| * DocScrutinizer05 facepalms | 01:44 | |
| chem|st | I know but that is not reggies deadline to leave us that is from what I understand 'comes handy to keep me admin account active for the next months' | 01:45 |
| DocScrutinizer05 | honestly I can't see relevance of any backup here | 01:45 |
| chem|st | DocScrutinizer05: sry to be so chilled on that matter :) | 01:45 |
| MentalistTraceur | Hold up, isn't talk.maemo.org like the only source of info for most community members? So if the rest of infra is having problems, and TMO ends up being down simultaneously, everyone's just left hanging with no clue as to when things will stop working again? | 01:45 |
| MentalistTraceur | (Expecially now that mailing list doesn't deliver mail properly for the time being) | 01:46 |
| MentalistTraceur | s/stop/start/ | 01:46 |
| DocScrutinizer05 | and moving a service from a working location to a non-working one without any particular urgent need is just silly | 01:47 |
| chem|st | then post om tmo that there are networking issues and there might be some downtime in the next couple of days as tmo will migrate to the same infra as the rest of m.o | 01:47 |
| chem|st | DocScrutinizer05: thought reggie canceld his server contract?! | 01:48 |
| DocScrutinizer05 | yeah, great, since we *need* further load on the already overloeded firewall that gives us all that pain, due to Reggie not bothering to shift the move one day or two towards the 20th, despite he maybe even had better time on thursday than tomorrow | 01:49 |
| chem|st | ? | 01:49 |
| DocScrutinizer05 | chem|st: please! you know pretty well that Reggie said the server will be up until Sunday | 01:49 |
| DocScrutinizer05 | actually until 20.1. | 01:49 |
| chem|st | I know... | 01:50 |
| DocScrutinizer05 | I *strongly* suggest to consider postponing tmo migration one or two days | 01:50 |
| DocScrutinizer05 | or at least put old server into r/o mode and migrate, but NOT change the DNS to point to new server | 01:52 |
| DocScrutinizer05 | though that wouldn't allow me to update http://talk.maemo.org/showthread.php?t=88659 | 01:53 |
| chem|st | not true... | 01:54 |
| DocScrutinizer05 | not helping... | 01:54 |
| chem|st | hehe | 01:54 |
| DocScrutinizer05 | could you please elaborate on your whole take on that? incl this "not true..." !? | 01:55 |
| MentalistTraceur | Btw, I'm not sure how posting on TMO that tomorrow TMO will quite possibly be not working along with the rest of the infra helps the problem of many community users being left out of the loop. | 01:55 |
| chem|st | we can set all tmo as r/o and deactivate thread creation and still have admins+mods be able to alter tmo | 01:56 |
| DocScrutinizer05 | sure | 01:56 |
| DocScrutinizer05 | I'd just prefer you helping a bit instead of antagonizing | 01:57 |
| chem|st | or you can proxy requests to new tmo if it is working and stay r/o (old server) if not | 01:57 |
| chem|st | but on the other hand what is the big deal with having some downtime? | 01:58 |
| DocScrutinizer05 | *I* can nothing of all that! | 01:58 |
| chem|st | and you are not the one doing it are you? | 01:58 |
| chem|st | for the page proxiing thing I would need to ask some friend for help | 01:59 |
| DocScrutinizer05 | honestly you're obviously trying to troll me | 01:59 |
| chem|st | ? | 01:59 |
| MentalistTraceur | DocScrutinizer05, I don't think he's trolling you, it's just a normal disagreement. | 01:59 |
| DocScrutinizer05 | [2013-01-16 00:47:14] <DocScrutinizer05> and moving a service from a working location to a non-working one without any particular urgent need is just silly <DocScrutinizer05> yeah, great, since we *need* further load on the already overloeded firewall that gives us all that pain -- i haven't seen any "disagreement" regarding my arguments before | 02:01 |
| chem|st | I am not trying to troll you I am trying to get you to understand that it is not a big deal, all m.o is down... if move of tmo proceeds you may or may not keep itt alive in ro... googling hits itt anyways so you may change the dns as well... | 02:01 |
| DocScrutinizer05 | that's incorrect | 02:01 |
| chem|st | oh did that change now? | 02:02 |
| chem|st | well wiki is ro for weeks now | 02:02 |
| DocScrutinizer05 | the moment you change the DNS it will add load to the broken firewall and render tmo inoperable | 02:02 |
| DocScrutinizer05 | I urgently request to NOT do the first, and suggest to not do the second | 02:03 |
| chem|st | called d.o.w.n.t.i.m.e and happens... first is proceed with move and second dns? | 02:04 |
| chem|st | reggies call | 02:04 |
| DocScrutinizer05 | sigh, first is >>it will add load to the broken firewall<< | 02:04 |
| DocScrutinizer05 | second is >>render tmo inoperable<< | 02:05 |
| chem|st | sounded mor like an isp issue from reading the logs | 02:05 |
| DocScrutinizer05 | and it's not exactly reggie's call, or at least it's even less my call, since *you* are our current tmo admin and it's up to *you* to ask reggie if he might pretty please postpone tmo migration one or two days | 02:06 |
| chem|st | like qos was set to what isp said is bandwidth but band collapses if used | 02:06 |
| chem|st | so this is becoming an official now? well, I'll ask him... | 02:07 |
| DocScrutinizer05 | my call however is to say "You MUST NOT add additional load to the already overloeded firewall" | 02:07 |
| chem|st | and stop calling me names in public! :) | 02:07 |
| DocScrutinizer05 | so your options are as elaborated above | 02:08 |
| chem|st | is dns record now a proper a record or is it some kind of fwd from elsewhere? | 02:08 |
| DocScrutinizer05 | either postpone, or put into r7o mode and either way don't touch DNS | 02:08 |
| DocScrutinizer05 | CNAME | 02:08 |
| chem|st | yeah just digged it... | 02:09 |
| DocScrutinizer05 | btw regarding backup, i'd appreciate if reggie would talk to brkn and send a backup to our backup location | 02:18 |
| DocScrutinizer05 | I don't mind if he wants or doesn't want to encrypt that backup, and hand the key tu uncrypt to you | 02:19 |
| DocScrutinizer05 | to* | 02:19 |
| DocScrutinizer05 | I just want a backup in a completely unrelated separate location | 02:20 |
| DocScrutinizer05 | in case something at Nemein blows up | 02:20 |
| DocScrutinizer05 | (if brkn isn't available, ask me) | 02:21 |
| *** kolp has quit IRC | 02:24 | |
| Woody14619 | FWIW: Personally, I think downtime will do us good in the long run, if it happens. | 02:28 |
| Woody14619 | The old addage "Don't know what you got till it's gone" doesn't work unless stuff actually goes away, even for a short while. | 02:29 |
| Woody14619 | If TMO and all that goes out for a day or two, even with fore-warning, like has already been said on TMO, it will make folks sit up and maybe realize how close to the brink this whole thing got. | 02:30 |
| Woody14619 | May even inspire a few of the whiners to STFU, and/or others to determin it's worth throwing $5/month at for the foreseable future... Not like we couldn't use the funding... | 02:31 |
| chem|st | DocScrutinizer05: that was the idea of backup | 02:31 |
| chem|st | I'm going to backup twice just to be sure | 02:31 |
| chem|st | in case of death my sister has access to the keys... | 02:31 |
| chem|st | that sort of backup? | 02:32 |
| DocScrutinizer05 | yes | 02:34 |
| DocScrutinizer05 | there's a semi-official central location for that sort of backups | 02:34 |
| Woody14619 | And while I sympathise with DocScrutinizer05 on this... I know it's not easy being seen as "in charge" and having "bad stuff" happen that you have no control over... I don't think this is quite a panic moment either. | 02:35 |
| DocScrutinizer05 | that i pay for from my own pocket | 02:35 |
| chem|st | Woody14619: yeah that is why I lost my "chicken" abilities a while ago | 02:36 |
| DocScrutinizer05 | Woody14619: my priorities are clearly set and stated above: 1.: DO NOT add load to the firewall; 2.: You SHOULD NOT do the migration tomorrow | 02:37 |
| chem|st | getting emails and pinged for hours that the server is down while you are trying to get it back up... | 02:37 |
| DocScrutinizer05 | ~2119 | 02:37 |
| Woody14619 | I do agree it would be nice to have a couple backups, in a couple locations... just to be safe. | 02:37 |
| chem|st | DocScrutinizer05: you are cc'ed | 02:37 |
| Woody14619 | I get that Doc, but I don't think the firewall is overloaded... Just misconfigured. I really don't think load will affect it much. | 02:37 |
| DocScrutinizer05 | thanks | 02:37 |
| Woody14619 | I'd be more concerened about the integrety of the transfer than screwing things up for reading users... | 02:38 |
| DocScrutinizer05 | Woody14619: it evidently does, since stuff like wiki worked like a charm 2 days ago | 02:38 |
| DocScrutinizer05 | Woody14619: ...and shit hit the fan when DNs switched | 02:38 |
| DocScrutinizer05 | DNS | 02:38 |
| Woody14619 | Or, they did something to the firewall at the same time and just aren't mentioning that... | 02:39 |
| Woody14619 | Or it was linking some static content from another area, and the firewall got confused over that transfering... | 02:39 |
| Woody14619 | Or one of 50 million other possabilities. | 02:39 |
| Woody14619 | Saying it's load is just as big an assuption as any of the above. | 02:40 |
| DocScrutinizer05 | Eero said FW is statistically dropping connections | 02:40 |
| DocScrutinizer05 | he also said "I guess you slashdotted us - CPU load is manageable but there's a bottleneck in network" (OWTTE) | 02:41 |
| Woody14619 | Could even be glitchy hardware, or some off packet rule... | 02:41 |
| DocScrutinizer05 | so whatever it been, it's most likely related to bandwidth | 02:41 |
| DocScrutinizer05 | Woody14619: nope, since stuff inproven when [2013-01-15 17:04:08] <mashiara> I enabled traffick-shaping on the maemo-firewall but I have no real data yet to be able set correct parameters so current config is based on guesswork | 02:42 |
| Woody14619 | Well, that's all fine... It could be. But again.. even if we transfer and find it's too constraind, that's fine. They'll fix the pipe eventually, and things will go back to normal. | 02:43 |
| DocScrutinizer05 | yes, everything will be fine, if we only wait for long enough | 02:43 |
| Woody14619 | If this happens too smoothly, the complacency in the community will continue unabated. I almost like the idea that things will get rocky for a bit. | 02:44 |
| DocScrutinizer05 | but until then, I try to manage stuff in a way that minimizes discomfort to users | 02:44 |
| DocScrutinizer05 | things already ARE quite rocky, since 36h | 02:45 |
| Woody14619 | It may actually shake a few people into droping some change on the project when they finally realize that it's not a trust-fund baby and could actually go away if they don't. | 02:45 |
| Woody14619 | No... For YOU they've been rocky. For a large chunk of the user base, things have been just fine. | 02:45 |
| DocScrutinizer05 | tmo is the only service that's not down | 02:45 |
| Woody14619 | Lots of people don't actually edit the wiki... they just read it. And read TMO... | 02:46 |
| Woody14619 | For active members, using builder, and garage, and wiki, yes... They noticed. | 02:46 |
| DocScrutinizer05 | and on tmo we will see downtime anyway, when Reggie does the actual move | 02:46 |
| Woody14619 | But the casual TMO browser and wiki reader has seen nothing but yammering about what's going on... | 02:46 |
| DocScrutinizer05 | I'm just demanding that we try to shift that tmo downtime out of the time of general maemo.org downtime we see right now | 02:47 |
| Woody14619 | Having some time where it's gone (not read only, not slow... GONE) I think will wake a few people up. | 02:47 |
| Woody14619 | Which has merit... | 02:47 |
| Woody14619 | I get why you want that... | 02:47 |
| Woody14619 | But really, I'm just confirming, if it does happen now, it won't be the end of the world. | 02:48 |
| Woody14619 | Will it be inconvenient? Maybe. But it could also be inconvenient to delay to those that have volunteered to help. | 02:49 |
| DocScrutinizer05 | if that general m.o downtime extends beyond tomorrow evening, I don't care anymore, and we'll face some more trouble no matter when tmo gets moved. But just *tomorrow* is a very BAD[TM] day to do that particular migration | 02:49 |
| Woody14619 | Well... I think the best day to do it is when we have people who know what they're doing willing to help with it. Right now, that's setup for tomorrow. I agree, it can't hurt to ask politely if they would mind delaying a day or two... | 02:51 |
| DocScrutinizer05 | (inconvenient to those who help) that's Reggie and that's the reason I *asked* if he pretty please might consider to postpone. If he can't then I have no problem with putting old tmo into r7o state and do the whole migration minus DNS change tomorrow | 02:51 |
| Woody14619 | But if it need to happen now or without as much help later, I'd vote do it now. | 02:51 |
| Woody14619 | So we're in violent agreement... :) | 02:52 |
| DocScrutinizer05 | yep :-) | 02:52 |
| Woody14619 | I just don't think it's worth getting riled up over... | 02:52 |
| Woody14619 | "Best" case, TMO goes read only, the move happens, DNS resolves magically in the next 6 hours, and after a day of r/o, it's all back to happ.y | 02:53 |
| DocScrutinizer05 | there's one more info you might miss: iirc Reggie said he has time to do the migration from Wednesday *on* | 02:53 |
| Woody14619 | Worst case, TMO goes black, doesn't resolve for a couple days, a few people wake the fk up and realize what they'll miss if they don't start helping. | 02:54 |
| DocScrutinizer05 | and DNS change done by Nokia yesterday been a change from A <tmo-IP> to CNAME itt | 02:54 |
| Woody14619 | I don't see a losing situation either way here. | 02:54 |
| Woody14619 | Alternate best case, same as the first, but delayed a couple days. | 02:56 |
| Woody14619 | Alternate worst: We're stuck trying to migrate without as much time from those who volunteered to help.... | 02:56 |
| Woody14619 | I personally think alternate worst case is the worst of worst cases... ;) | 02:57 |
| DocScrutinizer05 | sure | 02:57 |
| Woody14619 | An improper/hurried move is worse than a move that may cause a limited finite down time. | 02:57 |
| Woody14619 | But yeah... :) In any case... We'll survive. | 02:58 |
| DocScrutinizer05 | and if shit happens, Nokia *again* touches tmo DNS tomorrow 1000UTC, just like requested by Eero, and points it to the new vserver | 02:58 |
| * Woody14619 looks for the right cord to trip over to make TMO go out for a day or two... Almost seems like the best option over all. | 02:59 | |
| DocScrutinizer05 | i honestly don't have the faintest idea what made them change it from A record to CNAME record yesterday | 03:00 |
| DocScrutinizer05 | might be the rationale of "itt is Reggie's domain, so let him manage it whatever way he wants" | 03:01 |
| DocScrutinizer05 | Woody14619: you may ponder about the benefits of a 48h tmo downtime, I'm councilor right now and that prohibits any such considerations | 03:02 |
| Woody14619 | Agreed... Like I said above, I've been in your shoes, very recently. | 03:03 |
| DocScrutinizer05 | :nod: | 03:03 |
| Woody14619 | And sympahthize. Just don't stress enough to have a cardiac over this. ;) | 03:04 |
| DocScrutinizer05 | nah, I'm loving to get upsat ;-P | 03:04 |
| DocScrutinizer05 | set even | 03:04 |
| DocScrutinizer05 | GeneralAntilles: it just occured to me that we have another site that's quite related to maemo and not on peril to go down during migration - so a site ideal to share latest info: http://hildonfoundation.org/news/ | 03:07 |
| GeneralAntilles | Aye, I don't have access, though. | 03:08 |
| DocScrutinizer05 | still no access? duh! | 03:08 |
| GeneralAntilles | Gotta be a vote first. | 03:08 |
| DocScrutinizer05 | I thought that been scheduled for yesterday | 03:08 |
| GeneralAntilles | Well, nobody told me anything. :D | 03:09 |
| DocScrutinizer05 | lol | 03:09 |
| DocScrutinizer05 | i might be mistaken as well | 03:09 |
| DocScrutinizer05 | actually former board meetings seem to not follow any fixed schedule | 03:11 |
| *** freemangordon has quit IRC | 03:20 | |
| DocScrutinizer05 | chem|st: many thanks for smooth management :-) | 03:31 |
| chem|st | DocScrutinizer05: as of now I think m.o is not on the same route as tmo | 03:35 |
| chem|st | new servers | 03:35 |
| chem|st | tmo renders just flawless | 03:35 |
| DocScrutinizer05 | o.O | 03:37 |
| DocScrutinizer05 | the NEW tmo? | 03:37 |
| chem|st | yes | 03:38 |
| DocScrutinizer05 | mompls | 03:38 |
| chem|st | remember it is a vm | 03:38 |
| chem|st | somewhere... | 03:38 |
| chem|st | it fits in the ip scheme but is not on the same firewall | 03:39 |
| DocScrutinizer05 | they have identical route til last resovable node 83.145.255.35 (83.145.255.35) which i guess is firewall | 03:41 |
| DocScrutinizer05 | probably firewall doesn't drop packets/sessions for 188.117.59.209 yet, since there's not much traffic | 03:42 |
| DocScrutinizer05 | to that IP | 03:42 |
| chem|st | that is not a nemein address | 03:43 |
| chem|st | still ISP | 03:43 |
| DocScrutinizer05 | sure, the FW is not Nemein's FW aiui | 03:45 |
| DocScrutinizer05 | Eero talked to Nemein's ISP about FW issue | 03:45 |
| chem|st | hmm ok tomorrow more | 03:48 |
| DocScrutinizer05 | night chem|st | 03:49 |
| *** reggie_ has joined #maemo-meeting | 04:20 | |
| *** M4rtinK has quit IRC | 04:37 | |
| *** reggie_ has left #maemo-meeting | 05:22 | |
| *** DocScrutinizer05 has quit IRC | 06:02 | |
| *** DocScrutinizer05 has joined #maemo-meeting | 06:02 | |
| *** ShadowJK has joined #maemo-meeting | 06:35 | |
| MentalistTraceur | Alright, I have to sleep, so I'll be dropping out of here now. | 07:00 |
| MentalistTraceur | Hopefully no fecal matter hits any rotating blade-like surfaces more than it already has for the remainder of the migration. | 07:01 |
| *** MentalistTraceur has quit IRC | 07:01 | |
| *** thedead1440 has quit IRC | 07:03 | |
| *** thedead1440_ has joined #maemo-meeting | 07:05 | |
| DocScrutinizer05 | let's see :-) | 07:06 |
| * DocScrutinizer05 google a bit, for "traffic shaping" | 07:10 | |
| *** thedead1440_ has quit IRC | 07:11 | |
| *** ZogG_laptop has joined #maemo-meeting | 07:55 | |
| DocScrutinizer05 | latest update: wiki, maemo.org, garage, all seems quite responsible now | 07:57 |
| DocScrutinizer05 | err responsive | 07:58 |
| DocScrutinizer05 | or I only cought the bright 30 seconds | 07:58 |
| DocScrutinizer05 | well, login on maemo.org already loading 49% after 2 minutes | 08:16 |
| *** mashiara has joined #maemo-meeting | 09:05 | |
| mashiara | I'm back at work, I have not yet received replies to some emails I sent yesterday (not surprising it's only 9am) requesting more info from various sources. | 09:06 |
| *** Pali has joined #maemo-meeting | 09:07 | |
| DocScrutinizer05 | mashiara: good morning :-D | 09:27 |
| DocScrutinizer05 | mashiara: any info about static.maemo.org? | 09:28 |
| DocScrutinizer05 | it seems the vserver isn't working | 09:29 |
| DocScrutinizer05 | (or the DNS isn't switched yet, sorry last time I checked been with etc/hosts patch two days ago, and back then it didn't work) | 09:30 |
| DocScrutinizer05 | mashiara: btw some 2..3h ago stuff semi-worked for a while | 09:32 |
| *** Pali has quit IRC | 09:33 | |
| *** freemangordon has joined #maemo-meeting | 10:05 | |
| *** ivgalvez has joined #maemo-meeting | 10:07 | |
| mashiara | info back from the ISP, our racks uplink (100mbit/s) is peaked, more bandwidth is being procured | 10:11 |
| mashiara | but I wonder if there is something else than active community at play since I QoS:ed (from the level i could affect way below the uplink) the maemo firewall to something way lower and it did not help with other services on the rack at all... | 10:14 |
| mashiara | I checked the DNS about 30min ago it's all done, execpt talk.maemo.org (I also got an email yesterday saying basically the same) | 10:17 |
| mashiara | but talk is supposed to be switched in two hours. | 10:18 |
| DocScrutinizer51 | ok, when it actually gets switched then we will have to deal with that | 10:19 |
| ivgalvez | hi all | 10:20 |
| DocScrutinizer51 | right now it's a CNAME and thus Reggie could change his ITT DNS any time | 10:20 |
| kerio | are we really using up 100Mbits for *.maemo.org? wtf | 10:46 |
| mashiara | my thoughts exactly... | 10:46 |
| mashiara | it might be a coincidence | 10:46 |
| kerio | who's downloading all the firmwares again? | 10:48 |
| mashiara | if it was a single source the QoS would take care of it and other stuff on the rack would not suffer | 10:48 |
| DocScrutinizer51 | the only suspect I could think of was auto-update-check of several 100k N900 on extras and extras-devel repo | 10:56 |
| DocScrutinizer51 | and the Nokia repos of course | 10:57 |
| DocScrutinizer51 | since according to mashiara even a dozen fools running apt-mirror would get throttled by traffic shaper | 10:58 |
| DocScrutinizer51 | mashiara: could you shut down repository.maemo.org for an hour, to check if load is related to repos? | 11:01 |
| kerio | do several hundred thousands n900s even exist? | 11:01 |
| DocScrutinizer51 | sure | 11:01 |
| Stskeeps | this is a good time to collect how many n900s are actually still in field | 11:01 |
| Stskeeps | :P | 11:01 |
| DocScrutinizer51 | in theory definitely | 11:01 |
| kerio | and they're HAMming on maemo.org at the same time? | 11:01 |
| mashiara | ah fsck... I had completely forgotten about the device auto-updates... | 11:02 |
| DocScrutinizer51 | well, I think default auto-update period is 24h | 11:02 |
| kerio | :s | 11:02 |
| mashiara | X-Fade: was repository.maemo.org pointed to Akamai previously ? | 11:02 |
| DocScrutinizer51 | yes | 11:03 |
| mashiara | if that is the source of our problems HiFo is going to have a problem... | 11:04 |
| mashiara | anyway, repository.maemo.org is going down for a while so we can test this hypothesis | 11:04 |
| DocScrutinizer05 | repository.maemo.org.edgesuite.net. 21600 IN CNAME a515.g.akamai.net. | 11:04 |
| DocScrutinizer05 | mashiara: I think we *might* handle that, by renaming repo URL and requiring users to tweak their HAM settings | 11:06 |
| DocScrutinizer05 | would instantly disable all auto-updates | 11:07 |
| kerio | this shit right here is why you should enable pdiffs | 11:07 |
| mashiara | though even if the server is down tons of devices trying to just open the connection will still hammer any single location. | 11:07 |
| DocScrutinizer05 | yep | 11:08 |
| Stskeeps | or perhaps that akamai is pulling all the repository contents as it thinks it's all new? | 11:08 |
| Stskeeps | :P | 11:08 |
| DocScrutinizer05 | once the DNS record vanished, this will decay though | 11:08 |
| DocScrutinizer05 | HAHA akamai syncing, golden! | 11:09 |
| mashiara | so shall I request repository.maemo.org to be removed from DNS and slowrepo.maemo.org to be added in stead ? | 11:09 |
| DocScrutinizer05 | not yet | 11:09 |
| DocScrutinizer05 | let's first see what temporary shutdown of repository.maemo.org will result in | 11:10 |
| DocScrutinizer05 | the server, not the DNS | 11:10 |
| mashiara | yes, it's down now. | 11:11 |
| DocScrutinizer05 | hmm, can't see much difference yet | 11:12 |
| DocScrutinizer05 | sorry I'm afk for a few hours, other tasks pending :-/ | 11:13 |
| DocScrutinizer05 | maybe wireshark or similar analysis tools on firewall could help to get an idea about the URLs that are requested most frequently? | 11:14 |
| mashiara | 1:17 <X-Fade> I see we requested dns for stage.maemo.org, we should not have done that. | 11:27 |
| mashiara | 11:18 <X-Fade> As now akamai is also trying to refresh everything from our servers. | 11:27 |
| mashiara | 11:18 <X-Fade> Looking at akamai stats, there are about 30k devices still alive. | 11:27 |
| mashiara | 11:19 <X-Fade> maybe 40k if you include older firmwares. | 11:27 |
| mashiara | "oops" | 11:27 |
| * Stskeeps wins | 11:27 | |
| kerio | /dcc send Stskeeps internetpacks/5internets | 11:29 |
| mashiara | oh well, mistakes get made when we get the go-ahead for migration a couple of days before chrismass... | 11:29 |
| mashiara | lunch now | 11:33 |
| DocScrutinizer51 | don't worry, it's fine we got a clue now | 11:33 |
| kerio | can we just tell akamai to stop updating everything? | 11:34 |
| kerio | how does akamai work? who's responsible for it? | 11:34 |
| mashiara | X-Fade seems to have some access to the stats at least but I have no idea how it's managed. | 11:34 |
| DocScrutinizer51 | shut down stage.maemo.org | 11:35 |
| DocScrutinizer51 | or even better, 404 | 11:35 |
| DocScrutinizer51 | or ask nokia to emergency change DNS to 127.0.0.1 | 11:37 |
| kerio | but then akamai will start mirroring akamai | 11:38 |
| DocScrutinizer51 | 40k devices should"mt tear down repo.m.o | 11:40 |
| DocScrutinizer51 | one query each other second | 11:40 |
| kerio | especially because they shouldn't actually download anything | 11:41 |
| kerio | until the first -devel update | 11:41 |
| DocScrutinizer51 | yep, only the package list | 11:41 |
| kerio | not even that, they're cached | 11:41 |
| kerio | it would just check the headers | 11:41 |
| DocScrutinizer51 | even better | 11:42 |
| mashiara | shutting down stage | 11:42 |
| kerio | what's stage, btw? | 11:42 |
| mashiara | <•mashiara> ok, where should stage.maemo.org point to ? | 11:42 |
| mashiara | 11:39 <X-Fade> oblivion.. | 11:42 |
| mashiara | 11:40 <X-Fade> That was only the backend for akamai.. | 11:42 |
| mashiara | 11:40 <X-Fade> So you had repository.maemo.org -> akamai -> stage. | 11:42 |
| mashiara | so I will request that to be removed from DNS shortly | 11:43 |
| kerio | i see | 11:43 |
| ivgalvez | most people might be using devel which downloads a huge amount of data on each update | 11:47 |
| kerio | ivgalvez: not really, -devel has been the same for a while now | 11:49 |
| *** M4rtinK has joined #maemo-meeting | 11:50 | |
| DocScrutinizer51 | ivgalvez: we'll deal with that issue when we face a problem | 11:54 |
| kerio | ivgalvez: we'll deal with that issue by using pdiffs | 12:00 |
| *** kolp has joined #maemo-meeting | 12:04 | |
| DocScrutinizer05 | kerio: pdiffs are useless for the "normal2 update check | 12:07 |
| kerio | why? | 12:09 |
| DocScrutinizer05 | kerio: please discuss that over at #maemo-ssu | 12:10 |
| *** qwazix has joined #maemo-meeting | 12:53 | |
| mashiara | after shutting down stage things seem to have normalized quite a bit. I will request some more dns changes (since the VM that hosts stage also contains the repository data it's simplest to point repository.maemo.org to that machine) | 13:22 |
| freemangordon | repository.maemo.org seems to work, main page is there, but repos cannot be browsed | 13:31 |
| *** brkn has joined #maemo-meeting | 13:41 | |
| mashiara | I thought i shut down apache on that machine, maybe ferenc/niels restarted it to test something | 13:41 |
| mashiara | "that machine" being stage the VM that repository.maemo.org pointed to was actually doing nothing due to this whole Akamai mess | 13:45 |
| kerio | mashiara: echo keep this shit down > /etc/issue | 14:02 |
| kerio | (don't do that) | 14:02 |
| mashiara | anyway things seem to have normalized a bit, I will try to get some of the work I was supposed to get done today while we wait for the new DNS changes (remove stage, readdress repository, add downloads) to get done by Nokia | 14:09 |
| *** thedead1440 has joined #maemo-meeting | 14:49 | |
| *** lbt has quit IRC | 14:57 | |
| *** lbt has joined #maemo-meeting | 15:04 | |
| *** lbt has quit IRC | 15:04 | |
| *** lbt has joined #maemo-meeting | 15:04 | |
| *** Pali has joined #maemo-meeting | 15:13 | |
| *** ivgalvez has quit IRC | 17:24 | |
| *** Pali has quit IRC | 17:26 | |
| mashiara | seems things have detoriated again, I will shut down stage until those DNS changes have taken effect (as soon as I can get the VPN up...) | 18:26 |
| mashiara | aand I can't, timeouts... I'll try again when I'm at home, now I have a train to catch. | 18:36 |
| *** brkn has left #maemo-meeting | 18:59 | |
| *** freemangordon has quit IRC | 19:12 | |
| kerio | huh, maemo.org is quite fast now | 19:15 |
| DocScrutinizer05 | ok, now if static.maemo.org was helping to render stuff like http://wiki.maemo.org/Community_SSU in a digestible way.... or is that just me? | 20:11 |
| *** Pali has joined #maemo-meeting | 20:13 | |
| kerio | DocScrutinizer05: borked for me too | 20:18 |
| kerio | but i guess it's just a misconfiguration of static.maemo.org | 20:18 |
| DocScrutinizer05 | yep, something wrong with static.m.o | 20:19 |
| DocScrutinizer05 | repo still shut down? | 20:20 |
| kerio | DocScrutinizer05: http://static.maemo.org/style_maemo2009/css/master.css is a 404 | 20:21 |
| DocScrutinizer05 | :nod: | 20:21 |
| kerio | http://static.maemo.org/style_maemo2009/css/reset.css too | 20:21 |
| kerio | i think repository. was shut down to test the akamai theory | 20:21 |
| DocScrutinizer05 | thanks for confirming | 20:21 |
| DocScrutinizer05 | yes, I'm asking if it could get started again, or even been | 20:22 |
| DocScrutinizer05 | oooh, repo is unrelated to akamai, stage been the akamai bait | 20:23 |
| DocScrutinizer05 | aiui | 20:23 |
| kerio | mashiara: can you restart repository. ? | 20:26 |
| ShadowJK | This morning when i loaded maemo.org, it had content referenced from static.maemo.org:81 | 20:27 |
| DocScrutinizer05 | you checked the page for the real source of those elements? | 20:28 |
| ShadowJK | no I just randomly noticed because my network didn't allow it | 20:29 |
| DocScrutinizer05 | ShadowJK: btw that hasn't changed | 20:29 |
| DocScrutinizer05 | maemo.org renders in 0.5s now, as does wiki | 20:29 |
| DocScrutinizer05 | and garage | 20:29 |
| kerio | yeah, :81 sucks | 20:29 |
| DocScrutinizer05 | dang, now I re-read and understand your post. bear with me, coffee... | 20:32 |
| * DocScrutinizer05 just thought "old, yeah - but `81?? wtf!" | 20:34 | |
| DocScrutinizer05 | >>maemo.org service break on Sunday (2010-10-03)<< | 20:35 |
| DocScrutinizer05 | ;-D | 20:35 |
| DocScrutinizer05 | >>Q1 2010 Community Council election - voting open<< | 20:36 |
| mashiara | kerio: we need those DNS changes to go through first. | 21:05 |
| *** Woody14619 has quit IRC | 21:07 | |
| mashiara | also send an email to me about the wrong url for static files on wiki | 21:08 |
| mashiara | though I will make a workaround now as well | 21:09 |
| *** Woody14619 has joined #maemo-meeting | 21:09 | |
| *** Woody14619 has quit IRC | 21:09 | |
| *** Woody14619 has joined #maemo-meeting | 21:09 | |
| DocScrutinizer05 | mashiara: great! We're quite back to "normal" :-D | 21:21 |
| DocScrutinizer05 | repo still down | 21:21 |
| DocScrutinizer05 | and static.m.o not working | 21:22 |
| mashiara | it is now | 21:22 |
| DocScrutinizer05 | :-D | 21:22 |
| * DocScrutinizer05 checks | 21:22 | |
| DocScrutinizer05 | \o/ | 21:22 |
| DocScrutinizer05 | ok, another issue killed | 21:23 |
| mashiara | repo will stay down untill dnsauthority@nokia.com implements my requests from earlier today | 21:23 |
| DocScrutinizer05 | mashiara: many thanks! | 21:23 |
| DocScrutinizer05 | tbh my feeling is: wiki.maemo.org never been that fast :-D | 21:25 |
| DocScrutinizer05 | http://maemo.org/ "Announcements" still looks pretty out of date | 21:27 |
| DocScrutinizer05 | though tbh I couldn't say how it looked 2 weeks ago - I have a hard time it was about 2010 events thoug | 21:28 |
| DocScrutinizer05 | hard time to assume...* | 21:29 |
| DocScrutinizer05 | which brings us to our next question: who are the current "maintainers"/admins for the web frontend in maemo.org, for wiki etc | 21:31 |
| mashiara | btw, my overtime is overflowing already so I will be back tomorrow | 21:31 |
| *** mashiara has left #maemo-meeting | 21:32 | |
| Pali | WHY 81? | 21:32 |
| DocScrutinizer05 | mashiara: were those accounts migrated - whatever they been? | 21:32 |
| DocScrutinizer05 | well, he deserves his off time :-D | 21:32 |
| DocScrutinizer05 | I'm quite happy with our today's results in migration disaster recovery | 21:33 |
| DocScrutinizer05 | thanks to everybody involved, for today. Tomorrow we'll clean up the litter, so please stay tuned | 21:34 |
| DocScrutinizer05 | special thanks to bergie for showing up and coordinating stuff when we needed it most | 21:36 |
| DocScrutinizer05 | Damn, I hoped for a new task in my daywork taking me out of maemo.org business and giving me some "holiday" and chillout. | 21:50 |
| DocScrutinizer05 | Now the opposite happened, I'm vacant and 24/7 exposed to this battleground ;-) | 21:50 |
| kerio | :D | 21:56 |
| kerio | do we have an ip for repository.maemo.org? | 21:56 |
| DocScrutinizer05 | host -a ? | 22:11 |
| kerio | oh, it's still in the records | 22:12 |
| DocScrutinizer05 | yep, it's in DNS but "switched off" | 22:12 |
| DocScrutinizer05 | aiui mashiara wants that DNs to point to the stage.maemo.org "master" | 22:12 |
| DocScrutinizer05 | DNSSSSSS | 22:13 |
| DocScrutinizer05 | sorry | 22:13 |
| DocScrutinizer05 | grrr | 22:13 |
| kerio | i thought we were going to drop akamai | 22:14 |
| kerio | why do we need stage. again? | 22:14 |
| Pali | on stage.maemo.org was central svn repo | 22:16 |
| Pali | what is files.maemo.org? | 22:16 |
| kerio | merlin1991: the changelog on the wiki is outdated! | 22:18 |
| Stskeeps | files.* was something used for file exchange in maemo organisation, probably not useful anymore | 22:18 |
| DocScrutinizer05 | kerio: this is no general discussion channel, please keep off-topic stuff on #maemo | 22:20 |
| * DocScrutinizer05 says that ;-P | 22:21 | |
| DocScrutinizer05 | Stskeeps: thanks for the info! :-) | 22:21 |
| DocScrutinizer05 | seems we need to do quite a bit of weeding out | 22:21 |
Generated by irclog2html.py 2.15.1 by Marius Gedminas - find it at mg.pov.lt!