IRC log of #maemo for Wednesday, 2017-06-07

DocScrutinizer05yes, writing to techstaff (incl your IP), or pinging chem|st (or yes, or warfare?) in here is all you can do. Except of getting a better IP00:02
DocScrutinizer05s/or yes/or xes/00:02
infobotDocScrutinizer05 meant: yes, writing to techstaff (incl your IP), or pinging chem|st (or xes, or warfare?) in here is all you can do. Except of getting a better IP00:02
DocScrutinizer05but I think already told ya, no?00:03
L29Ah_told what?00:04
DocScrutinizer05it's a royal PITA00:04
L29Ah_how do i get root on
DocScrutinizer05didn't tmo rise a page with intructions what to do about that 'ban'?00:04
DocScrutinizer05L29Ah_: WUT?00:04
* DocScrutinizer05 hands L29Ah_ a carrot00:05
DocScrutinizer05here's your root00:05
L29Ah_ok enjoy the royal pita00:05
*** dafox has joined #maemo00:05
DocScrutinizer05there's only 3 people getting root on all maemo servers afaik. Xes and warfare are the other two00:06
DocScrutinizer05when you need root on a particular VM like, repo.m.o or whatever, talk to council and techstaff and explain what you want to contribute00:07
DocScrutinizer05odds are you won't get root permissions nevertheless, rather a usual unix user account and a communication channel to the master sysops why will integrate your stuff, after checking it00:09
*** Venemo has quit IRC00:11
*** atk is now known as Arch-TK00:19
*** Arch-TK is now known as atk00:19
*** Pali has quit IRC00:25
*** dafox has quit IRC00:35
*** HRH_H_Crab has joined #maemo00:42
*** xes__ is now known as xes00:46
*** gry has joined #maemo00:58
*** Vajb has quit IRC01:16
*** Vajb has joined #maemo01:16
*** Wikiwide has joined #maemo01:51
*** Wikiwide has quit IRC01:53
*** N-Mi has quit IRC01:54
*** Wikiwide has joined #maemo01:55
*** wnd has quit IRC02:09
*** wnd has joined #maemo02:09
*** LjL has quit IRC02:53
*** LjL has joined #maemo03:01
*** infobot has quit IRC03:18
*** infobot has joined #maemo03:20
*** gry has quit IRC04:15
*** florian has quit IRC04:17
*** Wikiwide has left #maemo04:42
*** Wikiwide has joined #maemo05:08
*** Vajb has quit IRC05:09
Wikiwideel-v1.db is borked again. Wrong nick for some quotes. Will have to SQL Update Where again, to fix it. Somewhen in offline.05:11
*** Vajb has joined #maemo05:19
* Maxdamantus doesn't think he's had that corrupted since switching to ext4 around two years ago05:39
*** Wikiwide has left #maemo05:57
*** pagurus` has joined #maemo05:58
*** pagurus has quit IRC06:00
*** Wikiwide has joined #maemo06:11
WikiwideHow large is your el-v1.db? Mine is 50MB+06:12
MaxdamantusMm .. mine's only 833 K06:25
*** povbot has joined #maemo06:34
*** Kilroo has quit IRC06:39
MaxdamantusNo, I just ssh into my computer at home, which is always running irssi in tmux.06:44
WikiwideFun :-)  Anyway, telepathy-idle should speak with NickServ more, because the way it handles nicks makes it difficult to work across nick changes, as well as integrate with Contacts.06:51
*** L29Ah_ has quit IRC07:05
Maxdamantusso it messages nickserv `info ..` for lots of nicks and tries to parse out the account name?07:06
WikiwideNay, it doesn't speak with NickServ at all atm.07:07
* Maxdamantus imagines the WHOIS information on freenode would be better to deal with .. in either case, the logic is kind of network-specific.07:07
MaxdamantusOh, you're saying it doesn't spak with nickserv, but should.07:07
WikiwideSo I have to keep "svetlana", "svetlana-gry", "gry", "gry_" and "evilgry" in Address Book, and it isn't perfect at all.07:08
WikiwideBecause 1) Gry may use yet-another-nick; 2) somebody else can use one of her nicks, especially not protected ones (like gry_ )07:09
WikiwideAlso, it doesn't notice servers. It thinks that gry on freenode, moznet and whatever server has to be the same person, regarding integration with Contacts at least.07:10
*** Wikiwide has quit IRC07:34
*** Wikiwide has joined #maemo07:43
*** sunshavi has quit IRC07:53
*** DocScrutinizer05 has quit IRC07:55
*** DocScrutinizer05 has joined #maemo07:55
*** ceene has quit IRC08:21
*** Smily has quit IRC08:23
MaxdamantusPresumably another good way of avoiding corruption would be to just append messages to log files.08:23
*** Smily has joined #maemo08:24
MaxdamantusHmm .. just wondering, is your filesystem mounted with data=ordered? Mine is .. presumably there isn't a practical way for sqlite to maintain consistency without it.08:25
WikiwideHow to check?08:26
Maxdamantuscat /proc/mounts08:26
MaxdamantusLook for the line mentioning your main ext3 mount08:26
MaxdamantusProbably /home08:26
Maxdamantus/dev/mmcblk0p2 /home ext4 rw,noatime,nodiratime,commit=20,barrier=1,data=ordered 0 008:27
KotCzarnyor just finding a way to minimize that db08:27
Wikiwide/dev/mmcblk0p2 /home ext3 rw,noatime,nodiratime,errors=continue,commit=1,data=writeback 0 008:27
MaxdamantusMm .. I suspect that should really be ordered.08:28
WikiwideHow do I change from writeback to ordered? And what's the difference?08:28
* Maxdamantus wonders if that's something he changed at about the time he switched to ext408:28
*** ceene has joined #maemo08:29
Maxdamantuswriteback refers to a writeback cache, the important part is that there isn't much guarantee about what order stuff within files is written in.08:29
Maxdamantuswith data=ordered, if a process performs two writes to a file and the system crashes, it won'at be left in a state where nthe second write seemed to happen but the first didn't.08:30
Wikiwide/me just wants to check whether it is typical for CSSU-thumb to have data=writeback... Or is it just-me problem?08:31
MaxdamantusI think it's probably typical. I probably would have changed it myself if it were the default.08:31
Maxdamantusanyway, to change it you need to modify some scriupt in /etc so it doesn't generate fstab on boot, then you can just modify the fstab08:32
Maxdamantusor else modify the script so it adds data=ordered08:33
KotCzarnymine has writeback too08:33
*** spiiroin has quit IRC08:33
WikiwideWouldn't it be better to modify script to generate data=ordered fstab ?08:33
MaxdamantusMaybe. It seems kind of silly to me generating the same fstab every boot though anyway.08:35
WikiwideIn case it gets damaged somehow?08:36
KotCzarnyfor dynamic mounts it should be written as an udev rule08:37
KotCzarnynot rewriting fstab O.o08:37
WikiwideIs /home a dynamic mount ?08:38
KotCzarnyunless that script only runs once08:39
MaxdamantusActually, I'm not even mounting based on that fstab .. I have a script that mounts it  before preinit08:41
WikiwideNo idea. /etc/default/mount-opts ext3_opts="data=writeback"08:42
MaxdamantusEventually we should all just be using CoW filesystems, which work perfectly well with writeback caching (as long as there's an underlying barrier mechanism)08:47
Maxdamantusotherwise you need to do journalling, but noone uses data journalling on ext (though it is supported)08:48
Maxdamantus(journalling = write everything twice)08:48
WikiwideWhat is CoW?08:53
keriosadly everyone seems to be fucking it up, as of late :<08:59
kerioapple's brand new apfs has no data checksums08:59
*** chainsawbike has quit IRC09:00
keriohow do you go "yep, this zfs thing is good, i'ma copy it but i'll remove the best feature"09:00
KotCzarnymaybe its patented :P09:01
*** dafox has joined #maemo09:12
*** dafox has quit IRC09:27
MaxdamantusDoes their apfs still do unicode normalisation in the filesystem?09:28
MaxdamantusOh good, it doesn't.09:28
MaxdamantusSo I guess it's no longer worse than Microsoft.09:28
*** chainsawbike has joined #maemo09:31
kerioMaxdamantus: no but they moved unicode normalization in half the APIs09:37
kerioso you can have two files with names that normalize to the same thing09:38
keriobut you can only ever see and access one in guis09:38
KotCzarnywhoever thought that using anything else than ascii for file names should sort files in hell09:40
*** dmth|intevation has joined #maemo09:40
*** spiiroin has joined #maemo09:59
WikiwideUnicode is Unicode. At long as everybody uses the same encoding, it shouldn't a problem. Anyway, file name doesn't matter for tracker-ed files.10:02
KotCzarnysure, and then are no-printable characters10:02
KotCzarnyand whole lots of problems10:02
WikiwideHmm, I understand that most programs don't treat filenames as "text document" aka "let's change text direction, breakline, and so on", but it should be doable.10:13
WikiwideGranted, non-standard.10:13
*** florian_kc has joined #maemo10:16
MaxdamantusWell, text documents aren't stored as some magical normalised unicode object in the filesystem.10:17
MaxdamantusWhatever happens, neither should filenames.10:17
MaxdamantusThey should both be treated the same way by the filesystem: a sequence of bytes.10:18
KotCzarnywikiwide: i'm not saying unicode is bad, i'm saying it shouldnt be used for file names10:18
MaxdamantusSomething outside of the kernel is responsible for interpreting the bytes in some way so as to have various text directions.10:18
KotCzarnyunless you want confusion, collisions and errors10:18
Maxdamantusall the kernel has is byte 0, byte 1, byte 2, .. byte n10:18
MaxdamantusI think unicode is perfectly fine for filenames, just make sure the assumptions about unicode stay in userspace, preferably at a high level that only graphical applications interact with.10:20
MaxdamantusSo if you call `open(2)` on the string ".GIT", you shouldn't be able to read that file back by calling `open(2)` on the string ".git"10:21
*** geaaru has joined #maemo10:21
MaxdamantusOtherwise this happens:
Maxdamantusgit shouldn't be responsible for handling arbitrary normalisations, since it's not trying to render text.10:23
MaxdamantusAll it should care about is arrays of bytes, possibly terminated with 010:24
*** vahe has joined #maemo10:26
MaxdamantusI've been wondering how OS X behaves regarding unicode normalisation across OS versions.10:26
Maxdamantusbecause the effects of unicode normalisation change across revisions of unicode.10:27
MaxdamantusSo what happens if you transfer files back and forth between systems that do and don't use Unicode 9.0?10:28
Maxdamantusor worse, if you plug a harddrive into various versions of OS X?10:29
Maxdamantussince then you don't really get a chance to do the translations at the protocol level (which people probably don't bother doing anyway, like the CRLF crap in Windows)10:30
Maxdamantus(note on the CRLF point: protocols like FTP have a "text" mode that you're meant to use when transferring text files, which will correctly translate to/from things like CRLF form; the idea is that CRLF is just a way of encoding text, so as long as you transcode it every time you move it to a different system, there shouldn't be a problem)10:40
MaxdamantusBut that's just silly. It should just be stored in the simplest way possible.10:42
infobotMaxdamantus meant: But that's just silly. It should just be stored in the simplest way plausible.10:42
*** L29Ah_ has joined #maemo11:16
*** jskarvad has joined #maemo11:17
*** florian_kc is now known as floriam11:46
*** floriam is now known as florian11:47
*** N-Mi has joined #maemo12:06
*** N-Mi has joined #maemo12:06
*** dreamer has quit IRC12:15
*** mickname has quit IRC12:24
*** mickname has joined #maemo12:29
*** spiiroin has quit IRC12:31
auenfx4you mean emoji in file names isnt a good thing? :P12:45
*** spiiroin has joined #maemo12:46
KotCzarnywe need emojis for basic unix tools!12:46
auenfx4is there a programming language that is entirely in emojis?12:47
auenfx4aparently ues12:47
auenfx4but, i'd probably use lolcode over it still12:48
KotCzarnyas if there would be a difference12:49
L29Ah_what about lolicode12:53
*** dingen has joined #maemo12:59
MaxdamantusWhat I'm saying has no effect on whether or not you can use emojis in filenames.13:13
MaxdamantusJust that the kernel shouldn't have to know about emojis.13:14
Maxdamantusor the definition of the filesystem format.13:14
MaxdamantusUnicode already gives a really good way to represent emojis as bytes: UTF-813:14
MaxdamantusA lot of the time when designing software, the best way to support things is not to cram lots of features in that specifically tend to that thing.13:16
Maxdamantusbut rather to be generic enough to allow it.13:16
MaxdamantusLinux filesystems are perfectly generic enough, since they all handle filenames as bytes, and UTF-8 is a very good way to represent Unicode as bytes, even versions of Unicode that came out after the filesystem was last updated.13:17
MaxdamantusThat seems to be the direction that Apple's new filesystem is going, since they're presumably well aware by now that what they did was completely wrong.13:18
L29Ah_another filesystem?13:18
MaxdamantusYeah, APFS13:19
MaxdamantusHFS+ does two really stupid things WRT unicode: use 16-bit code units (so things are meant to be encoded in UTF-16), and do unicode normalisation at the filesystem level.13:20
Maxdamantusafaict, APFS does neither.13:20
L29Ah_ i'd say there are a lot of features you don't usually see at the filesystem level13:21
MaxdamantusLike what? Encryption?13:22
MaxdamantusOther than encryption, all the other things listed there should be standard features of new filesystems.13:22
Maxdamantuseg, btrfs and ZFS are relatively new filesystems that do all of those things other than encryption.13:23
L29Ah_multikey encryption, snapshots, data deduplication13:23
Maxdamantus(ZFS might do encryption, dunno)13:23
Maxdamantusactually, ZFS might not have cloning of files .. btrfs does.13:23
L29Ah_i don't see the reason why snapshots should be done at the filesystem level instead of the lower one (as in lvm)13:24
L29Ah_data deduplication/CoW may have been as well (as seen in fossil + venti), yet it didn't go mainstream for some reason13:24
MaxdamantusWell, why not just do data integrity at the LVM level?13:24
MaxdamantusIt's perfectly feasible for something like LVM to be CoW and provide total data ordering to arbitrary filesystems that sit on top of it.13:25
L29Ah_data integrity is done at the hardware level :]13:25
MaxdamantusI don't trust data integrity done at the hardware level.13:25
L29Ah_then it's time for you to use raid13:26
Maxdamantuser, when I said integrity before, I more meant "crash protection"13:26
L29Ah_i don't see what task the extra checksums at filesystem level would solve13:26
L29Ah_maybe you wanted journaling?13:26
MaxdamantusNo, journalling is a really bad way of solving that.13:27
MaxdamantusJournalling means you have to write everything twice.13:27
MaxdamantusWhich means your speed is at least halved.13:27
MaxdamantusCoW is a much better way to do it.13:27
L29Ah_it doesn't mean that13:27
MaxdamantusYes, it does. That's exactly what journalling is.13:27
L29Ah_CoW doesn't impede journaling13:27
MaxdamantusWrite the new stuff into a temporary space, then write it to the actual space.13:27
L29Ah_and you don't need to store checksums for CoW13:28
MaxdamantusThat way if you crash in the middle, you can ignore the stuff in the temporary space.13:28
Maxdamantusor ignore the stuff in the actual space (and copy everything from the temporary space)13:28
Maxdamantusnote: practically noone actually uses full journalling on ext, only metadata.13:28
MaxdamantusThe best crash protection you can get from ext without full journalling is data=ordered, which has its own performance issues.13:29
L29Ah_well, i meant CoW for data, journaling for metadata13:29
MaxdamantusSo you're writing the metadata twice? Why?13:30
L29Ah_to avoid data loss on crashes (:13:31
MaxdamantusYou get that with CoW.13:31
Maxdamantusand you don't have to write everything twice.13:31
L29Ah_CoW over metadata?13:31
MaxdamantusAlso, you mentioned RAID .. RAID is a really bad solution to this as well.13:32
L29Ah_where do i store the metadata about metadata then?13:32
MaxdamantusIt practically requires .. again .. journalling.13:32
MaxdamantusSo with RAID you have to write everything twice.13:32
MaxdamantusIf you have a really expensive hardware RAID controller, it might have hundreds of megabytes of really fast NVRAM to work as its journal.13:32
Maxdamantusif you're using a less expensive RAID controller or software RAID, you're just fucked in terms of crash protection.13:33
MaxdamantusEvery RAID setup that doesn't have a journal is vulnerable to the write hole issue.13:33
MaxdamantusAgain, you need CoW to do this without journalling.13:33
MaxdamantusThings like btrfs and ZFS offer RAID-like features (striping, parity, duplicating, etc), but using CoW instead of journalling.13:34
MaxdamantusWell, btrfs' parity mode supposedly still has the write hole, last I checked.13:34
Maxdamantus22:32:03 < L29Ah_> where do i store the metadata about metadata then?13:35
MaxdamantusYou don't store it until you know all of the metadata is stored.13:35
Maxdamantus"it" being the information in the superblocks.13:36
MaxdamantusThe superblock is a small block of information that is replicated multiple times on a single device.13:36
MaxdamantusWhen mounting the filesystem, you read all the instances of the superblock and look for the latest one that matches its own checksum.13:37
Maxdamantus(this is in btrfs, other CoW filesystems might have other solutions, but they'll probably be essentially the same)13:38
siceloOT: what is currently best RSS reader for N900?13:43
* L29Ah_ uses
siceloi'm not looking for something web based13:45
bencohI'd tend to say cutenews, but...13:45
L29Ah_it is not web based13:46
sicelobencobut ...13:47
siceloL29Ah_: i also don't want them in my mailbox13:48
bencohsicelo: I thought about using it with miniflux (since miniflux would prepare content remotely then provide me with a clean rss feed / clean articles)13:49
bencohI eventually ditched the whole idea and don't use any rss reader13:50
bencohjust that cutenews seemed the most clean/efficient at that time13:50
bencoh(compared to, say, feedingit)13:50
sicelothanks bencoh. will use cutenews then13:51
siceloi think i do need an rss reader. stock one is fine for me, but it has problems with others13:52
sicelowhat did you end up using?14:01
bencohno rss, no articles on the go14:02
bencohnot a great loss anyway14:02
WikiwideWhat's the problem with built-in rss reader?14:17
siceloi do use it just fine most of the time - however it can't open for example
*** vahe has quit IRC14:38
*** cyphase has quit IRC15:05
DocScrutinizer05Wikiwide: everything >1MB for el-v1.db is considered to bring down whole system to a grinding halt, due to sqlite accessing all that stuff and thus bringing swap hell to a system with only 250MB of RAM with only ~50MB of that available to userland15:09
WikiwideHmm... Interesting... My point of view is that el-v1.db should be separated into many small files (though it would be hell to maintain), so that when user wants the last 10 messages, he doesn't have to load 1million-heavy history into RAM.15:13
L29Ah_does sqlite always read the whole database to ram?15:13
L29Ah_also i see 100MB free on my pretty much stock maemo15:14
WikiwideHow do other programs do that? Log and load logs into GUI?15:14
*** cyphase has joined #maemo15:14
DocScrutinizer05Wikiwide: it's one of the known problems in maemo that Nokia devels never tested maemo for long term sustainability, neither with 1000s of contacts in database, nor for cleaning up any 'logging' files like particularly el-v1.db. You *could* write your own SQL cleanup jobs if you know SQL and understand how the db is structured15:14
DocScrutinizer05for all those questions please refer to RDBMS-10115:15
WikiwideWhat would cleanup entail? I know basic SQL, but structure of el-v1.db is confusing.15:15
DocScrutinizer05yes, that's a problem15:16
DocScrutinizer05luckily I think the data isn't incremental at all, in a sense of "you can delete old records without any impact to new data"15:17
WikiwideI don't want to delete records.15:17
DocScrutinizer05it's probably safe to delete just records with old timestamps, out of a set of very many similar types of records15:18
WikiwideImagine life-logger.15:18
DocScrutinizer05how would you shrink your el-v1.db without doing that?15:18
WikiwideAlso, some time in distant future it could be possible to implement Find, and search All Messages for a word, sender, or something.15:19
WikiwideHow to shrink? Split it, either chronically or channel-y or both, and load only pieces needed.15:19
DocScrutinizer05you can't keep 1TB of data on a 512MB uSD, and you can'tkeep the data of a 50MB el-v1.db in a el-v1.db <1MB15:20
Maxdamantusimo that data shouldn't even all be stored in sqlite.15:20
MaxdamantusSimple log files for actual conversation data seems more sensible to me .. easier to maintain/trim.15:21
DocScrutinizer05you can mv el-v1.db el-v1_1.db any time. Instantly reboot after that!15:21
WikiwideMmm. Sqlite has good tools to go through data. Where and the like. Text file wouldn't be as easy to edit.15:21
Maxdamantusand don't need special tools to search .. grep should just work.15:21
WikiwideGrep wouldn't do field search, like who-sender or which-channel or SMS-vs-IM15:23
L29Ah_googling suggests sqlite won't load the whole database into ram unless explicitly requested to15:23
WikiwideI think that Quassel has similar Sql database for logs, or something.15:24
DocScrutinizer05of course not15:24
Maxdamantusgrep who-sender ~/irclogs/Freenode/#maemo.log15:24
DocScrutinizer05read RDBMS-10115:24
L29Ah_so i doubt it makes sense to rotate it15:24
Maxdamantusgrep '^..:.. <.Wikiwide>' ~/irclogs/Freenode/#maemo.log15:24
bencohWikiwide: tbh when I'm looking for some information in my (say) text messages I just launch sqlite and fire a SELECT query15:24
WikiwideOkay, RDBMS-101. Will look it up.15:25
DocScrutinizer05I doubt this chat makes sense to me, I think I'm missing parts thanks to /ignore15:25
DocScrutinizer05  :-)15:26
Wikiwidebencoh: nice! SQL SELECT aka search is a good example of how human-friendly sqlite database is.15:26
DocScrutinizer05a RDBMS is usually b-tree or somesuch, with indizes. depending on the operation you perform, it's necessary to sometimes walk the complete dataset which means you load every part of dataset to RAM, sequentially. Since linux fs and RDBMS both keep buuffers, you will inevitably hog some RAM with that15:27
*** vahe has joined #maemo15:29
DocScrutinizer05and mere experience holds evidence that el-v1.db >1or2 MB brings device to a state that's close to "grinding halt", with delays in the range of 10 to 30 seconds to events that are supposed to be realtime, like dialer popping up on inbound call. You don't want that to happen 30s *after* inbound call started15:31
*** Wikiwide has quit IRC15:31
* Maxdamantus has IRC logs that are gigabytes long15:32
Maxdamantusirssi doesn't care .. all it does is open them in append mode and write to them.15:32
DocScrutinizer05bencoh: make that a script, and augemt it so it could work across multiple database files, so users could keep archive in el-v1_*.db ?15:33
Maxdamantus(on PC, not on N900, though N900 probably wouldn't suffer from having those large logs either)15:33
DocScrutinizer05bencoh: that would be an extremely useful little app15:33
bencohDocScrutinizer05: it probably would, yeah15:33
DocScrutinizer05my SQL is pretty dusty, thugh I have a comprehensive understanding of how things work and should work15:34
DocScrutinizer05IOW I could do SQL but it takes ages15:35
Maxdamantusimo if there were to be more advanced search features, it should just use the logs as a source of truth.15:35
bencoh"the logs"?15:35
Maxdamantusie, once you've got logs that are >100 MiB long, you can generate extra search information that's stored in a cache.15:36
bencohsome kind of index?15:36
MaxdamantusIf you think that search information might be corrupted, just delete it and have it regenerated.15:36
KotCzarnymaxd: that's assuming world is static, ie. doesn't change15:36
bencohKotCzarny: not necessarily15:36
bencohsee mairix, notmuch, xapian, the likes15:37
MaxdamantusKotCzarny: well, the index data would still be generated incrementally.15:37
bencoh(xapian is actually the engine. mairix/notmuch are frontends)15:37
KotCzarnythere is a saying 'only cow doesnt change it's opinions'15:37
bencohonly CoW?15:37
KotCzarnymoo :P15:38
infobotmooooooo! I am cow, hear me moo, I weigh twice as much as you. I'm a cow, eating grass, methane gas comes out my ass. I'm a cow, you are too; join us all! type apt-get moo.15:38
DocScrutinizer05((<Wikiwide> bencoh: nice! SQL SELECT aka search is a good example of how human-friendly sqlite database is.)) also a good example for the RDBMS needing to walk the *complete* dataset if it can't optimize the SELECT condition to use indizes15:41
DocScrutinizer05and I think what wikiwide meant is "human friendly" of 4th generation languges like SQL at large15:42
DocScrutinizer05the tricky part however starts with defining and understanding the data model15:43
DocScrutinizer05*using* it is simple, after that15:43
DocScrutinizer05SELECT responsetext FROM ideal-database WHERE (is-relevant-to-wikiwide = TRUE);  ;-P15:45
DocScrutinizer05now is-relevant-to-wikiwide could be a stored procedure AIUI ;-)15:46
*** frals has quit IRC15:46
*** frals has joined #maemo15:46
DocScrutinizer05still ineffective until you get an index for it15:47
DocScrutinizer05I prefer to use grep for such stuff, generally15:47
DocScrutinizer05XML isn't all bad for those purposes15:48
DocScrutinizer05not compatible with simple grep usually, but still simpler and more lightweight than the usual RDBMS15:49
DocScrutinizer05for IRC I have plaintext logfiles (not on N900) and grepping in them is a matter of milliseconds, even in the 20+MB sized ones15:51
DocScrutinizer05anyway maemo fremantle lacks any useful means (except for SMS iirc) to clean out old records form el-v1.db15:52
infobotDocScrutinizer05 meant: anyway maemo fremantle lacks any useful means (except for SMS iirc) to clean out old records from el-v1.db15:52
DocScrutinizer05heck, I even don't lnow of any simple archiver or export tool for the data in there15:53
MaxdamantusI'm pretty sure grep is even going to be faster at most searches you'd want to do.15:56
Maxdamantusbecause of DocScrutinizer05's point; unless you're doing some operation that the schema was specifically designed to make optimal, it's probably just searching through all the rows.15:57
MaxdamantusI imagine in the best case you might be able to get faster searches for a particular time period, but if every message is logged with a timestamp and there are no clock skews, you can already do that using `look` from util-linux15:58
Maxdamantus(`look` does a line-wise binary search on a file)15:59
* Maxdamantus hacked together some program himself that did that, before realising `look` already existed15:59
MaxdamantusI suspect still in that case, running an entire look process will be far quicker than simply opening an SQLite database16:01
MaxdamantusHm, though `look` doesn't seem to have an option to just produce an offset into the file.16:03
MaxdamantusMy program did that, so you could just pass the result as an argument to something like `less` to seek to the right position.16:04
MaxdamantusOh hey, my program can actually exec less for you.16:06
MaxdamantusFunnily enough, I also have that mechanism built into my terminal.16:08
KotCzarnyyeah, probably set PAGER variable in bash16:09
Maxdamantusin rxvt-unicode you can hold ctrl-shift and drag to show information about a character .. I modified my rxvt-unicode so whenever it calculates that information it does a binary search on UnicodeData.txt to show more interesting stuff.16:09
MaxdamantusNo noticable latency, because each lookup takes probably less than 1 ms16:10
Maxdamantusonly a 1.4 MiB file, but obviously binary search scales extremely well.16:10
Maxdamantus$ time ./findline /usr/share/unicode-data/UnicodeData.txt 000{0..9}\; >/dev/null16:13
Maxdamantusreal    0m0.001s16:13
Maxdamantus in case anyone's interested16:14
DocScrutinizer05SELECT responsetext FROM ideal-database WHERE (is-relevant-to-wikiwide = TRUE) gets you a dozen or a few 10000s of lines/records out of a 500MB database in only a few milliseconds, while  SELECT responsetext FROM the-usual-database WHERE (responsetext LIKE "this is what I search for"); takes ages and gets beat by a simple grep hands down16:14
* Maxdamantus just adapted it from his urxvt code one day at work when analysing some stuff from a database dump16:15
DocScrutinizer05database dudes have a hard time eachs single time I show them a grep is faster than their RDBMS in particular cases ;-P16:23
DocScrutinizer05"but but but...! it's not... errr... wel you can't.... ERRRR but it's NOT SQL!"16:24
*** Kabouik has joined #maemo16:36
Maxdamantusfun test: sha1sum <(./findline -e -i /tmp/unicodedata-sorted.txt </tmp/unicodedata-sorted.txt) /tmp/unicodedata-sorted.txt16:48
*** vahe has quit IRC16:48
Maxdamantus(for every line in the file, looks for that line in the file using binary search)16:49
Maxdamantusabout 170k independent searches per second16:50
Maxdamantuscould easily be optimised, since there are particular hotspots that the binary searches all touch that could remain in memory (ie, the lines at 0.5, 0.25, 0.75, ...)16:51
*** vahe has joined #maemo17:07
*** vahe has quit IRC17:17
*** vahe has joined #maemo17:18
parazydfreemangordon: amprolla is finished i believe :)17:37
parazydshould undergo some testing now by the 'big bosses'17:37
*** vahe has quit IRC17:37
parazydwill find out more when they wake up (also got a devuan meeting tonight i think)17:37
*** vahe has joined #maemo17:38
*** Kabouik_ has joined #maemo17:40
*** Kabouik has quit IRC17:40
*** Kabouik_ has quit IRC17:45
*** Kabouik_ has joined #maemo17:45
*** Kabouik_ has quit IRC18:22
*** Kabouik has joined #maemo18:22
*** florian has quit IRC18:25
*** Pali has joined #maemo18:35
*** Kabouik has quit IRC18:44
*** dafox has joined #maemo18:52
freemangordonparazyd: great :)18:57
*** jskarvad has quit IRC19:00
*** spiiroin has quit IRC19:02
*** jskarvad has joined #maemo19:03
*** vahe has quit IRC19:04
*** jskarvad has quit IRC19:10
*** jskarvad has joined #maemo19:12
Wizzupparazyd: thanks for going the extra mile and getting it into devuan. onwards19:13
infobotWizzup meant: parazyd: thanks for going the final mile and getting it into devuan. onwards19:13
*** LauRoman has quit IRC19:14
*** LauRoman has joined #maemo19:15
*** spiiroin has joined #maemo19:16
*** ff_ has joined #maemo19:22
ff_ el-v1.db might help19:23
*** florian has joined #maemo19:24
ff_I done it, and el-v1.db is 14% smaller now, no data loss, I don't know about speed19:25
*** ff_ has quit IRC19:26
*** dafox has quit IRC19:31
*** hurrian_ has joined #maemo19:40
*** hurrian has quit IRC19:41
*** N-Mi has quit IRC19:47
*** Milhouse has quit IRC20:03
*** dmth|intevation has quit IRC20:07
*** geaaru has quit IRC20:08
*** Milhouse has joined #maemo20:16
*** dafox has joined #maemo20:19
siceloback to my rss issue - cutenews doesn't load on my N900.20:21
siceloanyway, guess should try all of them one by one20:23
siceloah, silly problem.
parazydWizzup: thank you for the help :)20:41
sixwheeledbeastsicelo: seems a bit odd for it to be file permissions issue.21:10
siceloindeed, but at least on my installation that worked. i actually first adjusted the .desktop file, which didn't work21:11
*** jskarvad has quit IRC21:24
*** Venemo has joined #maemo21:39
*** cyteen has quit IRC21:42
sixwheeledbeastThanks I'll try it later if I get a minute21:49
*** heroux has quit IRC21:52
*** heroux has joined #maemo21:53
*** spiiroin has quit IRC22:42
*** parazyd has quit IRC22:43
*** how900 has quit IRC22:43
*** how900 has joined #maemo22:45
*** how900 has joined #maemo22:45
*** parazyd has joined #maemo22:45
*** parazyd has joined #maemo22:45
*** how900 has quit IRC22:46
*** parazyd has quit IRC22:46
*** how900 has joined #maemo22:46
*** how900 has joined #maemo22:46
*** parazyd has joined #maemo22:47
*** parazyd has joined #maemo22:47
*** parazyd has quit IRC22:50
*** how900 has quit IRC22:50
*** how900 has joined #maemo22:51
*** how900 has joined #maemo22:51
*** parazyd has joined #maemo22:51
*** parazyd has joined #maemo22:51
*** spiiroin has joined #maemo23:08
*** dafox has quit IRC23:11
*** cyteen has joined #maemo23:38
*** Venemo has quit IRC23:53

Generated by 2.15.1 by Marius Gedminas - find it at!