IRC log of #maemo for Saturday, 2017-02-11

ff_with 35000 iops it could works like RAM, so, n900 with extremely expensive 32GB RAM/ROM :)00:05
L29Ah23:15:04]<KotCzarny> most noname cards have iops in 0.1-1/s range00:05
L29Ahwhat00:05
ff_I couldn't be surprised if Chinese people could produce something like that00:29
ff_anyone know speed of microSD data bus in n900?00:33
ff_maybe it's not worth the bother with fast microSD00:34
MaxdamantusI'm getting around 12 MB/s read.00:36
MaxdamantusI have a feeling that the maximum speed does kind of matter.00:36
ff_what are specs of sd?00:37
Maxdamantusiirc I would only get around 4 MB/s on a lower-end Kingston microSD, which was rated at something like 30 MB/s.00:37
ff_speed of read or write?00:38
MaxdamantusSanDisk 128GB Ultra® microSDXC™ UHS-I Class 10 card00:38
MaxdamantusSD cards don't really have a particular speed.00:39
MaxdamantusIt depends on things like how the controller has laid out the data.00:40
ff_with it you get 12MB/s in n900?00:41
MaxdamantusYes.00:41
Maxdamantus236978176 bytes (237 MB) copied, 20.236 s, 11.7 MB/s00:41
ff_and how with write?00:41
Maxdamantuswrite is trickier to test.00:42
Maxdamantuscan try after remounting with -o sync, I guess.00:43
ff_not pressure, I'm just curious00:44
MaxdamantusI'm getting 10.6 MB/s writing to ext4 mounted with -o sync using dd with conv=fsync on00:45
Maxdamantus(so that's to a filesystem, not just a block device)00:45
Maxdamantusthough I'm only writing zeroes, dunno if the card does something special with that.00:45
MaxdamantusWell, I imagine the card should handle more than 10.6 MB/s anyway .. it'll at least be pushing that 10.6 MB/s in SDIO packets.00:46
ff_I don't know also, thanks for real data with n90000:47
MaxdamantusThe sync options are also going to usually reduce overall throughput00:47
ff_it seems that n900 has 10MB/s speed limit on sd bus00:48
MaxdamantusThese speeds are all sequential btw, not random.00:48
ff_random should be lower than this 10MB/s cameramemoryspeed.com/reviews/sd-cards/sandisk-extreme-plus-95-90-mbs-64gb-micro-sdxc-memory-card/00:50
ff_9MB/s for read and 3MB/s for write if that benchmark is with the same class card00:51
ff_microSDXC UHS-I U3, I'm not sure for that U3 symbol00:55
*** Oksanaa has quit IRC00:55
*** shentey has quit IRC00:56
*** Oksanaa has joined #maemo00:57
*** ff_ has quit IRC01:14
*** jskarvad has quit IRC01:15
DocScrutinizer05a pty ff_ left. Was absolutely right it doesn't really make sense to bother, since databus is limited to 4bit buswidth * (iirc) 40MHz clockrate max, so 20MB/s01:34
DocScrutinizer05*might* even be just 25MHz01:35
OksanaaQuestion is, which cards actually work at maximum limit of the databus.01:35
DocScrutinizer05all01:38
DocScrutinizer05:-)01:38
*** RzR has quit IRC01:38
Maxdamantusexcept for my Kingston one.01:38
DocScrutinizer05it's a pretty ancient databus protocol. Recent Ultra card only work since they support legacy protocols, the OMAP3 doesn't support those new speeds01:39
DocScrutinizer05UHS-I and UHS-II are not supported by OMAP301:39
DocScrutinizer05for write, the real show stopper is page erase needed to rewrite a block with new data. It can take up to some 100s of MILLIseconds01:40
DocScrutinizer05at very least 10ms01:41
DocScrutinizer05per page erase01:41
OksanaaIs it possible to have faster databus with OMAP3? And what is page erase, is it like going over the card with eraser to be sure that write to card goes well?01:42
DocScrutinizer05yes @ #201:43
DocScrutinizer05no @ #1, it's a physical interface that can't do any faster01:43
DocScrutinizer05OMAP has a 4bit and a 8bit storage bus, both with same max clock frequency. In N900 the 8bit is for eMMC01:44
OksanaaLike, would Neo900 be able to have faster databus than N900?01:44
*** ketas is now known as ketas-01:44
DocScrutinizer05no01:44
DocScrutinizer05alas not01:44
DocScrutinizer05that's also one killer argument why Neo900 must have eMC despite many people would prefer two uSD slots: eMMC supports 8bit01:45
OksanaaAbout page erase: is it possible to do it beforehand, like use idle time to erase empty blocks on the card so that by the time a write is needed, card already has several blocks ready?01:46
MaxdamantusI imagine that's something the card will already do.01:46
OksanaauSD slot supports only 4bit?01:46
MaxdamantusHosts don't interact with SD cards as flash devices.01:46
MaxdamantusThat's all stuff that's handled by the computer inside the SD card.01:46
OksanaaMaxdamantus: that would be neat, if it's implemented already.01:47
*** ketas has joined #maemo01:48
DocScrutinizer05(do it beforehand) that's how all slightly smarter cards work, as long as the OS supports TRUNCATE01:49
MaxdamantusTRUNCATE? Do you mean TRIM?01:49
Maxdamantusif so, that has nothing no do with SD cards.01:50
DocScrutinizer05the card needs to know which blocks can get erased beforehand, even writing all zeroes to a whole page doesn't automatically tell card "this is no data, this is empty storage free to use"01:50
Maxdamantuswell, that's not what SD cards typically do.01:50
DocScrutinizer05uSd has only 4 bit data, yes. count the pins on your uSD card ;-)01:51
DocScrutinizer05you need GND, VCC, clock r/w whatnot else01:51
MaxdamantusDocScrutinizer05: can you provide a link to some information about this magical "TRUNCATE" thing you're talking about?01:51
Maxdamantusor is this another case where you're just saying things about technologies you don't understandL01:51
DocScrutinizer05Oksanaa: https://wiki.debian.org/SSDOptimization01:55
MaxdamantusWe're talking about SD cards, not SSDs.01:55
MaxdamantusSD cards don't have TRIM (or "TRUNCATE" if you like making up names for things)01:55
DocScrutinizer05basically all SSD rationale also applies to SD cards01:56
MaxdamantusSD is a completely different protocol.01:56
Maxdamantusand it doesn't have anything like TRIM in it.01:56
DocScrutinizer05sorry, DISCARD, not TRUNCATE01:56
Maxdamantusapparently it's actually ERASE01:58
DocScrutinizer05the best way is to actually never use a maybe 10% of the SD card, by downsizing the partition(s)02:02
DocScrutinizer05once the system wrote to all available blocks (and didn't issue a DISCARD or whatever the equivalent name on SD bus, for all the no longer needed blocks, and made sure the non-used blocks concatenate to a whole erase page), the card will significantly drop on write performance02:04
*** sunshavi has joined #maemo02:04
OksanaaSo, use up to 90% or so of the card.02:05
DocScrutinizer05since the card controller can't fortune tell which pages to erase "preemptively"02:05
DocScrutinizer05card always have a few hidden spare pages, but not that many02:06
DocScrutinizer05Oksanaa: (90%) I mere guess by me, but sounds good02:06
OksanaaSo, if somebody made a card with high enough quantity of hidden spare pages, that would be easier on user.02:07
DocScrutinizer05yes02:08
MaxdamantusWell, presumably it'll usually have around 73 MB per GB of spare blocks.02:08
Maxdamantusif its capacity is given in GB.02:09
Maxdamantussince 1 GiB is a bit more than 73 MB more than a GB.02:09
Maxdamantusflash usually comes in a power-of-2 number of bits, then the SD card will offer a power-of-2 number of GBs or MBs.02:10
DocScrutinizer05maybe interesting (didn't read it completely): https://blogofterje.wordpress.com/2012/01/14/optimizing-fs-on-sd-card/02:11
*** sunshavi has quit IRC02:14
DocScrutinizer05https://haydenjames.io/increase-performance-lifespan-ssds-sd-cards/  here it's called TRIM02:16
JuestoDo compactation?02:16
DocScrutinizer05though >>Therefore, when you delete a file, your SSD is now able to write data to blocks as if it they were brand new without having to perform the cumbersome deletion process.<< obviously is utter bullshit02:16
*** DarioAlejandro has quit IRC02:25
DocScrutinizer05MEH! I need to avoid looking into chanlog, always an annoyance to read that crap my /ignore luckily usually saves me to read02:27
*** sunshavi has joined #maemo02:30
DocScrutinizer05Juesto: one of the major problems about uSD embedded controller is: it doesn't know about ext-fs or anything else than VFAT basically. On ext[234] you have inodes scattered all over the complete storage02:31
DocScrutinizer05it's assumed that the SD controller even knows about File allocation Table in VFAT and optimizes using VFAT properties it knows about. Can't do same for any other filesystem02:32
*** dafox has joined #maemo02:33
DocScrutinizer05Juesto: anyway SSD/SD analysis and optimization is absolutely non-trivial, due to the complex way a flash storage works and the obscure SEKRIT sauce the uSD controller adds on top02:36
luke-jrMaxdamantus: SD cards have a virtual block layer on top of the real flash, which adds overhead for remapping bad blocks02:36
DocScrutinizer05that's why for really tight control and fast r/w access you want NAND and a filesystem like ubifs02:36
DocScrutinizer05oh hi luke-jr :-)02:37
luke-jrhi DocScrutinizer0502:37
* DocScrutinizer05 seems to recall there were plans or even initiatives to reflash the uSd controller firmware to allow using the uSD like it was normal un-managed NAND storage02:38
*** ketas- has quit IRC02:39
DocScrutinizer05obviously such a project would be highly brand&make specific on the particular uSD02:39
DocScrutinizer05Oksanaa: re "90%" don't get fooled: see https://www.sandisk.com/home/memory-cards/sd-cards/extremeplus-sd-uhs-i "DISCLOSURES: ... 1MB=1,000,000 bytes. ** 1GB=1,000,000,000 bytes"02:50
DocScrutinizer05*plus*:  "Actual user storage less."02:51
DocScrutinizer05so a 32GB uSD has a 32E9 bytes, minus a non-marginal fraction for overhead like spare blocks and prolly also flash controller metadata etc02:55
*** Kabouik_ has joined #maemo02:58
*** DarioAlejandro has joined #maemo03:02
*** florian has quit IRC03:07
*** dafox has quit IRC03:12
*** cyphase has quit IRC03:16
DocScrutinizer05AAAH and there's the term I missed: FTL aka Flash Translation Layer (aka "the uSD embedded controller)  http://www.linux-mtd.infradead.org/doc/ubifs.html03:19
DocScrutinizer05s/"the uSD embedded controller("the uSD embedded flash controller"/03:19
DocScrutinizer05http://www.linux-mtd.infradead.org/doc/ubifs.html#L_raw_vs_ftl03:21
Maxdamantus13:55:39 < DocScrutinizer05> so a 32GB uSD has a 32E9 bytes, minus a non-marginal fraction for overhead like spare blocks and prolly also flash controller metadata etc03:21
MaxdamantusNo.03:21
*** cyphase has joined #maemo03:21
MaxdamantusIt has 32 GiB inside, but provides at least 30 GB that you can write to.03:21
MaxdamantusYou practically can't get 32 GB in flash chips .. a 32 GB SD card will have 32 GiB of flash inside, but the block device you end up with will be around 32 GB.03:23
MaxdamantusHm, actually, my 128 GB card only has 127865454592 bytes.03:25
DocScrutinizer05the trick with optimizing uSD is:http://www.linux-mtd.infradead.org/faq/general.html#L_mtd_vs_hdd you have an interface/protocol for a block device (left side of table, with extensions like TRIM for example), but you try to implement a "usage pattern" aka filesystem and driver that takes care about properties described on right side half of the table03:26
*** Kabouik_ has quit IRC03:27
DocScrutinizer05you need a priori knowledge (or assumptions) about how the FTL firmware works and what it does in response to what you do on the block device interface03:27
MaxdamantusThe eMMC on my N900 has 32015122432 bytes though.03:33
DocScrutinizer05http://www.linux-mtd.infradead.org/faq/general.html#L_ext2_mtd >>But in many cases using mtdblock is a very bad idea because what it basically does if you change any sector of your mtdblockX device, it reads the whole corresponding eraseblock into the memory, erases the eraseblock, changes the sector in RAM, and writes the whole eraseblock back. This is very straightforward. If you have a power failure when the eraseblock is being erased,03:33
DocScrutinizer05you lose all the block device sectors in it. The flash will likely decay soon because you will wear few eraseblocks out - most probably those ones which contain FAT/bitmap/inode table/etc.<<03:33
DocScrutinizer05eraseblock (I called it erase page, sorry) is as large as several dozen or even hundreds of kB03:34
DocScrutinizer05and to change a singke bit in it, you need to read modify erase write the whole eraseblock03:35
MaxdamantusDocScrutinizer05: yeah, that's why you shouldn't use something like mtdblock, as mentioned above.03:35
MaxdamantusDocScrutinizer05: something sensible would keep writing to new blocks.03:35
Maxdamantusthat's basically the principle behind log-based filesystems.03:36
MaxdamantusThe FTLs in SD cards probably do something similar.03:36
DocScrutinizer05*unless* the bit change is 0->1, assuming all-0s is the result of block-erase (which takes several milliseconds as I already explained)03:36
Maxdamantushttps://en.wikipedia.org/wiki/List_of_log-structured_file_systems03:37
MaxdamantusThe filesystems at the bottom are all the standard ones for flash.03:38
Maxdamantusincluding UBIFS, which is used by maemo.03:38
DocScrutinizer05so stuff like FTL and ubifs implements very nifty ways to not always read-modify-write for a few bytes that need to be changed, rather it uses journals etc that simply get extended at end, writing to the already/still all-0 region of a erase block03:38
Maxdamantusan SD card or MMC has to do the same sort of stuff, but it just provides a block device instead of a filesystem.03:38
DocScrutinizer05and TRIM deals with telling FTL that a complete eraseblock is 'clean' again and ready to get erased when there's some spare time in background for doing so03:39
DocScrutinizer05(simplified image, since TRIM doesn't really care about erase blocks, it has no notion about them afaik. It's up to the FTL to deduce erase blocks needing a background erase, from what  fs blocks (512) TRIM told the FTL are unused03:43
DocScrutinizer05)03:43
*** Pali has quit IRC04:01
*** Oksanaa has left #maemo04:11
*** RedW has quit IRC05:43
*** RedW has joined #maemo05:43
*** RedW has quit IRC05:50
*** utanapischti has joined #maemo05:51
*** drawkula has quit IRC05:51
*** F3l1x_10m is now known as Yuri_10m07:05
*** DocScrutinizer05 has quit IRC07:29
*** DocScrutinizer05 has joined #maemo07:29
*** chainsawbike has quit IRC07:47
*** chainsawbike has joined #maemo07:49
KotCzarnyl29ah: yes, see above eraseblock explanation08:30
KotCzarnysd cards were created for mobile capture devices08:30
KotCzarnynowadays people started using them as rootfs etc, so sandisk/samsung started optimizing 4k random r/w08:31
KotCzarnyif you want to measure correctly run iozone test, you will see for yourlsef08:32
*** RedW has joined #maemo09:28
*** florian has joined #maemo10:15
*** Kabouik_ has joined #maemo13:08
*** Pali has joined #maemo13:15
*** xorly has joined #maemo13:16
*** rosseaux has joined #maemo13:28
*** florian has quit IRC13:41
*** Kabouik_ has quit IRC14:45
*** Kabouik_ has joined #maemo15:34
*** ced117_ is now known as ced11715:54
*** ced117 has joined #maemo15:54
*** florian has joined #maemo16:09
*** eMHa has quit IRC16:37
*** eMHa has joined #maemo16:40
*** dafox has joined #maemo16:45
*** dafox has quit IRC16:52
*** shentey has joined #maemo18:28
*** shentey has quit IRC18:48
L29AhOne more step18:55
*** shentey has joined #maemo18:56
*** fuz_ has quit IRC19:04
*** shentey has quit IRC19:12
*** fuz_ has joined #maemo19:35
*** L29Ah has quit IRC19:46
*** lobito has quit IRC20:49
*** cyphase has quit IRC21:36
*** cyphase has joined #maemo21:41
*** Roth has joined #maemo22:05
*** louisdk has joined #maemo22:07
*** Roth_ has joined #maemo22:08
*** Roth has quit IRC22:11
*** Roth has joined #maemo22:14
*** Roth_ has quit IRC22:16
*** Roth_ has joined #maemo22:19
*** xes_ has joined #maemo22:20
*** Roth has quit IRC22:21
*** xes has quit IRC22:23
*** Roth has joined #maemo22:24
*** Roth_ has quit IRC22:27
*** Roth has quit IRC22:32
sunshavi~info sunshavi23:00
*** RzR has joined #maemo23:25
*** sunshavi has quit IRC23:53

Generated by irclog2html.py 2.15.1 by Marius Gedminas - find it at mg.pov.lt!